Open-Vocabulary Detection: Zero-Shot Defect Finding

Open-vocabulary detection is a computer-vision technique that finds and locates objects described in plain language, rather than only the fixed set of categories a model was trained on. Instead of building a bespoke detector for "leading-edge erosion" or "weld undercut"—and gathering thousands of labelled images first—you type the phrase and the model returns candidate detections straight away. For inspection teams working with scarce labelled data, that changes the starting point: you can stand up a first-pass baseline on your own images with zero training data, review what it finds, and only then decide what is worth labelling. This guide explains how it works and where it fits in an engineering inspection workflow.

What is open-vocabulary detection?

Open-vocabulary detection is object detection that is not tied to a fixed list of classes. A vision-language model compares regions of an image against arbitrary text prompts, so it can localise categories it was never explicitly trained to detect—for example "corrosion", "spalled concrete" or "missing bolt"—described in your own words.

The important word is arbitrary. A conventional detector can only ever return the classes in its training set. An open-vocabulary detector takes your vocabulary as an input, at the moment you run it, and does its best to find matches. That difference is what makes it so useful when your "classes" are defect types that change from asset to asset.

How does open-vocabulary detection differ from closed-set detection?

Traditional detectors—the Faster R-CNN and YOLO families most engineers will have heard of—are closed-set. They are trained on a fixed label list, such as the 80 everyday categories in the COCO dataset, and can only output those. Adding a new class means collecting and labelling a fresh dataset and retraining the model. For inspection, where the categories are defect types that vary by material, asset and acceptance standard, that retraining loop is slow and expensive.

Open-vocabulary detection breaks the fixed label list. It is built on foundation models: large models pre-trained on huge, diverse image-and-text datasets that learn a shared representation of pictures and language. CLIP, which learned to match images to captions across hundreds of millions of pairs, is the best-known example. Because images and text live in the same representation, a text prompt and an image region can be compared directly—no bespoke class head required.

The models doing the work

A few families of model turn that idea into practical detection and segmentation:

• Grounding DINO is an open-set, language-conditioned detector. It localises any user-specified phrase zero-shot, fusing image and text features throughout the network rather than only at the end. On public zero-shot transfer benchmarks it sets strong records, comfortably ahead of earlier open-vocabulary detectors.
• OWL-ViT, from Google, attaches detection heads to a Vision Transformer with CLIP-style image-text pre-training, enabling zero-shot detection driven by text queries.
• Segment Anything (SAM), from Meta, is a promptable segmentation foundation model. It produces high-quality masks from a click, a box, or another model's output, and generalises to new image distributions without retraining. SAM 2 extends the same promptable behaviour to video.

In practice these are often chained: a text-prompted detector such as Grounding DINO proposes boxes from your phrase, and SAM turns each box into a pixel-accurate mask. That "describe it, then segment it" pattern is how a phrase like pitting corrosion becomes a measured, outlined region on a photograph.

Zero-shot detection: describe it, don't train it

Zero-shot detection means the model handles a category for which it has seen zero labelled examples for your task. You supply the vocabulary at inference time as text—"hairline crack", "flange corrosion", "coating blister", "delamination"—and the model attempts to find and localise each one. Nothing is trained, nothing is fine-tuned; you are querying what the foundation model already knows about the visual world and steering it with language.

For an engineer, the mental model is simple: you are not programming a detector, you are briefing one. The quality of the prompt matters, just as the wording of an inspection instruction matters, and part of using these tools well is learning which phrasings pull out the features you care about.

Why does open-vocabulary detection matter for engineering inspection?

The single biggest barrier to machine vision on real assets is not the algorithm—it is data. Inspection imagery is expensive to gather because it often means gaining access to a live plant, a subsea structure or a height-restricted span. The defects that matter most are, by definition, rare, so a representative labelled set can take months to assemble. And the labelling itself has to be done by scarce, expensive experts who know the difference between benign surface staining and active corrosion.

Open-vocabulary detection attacks that barrier directly. Because it needs zero training data to produce a first result, you can point it at whatever images you already have and get a baseline the same afternoon. That baseline is not the finished detector, but it is an enormously useful starting point: it tells you roughly how visible your defects are, where the easy wins and hard cases sit, and what a bespoke model would eventually need to learn. It reframes the first question from "can we afford to build a detector?" to "here is what a baseline already finds—what is worth improving?"

How do you get from a baseline to a trusted, trained model?

An open-vocabulary baseline is the beginning of the workflow, not the end—and being honest about its limits is what makes it useful.

Foundation models are trained largely on everyday photographs, so they carry a domain gap. On specialised imagery—radiographs, thermal frames, subsea or borescope footage—expect a mix of confident correct calls, near misses and false positives on subtle, domain-specific defects. Treating the raw output as gospel would be a mistake in any setting, and unacceptable in a safety-critical one.

That is exactly why confidence has to be part of the output. A baseline you can trust does not just say crack / no crack; it tells you how sure it is, and routes the low-confidence cases to a qualified human rather than quietly passing or failing them. Uncertainty-aware review keeps expert attention on the ambiguous few instead of the obvious many, which is the only way risk-based inspection scales.

From there, the efficient path to a bespoke, dependable detector is active learning: the baseline proposes labels, an expert corrects only the most informative examples, and a domain-specific model is trained on a small fraction of the images you would otherwise have had to annotate by hand. We cover that loop in Label-Efficient Computer Vision: Cutting Image-Labelling Effort with Active Learning. For a full worked run—an open-vocabulary baseline for infrastructure defects built in a single afternoon—see our forthcoming worked example on prompt-to-detection baselines.

Where it fits alongside the standards

None of this replaces the qualified inspector or the standard they work to. Visual testing of welded joints, for instance, is governed by BS EN ISO 17637, which sets out inspector qualification, viewing conditions and even minimum illuminance—350 lx at the surface, 500 lx recommended—for a valid examination (ISO 17637; see also the British Institute of Non-Destructive Testing, bindt.org). Open-vocabulary detection sits upstream of and alongside that process as part of an automated visual inspection pipeline: it triages the imagery, flags candidates, and quantifies how confident it is, so that expert time—and formal sign-off against the standard—is spent where it counts.

The pattern is consistent: describe the defect, get an instant baseline, review with confidence scores, and only label the cases that teach the model something new. That is how you get from "we have no training data" to a trusted, asset-specific detector without an eye-watering labelling bill.

Try it on your own images

You do not need a dataset to start. Describe a defect in words and get a baseline on your own inspection photographs, then decide what is worth training. Run an open-vocabulary baseline with VisionEngine.

Hero image: “Pitting Corrosion on a pipe” by Vsolymossy, CC BY 3.0, via Wikimedia Commons.