use-libreyolo-zero-shot

Zero-shot and promptable LibreYOLO

Four tiers answer "no training data, still want results". They are separate factories from LibreYOLO(...), with snapshot-style weights (a downloaded directory, not a Libre*<size>.pt checkpoint).

Pick the tier by question

Combinations are normal: LibreOpenVocab boxes fed to LibreSAM as box prompts gives text-to-mask ("segment the forklifts") without training anything.

If the classes are known in advance and latency matters, a fine-tuned regular detector beats all of these at runtime; zero-shot is for open or changing vocabularies and for prototyping before labeling.

LibreOpenVocab (text-vocabulary detection)

Discriminative text-conditioned detectors (not VLMs): real boxes, real calibrated-ish scores, no text generation. Contract: docs/adr/0008 and docs/openvocab_design.md.

from libreyolo import LibreOpenVocab

model = LibreOpenVocab("grounding-dino")      # or "owlv2"; aliases in
model.set_classes(["cat", "dog", "remote control"])   # sticky vocabulary
r = model.predict("a.jpg", conf=0.25)         # standard Results: r.boxes, r.names

• Vocabulary is sticky across predicts; defaults to COCO-80 when set_classes was never called. set_classes rejects a bare string, empties, and case-insensitive duplicates.
• Class names in prompts work best as short noun phrases; "remote control" outperforms "remote".
• Grounding DINO extras: text_threshold= (token threshold) on predict; long vocabularies are automatically chunked into multiple prompts and merged, so big label lists work but cost one forward per chunk. Phrase to class mapping is deliberately conservative: unmatched/ambiguous phrases are dropped rather than guessed, so a missing detection can be a mapping drop, lower conf/text_threshold and check labels before blaming the detector.
• OWLv2 has no text_threshold and rejects it. Ensemble models, strong on rare-object recall.
• Scores across the two families are not comparable; tune conf per family, per vocabulary.
• Sizes: grounding-dino t/b; owlv2 b16/l14. Weights auto-download from the LibreYOLO HF mirrors into weights/Libre<Family><size>/ directories.

LibreSAM (promptable segmentation)

Contract: docs/adr/0007. Point/box prompts to masks, or segment-everything.

from libreyolo import LibreSAM

model = LibreSAM("base")                            # SAM-1 (Apache-2.0), default tier
r = model.predict("img.jpg", points=[900, 370], labels=[1])   # 1=foreground, 0=background
r = model.predict("img.jpg", bboxes=[100, 100, 200, 200])
r = model.predict("img.jpg")                        # segment everything (slow)
r.masks.xy; r.boxes.xyxy                            # boxes derived from masks

model.set_image("img.jpg")                          # encode once (the expensive part)...
a = model.predict(points=[500, 375], labels=[1])    # ...prompt many, cheap
model.reset_image()

• Family pick: LibreSAM("base"/"large"/"huge") = SAM-1; LibreSAM2 aliases for SAM-2 (better masks, video-capable lineage); LibreMobileSAM (tiny encoder, edge/CPU-friendly, same prompt API). The alias table in libreyolo/models/sam/model.py is authoritative.
• Interactive loops: always set_image once, then prompt; re-passing the image per predict re-encodes and dominates latency.
• Prompt coordinates are pixels on the original image. Multiple points with mixed labels refine one object; foreground-only points on different objects need separate predicts.

LibreCLIP (zero-shot classify)

from libreyolo import LibreYOLO
model = LibreYOLO("LibreCLIPb32-cls.pt")
model.set_classes(["a forklift", "an empty aisle", "a spill"])
r = model("frame.jpg"); r.probs.top1

Same Results.probs contract as trained classifiers; defaults to ImageNet-1k labels until set_classes. Descriptive phrases ("a photo of a ...") often score better than bare nouns; the model re-derives text embeddings on each set_classes, so set once, not per frame.

LibreVLM (generative, last resort for detection)

LibreVLM(<alias>) prompts a vision-language model and parses its text into boxes. Flexible (arbitrary instructions via prompt=), but slow, uncalibrated, and parse-dependent; treat outputs as suggestions, not metrics-grade detections, and prefer LibreOpenVocab when the task is "boxes for named classes". Zero-shot vocab via set_classes works by prompting, so any string "works", with matching honesty caveats.

Honest limits (say these to users)

• Zero-shot mAP is well below a fine-tuned detector on a fixed class list; these tiers trade accuracy and speed for vocabulary freedom.
• All four tiers are inference-only in LibreYOLO: no train()/fine-tuning, and val() support varies (CLIP has a classify validator; open-vocab eval runs through the standard detect path with a fixed vocabulary).
• Everything runs through the transformers extra stack; first use downloads snapshot weights (hundreds of MB to GB). libreyolo checks confirms the extra is present.
• Export of these tiers is not supported like factory families; don't promise ONNX/TensorRT for them without checking current e2e coverage.

Related

• skills/use-libreyolo/: the trained-model core library guide.
• docs/adr/0007-libresam-contract.md, docs/adr/0008-open-vocab-detector-contract.md, docs/openvocab_design.md, docs/librevlm_design.md: the contracts.
• skills/libreyolo-license-audit/: weight licenses per tier if rehosting questions come up.