libreyolo-upload-hf-dataset

Upload a LibreYOLO dataset repo to HuggingFace

Use this skill when publishing a dataset to https://huggingface.co/datasets/LibreYOLO/<repo>.

Scope: data repos consumed by model.train(...) / model.val(...) — a single .zip (or a download: script) in a LibreYOLO-supported layout. The companion skill libreyolo-upload-hf-model covers weights.

The golden rule

Never redistribute a third party's packaged artifact — their .zip, their tarball, their weights. Even when the underlying data is permissively licensed, their repackaging may carry a license they attached, and mirroring their file is redistributing their artifact.

Instead: rebuild from the canonical upstream and publish your own artifact with pinned provenance. scripts/build_imagenette.py is the reference example — it pulls fast.ai's Apache-2.0 tarball, pins its sha256, repackages to a LibreYOLO-layout .zip, and uploads. Copy that shape.

The redistribution license gate (do this FIRST)

Before downloading or hosting anything, establish the source license and decide. This gate is the whole point of the skill — the bug it prevents is someone wiring data="foo" to a URL nobody checked.

Worked precedent: imagenet10 (ImageNet-derived → not redistributable) was replaced by smoke10, a subset of Apache-2.0 Imagenette. When the license fails the gate, substitute a clean-sourced equivalent; do not host it anyway.

LibreYOLO format by task

The dataset must be in the layout the loader expects for its task. Full spec: docs/dataset_schema.md and the loaders under libreyolo/data/. Summary:

Only detection, instance segmentation, and OBB accept native COCO JSON via an annotations: block (docs/dataset_schema.md). Pose, semantic, depth, and restore have no COCO-JSON loader path — convert to their native layout offline.

Converting into these (target on the right):

• COCO JSON for detection / instance-seg / OBB → keep it; wire with an annotations: block (no conversion).
• COCO keypoints JSON → there is no annotations: path for pose; convert offline to YOLO-pose TXT with libreyolo.data.convert_coco_keypoints_json_to_yolo_pose (or convert_coco_keypoints_splits for train+val at once), then ship the resulting labels/ tree.
• A competitor's YOLO-TXT export → already compatible (data.yaml + labels/).
• Roboflow → export as YOLO or COCO; both load.
• Raw class masks → single-channel PNGs under masks_dir; remap source IDs with label_mapping.
• Paired degraded/clean images (restore) → inputs/** + targets/** with matching stems.
• Folder of labelled images → ImageFolder train/val/<class>.

The dataset-repo contract

A dataset repo contains the artifact plus a card. No bare zips.

<repo>/
├── README.md                # dataset card: frontmatter + Provenance / Contents / Use / License
└── <name>.zip               # the data in LibreYOLO layout (or split .zips)

Large sets may ship split archives; keep the card + a manifest. Do not upload the raw upstream tarball alongside the rebuilt zip.

Dataset card template

Frontmatter keys are HF dataset-card standard. license is the source license (see the gate). Body sections below the ---:

---
license: apache-2.0
task_categories:
  - image-classification        # object-detection / image-segmentation /
                                # keypoint-detection / depth-estimation /
                                # image-to-image (restore)
tags:
  - libreyolo
  - <dataset-tag>
pretty_name: <Pretty Name> (LibreYOLO)
size_categories:
  - 10K<n<100K
---

license: is mandatory — never publish a card without it. It is the source license the gate resolved; if you cannot name one, you have not passed the gate, so stop and ask rather than upload. The guard tests check hosting only, not the license field, so this rule is enforced by you, not by CI. A license-less repo shows as "unknown" on HF and downstream users cannot tell whether the data is safe to train a shippable model on.

• # <Pretty Name> (LibreYOLO) — one sentence: what it is, which task, why hosted.

• ## Provenance — source URL + org + license; pinned source sha256; the transform applied (e.g. "extracted .tgz → .zip", "converted COCO → YOLO TXT"); the line "No third-party repackaging used."

• ## Contents — the layout and per-split counts; class list or class count.

• ## Use with LibreYOLO — a short snippet, e.g.:

  from libreyolo import LibreYOLO
  model = LibreYOLO("<checkpoint>")
  model.val(data="<name>")      # auto-downloads this dataset
  

• ## License — the source license, inherited, with attribution.

Wiring auto-download

Two mechanisms — pick by task:

1. YAML datasets (detection / segment / obb / pose / semantic / depth / restore) — libreyolo/config/datasets/<name>.yaml with a download::
   • a single URL to the HF zip → auto-downloads on first use, OR
   • a download: | Python script (multi-line) for multi-step fetch/convert — runs only under allow_download_scripts=True. Scripts must from libreyolo.data.utils import download and pull from huggingface.co/datasets/LibreYOLO/... or official sources (cocodataset.org) — never a competitor host.
2. Classification known-names — add to _KNOWN_DATASETS in libreyolo/data/classify_dataset.py, value = the HF resolve/main/<name>.zip URL.

Name is the cache key — new content needs a new name. Both resolvers cache by name (classify under DATASETS_DIR/<name>, yaml under the dataset dir) and check that cache before the URL, so re-pointing a name already in the wild at a different artifact silently keeps serving the old bytes to everyone who downloaded once. When the data changes, publish under a new name (as imagenet10 → smoke10 did) or a version suffix (<name>-v2); never swap the artifact behind a name in use.

Both are guarded:

• tests/unit/test_data_utils.py::test_builtin_dataset_yamls_do_not_use_ultralytics_assets
• tests/unit/test_data_utils.py::test_known_classify_datasets_are_libre_hosted

Run them after wiring; they must stay green.

Upload workflow

1. License gate (section above). If it fails: substitute a clean source or stop and ask.

2. Fetch the canonical upstream and pin its sha256 (reproducibility + tamper check).

3. Convert to the LibreYOLO layout for the task; verify counts and a few samples.

4. Zip with ZIP_STORED (images are already compressed — deflate wastes CPU for ~0 gain).

5. Write the dataset card (provenance + license). Confirm the license: frontmatter field is present and non-empty before uploading — it is mandatory and unenforced by CI.

6. Create + upload with huggingface_hub (repo_type="dataset"), token from env only (HF_TOKEN), never committed:

   from huggingface_hub import HfApi
   api = HfApi(token=os.environ["HF_TOKEN"])
   api.create_repo("LibreYOLO/<name>", repo_type="dataset", private=False, exist_ok=True)
   api.upload_file(path_or_fileobj=card.encode(), path_in_repo="README.md",
                   repo_id="LibreYOLO/<name>", repo_type="dataset")
   api.upload_file(path_or_fileobj="<name>.zip", path_in_repo="<name>.zip",
                   repo_id="LibreYOLO/<name>", repo_type="dataset")
   

7. Verify the resolve URL serves real bytes, not an LFS pointer: https://huggingface.co/datasets/LibreYOLO/<name>/resolve/main/<name>.zip

8. Wire auto-download (section above) and run the guard tests.

9. End-to-end check: resolve_classify_data("<name>") (classify) or model.val(data="<name>.yaml") (yaml) on a cleared cache.

Prefer a committed build script (like scripts/build_imagenette.py) over one-off commands, so the artifact is reproducible from source.

Ask the user when

• The source license is CC-BY-NC, research-only, unknown, or absent.
• There is no pinnable upstream version/commit (reproducibility needs it).
• The data contains faces / PII / sensitive content (privacy layers on top of license).
• The dataset is large (multi-GB) — hosting + egress cost is a decision.
• A dataset repo with that name already exists (overwrite is destructive).
• No supported layout fits the task cleanly.

Common traps

• Mirroring a competitor's packaged zip — the exact bug this skill exists to prevent. Rebuild from upstream.
• Re-hosting ImageNet-derived data (redistribution restricted) — substitute Imagenette (Apache-2.0).
• CC-BY-NC data in a commercially-usable project — the -NC blocks it.
• Not pinning the source sha256 — a silent upstream re-cut changes the data under you.
• Zipping already-JPEG images with deflate — use ZIP_STORED.
• Renaming class folders — class index is the sorted folder name; reordering changes label IDs. Keep upstream naming or document the mapping.
• Uploading without a card / license — HF flags "no license" and users can't tell if it is safe to use.
• Re-pointing an existing dataset name at new content — the resolver caches by name and checks the cache before the URL, so everyone who already downloaded keeps the old bytes forever. Ship changed data under a new name or version suffix (see "Wiring auto-download").
• Assuming pose/semantic/depth/restore load COCO JSON — only detection, instance-seg, and OBB have an annotations: path; the rest need offline conversion to their native layout.
• Pointing the classify resolver at a .tgz — it extracts .zip only; convert first (yaml download: scripts can handle either via the download() helper).