OpenParable
Skip to content
← Back to skills

Hf Quant And Layer Package Jobs

Development
hf-quant-and-layer-package-jobs

Hf Quant And Layer Package Jobs is an Agent Skill for development published in the Mesh-LLM/mesh-llm repository on GitHub: a SKILL.md instruction file an AI agent can load to do this task.

Use when running quantization of a BF16/FP16 GGUF repo and Skippy layer-package creation as one local or Hugging Face Jobs workflow, publishing both artifacts to Hugging Face.

View on GitHub ↗
QUICK START

How to use this skill

Bring this guide into your coding agent with a prompt tailored to the tool you use.

  1. Open your project in Codex.
  2. Copy and paste the prompt below.
  3. Review files and risks before approving any action.
Prompt to paste
Inspect the Agent Skill at https://github.com/Mesh-LLM/mesh-llm/blob/HEAD/.agents/skills/hf-quant-and-layer-package-jobs/SKILL.md and all supporting files. Report provenance, permissions, scripts, dependencies, and security risks. Propose an exact installation into .agents/skills/hf-quant-and-layer-package-jobs, but wait for my approval before writing files or running scripts.

Copying this prompt does not install or run the skill. Review third-party files before use.

HF Quant And Layer Package Jobs

Use this skill when a workflow should produce both a quantized GGUF repo and a Skippy layer package from an existing BF16/FP16 GGUF repo. The quantization phase must use skippy-quantize; do not use llama-quantize, llama-quantise, convert_hf_to_gguf.py, hf_to_gguf.py, or the misspelled old notes form hf_to_gguff.py.

Preconditions

  • Source BF16/FP16 GGUF repo is complete and has a known selector/prefix.
  • Target quant repo, quant selector, tensor-type file, output basename, expected split count, and memory budget are known.
  • Target layer-package repo is known or intentionally auto-derived by mesh-llm models package.
  • The layer package phase starts only after skippy-quantize verify-job succeeds for the quantized artifact.

Local Workflow

Quantize first:

target/release/skippy-quantize init-quant \
  --source /mnt/bf16 \
  --source-prefix BF16 \
  --target /mnt/quant \
  --target-prefix <quant-selector> \
  --output-basename <model>-<quant-selector> \
  --quant <quant-selector> \
  --tensor-type-file /mnt/recipe/tensor-types.txt \
  --window-size 1 \
  --manifest /tmp/skippy-quantize.json

target/release/skippy-quantize run-quant \
  --manifest /tmp/skippy-quantize.json \
  --backend llama-api \
  --max-memory 32G \
  --work-dir /tmp/skippy-quantize-work \
  --spool-dir /tmp/skippy-quantize-output \
  --record-dir /tmp/skippy-quantize-records \
  --json-event-file /tmp/skippy-quantize-status.json \
  --json-event-interval-seconds 120 \
  --json-event-window 8

target/release/skippy-quantize verify-job \
  --manifest /tmp/skippy-quantize.json \
  --llama-load

Before the real run, dry-run the same quant job and confirm it reports the expected source, target, tensor recipe, backend, memory budget, and next window:

target/release/skippy-quantize quant-job \
  --source /mnt/bf16 \
  --source-prefix BF16 \
  --target /mnt/quant \
  --target-prefix <quant-selector> \
  --output-basename <model>-<quant-selector> \
  --quant <quant-selector> \
  --tensor-type-file /mnt/recipe/tensor-types.txt \
  --window-size 1 \
  --manifest /tmp/skippy-quantize.json \
  --backend llama-api \
  --max-memory 32G \
  --dry-run

Publish the quant repo if the target is not already a mounted Hub repo:

hf repo create <org>/<quant-repo> --type model --private
hf upload <org>/<quant-repo> /mnt/quant . --repo-type model

Package the published quant:

mesh-llm models package <org>/<quant-repo>:<quant-selector> --dry-run
mesh-llm models package <org>/<quant-repo>:<quant-selector> --confirm --follow

Or package locally and publish:

target/debug/skippy-model-package write-package \
  <org>/<quant-repo>:<quant-selector> \
  --out-dir /tmp/<model>-layers

target/debug/skippy-model-package preflight \
  /tmp/<model>-layers \
  --verify-sha256

hf repo create <org>/<layer-package-repo> --type model --private
hf upload <org>/<layer-package-repo> /tmp/<model>-layers . --repo-type model

HF Jobs Workflow

When combining both phases in one HF Job, keep the quantized GGUF repo as the durable boundary:

  1. Mount the BF16/FP16 source repo read-only.
  2. Mount the target quant repo read/write.
  3. Run skippy-quantize init-quant if the manifest is missing.
  4. Run skippy-quantize run-quant until complete.
  5. Run skippy-quantize verify-job; stop if it fails.
  6. Submit or run the mesh-llm models package <quant-repo>:<selector> package phase.
  7. Record both the quant repo commit and the layer-package repo commit.

Template:

hf jobs uv run \
  --namespace meshllm \
  --flavor cpu-upgrade \
  --timeout 4d \
  --secrets HF_TOKEN \
  --volume hf://models/<bf16-repo>:/mnt/bf16 \
  --volume hf://models/<quant-repo>:/mnt/quant \
  --env SKIPPY_QUANTIZE_OUTPUT=json \
  --env PYTHONUNBUFFERED=1 \
  --detach \
  /path/to/skippy_quant_then_package_job.py \
  -- \
  --source /mnt/bf16 \
  --source-prefix BF16 \
  --target /mnt/quant \
  --target-prefix <quant-selector> \
  --output-basename <model>-<quant-selector> \
  --quant <quant-selector> \
  --tensor-type-file /mnt/recipe/tensor-types.txt \
  --package-ref <org>/<quant-repo>:<quant-selector> \
  --max-memory 32G

Resume Rules

  • If quant shards already exist, skippy-quantize resumes at the first missing shard.
  • If the quant repo verifies successfully, skip quantization and run or inspect the package job.
  • Do not delete a verified quant repo to force a clean package run. Package jobs should consume the published quant artifact as the source of truth.

Validation

Before promoting the combined run, record:

  • source BF16/FP16 repo revision;
  • quant repo commit, quant selector, tensor recipe, split count, and verify output;
  • layer-package job id, target repo, target commit, and package certification;
  • total HF job cost and whether the combined workflow saved time or only saved operator steps.