Skippy Cache Family Bench
Testing & Qualityskippy-cache-family-benchSkippy Cache Family Bench is an Agent Skill for testing & quality published in the Mesh-LLM/mesh-llm repository on GitHub: a SKILL.md instruction file an AI agent can load to do this task.
Use this skill when benchmarking Skippy exact-prefix cache across model families, comparing Skippy against llama-server, producing README benchmark tables, updating crates/skippy-cache/README.md evidence, or diagnosing cache benchmark gaps by family or Hugging Face use case.
How to use this skill
Bring this guide into your coding agent with a prompt tailored to the tool you use.
- Open your project in Codex.
- Copy and paste the prompt below.
- Review files and risks before approving any action.
Inspect the Agent Skill at https://github.com/Mesh-LLM/mesh-llm/blob/HEAD/.agents/skills/skippy-cache-family-bench/SKILL.md and all supporting files. Report provenance, permissions, scripts, dependencies, and security risks. Propose an exact installation into .agents/skills/skippy-cache-family-bench, but wait for my approval before writing files or running scripts.
Copying this prompt does not install or run the skill. Review third-party files before use.
skippy-cache-family-bench
Use this skill for reproducible Skippy cache benchmark evidence. The goal is to
compare production cache payloads only: ResidentKv for dense families and
KvRecurrent for recurrent/hybrid families. Do not report FullState as a
production cache mode.
Workflow
-
Run the reproducible wrapper. It builds by default, then runs full-GGUF baselines, Hugging Face use-case prompts, and the README report renderer:
evals/skippy-cache-family-bench.sh /tmp/skippy-cache-family-bench -
For fast iteration after a build, skip the build step:
SKIPPY_CACHE_SKIP_BUILD=1 evals/skippy-cache-family-bench.sh /tmp/skippy-cache-family-bench -
If running the pieces manually, keep these matched settings for full-GGUF family baselines:
LLAMA_STAGE_BUILD_DIR=.deps/llama-build/build-stage-abi-cpu \ python3 evals/skippy-cache-production-bench.py \ --output-dir /tmp/skippy-cache-family-bench/full-gguf \ --runtime-lane-count 1 \ --llama-parallel 1 \ --prefix-tokens 128 -
Run the use-case benchmark matrix against the same family set:
LLAMA_STAGE_BUILD_DIR=.deps/llama-build/build-stage-abi-cpu \ python3 evals/skippy-cache-production-bench.py \ --output-dir /tmp/skippy-cache-family-bench/use-cases \ --runtime-lane-count 1 \ --llama-parallel 1 \ --prefix-tokens 128 \ --use-case all -
Render README-ready tables from the combined JSON outputs:
python3 evals/skippy-cache-family-report.py \ --input /tmp/skippy-cache-family-bench/full-gguf/production-cache-bench.json \ --input /tmp/skippy-cache-family-bench/use-cases/production-cache-bench.json \ --output /tmp/skippy-cache-family-bench/readme-tables.md
Reporting Rules
- Keep rows and columns ordered by related family: Qwen3Next, Falcon-H1, Llama, Qwen3 dense, DeepSeek2, GLM-4.7 Flash, GLM4, Gemma4 A4B, Gemma4 E4B, Gemma3, Gemma2, OLMo, MiniMax M2.7.
- Always report Skippy versus llama-server for full-GGUF rows.
- Keep DeepSeek3 in package-only evidence unless a machine can run a monolithic full-GGUF llama-server baseline.
- Use one generated token and matched prefix tokens for apples-to-apples rows.
- If a family fails correctness, leave the benchmark row out of promoted README
evidence and explain the failure in
crates/skippy-cache/TODO.md. - Preserve raw outputs under
/tmp/...or another explicit run directory; do not paste ad hoc numbers without the backingproduction-cache-bench.json.