Rank LLM Verify
Testing & Qualityrank-llm-verifyRank LLM Verify is an Agent Skill for testing & quality published in the castorini/rank_llm repository on GitHub: a SKILL.md instruction file an AI agent can load to do this task.
Use when validating rank_llm artifacts after rerank, retrieve-cache, or related CLI workflows. Checks JSONL integrity, TREC formatting, candidate shape, invocation-history structure, and duplicate query identifiers. Wraps rank-llm view plus custom assertions.
How to use this skill
Bring this guide into your coding agent with a prompt tailored to the tool you use.
- Open your project in Codex.
- Copy and paste the prompt below.
- Review files and risks before approving any action.
Inspect the Agent Skill at https://github.com/castorini/rank_llm/blob/HEAD/.claude/skills/rank-llm-verify/SKILL.md and all supporting files. Report provenance, permissions, scripts, dependencies, and security risks. Propose an exact installation into .agents/skills/rank-llm-verify, but wait for my approval before writing files or running scripts.
Copying this prompt does not install or run the skill. Review third-party files before use.
rank_llm Verify
Validates stored RankLLM artifacts for correctness and structural consistency.
When to Use
- After
rank-llm rerank - After
rank-llm retrieve-cache - Before using rerank outputs for
evaluateoranalyze - When comparing artifacts across models or prompt templates
What It Checks
JSONL Integrity
- Every line is valid JSON
- No empty files
- No truncated or malformed records
Request Input
- Every record has
queryandcandidates - Every record has at least one candidate
- Candidate entries expose either
docortext
Rerank Output
- Every record has
queryandcandidates - Every candidate has
docid,score, anddoc - No duplicate query identifiers when
qidvalues are present
Invocation History
- Top-level file is a JSON list
- Each record has an
invocations_historylist - Every invocation entry is object-shaped
TREC Output
- Every non-empty line has 6 columns
- Rank field is an integer
- Score field is numeric
Usage
Run the verification script:
bash .claude/skills/rank-llm-verify/scripts/verify.sh <artifact-path> [artifact-type]
Supported artifact types:
request-inputrerank-outputinvocations-historytrec-output
If no artifact type is provided, the script attempts to auto-detect it with rank-llm view.
Verification Script
See scripts/verify.sh for the runnable verification wrapper.
Gotchas
rank-llm validate rerankchecks input contracts before execution. The verify script checks stored output artifacts after execution.rank-llm viewonly detects supported artifact families. If a file is not one of those shapes, pass the artifact type manually or inspect the file directly.- A rerank JSONL file can be structurally valid and still be low quality. Use
evaluateandanalyzefor score and response diagnostics.