OpenParable
Skip to content
← Back to skills

Litert Model Equivalence Test

Testing & Quality
litert-model-equivalence-test

Litert Model Equivalence Test is an Agent Skill for testing & quality published in the google-ai-edge/litert-torch repository on GitHub: a SKILL.md instruction file an AI agent can load to do this task.

Validates equivalence between LiteRT models (litert_lm) and PyTorch models (transformers). Use when you need to verify that an exported LiteRT model produces the same outputs as the original Hugging Face model. Supports multi-turn conversations and custom prompts.

View on GitHub ↗
QUICK START

How to use this skill

Bring this guide into your coding agent with a prompt tailored to the tool you use.

  1. Open your project in Codex.
  2. Copy and paste the prompt below.
  3. Review files and risks before approving any action.
Prompt to paste
Inspect the Agent Skill at https://github.com/google-ai-edge/litert-torch/blob/HEAD/litert_torch/generative/export_hf/experimental/validation/SKILL.md and all supporting files. Report provenance, permissions, scripts, dependencies, and security risks. Propose an exact installation into .agents/skills/litert-model-equivalence-test, but wait for my approval before writing files or running scripts.

Copying this prompt does not install or run the skill. Review third-party files before use.

LiteRT Model Equivalence Test

This skill provides instructions for running equivalence tests between LiteRT models and their PyTorch source models.

Usage

Use the equivalence_test script to compare the outputs of a Hugging Face model and its exported LiteRT version.

Running the Test

Run the test using bazel run from your workspace:

bazel run \
  //third_party/py/litert_torch/generative/export_hf/experimental/validation:equivalence_test \
  -- \
  --model_id={model_id} \
  [--prompt={prompt}] \
  [--prompt_file={prompt_file}] \
  [--max_new_tokens={max_new_tokens}] \
  [--max_num_tokens={max_num_tokens}] \
  [--work_dir={work_dir}] \
  [--externalize_embedder] \
  [--single_token_embedder] \
  [--split_cache] \
  [--backend={backend}]

Flags

  • --model_id: The Hugging Face model ID to validate (e.g., google/gemma-3-270m-it).
  • --prompt: Prompt to test. Specify multiple times for multi-turn conversations.
  • --prompt_file: Path to a file containing one (complex) prompt. Overrides --prompt.
  • --max_new_tokens: Maximum new tokens to generate per turn (default: 20).
  • --max_num_tokens: KV cache length for the model (default: 2048).
  • --work_dir: Base directory for model export. If not specified, a temporary directory under HOME is used.
  • --externalize_embedder: Externalize the embedder during export (default: False).
  • --single_token_embedder: Use single token embedder during export (default: False).
  • --split_cache: Split KV cache during export (default: False).
  • --backend: Hardware backend to use for LiteRT LM (cpu | npu, default: cpu).

Examples

Single-turn test with custom prompt:

bazel run \
  //third_party/py/litert_torch/generative/export_hf/experimental/validation:equivalence_test \
  -- \
  --model_id=google/gemma-3-270m-it \
  --prompt="What is the capital of France?"

Multi-turn test:

bazel run \
  //third_party/py/litert_torch/generative/export_hf/experimental/validation:equivalence_test \
  -- \
  --model_id=google/gemma-3-270m-it \
  --prompt="What's the capital of France?" \
  --prompt="How about Germany?"

Testing with a prompt file:

bazel run \
  //third_party/py/litert_torch/generative/export_hf/experimental/validation:equivalence_test \
  -- \
  --model_id=google/gemma-3-270m-it \
  --prompt_file=/path/to/prompts.txt

Testing with externalized embedder:

bazel run \
  //third_party/py/litert_torch/generative/export_hf/experimental/validation:equivalence_test \
  -- \
  --model_id=google/gemma-3-270m-it \
  --externalize_embedder \
  --single_token_embedder

Testing NPU export variant:

bazel run \
  //third_party/py/litert_torch/generative/export_hf/experimental/validation:equivalence_test \
  -- \
  --model_id=google/gemma-3-270m-it \
  --externalize_embedder \
  --split_cache \
  --backend=npu