OpenParable
Skip to content
← Back to skills

Update LLM Model List

Development
update-llm-model-list

Update LLM Model List is an Agent Skill for development published in the Agenta-AI/agenta repository on GitHub: a SKILL.md instruction file an AI agent can load to do this task.

Audit and update the supported LLM model list in assets.py against litellm's registry (models.litellm.ai). Use when adding new models, pruning outdated ones, or verifying the list is correct.

View on GitHub ↗
QUICK START

How to use this skill

Bring this guide into your coding agent with a prompt tailored to the tool you use.

  1. Open your project in Codex.
  2. Copy and paste the prompt below.
  3. Review files and risks before approving any action.
Prompt to paste
Inspect the Agent Skill at https://github.com/Agenta-AI/agenta/blob/HEAD/.claude/skills/update-llm-model-list/SKILL.md and all supporting files. Report provenance, permissions, scripts, dependencies, and security risks. Propose an exact installation into .agents/skills/update-llm-model-list, but wait for my approval before writing files or running scripts.

Copying this prompt does not install or run the skill. Review third-party files before use.

Update LLM Model List

Overview

The canonical model list lives in sdks/python/agenta/sdk/utils/assets.py → supported_llm_models. It drives the model dropdown in the playground, cost metadata, and the model_to_provider_mapping.

The authoritative external source is litellm.model_cost (2 600+ entries), which mirrors https://models.litellm.ai/.

A pytest guard lives at: sdks/python/oss/tests/pytest/unit/test_supported_llm_models.py


Key rules

  1. Every model must exist in litellm.model_cost (direct key, or with provider prefix stripped).
    • anthropic/claude-* → litellm stores as claude-* (prefix is intentional for routing, stripped for cost lookup)
    • cohere/command-* → litellm stores as command-*
    • All other providers keep their full prefix (e.g. gemini/, groq/, together_ai/)
  2. Provider key ("anthropic", "gemini", …) must match the Secrets API enum in api/oss/src/core/secrets/enums.py (StandardProviderKind).
  3. No duplicates within a provider list.

Step 1 — Check which current models are outdated / wrong

Run this with uvx (no local install needed):

cat > /tmp/check_agenta_models.py << 'SCRIPT'
# /// script
# requires-python = ">=3.11"
# dependencies = ["litellm"]
# ///
import litellm, sys

# paste supported_llm_models here or import it
from agenta.sdk.utils.assets import supported_llm_models

mc = set(litellm.model_cost.keys())

def exists(m):
    if m in mc: return True
    if "/" in m and m.split("/", 1)[1] in mc: return True
    return False

fails = []
for provider, models in supported_llm_models.items():
    for model in models:
        if not exists(model):
            fails.append((provider, model))

total = sum(len(v) for v in supported_llm_models.values())
print(f"Total models checked: {total}")
if fails:
    for p, m in fails:
        print(f"  MISSING [{p}] {m}")
    sys.exit(1)
else:
    print("All models valid ✓")
SCRIPT
uvx --with litellm python /tmp/check_agenta_models.py 2>/dev/null

Alternatively, run the pytest unit test directly (requires agenta installed):

pytest sdks/python/oss/tests/pytest/unit/test_supported_llm_models.py -v

Step 2 — Find models missing from Agenta (big-3 audit)

This script finds models in litellm that Agenta doesn't list yet, filtered to remove noise (audio, video, embeddings, codex, snapshots):

cat > /tmp/find_missing.py << 'SCRIPT'
# /// script
# requires-python = ">=3.11"
# dependencies = ["litellm"]
# ///
import litellm, re

AGENTA_ANTHROPIC = set()   # fill from assets.py (bare names, no prefix)
AGENTA_OPENAI    = set()   # fill from assets.py
AGENTA_GEMINI    = set()   # fill from assets.py (with gemini/ prefix)

mc = set(litellm.model_cost.keys())

NOISE = [
    "audio","tts","speech","whisper","transcri","realtime","diarize",
    "dall-e","image","video","veo","embed","moderat","search",
    "babbage","davinci","ada","instruct","codex","computer-use",
    "robotics","learnlm","gemma","live","v1:0",
]
KEEP = {"gpt-4o","gpt-4o-mini"}
DATED = re.compile(r"-\d{4}-\d{2}-\d{2}$")
EXP   = re.compile(r"exp-\d{4}|\d{2}-\d{2}$")

def noise(m):
    if m in KEEP: return False
    return any(kw in m.lower() for kw in NOISE)

def dated(m):
    return bool(DATED.search(m)) or bool(EXP.search(m))

def report(label, candidates, known, prefix=""):
    print(f"\n=== {label} ===")
    for m in sorted(candidates):
        bare = m[len(prefix):] if prefix else m
        if bare in known or m in known: continue
        tag = "[dated/exp]" if dated(m) else "[alias]" if m.endswith("-latest") else "*** MISSING ***"
        print(f"  {m}  {tag}")

# Anthropic
report("ANTHROPIC", [m for m in mc if m.startswith("claude-") and not noise(m)],
       AGENTA_ANTHROPIC)

# OpenAI (no slash, starts with gpt- / o1 / o3 / o4)
OAI = [m for m in mc if any(m.startswith(p) for p in ("gpt-","o1","o3","o4","chatgpt"))
       and "/" not in m and not noise(m)]
report("OPENAI", OAI, AGENTA_OPENAI)

# Gemini
report("GEMINI", [m for m in mc if m.startswith("gemini/") and not noise(m)],
       AGENTA_GEMINI, prefix="gemini/")
SCRIPT
uvx --with litellm python /tmp/find_missing.py 2>/dev/null

Fill in the AGENTA_* sets from the current assets.py before running.


Step 3 — Edit assets.py

File: sdks/python/agenta/sdk/utils/assets.py

  • Add models inside the correct provider list, newest first.
  • For Gemini 1.5 models (still widely used): add under "gemini".
  • For OpenAI o-series pro tiers (o1-pro, o3-pro): add after their base model.
  • For Groq: always cross-check litellm.groq_models — Groq rotates its model catalogue frequently.
  • For DeepInfra / Together AI: check litellm.deepinfra_models / litellm.together_ai_models for current names.

Provider prefix conventions

Provider keyAgenta prefixlitellm cost key prefix
anthropicanthropic/claude- (no prefix)
coherecohere/command- (no prefix)
geminigemini/gemini/
groqgroq/groq/
mistralmistral/mistral/
openai(none)(none)
openrouteropenrouter/openrouter/
perplexityaiperplexity/perplexity/
together_aitogether_ai/together_ai/
deepinfradeepinfra/deepinfra/

Step 4 — Run ruff then the test

# Format + lint
uvx --from ruff==0.14.0 ruff format sdks/python/agenta/sdk/utils/assets.py
uvx --from ruff==0.14.0 ruff check --fix sdks/python/agenta/sdk/utils/assets.py

# Validate all models against litellm (no agenta install needed)
uvx --with litellm python /tmp/check_agenta_models.py 2>/dev/null

All checks must pass before committing.


Related files

FilePurpose
sdks/python/agenta/sdk/utils/assets.pyCanonical model list + cost metadata builder
sdks/python/oss/tests/pytest/unit/test_supported_llm_models.pyPytest guard (parametrized per model)
api/oss/src/core/secrets/enums.pyProvider keys — must stay in sync
api/oss/src/resources/evaluators/evaluators.pySeparate (shorter) model list for evaluator dropdown