kiln-check-finetune-deprecation

Check Fine-Tune Model Deprecation in Kiln

Audit the fine-tunable models listed in Kiln to find base models that are no longer supported for fine-tuning by their providers. This is separate from general model deprecation — a model can still be available for inference but no longer supported as a fine-tuning base.

---

Global Rules

• Sandbox: All curl, uv run, and python3 commands that hit the network MUST use required_permissions: ["all"]. The sandbox blocks network access.
• Env vars: Source .env before running check scripts: export $(grep -v '^#' .env | xargs)
• Vertex AI auth: The Vertex check requires gcloud CLI authentication. Before running, ask the user to run gcloud auth login if they haven't recently. If gcloud auth print-access-token fails, prompt the user to authenticate.
• Non-destructive: This skill only reports findings. It does not automatically remove or deprecate models.

---

Supported Providers

---

Background

Kiln has two types of fine-tunable model entries:

1. Static entries — Models in libs/core/kiln_ai/adapters/ml_model_list.py with provider_finetune_id set. Currently used by Together AI and Vertex AI.
2. Dynamic entries (Fireworks) — Fetched at runtime from api.fireworks.ai/v1/accounts/fireworks/models filtering by tunable=True. This list is managed by Fireworks and may include stale/unsupported models.

The API endpoint that serves the fine-tune dropdown is GET /api/finetune_providers in app/desktop/studio_server/finetune_api.py.

---

Phase 1 – Check Static Fine-Tune IDs

Run the check script from the repo root:

uv run python3 .agents/skills/kiln-check-finetune-deprecation/scripts/check_finetune.py static > /tmp/kiln_finetune_static.json

This extracts all provider_finetune_id entries from ml_model_list.py and checks each provider's docs/API to see if the model is still available for fine-tuning:

• Together AI: Scrapes the Together AI docs page for the list of supported fine-tune models. Together uses full HuggingFace-style IDs (e.g. meta-llama/Meta-Llama-3.1-8B-Instruct-Reference).
• Vertex AI: Checks the Vertex AI publisher models API. Vertex fine-tuning uses specific versioned model IDs (e.g. gemini-2.0-flash-001).

Output: JSON to stdout with per-provider results, human summary to stderr.

---

Phase 2 – Check Fireworks Dynamic Models

Fireworks fine-tunable models are fetched dynamically from their API, not stored in ml_model_list.py. The script checks the API's supervisedLoraTunable and supervisedFullParameterTunable fields (not the old tunable field, which is stale) and cross-references against our allowlist.

uv run python3 .agents/skills/kiln-check-finetune-deprecation/scripts/check_finetune.py fireworks > /tmp/kiln_finetune_fireworks.json

This script:

1. Fetches all models with supervisedLoraTunable=True or supervisedFullParameterTunable=True from api.fireworks.ai/v1/accounts/fireworks/models
2. Cross-references against FIREWORKS_SUPPORTED_FINETUNE_MODELS from libs/core/kiln_ai/adapters/fine_tune/fireworks_finetune.py — the same allowlist that filters the runtime fine-tune dropdown
3. Checks both directions:
   • Stale allowlist entries: models in our allowlist that the API no longer marks as supervised-tunable — these may need to be removed
   • Missing from allowlist: models the API marks as supervised-tunable but aren't in our allowlist — these are candidates to add

The canonical allowlist lives in fireworks_finetune.py and is shared between the runtime dropdown filter and this audit skill. There is a single place to update when Fireworks changes their supported models.

Interpreting results: The supervised API fields return ~43 models, but our allowlist intentionally filters to ~20 that match Fireworks' documented supported models. Seeing ~23 "models in API but NOT in allowlist" is normal — these are models Fireworks marks as technically tunable but doesn't officially document or support. Only flag these if a model appears in the Fireworks docs but is missing from our allowlist. The important direction is stale allowlist entries — models we ship that the API no longer marks as supervised-tunable.

Output: JSON to stdout, human summary to stderr.

---

Phase 3 – Report Findings

Read the JSON outputs and present findings in a clear table:

Fine-Tune Deprecation Audit Results
====================================

STATIC PROVIDERS (provider_finetune_id in ml_model_list.py):

  ✅ OpenAI: 5/5 found
     gpt-4.1-2025-04-14, gpt-4.1-mini-2025-04-14, gpt-4.1-nano-2025-04-14,
     gpt-4o-2024-08-06, gpt-4o-mini-2024-07-18

  ✅ Vertex AI: 2/2 found
     gemini-2.0-flash-001, gemini-2.0-flash-lite-001

  ❌ Together AI: 2/4 missing
     ✅ Qwen/Qwen2.5-72B-Instruct — found
     ✅ Qwen/Qwen2.5-14B-Instruct — found
     ❌ meta-llama/Meta-Llama-3.1-8B-Instruct-Reference — NOT FOUND
     ❌ meta-llama/Meta-Llama-3.1-70B-Instruct-Reference — NOT FOUND

  ⏭️ SKIPPED (no credentials):
     - vertex (VERTEX_PROJECT_ID not set)

DYNAMIC PROVIDER (Fireworks):

  ⚠️ 167 models marked tunable in API
  ⚠️ 15 models in known-good docs list
  ✅ 11 models in both API and docs
  ❌ 156 models in API but NOT in docs (likely stale)
     - accounts/fireworks/models/qwen2-72b-instruct
     - accounts/fireworks/models/code-llama-7b
     - ...
  ⚠️ 4 models in docs but NOT in API
     - accounts/fireworks/models/gemma-2-9b-it
     - ...

Note: The example above is illustrative. Actual model counts and entries will vary.

---

Phase 4 – Remediation

Based on findings, recommend specific actions:

For static entries (provider_finetune_id):

• If a model's fine-tune ID is no longer valid, either:
  • Update the provider_finetune_id to a newer version if one exists
  • Remove the provider_finetune_id field to stop offering fine-tuning for that model on that provider
  • Set deprecated=True on the provider entry if the model is fully gone

For Fireworks dynamic entries:

• The stale models come from Fireworks' API, not our code. Options:
  • Filter in our code — update fetch_fireworks_finetune_models() in finetune_api.py to filter against a known-good list
  • Report to Fireworks — flag the stale models to Fireworks support
  • Both — filter now, report later

Always ask the user to confirm before making any code changes.

---

Phase 5 – Verify

After any changes, run the relevant tests:

uv run python3 -m pytest app/desktop/studio_server/test_finetune_api.py -q

---

Checklist

• Env vars sourced from .env
• Vertex auth confirmed (gcloud auth print-access-token works, or skip Vertex)
• Static fine-tune IDs checked against provider APIs (Together, Vertex)
• Fireworks dynamic models cross-referenced against known-good list
• Findings reported to user with clear table
• User confirmed remediation approach
• Code changes made (if any)
• Tests pass after changes
• FIREWORKS_SUPPORTED_FINETUNE_MODELS in fireworks_finetune.py updated if docs have changed