dust-llm

Adding Support for a New LLM Model

This skill guides you through adding support for a newly released LLM.

Quick Reference

Files to Modify

Prerequisites

Before adding, gather:

• Model ID: Exact provider identifier (e.g., gpt-4-turbo-2024-04-09)
• Context size: Total context window in tokens
• Pricing: Input/output cost per million tokens
• Capabilities: Vision, structured output, reasoning effort levels
• Tokenizer: Compatible tokenizer for token counting

Step-by-Step: Adding an OpenAI Model

Step 1: Add Model Configuration

IMPORTANT: Always verify model specs from the official provider docs before adding. Key properties to verify: context window, max output tokens, vision support, structured output support. Add the doc URL as a comment above the config:

• OpenAI: https://developers.openai.com/api/docs/models/{model-id}
• Anthropic: https://docs.anthropic.com/en/docs/about-claude/models/overview
• Google: https://ai.google.dev/gemini-api/docs/models
• Mistral: https://docs.mistral.ai/getting-started/models/models_overview/

Edit front/types/assistant/models/openai.ts:

export const GPT_4_TURBO_2024_04_09_MODEL_ID = "gpt-4-turbo-2024-04-09" as const;

// https://developers.openai.com/api/docs/models/gpt-4-turbo
export const GPT_4_TURBO_2024_04_09_MODEL_CONFIG: ModelConfigurationType = {
  providerId: "openai",
  modelId: GPT_4_TURBO_2024_04_09_MODEL_ID,
  displayName: "GPT 4 turbo",
  contextSize: 128_000,
  recommendedTopK: 32,
  recommendedExhaustiveTopK: 64,
  largeModel: true,
  description: "OpenAI's GPT 4 Turbo model for complex tasks (128k context).",
  shortDescription: "OpenAI's second best model.",
  isLegacy: false,
  isLatest: false,
  generationTokensCount: 2048,
  supportsVision: true,
  supportedReasoningEfforts: {
    none: true,
    light: true,
    medium: true,
    high: true,
  },
  defaultReasoningEffort: "none",
  supportsResponseFormat: false,
  tokenizer: { type: "tiktoken", base: "cl100k_base" },
};

Step 2: Add Pricing

IMPORTANT: Always verify pricing from the official provider page before adding:

• OpenAI: https://openai.com/api/pricing/
• Anthropic: https://www.anthropic.com/pricing#anthropic-api
• Google: https://ai.google.dev/pricing
• Mistral: https://mistral.ai/technology/#pricing

Add a comment with the source URL above each model's pricing entry.

Edit front/lib/api/assistant/token_pricing.ts:

const CURRENT_MODEL_PRICING: Record<BaseModelIdType, PricingEntry> = {
  // ... existing
  // https://openai.com/api/pricing/
  "gpt-4-turbo-2024-04-09": {
    input: 10.0,  // USD per million input tokens
    output: 30.0, // USD per million output tokens
    cache_read_input_tokens: 1.0,      // Optional: cached reads
    cache_creation_input_tokens: 12.5, // Optional: cache creation
  },
};

Step 3: Register in Central Registry

Edit front/types/assistant/models/models.ts:

export const MODEL_IDS = [
  // ... existing
  GPT_4_TURBO_2024_04_09_MODEL_ID,
] as const;

export const SUPPORTED_MODEL_CONFIGS: ModelConfigurationType[] = [
  // ... existing
  GPT_4_TURBO_2024_04_09_MODEL_CONFIG,
];

Step 4: Update Router Whitelist

Edit front/lib/api/llm/clients/openai/types.ts:

export const OPENAI_WHITELISTED_MODEL_IDS = [
  // ... existing
  GPT_4_TURBO_2024_04_09_MODEL_ID,
] as const;

Step 5: Update SDK Types

Edit sdks/js/src/types.ts:

const ModelLLMIdSchema = FlexibleEnumSchema<
  // ... existing
  | "gpt-4-turbo-2024-04-09"
>();

Step 6: Add to UI (Optional)

Edit front/components/providers/types.ts:

export const USED_MODEL_CONFIGS: readonly ModelConfig[] = [
  // ... existing
  GPT_4_TURBO_2024_04_09_MODEL_CONFIG,
] as const;

Step 7: Test (Mandatory)

Edit front/lib/api/llm/tests/llm.test.ts:

const MODELS = {
  // ... existing
  [GPT_4_TURBO_2024_04_09_MODEL_ID]: {
    runTest: true,  // Enable for testing
    providerId: "openai",
  },
};

Run test:

RUN_LLM_TEST=true npx vitest --config lib/api/llm/tests/vite.config.js lib/api/llm/tests/llm.test.ts --run

After test passes, set runTest: false to avoid expensive CI runs.

Adding Anthropic Models

Same pattern with Anthropic-specific files:

1. front/types/assistant/models/anthropic.ts - Add CLAUDE_X_MODEL_ID and config
2. front/lib/api/llm/clients/anthropic/types.ts - Add to ANTHROPIC_WHITELISTED_MODEL_IDS
3. front/types/assistant/models/models.ts - Register in central registry
4. front/lib/api/assistant/token_pricing.ts - Add pricing
5. sdks/js/src/types.ts - Update SDK types
6. Test and validate

Model Configuration Properties

Validation Checklist

• Model config added to provider file
• Pricing updated (input, output, cache if applicable)
• Registered in central registry (MODEL_IDS + SUPPORTED_MODEL_CONFIGS)
• Router whitelist updated
• SDK types updated
• UI config added (if needed)
• Integration test passes
• Test disabled after validation

Troubleshooting

Model not in UI: Check USED_MODEL_CONFIGS in front/components/providers/types.ts

API calls failing: Verify model ID matches provider's exact identifier, check router whitelist

Token counting errors: Validate context size and tokenizer configuration

Pricing issues: Ensure prices are per million tokens in USD

Reference

• See front/types/assistant/models/openai.ts and anthropic.ts for examples
• Provider docs: OpenAI, Anthropic, Google, Mistral