my-llm-api

MyLLMAPI

A two-surface LLM gateway:

• Raw — complete / embed / models. You pick a self-hosted catalog model (run myapi llm models for the live list). Proprietary models are not callable here.
• Verbs — classify / extract / summarize / draft. You ask for a task done; the model is implementation detail and is never named in the response.

Pricing is in cents per 1M tokens at the actual upstream rate; cost is debited from your MyAPI balance.

Capabilities

Use this for workflow tasks — summarize a doc, classify an inbound email, extract structured data, draft a reply. The reply goes to stdout; a one-line usage footer (tokens + cost + finish reason) goes to stderr, so myapi llm complete ... | jq and similar pipelines work as expected.

Don't use this as your own model. If you're an agent reading this, you already have a more capable model than what's exposed here. Reach for the LLM verbs when you're scripting a recurring step where a small/cheap model is the right tool — not for one-shot reasoning you can just do yourself.

Raw vs. verbs

• Raw complete — full control: pick the model, build the messages array, set max_tokens/temperature/stop. Use when shape matters.
• Verbs — you want a result (label, JSON, summary, draft). Use when you don't care which model runs underneath.

Raw complete request

{
  "model": "<model-id>",
  "messages": [
    { "role": "system", "content": "You are a terse editor." },
    { "role": "user",   "content": "Tighten this paragraph: ..." }
  ],
  "max_tokens": 200,
  "temperature": 0.2,
  "stop": ["\n\n"]
}

Roles: system | user | assistant. Multiple system messages collapse to one instruction. max_tokens, temperature, and stop are optional — per-model defaults apply.

Raw complete response

{
  "model": "<model-id>",
  "content": "...assistant reply...",
  "finish_reason": "stop",
  "usage": { "input_tokens": 42, "output_tokens": 87, "cost_cents": 0.029 }
}

finish_reason is one of stop (normal), length (hit max_tokens), filter (blocked).

If model isn't in the self-hosted catalog the server returns MODEL_NOT_IN_RAW_CATALOG — that's the signal to use a verb instead, not to retry with a different --model.

Raw embed

Embed text into a dense vector. --model is optional when the catalog serves exactly one embed model; else pass one from myapi llm models --kind embed. Returns EMBED_NOT_AVAILABLE when none is served.

Model catalog

• Chat models: id, kind: 'chat', context_window, input_cost_per_1m_cents, output_cost_per_1m_cents
• Embed models: id, kind: 'embed', dimensions, input_cost_per_1m_cents

The catalog is live — it reflects what the inference gateway actually serves, refreshed every 15 minutes. Always query models rather than hard-coding ids.

Verb requests + responses

Every verb takes an optional tier (fast | reasoning | cheap) — opaque routing hint, server picks the model. Response usage block is identical across verbs.

Shared usage block on every verb:

{ "tier_used": "fast", "tokens_in": 65, "tokens_out": 37, "cost_cents": 0.005 }

The model/provider is never named in the verb response. That's the point — the verb is the contract, the model is implementation.

OpenAI-compatible drop-in

POST /llm/orgs/{org_id}/chat/completions (and /v1/chat/completions alias) accepts the OpenAI request shape and returns the OpenAI response shape — no envelope. Same catalog and pricing as raw complete. Use it when existing OpenAI SDK / LangChain / any base_url-configurable tooling should point at MyAPI without rewriting.

from openai import OpenAI
client = OpenAI(
    api_key="hq_live_…",
    base_url="https://api.myapihq.com/llm/orgs/<org_id>/v1",
)
r = client.chat.completions.create(model="<model-id>",
    messages=[{"role":"user","content":"Hi"}])

Use raw complete for first-party code; the compat path for existing OpenAI/LangChain tooling.

Commands

Pass - as the prompt/input to read from stdin. Pass --file <path> to read longer content from disk.

Examples

# List the live catalog
myapi llm models
myapi llm models --kind chat --json | jq '.models[].id'

# Raw completion — picks the first chat model from the catalog
myapi llm complete "Summarize in 12 words: $(cat README.md)"

# Pin a specific model (ids come from `myapi llm models`)
myapi llm complete "Refactor this function: ..." \
  --model <model-id> \
  --system "You are a careful Go reviewer." \
  --max-tokens 600

# ── Verbs (recommended for workflow steps) ──────────────────────────────

myapi llm classify "I was charged twice — please refund." \
  --labels billing,technical,sales,spam

myapi llm extract "Acme Corp employs 250 people in Berlin." \
  --schema '{"type":"object","properties":{"company":{"type":"string"},"employees":{"type":"integer"}}}'

myapi llm summarize --file long-thread.txt --style bullet

myapi llm draft --kind email \
  --prompt "Friendly welcome, under 60 words." \
  --context '{"recipient":"a new signup","product":"MyAPI"}'

# Classify + route an inbound webhook delivery
BODY=$(myapi webhook delivery <id> --json | jq -r '.body')
INTENT=$(printf '%s' "$BODY" | myapi llm classify - \
  --labels support,sales,spam --json | jq -r '.data.label')

Notes

• draft context safety. context fields are quoted into the prompt verbatim; sensitive-named keys (secret, api_key, password, …) are NOT redacted. Two guards on top: (a) injection-defense strips instructions/system/prompt/override keys and surfaces them in meta.warnings; (b) output guardrail substring-scans fact values (length ≥ 4) in the response and lists matches in meta.guardrails.facts_in_output (signal, not redaction). Rule of thumb: never put credentials, PII, or internal metadata in context — pass identifiers, reference them indirectly.
• Self-hosted raw, server-picked verbs. Raw runs on MyAPI's TPU; verbs route wherever the server picks.
• Cost + latency. usage.cost_cents is authoritative — no markup. Varies by tier: 200–600 ms to first token, 1–3 s end-to-end.
• Live catalog, no streaming, no BYOK. Don't hard-code ids — models is truth (CLI auto-picks if --model omitted). Full reply only.