local-llm

Local LLM Management — M5 Max 128GB

Manage local LLM inference on Apple Silicon. Covers model selection, server management, harness configuration, and performance optimization.

Hardware Context

• M5 Max 128GB: 614 GB/s bandwidth, 40-core GPU with Neural Accelerators, ~90-100 GB usable for models
• LLM inference is memory-bandwidth-bound — MoE models are the sweet spot (small active params, full knowledge)
• Neural Accelerators deliver 3.3-4x faster prefill vs M4 — long-context agent workflows finally work

Quick Commands

# Start a model with Ollama (MLX backend auto-enabled on Apple Silicon 32GB+)
ollama run qwen3.5:27b
ollama run glm-4.7-flash

# Launch a coding agent with local model
ollama launch claude --model gemma4:31b
ollama launch opencode --model gemma4:26b

# Start MLX server (fastest, OpenAI-compatible)
mlx_lm.server --model mlx-community/Qwen3.5-35B-A3B-4bit --port 8080

# Preferred: Qwen3.6-35B-A3B via llama.cpp (BF16 lossless, 256K ctx, vision)
# -hf auto-downloads the sibling mmproj; do NOT pass --mmproj-url with an HF shorthand (needs https://)
llama-server -hf unsloth/Qwen3.6-35B-A3B-GGUF:BF16 \
  -c 262144 --cache-type-k bf16 --cache-type-v bf16 \
  --jinja --reasoning off \
  --temp 0.7 --top-p 0.8 --top-k 20 --min-p 0.0 \
  -ngl 99 -fa on --host 127.0.0.1 --port 8080

# Start llama.cpp server (max quant control)
llama-server -hf unsloth/gemma-4-26B-A4B-it-GGUF:UD-Q4_K_XL -c 65536 --port 8080

# Check what's running
ollama ps
curl -s http://localhost:11434/api/ps | jq

Model Selection

Preferred default: Qwen3.6-35B-A3B. 35B total / 3B active MoE. 256K native context (1M via YaRN). Multimodal (vision). Rare property at 128GB: runs at BF16 (69 GB) losslessly with headroom for long context. Use Unsloth GGUFs via llama.cpp or MLX — Ollama is currently broken for this model (mmproj vision files not handled). Thinking mode is on by default; disable with --chat-template-kwargs '{"enable_thinking":false}' for agentic work.

Dual-model daily driver setup (~66 GB, 52 GB free):

• Qwen3.6-35B-A3B UD-Q5_K_XL for primary reasoning + vision (~27 GB)
• GLM-4.7-Flash Q8 for fast agentic coding (~38 GB)

Inference Engine Decision

Default to Ollama for agent workflows. Use raw MLX for max throughput. Use llama.cpp for exotic quants or models not yet in MLX format.

Coding Agent Harnesses

Performance Tuning

• Context window: Ollama defaults to 4096. Create a Modelfile with num_ctx 65536 for agentic work
• Thinking mode: /set nothink in Ollama for faster responses in agent loops
• Strip <think> blocks from multi-turn history — bloats context, hurts quality
• MoE loads all experts into RAM regardless of active params (4-bit Qwen3.5-35B-A3B = ~20 GB resident)
• Temperature: 0.6 for coding, 0.7 for instruct, 1.0 for thinking mode

Quantization Quick Reference

Rule: Use the highest quant that fits with room for your desired context length.

Troubleshooting

• Slow prefill? Check Activity Monitor for memory pressure / thermal throttling
• Tool calls failing? Usually the harness, not the model — try a different harness
• Model not found in Ollama? ollama list to verify, ollama pull <model> to download
• KV cache invalidation in Claude Code? Set CLAUDE_CODE_ATTRIBUTION_HEADER=0
• < 80 t/s on MoE models? Something is wrong — check for other Metal consumers

Reference Files

Detailed docs for each inference engine, harness, and hardware analysis:

References: ollama-api References: mlx-lm References: llama-cpp References: hermes-agent References: hardware References: harnesses References: models