Fastllm Triton Ops
Developmentfastllm-triton-opsFastllm Triton Ops is an Agent Skill for development published in the ztxz16/fastllm repository on GitHub: a SKILL.md instruction file an AI agent can load to do this task.
Guide for adding Triton-backed CUDA operators to FastLLM. Use when modifying FastLLM CUDA op code to add, extend, debug, validate, or benchmark Triton-generated kernels through tools/fastllm_triton_server.py, src/devices/cuda/cudadevice.cpp, src/devices/cuda/fastllm-triton-cuda.cu, include/devices/cuda/fastllm-cuda.cuh, or related CMake wiring.
How to use this skill
Bring this guide into your coding agent with a prompt tailored to the tool you use.
- Open your project in Codex.
- Copy and paste the prompt below.
- Review files and risks before approving any action.
Inspect the Agent Skill at https://github.com/ztxz16/fastllm/blob/HEAD/.codex/skills/fastllm-triton-ops/SKILL.md and all supporting files. Report provenance, permissions, scripts, dependencies, and security risks. Propose an exact installation into .agents/skills/fastllm-triton-ops, but wait for my approval before writing files or running scripts.
Copying this prompt does not install or run the skill. Review third-party files before use.
FastLLM Triton Ops
Overview
Use the existing Linear Triton prototype as the reference architecture: C++ decides whether an op is eligible, starts a local Python compiler server only on demand, asks it to emit a cached cubin plus metadata, then launches that cubin through the CUDA Driver API. Every Triton path must be environment-gated and must fall back to the original CUDA implementation on unsupported inputs or compile/launch failure.
Workflow
-
Inspect the existing CUDA op path in
src/devices/cuda/cudadevice.cpp.- Find the op's
Reshape,CanRun,Run, and lower-level helper functions. - Identify the exact tensor layout, dtype combinations, shape variables, optional bias/scale tensors, and output aliasing behavior.
- Keep the original implementation as the fallback path.
- Find the op's
-
Add or extend the Python compile path in
tools/fastllm_triton_server.py.- Add a
@triton.jitkernel for the new op. - Add
<op>_cache_paths(payload)with a deterministic filename that includes op name, dtype/layout variants, SM arch, compile-time tile sizes, and feature flags. - Add
compile_<op>(payload)that validates payload fields, compiles withASTSource, writes.cubin, writes.jsonmetadata, and returns the metadata. - Extend
handle_compile(payload)by dispatching onpayload["op"]. - Keep
/healthand/compilestable; do not break existing"op": "linear"requests.
- Add a
-
Add a CUDA launch wrapper in
src/devices/cuda/fastllm-triton-cuda.cu.- Reuse
LoadTritonKernelfor cubin/module/function caching. - Add one
extern "C"wrapper per op, for exampleFastllmCudaTriton<Op>(...). - Use
FastllmCudaPrepareInput,FastllmCudaPrepareOutput,FastllmCudaFinishInput, andFastllmCudaFinishOutputwhen passingDatabuffers. - Match the Triton kernel argument order exactly, including Triton's hidden
global_scratchandprofile_scratchpointer arguments when needed by AOT metadata. - Return
falseon load or launch failure so the C++ caller can fall back.
- Reuse
-
Declare the wrapper in
include/devices/cuda/fastllm-cuda.cuh.- Keep the signature close to the CUDA fallback helper's shape arguments.
- Pass metadata fields needed for launch, such as
kernelName,shared,numWarps, and tile sizes.
-
Wire the op in
src/devices/cuda/cudadevice.cpp.- Add a small metadata struct for the op's
.jsonfields. - Reuse common helpers:
CudaTritonCacheDir,CudaTritonDataTypeName,CudaTritonHttpRequest, andCudaTritonEnsureServer. - Add
CudaTriton<Op>BaseName,CudaTritonRead<Op>Meta,CudaTritonRequest<Op>Kernel, andTryCudaTriton<Op>. - Gate with global
FASTLLM_CUDA_TRITON; add an op-specific override such asFASTLLM_CUDA_TRITON_<OP>=0. - Validate device, pointer presence, dtype, layout, shape, arch, and feature constraints before requesting a kernel.
- In the original op helper, call
TryCudaTriton<Op>(...)immediately before the original CUDA implementation.
- Add a small metadata struct for the op's
-
Update build wiring only when needed.
src/devices/cuda/fastllm-triton-cuda.cuis already inCMakeLists.txt.- If adding new files, update
CMakeLists.txtand link dependencies without changing unrelated targets.
Current Contract
The existing Triton infrastructure uses these environment variables:
FASTLLM_CUDA_TRITON=1: enable Triton-backed CUDA ops globally.FASTLLM_CUDA_TRITON_<OP>=0: disable one op while keeping the global flag on, for exampleFASTLLM_CUDA_TRITON_LINEAR=0.FASTLLM_CUDA_TRITON_CACHE_DIR: override cubin/json cache directory.FASTLLM_CUDA_TRITON_SERVER_HOST,FASTLLM_CUDA_TRITON_SERVER_PORT: choose compiler server endpoint.FASTLLM_CUDA_TRITON_PYTHON: choose Python interpreter.FASTLLM_CUDA_TRITON_SERVER_SCRIPT: choose server script path.FASTLLM_CUDA_TRITON_SERVER_LOG: choose compiler server log path.FASTLLM_CUDA_TRITON_SERVER_WAIT_MS: choose startup wait timeout.- Per-op tile knobs should use
FASTLLM_CUDA_TRITON_<OP>_<PARAM>, matching Linear'sBLOCK_M,BLOCK_N,BLOCK_K,NUM_WARPS, andNUM_STAGES.
The metadata JSON returned by the compiler server should include at least:
{
"ok": true,
"op": "op_name",
"cubin": "/path/to/kernel.cubin",
"kernel": "compiled_kernel_name",
"shared": 0,
"num_warps": 4
}
Add op-specific launch fields, such as tile sizes, only when the C++ launcher needs them.
C++ Pattern
Keep the C++ control flow shaped like this:
if (!CudaEnvFlagEnabled("FASTLLM_CUDA_TRITON")) {
return false;
}
const char *opEnv = std::getenv("FASTLLM_CUDA_TRITON_MYOP");
if (opEnv != nullptr && opEnv[0] != '\0' && !CudaEnvFlagEnabled("FASTLLM_CUDA_TRITON_MYOP")) {
return false;
}
if (!inputs_are_supported) {
return false;
}
Meta meta;
if (!ReadMeta(metaPath, meta)) {
if (!RequestKernel(..., meta)) {
return false;
}
}
return FastllmCudaTritonMyOp(meta.cubinPath.c_str(), meta.kernelName.c_str(), ...);
Do not throw or call ErrorInFastLLM from the Triton trial path unless the original CUDA path would also fail. Unsupported Triton cases should return false.
Validation
Run validation in layers:
- Build:
bash install.sh -DUSE_CUDA=ON
- Unit or op test, with Triton enabled and an isolated cache:
FASTLLM_CUDA_TRITON=1 \
FASTLLM_CUDA_TRITON_CACHE_DIR=/tmp/fastllm-triton-optest \
../optest --op linear --device cuda:0 --param batch=4 --param in=8 --param out=6
Adjust the optest command for the op being added.
- End-to-end server smoke test:
FASTLLM_CUDA_TRITON=1 \
FASTLLM_CUDA_TRITON_CACHE_DIR=/tmp/fastllm-triton-qwen \
ftllm server ~/hfmodels/Qwen3-8B/ --device cuda:0 --host 127.0.0.1 --port 18080 --tokens 8192 --hide_input
Send one non-streaming request to /v1/chat/completions and confirm it completes.
- Benchmark after warmup.
- Always exclude first compile/server startup from performance numbers.
- Compare against the same command without
FASTLLM_CUDA_TRITON. - Report token throughput and wall time; state whether the measurement is kernel-only or end-to-end server throughput.
Guardrails
- Keep Triton optional: default behavior must stay unchanged when
FASTLLM_CUDA_TRITONis unset. - Prefer compile keys that are independent of runtime shapes when the kernel supports dynamic dimensions.
- Include every compile-time specialization in the cache filename to prevent stale cubin reuse.
- Serialize compilation in the Python server with the existing lock unless proving concurrent compilation is safe.
- Keep C++ JSON parsing defensive; missing or invalid metadata should fall back.
- Clean up test servers and compiler-server processes after benchmarks.