Add Model 03 Port Dit
Developmentadd-model-03-port-ditAdd Model 03 Port Dit is an Agent Skill for development published in the hao-ai-lab/FastVideo repository on GitHub: a SKILL.md instruction file an AI agent can load to do this task.
Use during /add-model Phase 4 or Phase 6 to prototype or parity-debug one FastVideo-native DiT/transformer component.
How to use this skill
Bring this guide into your coding agent with a prompt tailored to the tool you use.
- Open your project in Codex.
- Copy and paste the prompt below.
- Review files and risks before approving any action.
Inspect the Agent Skill at https://github.com/hao-ai-lab/FastVideo/blob/HEAD/.agents/skills/add-model-03-port-dit/SKILL.md and all supporting files. Report provenance, permissions, scripts, dependencies, and security risks. Propose an exact installation into .agents/skills/add-model-03-port-dit, but wait for my approval before writing files or running scripts.
Copying this prompt does not install or run the skill. Review third-party files before use.
Add Model Port DiT
Goal
Prototype or parity-debug one diffusion transformer in FastVideo-native code. This skill is for one component only; do not work on the VAE, encoders, pipeline, or unrelated conversion code unless the current component cannot load without a minimal fix there.
Inputs
Follow ../add-model/shared/component_skill_common.md and require the complete
packet from ../add-model/contracts/component_context.md.
DiT-specific packet fields:
component: transformer or DiT name.parity_test:tests/local_tests/<bucket>/test_<family>_<component>_parity.py.weights: converted transformer dir or local official path.target_files:fastvideo/models/dits/<family>.pyandfastvideo/configs/models/dits/<family>.py.
Modes
Use the common prototype and parity-debug modes from
../add-model/shared/component_skill_common.md.
DiT-specific prototype concerns include ambiguous official flags, shape mismatches, missing FastVideo layer equivalents, and dedicated output heads.
Reuse Proof
Apply the shared reuse proof. DiT-specific comparison must include attention algorithm, positional embeddings, RoPE/patching, timestep/guidance embeddings, scaling constants, dtype casts, state-dict names, and every output head.
Existing FastVideo Patterns
- Base class:
fastvideo/models/dits/base.py::BaseDiT. - Config bases:
DiTConfigandDiTArchConfiginfastvideo/configs/models/dits/base.py. - Use the matching DiT config bucket. Wrong bucket inheritance can typecheck but fail during pipeline wiring.
- Config export: add the config to
fastvideo/configs/models/dits/__init__.py. - Registry discovery: set
EntryClass = <ClassName>in the model file. - Loader path:
TransformerLoaderreadstransformer/config.json, callsdit_config.update_model_arch(config), resolves_class_namethroughModelRegistry, and constructs the class withconfigandhf_config. - Reference examples:
stable_audio.py,wanvideo.py,sd3.py,longcat.py, andltx2.py. - Layer guidance:
fastvideo/layers/AGENTS.md.
Implementation Rules
- Use FastVideo-native layers by default:
ReplicatedLinearfor DiT hot-path linears,DistributedAttentionfor standard full-sequence attention, andLocalAttentionfor local/window attention or simple single-GPU parity paths. - Raw SDPA is acceptable for cross-modality flat streams when no FastVideo distributed equivalent exists; document the SP gap in the module docstring.
- Mirror official tensor contracts exactly: latent packing, patch ordering, timestep embedding scale, RoPE/positional embedding, guidance embedding, cross-attention context order, output head order, and dtype casts.
- Preserve all output heads that the official DiT emits. Do not silently drop audio, depth, pose, mask, or auxiliary heads.
- Put architecture fields on
DiTArchConfig; keep inference steps, CFG scales, FPS, flow shift, and sampling defaults out of the arch config. - Define
_fsdp_shard_conditions,_compile_conditions,param_names_mapping, andreverse_param_names_mappingwhere needed. - Follow the production import boundary in
../add-model/shared/common_rules.md.
Prototype Checks
Follow the shared prototype success criteria. A useful one-off check is:
python - <<'PY'
# Import the target config/class, instantiate with random weights, and print
# state_dict names/shapes for the conversion mapping.
PY
Parity-Debug Loop
Run the shared parity-debug loop. The component test command is:
pytest <parity_test> -v -s
For numerical drift, use ../add-model-08-trace/SKILL.md before writing bespoke
hooks. Start with FastVideo's activation trace (fastvideo/hooks/activation_trace.py;
docs/contributing/activation_trace.md) and a block-level regex such as
FASTVIDEO_TRACE_LAYERS="^block\.layers\.[0-9]+$". Only fall back to custom
per-block hooks if the needed boundary or statistic is not exposed by
FASTVIDEO_TRACE_STATS.
Escape Hatches
Follow ../add-model/shared/common_rules.md and the component-specific guidance
in ../add-model/shared/component_skill_common.md. DiT-specific ask cases include
dropping an output head/modality, accepting an unsupported kernel/private op, or
choosing between incompatible official transformer definitions.
Handoff
Return ../add-model/contracts/component_skill_handoff.md following the common
handoff rules in ../add-model/shared/component_skill_common.md.