mindspeed-mm-pipeline

MindSpeed-MM End-to-End Multimodal Training Pipeline

This Skill is the routing entry point for all MindSpeed-MM Skills. It determines the model type based on user intent, routes to the corresponding Skill, and provides a complete pipeline overview.

Model Type Router

User Intent → Model Type Detection → Target Skill
  "Train understanding model / VLM"       → mindspeed-mm-vlm
  "Train generative model / video / image" → mindspeed-mm-generative
  "Train omni model"                       → See examples/qwen2.5omni/README.md
  "Train speech / TTS model"               → See examples/whisper/ or examples/cosyvoice3/README.md

Routing Criteria:

Complete Workflow Overview

VLM Workflow

1. Environment Setup (mindspeed-mm-env-setup)
→ 2. Model Dependency Installation (mindspeed-mm-vlm Step 0)
→ 3. Weight Download + HF→MM Conversion (mindspeed-mm-weight-prep)
→ 4. Data Preprocessing (MLLM JSON)
→ 5. Training (pretrain_vlm.py)
→ 6. Inference Validation (inference_vlm.py)
→ 7. Evaluation (evaluate_vlm.py)
→ 8. Weight Export MM→HF (optional)

Inter-Stage Data Flow:

model_from_hf/Qwen2.5-VL-7B-Instruct/   ← Step 3 download
    ↓ mm-convert hf_to_mm
ckpt/mm_path/Qwen2.5-VL-7B-Instruct/    ← Step 3 output
    ↓ Used as the load path in model.json
    ↓
dataset/train.json + images/             ← Step 4 input (MLLM JSON format)
    ↓ Used directly, no binary preprocessing needed
    ↓
saved_ckpt/                              ← Step 5 output
    ↓ mm-convert mm_to_hf (optional)
model_from_hf/.../converted/             ← Step 8 output

Generative Model Workflow

1. Environment Setup (mindspeed-mm-env-setup)
→ 2. Model Dependency Installation (mindspeed-mm-generative Step 0)
→ 3. Weight Download + HF→MM Conversion (mindspeed-mm-weight-prep)
→ 4. Data Preprocessing (video/image + caption JSON)
→ 5. Feature Extraction (VAE + TextEncoder)  ← VLM does not have this step
→ 6. Training (pretrain_sora.py)
→ 7. Inference Generation (inference_sora.py)
→ 8. Weight Export MM→HF (optional)

Inter-Stage Data Flow:

weights/Wan-AI/Wan2.1-T2V-1.3B-Diffusers/  ← Step 3 download
    ↓ mm-convert WanConverter hf_to_mm
weights/.../transformer/                     ← Step 3 output (in-place conversion)
    ↓
dataset/videos/ + dataset/train.json         ← Step 4 input
    ↓ Feature extraction script
dataset/features/                            ← Step 5 output (VAE latents + text embeddings)
    ↓ Used as training data input
    ↓
saved_ckpt/                                  ← Step 6 output
    ↓ mm-convert WanConverter mm_to_hf (optional)
converted_weights/                           ← Step 8 output

Key difference between VLM and generative models: Generative models require an additional feature extraction step before training (VAE encodes video/images into latents, TextEncoder encodes text into embeddings). VLM does not have this step.

Full Model Index

Understanding Models (VLM)

Generative Models

Omni Models

Audio Models

Post-training

Entry Script Selection Rules

MindSpeed-MM has three entry script patterns:

1. Megatron-based unified entry: pretrain_vlm.py (VLM), pretrain_sora.py (generative) — most models use these
2. Megatron-based model-specific entry: pretrain_internvl.py, pretrain_deepseekvl.py, pretrain_whisper.py, pretrain_lumina.py — dedicated scripts for specific models
3. FSDP2-based entry: pretrain_transformers.py or mindspeed_mm/fsdp/train/trainer.py or mindspeed_mm/fsdp/tasks/<model>/train.py — newer models (Qwen3VL, Qwen3Omni, LTX2, CosyVoice3, Qwen3TTS, FunASR)

Always check the actual shell script in examples/<model_name>/ — do not assume from the model name.

New models should use the unified entry. Legacy models still use model-specific entries and are being migrated gradually.

Common Training Args Quick Reference

The following parameters apply to all model types. For full parameter descriptions, see references/common-args.md.

Parallelism Parameters

Batch and Sequence Parameters

Memory Optimization Parameters

Training Control Parameters

Docker Runtime

FSDP2 vs Megatron Backend Selection

MindSpeed-MM supports two distributed training backends:

Selection Guidelines:

• Use the Megatron backend for most scenarios (better documentation and examples)
• If the model's official examples provide FSDP2 configuration and fine-grained parallelism tuning is not needed, FSDP2 is an option
• FSDP2 uses --fsdp2-config-path to specify the configuration file, replacing Megatron's TP/PP/CP parameters

Parameter Consistency Rules

The following parameters must be consistent between weight conversion and training:

Inconsistent parameters will cause weight loading failures or shape mismatch errors.

Pre-flight Checklist

Verify each item before starting deployment:

• Docker container created with --privileged --ipc=host (or --shm-size=16g)
• Model-specific dependencies installed (diffusers version, decord for video models)
• No stale torchrun processes holding MASTER_PORT
• NPU available: python -c "import torch_npu; print(torch.npu.is_available())"
• CANN environment activated: npu-smi info
• MindSpeed-MM installed: pip show mindspeed-mm
• Megatron module copied: ls MindSpeed-MM/megatron/
• Model type determined (VLM / Generative / Omni / Audio)
• Target model and specs confirmed
• HF weights fully downloaded
• TP/PP configuration determined and documented
• Training data prepared

FAQ

Q: How do I determine which Skill to use?

Choose based on model type: use mindspeed-mm-vlm for VLM models, mindspeed-mm-generative for generative models. When in doubt, refer to the model index table above.

Q: What if different models have conflicting dependency versions?

MindSpeed-MM models have vastly different version requirements for transformers/diffusers/peft. It is strongly recommended to create a separate Docker container for each model. See the dependency conflict section in mindspeed-mm-env-setup.

Q: Where can I find training scripts and configurations for a specific model?

Example scripts and YAML configurations for each model are located in the MindSpeed-MM/examples/<model_name>/ directory.

Q: What is the difference between pretrain_vlm.py and pretrain_qwen2vl.py?

pretrain_vlm.py is the new unified entry point that differentiates models via YAML configuration. pretrain_qwen2vl.py is the legacy model-specific entry point. New models should use the unified entry; legacy models still use their dedicated entry points.

Q: Why do generative models need a feature extraction step?

Generative models (e.g., Wan, CogVideoX) do not directly ingest raw video/images during training. Instead, a VAE first encodes video into latent features, and a TextEncoder encodes text into embeddings. Training then loads these pre-extracted features directly. This avoids redundant encoding during training and significantly improves training efficiency.

Q: Training fails with Communication_Error_Bind_IP_Port

Stale process holding the port from a previous run. Kill zombie processes or change MASTER_PORT in the training script.

ps aux | grep torchrun | grep -v grep | awk '{print $2}' | xargs kill -9

Related Skills

• mindspeed-mm-env-setup - Base environment setup
• mindspeed-mm-weight-prep - Weight conversion (HF<->MM)
• mindspeed-mm-vlm - VLM model training (understanding)
• mindspeed-mm-generative - Generative model training (video/image)

Reference Resources

• Model Registry - Complete lookup table: model → backend, entry script, converter, feature extraction, requirements
• Common Training Args Reference - GPT_ARGS, MOE_ARGS, OUTPUT_ARGS, environment variables
• MindSpeed-MM Repository
• MindSpeed-MM Training Args Documentation