manim-video-with-tts

Manim + Volcengine TTS Showcase Pipeline

Trigger Conditions

Use this skill when the user requests:

• Manim animation with Chinese voiceover / narration
• Math/tech explainer video where audio timing must match visuals
• "用 Manim 做解释视频，并加上中文旁白"
• "确保语音和画面节奏匹配"
• 3Blue1Brown-style video with TTS integration

Pipeline Overview

PLAN → TTS → CODE → DRAFT → PRODUCTION → STITCH → MUX → DELIVER

Prerequisites

• Python 3.10+, Manim Community v0.20+
• ffmpeg
• Volcengine TTS credentials (VOLCENGINE_TTS_ACCESS_TOKEN, VOLCENGINE_TTS_APP_ID, VOLCENGINE_TTS_VOICE_TYPE_*)
• lark-cli for Feishu upload
• Web server directory /var/www/hermes.aigc.green/media/

Step 1: Plan

Write plan.md before any code. Include:

1. Narrative Arc: Core question → Aha moment → Emotional curve
2. Scene Breakdown: One table row per scene with timing, visuals, animation type
3. Voiceover Script: Chinese text per scene, with estimated character count
4. Color Palette: Background, target curve, approximation, text, accent, grid
5. Typography: Monospace font only (FreeMono/DejaVu Sans Mono), minimum size 18

Rule: Scene count = audio segment count. Each scene gets exactly one TTS clip.

Step 2: Generate TTS & Measure Durations

Generate one MP3 per scene using Volcengine TTS:

volc-tts --text '第N段旁白' --voice female --output ./audio/s0N.mp3

Then measure exact durations:

for f in audio/*.mp3; do
  dur=$(ffprobe -v error -show_entries format=duration -of csv=p=0 "$f")
  echo "$f: ${dur}s"
done

Record these durations — they are the master clock for all Manim timing.

Step 3: Write Manim Code

Sync Strategy: Audio-Driven Timing

Since Manim does not natively support Volcengine TTS, use the manual sync approach:

# In each Scene.construct(), the sum of:
#   all run_time values + all self.wait() values
# must equal the corresponding audio clip duration.

Cross-Scene Continuity

To avoid rebuilding axes/curves every scene (wasting precious audio time):

class S3_NextScene(Scene):
    def construct(self):
        self.camera.background_color = BG
        axes = Axes(...)
        sin_curve = axes.plot(...)
        prev_curve = axes.plot(...)
        
        # Carry over from previous scene — instant, costs 0s
        self.add(axes, sin_curve, prev_curve)
        
        # Only animate NEW elements
        self.play(ReplacementTransform(prev_curve, next_curve), run_time=2.0)
        self.wait(...)
        # ...

Critical: The last frame of Scene N must match the first frame of Scene N+1 for seamless ffmpeg concatenation. If Scene N fades out an element, Scene N+1 must not self.add() it.

Duration Budgeting Template

For a scene with audio duration D:

D = Σ(run_times) + Σ(waits)

Recommended allocation:

• Title/text appearance: 0.8–1.2s
• Curve drawing/creation: 1.5–2.5s
• Transform/morph: 1.5–2.0s
• Formula writing: 0.8–1.0s
• Indicator/effect: 0.5–0.8s
• FadeOut cleanup: 0.3–0.5s
• Breathing waits: distribute remainder

Color Constants (High-Contrast Showcase)

BG          = "#0A0A0F"      # deep void black
SIN_COLOR   = "#FF4D4D"      # coral red — target curve
APPROX_COLOR= "#00F5FF"      # electric cyan — approximation
TEXT_COLOR  = "#FFFFFF"      # pure white
ACCENT      = "#FFD93D"      # golden yellow
GRID_COLOR  = "#2A2A3A"      # dim purple-grey
MONO        = "FreeMono"     # or "DejaVu Sans Mono"

Step 4: Draft Render

manim -ql script.py Scene1 Scene2 Scene3 ...

Verify each clip duration matches audio:

ffprobe -v error -show_entries format=duration -of csv=p=0 media/videos/script/480p15/Scene1.mp4

If mismatch > 0.1s, adjust run_time or self.wait() in code and re-render.

Step 5: Production Render

manim -qh script.py Scene1 Scene2 Scene3 ...

Step 6: Stitch & Mux

# Video concat
cat > concat_video.txt << 'EOF'
file 'media/videos/script/1080p60/Scene1.mp4'
file 'media/videos/script/1080p60/Scene2.mp4'
...
EOF
ffmpeg -y -f concat -safe 0 -i concat_video.txt -c copy video_pro.mp4

# Audio concat
cat > concat_audio.txt << 'EOF'
file 'audio/s01.mp3'
file 'audio/s02.mp3'
...
EOF
ffmpeg -y -f concat -safe 0 -i concat_audio.txt -c copy audio_full.mp3

# Mux
ffmpeg -y -i video_pro.mp4 -i audio_full.mp3 -c:v copy -c:a aac -b:a 192k final.mp4

Step 7: Deliver

Web Server (Primary Link)

cp final.mp4 /var/www/hermes.aigc.green/media/<filename>.mp4
# URL: https://hermes.aigc.green/media/<filename>.mp4

Feishu Drive (Backup)

cd <project-dir>
lark-cli drive +upload --file ./final.mp4 --name "<Title>.mp4"
lark-cli drive metas batch_query --data '{"request_docs":[{"doc_token":"<TOKEN>","doc_type":"file"}],"with_url":true}'

Extract url from response for Feishu direct link.

Performance Targets

Audience Adaptation: Two Dimensions

When adapting a technical explainer, distinguish two independent dimensions:

1. Audience age: Adult → Kids (cognitive complexity)
2. Locale: Global English → China-localized (language & cultural context)

These require separate rewrites. Do not try to make one version serve both purposes.

Dimension 1: Age Adaptation (Kids vs Adult)

When adapting for younger audiences (elementary/middle school), do not tweak—rewrite entirely. The kids version requires new plan, new script, new TTS, and new code.

Kids-Optimized Color Palette:

BG          = "#0D1117"      # deep void (same)
SIN_COLOR   = "#FF6B6B"      # coral red — warmer, less aggressive
APPROX_COLOR= "#4ECDC4"      # mint cyan — softer than electric
TEXT_COLOR  = "#F0F6FC"      # off-white, easier on eyes
ACCENT      = "#FFE66D"      # golden yellow — magic/block highlights
GRID_COLOR  = "#30363D"      # dim grey
MONO        = "FreeMono"

Kids Script Guidelines:

• "展开" / "expansion" → "搭积木" (building blocks)
• "阶数" / "order" → "第几块积木" (which block)
• "导数" / "derivative" → "斜率" (slope) or "弯的程度" (how much it bends)
• "收敛" / "convergence" → "越来越像" (more and more alike)
• "假装" (pretend) — the straight line is "pretending" to be the curve
• "骗过眼睛" (fool the eye) — technical accuracy achieved through deception
• "双胞胎" (twins) — when two curves become indistinguishable
• "弯腰" (bend at the waist) — cubic term makes the line "bend"

Timing Adjustments for Kids:

# Adult version
self.wait(0.3)  # between phrases

# Kids version  
self.wait(0.5)  # longer breathing room

Add suspense pauses before reveals: self.wait(1.0) before "Here's the secret..."

Dimension 2: China Localization (中英双语注释)

When the audience is Chinese students, apply the Bilingual Annotation Principle:

Every English mathematical term must appear with its Chinese equivalent on first use.

This applies to both narration and on-screen text.

Bilingual Annotation Rules

Critical: The mathematical notation (MathTex) on screen should never be removed or translated. The formula y = x, x - x³/6, etc. must remain in standard mathematical notation. Only the narration and text labels are localized.

China-Localized On-Screen Text Mapping

Three-Version Iteration Pattern

In practice, producing a China-localized kids version requires three iterations:

Workflow for V3:

# 1. Create entirely new plan (do not reuse V1/V2 plan)
cat > plan-kids-cn.md << 'EOF'
## Audience: Chinese Elementary/Early Middle School
## Principle: Bilingual annotation (Chinese first, English in parens)
## Labels: 100% Chinese
## Voice: Female Chinese TTS
EOF

# 2. Write Chinese narration with bilingual annotations
# Example: "你有没有想过，计算器是怎么算出正弦（sine）的？"

# 3. Generate TTS with female voice
volc-tts --text '...' --voice female --output ./audio-kids-cn/s01.mp3

# 4. Create new script file with Chinese labels
# script-kids-cn.py — all Text() objects use Chinese strings

# 5. Render, stitch, deliver separately

Example: China-Localized Script Evolution

Pitfall: Symbol Reading in Chinese TTS

Chinese TTS cannot naturally read mathematical symbols. Never write:

• ❌ volc-tts --text 'x³/6' — TTS will say random characters or fail
• ❌ volc-tts --text 'y=x' — TTS may read as "y dengyu x" (awkward)

Always write the spoken Chinese description:

• ✅ volc-tts --text '减去 x 的立方除以六' — natural Chinese
• ✅ volc-tts --text 'y 等于 x' — natural Chinese

The screen still shows x - x³/6 and y=x via MathTex. The narration describes them in Chinese. This dual-channel approach (visual: standard notation, audio: Chinese description) is the core of bilingual annotation.

Workflow for Kids Version

# 1. Save adult plan as baseline
cp plan.md plan-adult.md

# 2. Create entirely new plan
cat > plan-kids.md << 'EOF'
# Math Magic: How Calculators Know Sine
## Audience: Elementary/Early Middle School
## Metaphor: Building Blocks
## Characters: 小红 (target), 小蓝 (building)
EOF

# 3. Generate new TTS with female voice
volc-tts --text '...' --voice female --output ./audio-kids/s01.mp3

# 4. Create new script file (do not reuse adult script)
# script-kids.py with adjusted timings and softer colors

# 5. Render, stitch, deliver separately from adult version

Critical Pitfalls

1. Font not found: Always use FreeMono or DejaVu Sans Mono. Menlo is macOS-only.
2. SurroundingRectangle on point: SurroundingRectangle requires a Mobject, not a coordinate. Use Rectangle(...).move_to(point) instead.
3. Frame quantization at 15fps: Draft (-ql) can have up to ~0.07s timing drift per animation. Production (-qh) at 60fps reduces this to ~0.017s.
4. Audio shorter than video: If sum(video) > sum(audio), the final clip will have silent tail. Acceptable if < 0.5s. If larger, trim video with ffmpeg or adjust waits.
5. Scene transition flash: If Scene N fades out elements that Scene N+1 self.add()s, ensure the last frame of N matches the first frame of N+1. Otherwise a "flash" occurs at the stitch point.
6. Polynomial explosion: Taylor polynomials diverge outside the convergence radius. Set y_range wide enough to show the "shrinking" effect visually, or clip x_range to safe bounds.
7. Kids version feels rushed: If adapting for kids, increase all waits by 50%. Children's processing speed is slower; they need time to connect the metaphor to the math.

Sync Quality Checklist

• Each scene's run_time + wait sum matches audio duration within 0.1s
• Production render durations verified with ffprobe
• No visual flash at scene stitch points
• Audio concat duration matches video concat duration within 0.5s
• Final mux outputs without -shortest truncation
• Both Feishu + Web links tested and accessible