carocut-media-audio

Audio Assets

Generates voiceovers via TTS, retrieves background music and sound effects. Creates durations.json for timing synchronization in Remotion.

Audio Types

---

Voiceover Generation

Generate speech audio from script using Edge TTS.

TTS Batch Processing Workflow

Processes entire script.md and generates all voiceover files:

1. Parse {project_dir}/manifests/script.md for all [VO_XXX] markers and their text content
2. For each VO segment, invoke Edge TTS with the configured voice and speed
3. Save each result as VO_XXX.wav in {project_dir}/raws/audio/vo/
4. After all segments complete, extract durations to create {project_dir}/manifests/durations.json

Voice Selection

完整 voice 列表和选择建议见 references/edge-tts.md。

Speed Adjustment Rules

• The default TTS voice may sound slow. Consider using speed 1.2 for more natural pacing.
• Speed range: 0.5 (half speed) to 2.0 (double speed)
• Recommended: 1.0 - 1.3 for narration, 0.8 - 1.0 for emotional/dramatic content

TTS Speed and Storyboard Pacing (语速与节奏对应)

When generating TTS, read the corresponding shot's pacing field from the storyboard to determine speed:

Rules:

• If the storyboard does not specify pacing for a shot, default to medium (speed 1.2)
• For pause pacing shots, skip TTS generation entirely — these are breathing shots with no voiceover
• The pacing value is per-shot; different VO segments in the same video may have different speeds
• After applying pacing-based speed, duration extraction must reflect the actual playback length at that speed

---

Duration Extraction

Extract audio durations for timing synchronization. This step is mandatory.

Output Format (durations.json)

{
  "VO_001": 3450,
  "VO_002": 2800,
  "VO_003": 4200,
  "VO_004": 3100
}

All durations are in milliseconds. This file is critical for shot timing in Remotion.

Duration Rules

• All values MUST be in milliseconds (integer)
• Every [VO_XXX] marker in script.md must have a corresponding entry
• Duration reflects the actual audio length at the configured speed (e.g., 1.2x)
• Always re-extract durations after any audio modification

durations.json and Audio-Visual Relation

The durations.json format itself remains unchanged (simple VO_XXX: milliseconds mapping). However, the Builder's timing.ts module uses the storyboard's audio_visual_relation field in combination with durations to calculate actual audio start frames:

The audioOffsetFrames function in timing.ts reads these values to apply frame-level offsets. The Audio agent does not need to modify durations.json for this — it is handled downstream by the Builder.

重要依赖说明：虽然 durations.json 格式简单（VO_XXX: 毫秒数），但 Builder 在计算实际音频起始帧时，会同时读取 storyboard.yaml 中对应 shot 的 audio_visual_relation 字段。Audio agent 只需确保 durations.json 中的时长准确；偏移量由 Builder 的 timing.ts → audioOffsetFrames() 函数自动计算。

---

BGM and SFX Retrieval

Download royalty-free audio from Freesound.

BGM Search Strategy

• Search for loops or long-form tracks matching the video mood
• Minimum duration: 60 seconds (to cover full video length)
• Keywords should describe mood, genre, and instrumentation

BGM Rhythm Matching (BGM 节奏匹配)

Select BGM tempo based on the dominant pacing of each chapter/section in the storyboard:

Rules:

• Determine a chapter's dominant pacing by the most frequent pacing value among its shots
• If a video has distinct chapters with different pacing, consider using multiple BGM tracks with crossfade transitions
• For breathing shots between chapters, use ambient/atmospheric sounds or fade the BGM to bridge the transition
• When searching Freesound, include BPM range in keywords (e.g., "ambient piano 70bpm", "upbeat electronic 130bpm")
• BGM should complement, not compete with, the voiceover. Lower BPM tracks naturally sit better under narration.

SFX Search Strategy

• Search for short, punchy sound effects for transitions and UI feedback
• Maximum duration: typically 3-5 seconds
• Keywords should describe the exact sound needed

Search Keywords Tips

License Information

For CC-BY sounds, record attribution in raws/audio/LICENSES.txt.

---

Output Structure

raws/
  audio/
    vo/
      VO_001.wav
      VO_002.wav
      ...
      durations.json       # Critical timing data
    bgm/
      ambient_loop.mp3
    sfx/
      whoosh.mp3
      notification.mp3
    LICENSES.txt           # Attribution for CC-BY sounds

---

Workflow

1. Generate Voiceovers

   • Process manifests/script.md for all [VO_XXX] markers
   • All [VO_XXX] markers become VO_XXX.wav files
   • Extract durations to create durations.json

2. Retrieve BGM

   • Search for appropriate background music
   • Download to raws/audio/bgm/

3. Retrieve SFX

   • Search for transition sounds, UI sounds
   • Download to raws/audio/sfx/

4. Verify and Report

   • Check all files generated correctly
   • Report results to user

---

Incremental Mode

当 dispatch context 中 mode: "incremental" 时：

输入

• amendment.affected_resources 中指定的 VO 段落 ID 列表（需要重新生成的语音片段）

执行规则

1. 读取 script.md，仅定位指定的 VO 段落
2. 仅对指定的 VO 段落重新生成 TTS 音频
3. 使用与原始生成相同的 character、tone、speed 配置
4. 替换 raws/audio/vo/ 中对应的 WAV 文件
5. 必须重新提取完整的 durations.json（因为任何音频变化都会影响时序）
6. 返回处理摘要（重新生成了哪些 VO、新旧时长对比）

不执行

• 不重新生成未指定的 VO 音频
• 不重新检索 BGM/SFX（除非 amendment 中明确要求）

关键注意

• 即使只修改了一个 VO 片段，也必须重新提取整个 durations.json
• 时长变化可能影响下游的 Remotion 时序计算

---

Audio Timing Rules

These rules are critical for correct Remotion synchronization:

---

Network Requirements

Edge TTS requires outbound internet access and the Python package edge-tts.

---

User Communication

Progress Report

音频素材生成进行中...

TTS 生成:
  进度: 15/45 条语音
  当前: VO_016
  音色: zh-CN-XiaoxiaoNeural

Completion Report

音频素材生成完成。

语音配音:
  - 45 条语音文件
  - 总时长: 4分23秒
  - 音色: zh-CN-XiaoxiaoNeural
  - 时长数据: raws/audio/vo/durations.json

背景音乐:
  - 1 个循环音频 (2分30秒)

音效:
  - 3 个过渡音效
  - 2 个提示音

输出目录: raws/audio/
请确认后继续下一步。

---

Error Handling

---

Critical Notes

Duration File

durations.json is mandatory for Remotion implementation. It contains timing data in milliseconds that drives:

• Shot duration calculations
• Audio layer positioning
• Voiceover synchronization

Always re-extract durations after any audio modification.

Default Voice Speed

The default TTS voice may sound slow. Consider using speed 1.2 for more natural pacing. When the storyboard provides a pacing field, use the TTS Speed and Storyboard Pacing table above to determine the appropriate speed for each VO segment.