carocut-media-visual

Visual Assets

Retrieves stock images and generates custom images for video production. This step fulfills visual resource requirements defined in manifests/resources.yaml.

Decision Flow

digraph visual_decision {
    "Resource from resources.yaml" [shape=doublecircle];
    "source = existing?" [shape=diamond];
    "Already in raws/" [shape=box];
    "source = retrieve?" [shape=diamond];
    "Search stock images" [shape=box];
    "source = generate?" [shape=diamond];
    "Generate with AI" [shape=box];
    "Needs transparency?" [shape=diamond];
    "Remove background" [shape=box];
    "Save to raws/" [shape=box];
    "Update resources.yaml" [shape=doublecircle];

    "Resource from resources.yaml" -> "source = existing?";
    "source = existing?" -> "Already in raws/" [label="yes"];
    "source = existing?" -> "source = retrieve?" [label="no"];
    "source = retrieve?" -> "Search stock images" [label="yes"];
    "source = retrieve?" -> "source = generate?" [label="no"];
    "source = generate?" -> "Generate with AI" [label="yes"];
    "Search stock images" -> "Needs transparency?";
    "Generate with AI" -> "Needs transparency?";
    "Needs transparency?" -> "Remove background" [label="yes"];
    "Needs transparency?" -> "Save to raws/" [label="no"];
    "Remove background" -> "Save to raws/";
    "Already in raws/" -> "Update resources.yaml";
    "Save to raws/" -> "Update resources.yaml";
}

Resource Source Types

资源类型完整定义见 carocut-shared-schema skill。Media 只处理 visual 类型（image/video/sprite），不处理 data 和 components 类型。

Sprite Sheets (Recommended for Character Animation)

When the storyboard includes animated characters, mascots, or any element with multiple poses/actions, prefer generating sprite sheets over individual images. Sprite sheets provide:

• Consistent character appearance across all animation frames
• Smooth frame-by-frame animation in Remotion
• Efficient asset management (one file per character instead of many)

Video Resources (视频素材)

When the storyboard requires motion footage (product demos, b-roll, ambient shots), use type: video in resources.yaml:

- id: demo_recording
  type: video
  description: "产品演示录屏"
  source: retrieve
  search_keywords: ["product demo", "screen recording"]
  duration_range: "5-15s"

Video Search Strategy

• Use Pexels for STOCK video retrieval when using tool videos_search
• Search keywords follow the same English-keyword rules as image search
• Filter by duration range when specified in the resource definition
• Prefer HD (1080p) or higher resolution

Video Output

• Storage path: raws/videos/
• Preferred format: MP4
• Directory structure:

  raws/
    videos/
      retrieved/
        product_demo_001.mp4
        aerial_city_002.mp4
  

Video in resources.yaml

- id: vid_001
  type: video
  description: "Aerial city timelapse"
  source: retrieve
  search_keywords: ["aerial city", "timelapse", "urban"]
  duration_range: "5-10s"
  path: videos/retrieved/aerial_city_001.mp4

---

Image Retrieval (Stock Photos)

Search royalty-free images from Pexels and Pixabay.

Search Keywords Tips

Image Search Strategy

• Use English keywords for all searches
• Keep search result count low (3-5) to avoid context bloat
• Use specific keywords with style descriptors for better relevance
• When initial results are poor, expand with synonyms (e.g., "server room" -> "data center", "network infrastructure")
• Filter by orientation when the layout requires it (landscape for backgrounds, portrait for mobile)
• Filter by color when brand consistency matters

Visual Metaphor Search (视觉隐喻搜索)

When the storyboard specifies audio_visual_relation: counterpoint, the visual should not literally illustrate the narration. Instead, switch to metaphorical/associative search:

Rules:

• Only activate metaphor search when audio_visual_relation: counterpoint
• For sync or lead-visual, use literal/direct search as usual
• For lead-audio, visuals can be more abstract/atmospheric but not necessarily metaphorical
• When using metaphor search, pick 2-3 keywords from the table and combine with style descriptors
• If the narrative concept is not in the table, derive a visual metaphor by analogy

Framing-Aware Search (构图感知搜索)

Adjust search keywords based on the storyboard's framing field to find images matching the intended shot composition:

Rules:

• Append the framing keywords to the existing search query (do not replace)
• For MCU and MS, no framing keywords are needed
• Combine with metaphor search when applicable (e.g., counterpoint + ECU → "close up texture tangled wires")

License Information

---

Image Generation (AI)

Generate custom images via OpenAI-compatible API.

Prompt Rules

1. Write all prompts in English
2. For subjects needing transparency: Include "on a solid white background" or "on a plain white background"
3. For Chinese text in image: Use format sign that reads "Chinese text here"
4. Be specific: Describe composition, style, colors, lighting, perspective

Prompt Examples

Background image:

Abstract gradient background, soft blue to purple transition, subtle geometric patterns, clean modern aesthetic, suitable for tech presentation

Character with transparency:

A friendly 3D cartoon scientist character, female, wearing white lab coat and glasses, standing pose facing camera, on a plain white background, Pixar style rendering

Sprite sheet:

Grid: 8 columns x 4 rows.
Row 1: Idle - character breathing subtly
Row 2: Walking - 8-frame walk cycle from left
Row 3: Running - 8-frame run cycle
Row 4: Jumping - jump up and land sequence

Character: cute robot with round blue body, small antenna, expressive digital eyes.
Style: 3D cartoon, soft lighting, clean design.

---

Background Removal

Remove background from generated images to create transparent subjects. Works best when the original image has a clean, solid-color background.

Use Cases

• Characters/subjects generated on white background that need transparency
• Stock photos where only the subject is needed
• Foreground elements that will be composited over video backgrounds

---

Sprite Sheet Validation (MANDATORY)

After generating a sprite sheet, validation is performed automatically by the images_generate tool when using mode: sprite. You can also run validation independently with the images_validate_sprite tool.

Automatic Validation (Built-in to Generate)

When using images_generate with mode: sprite:

1. Grid dimensions auto-detected from the prompt (patterns like "8 columns x 4 rows"), or explicitly via cols/rows parameters
2. Post-generation validation runs automatically:
   • Checks image dimensions are evenly divisible by cols/rows
   • Checks magenta chroma key coverage >= 15%
3. Auto-retry on failure (up to max_retries times, default 3):
   • Each retry enhances the prompt with progressively stronger dimension constraints
   • Adds explicit pixel resolution requirements (e.g., "output MUST be 4096x2048")
4. Reports final validation result in JSON

Recommended usage:

images_generate(
  prompt="Grid: 8 columns x 4 rows. ...",
  output="/path/to/sprite.png",
  mode="sprite",
  cols=8,     # explicit is more reliable than auto-inference
  rows=4,
)

Independent Validation Tool

Use images_validate_sprite for post-hoc validation or re-checking:

images_validate_sprite(
  image="/path/to/sprite.png",
  cols=8,
  rows=4,
  fix_chroma=true,        # optional: replace magenta with transparency
  output="/path/to/sprite_transparent.png",
  extract_frames="/path/to/frames/",  # optional: extract individual frames
)

Returns JSON with check results:

{
  "success": true,
  "checks": [
    {"check": "dimensions", "passed": true, "frame_size": "512x512"},
    {"check": "aspect_ratio", "passed": true, "frame_aspect_ratio": 1.0},
    {"check": "chroma_key", "passed": true, "magenta_coverage": 42.3}
  ]
}

Sprite Sheet Failure Handling

If sprite sheet generation fails after all retries:

1. Log failure details (prompt, dimensions, error) in execution summary
2. Do NOT silently skip — report the failure to orchestrator as partial_failure
3. Suggest alternatives to the user:
   • Retry with simplified prompt (fewer cells, simpler subjects)
   • Fall back to static images instead of sprite animation
   • Provide alternative asset manually
4. Continue processing remaining (non-failed) resources

Manual Fallback Validation

If the tools are unavailable, use Python/Pillow to verify:

from PIL import Image

img = Image.open("raws/images/generated/robot_sprite.png")
cols, rows = 8, 4
w, h = img.size

assert w % cols == 0, f"Width {w} not divisible by {cols} columns"
assert h % rows == 0, f"Height {h} not divisible by {rows} rows"

frame_w, frame_h = w // cols, h // rows
print(f"Sprite: {w}x{h}, Frame: {frame_w}x{frame_h}, Grid: {cols}x{rows}")

Validation Checklist

Sprite Metadata in resources.yaml

After validation, record sprite metadata for downstream consumption:

- id: vis_sprite_001
  type: sprite
  description: "Robot character animation"
  source: generate
  path: images/generated/robot_sprite.png
  sprite:
    cols: 8
    rows: 4
    frameWidth: 512
    frameHeight: 512
    chromaKey: "#ff00ff"
    animations:
      idle: { row: 0, frameCount: 8 }
      walk: { row: 1, frameCount: 8 }
      run: { row: 2, frameCount: 8 }
      jump: { row: 3, frameCount: 8 }

---

Output Structure

raws/
  images/
    existing/                # User-provided (unchanged)
      logo.png
    retrieved/               # From Pexels/Pixabay
      server_room_001.jpg
      abstract_bg_002.jpg
    generated/               # From Caro LLM API
      robot.png              # With transparency
      scientist.png          # With transparency
      background_tech.png    # No transparency needed
      robot_sprite.png       # Sprite sheet (magenta bg)
  videos/
    retrieved/               # From Pexels Video API
      product_demo_001.mp4
      aerial_city_002.mp4

---

Workflow

1. Read manifests/resources.yaml to identify visual resources
2. For each resource with source: retrieve:
   • Search stock images with appropriate keywords
   • Download to raws/images/retrieved/
3. For each resource with source: generate:
   • If character/mascot with animation needs: use sprite mode (recommended)
   • Craft appropriate prompt based on description
   • Generate the image:
     • Normal mode: Single generation, no validation
     • Sprite mode: Use images_generate with mode: sprite, cols, rows params. Built-in auto-validation and retry handles dimension/chroma issues.
   • If needs transparency (non-sprite), remove background
   • Sprite post-processing: After successful generation, optionally use images_validate_sprite with fix_chroma=true to convert magenta to transparency
   • Save to raws/images/generated/
4. Update resources.yaml with actual file paths (include sprite metadata for sprite sheets)
5. Report results to user

---

Incremental Mode

当 dispatch context 中 mode: "incremental" 时：

输入

• amendment.affected_resources 中指定的新增视觉资源 ID 列表

执行规则

1. 读取 resources.yaml，仅定位指定的新增资源条目
2. 按 source 类型（retrieve/generate）执行获取或生成
3. 如需透明背景，执行背景移除
4. 如为 sprite sheet，执行完整验证流程
5. 更新 resources.yaml 中对应条目的 path 字段
6. 返回处理摘要（获取/生成了哪些资源）

不执行

• 不重新扫描已有的视觉资源
• 不重新下载或重新生成已存在的素材
• 不修改未指定资源的 path 或元数据

---

Updating resources.yaml

After retrieving/generating images, update the resource entry with the actual path:

Before:

- id: vis_001
  type: image
  description: "Server room with blue lighting"
  source: retrieve

After:

- id: vis_001
  type: image
  description: "Server room with blue lighting"
  source: retrieve
  path: images/retrieved/server_room_001.jpg

---

User Communication

Progress Report

视觉素材增强进行中...

处理中: vis_003 (server room background)
  来源: Pexels
  状态: 下载完成

Completion Report

视觉素材增强完成。

检索素材:
  - 3 张图片 (Pexels)
  - 2 张图片 (Pixabay)

生成素材:
  - 2 张图片 (带透明背景)
  - 1 张背景图

已更新: manifests/resources.yaml
输出目录: raws/images/
  - retrieved/ (5 个文件)
  - generated/ (3 个文件)

请确认素材后继续下一步。

---

Common Issues

---

Notes

• Always use English for search queries and generation prompts
• Keep search result count low (3-5) to avoid context bloat
• For characters/subjects, always generate on white background then remove
• For animated characters, prefer sprite sheets -- they produce more consistent results than generating individual frames separately
• After generating sprite sheets, always validate frame splitting before proceeding
• Update resources.yaml with actual paths after processing
• All image paths in resources.yaml are relative to raws/