autofigure

AutoFigure: Image to Editable DrawIO

Convert a raster figure (PNG/JPEG) into an editable .drawio file where text, arrows, and shapes are real mxCell elements, and icons are embedded as transparent PNGs using DrawIO's image style.

Available Tools

Legacy SVG tools (generate_svg_template, replace_icons_svg) are also available if you need SVG output instead.

Workflow

Follow these steps in order. Each step's output feeds into the next.

Step 1 — Segment Figure

segment_figure(
  image_path="path/to/figure.png",
  output_dir="./output"
)

Detects icons, diagrams, and visual elements using SAM3. Produces:

• samed.png — figure with gray rectangles (#808080) + sequential labels (<AF>01, <AF>02, …)
• boxlib.json — coordinates of each detected region

Optional parameters:

• text_prompts: Comma-separated SAM3 prompts (default: "icon,robot,animal,person")
• min_score: Confidence threshold (default: 0.0)
• merge_threshold: Overlap merge threshold (default: 0.001)

Step 2 — Crop & Remove Background

crop_remove_bg(
  image_path="path/to/figure.png",
  boxlib_path="./output/boxlib.json",
  output_dir="./output"
)

For each detected region:

1. Crops the icon from the original figure
2. Removes the background using RMBG-2.0
3. Saves transparent PNG to icons/icon_AF01_nobg.png

Produces icon_infos.json with paths and coordinates for all icons.

Step 3 — Generate DrawIO Template

generate_drawio_template(
  figure_path="path/to/figure.png",
  samed_path="./output/samed.png",
  boxlib_path="./output/boxlib.json",
  output_dir="./output"
)

Sends the original figure and samed.png to a multimodal LLM, which reconstructs the figure as DrawIO mxGraph XML:

• Text → <mxCell style="text;..."> elements
• Arrows/edges → <mxCell edge="1" ...> with mxGeometry
• Shapes/boxes → <mxCell style="rounded=0;..."> rectangles, ellipses, etc.
• Icon areas → gray placeholder mxCells with id="AF01" and value="<AF>01"

Includes automatic XML validation and LLM-based repair.

Produces:

• template.drawio — raw LLM output
• optimized_template.drawio — validated/fixed drawio file

Optional parameters:

• optimize_iterations: LLM refinement iterations (default: 0)

Step 4 — Replace Icons in DrawIO

replace_icons_drawio(
  template_drawio_path="./output/optimized_template.drawio",
  icon_infos_path="./output/icon_infos.json",
  figure_path="path/to/figure.png",
  output_path="./output/final.drawio"
)

Replaces each gray placeholder mxCell with the corresponding transparent PNG icon embedded using DrawIO image style (image=data:image/png,{base64}). Matching strategy:

1. By cell id (e.g., AF01)
2. By cell value (e.g., <AF>01 or <AF>01)
3. Fallback: coordinate matching against original boxlib positions
4. Last resort: append a new image mxCell at the correct position

Produces final.drawio — the editable DrawIO file with embedded icons, openable in draw.io.

Example Interaction

User: "Convert this figure to an editable DrawIO file: figures/method.png"

Agent steps:
  1. segment_figure(image_path="figures/method.png", output_dir="./output")
     → samed.png, boxlib.json (e.g. 5 boxes detected)
  2. crop_remove_bg(image_path="figures/method.png", boxlib_path="./output/boxlib.json", output_dir="./output")
     → icon_infos.json, 5 transparent PNGs
  3. generate_drawio_template(figure_path="figures/method.png", samed_path="./output/samed.png", boxlib_path="./output/boxlib.json", output_dir="./output")
     → template.drawio, optimized_template.drawio
  4. replace_icons_drawio(template_drawio_path="./output/optimized_template.drawio", icon_infos_path="./output/icon_infos.json", figure_path="figures/method.png", output_path="./output/final.drawio")
     → final.drawio
  5. Reply with the final drawio path

DrawIO Format Notes

• Icon images are embedded as base64 in the mxCell style: image=data:image/png,{base64_data}
• All cells use parent="1" and vertex="1" (edges use edge="1" instead)
• Cell IDs 0 and 1 are reserved; content starts from ID 2
• The file can be opened directly in draw.io or VS Code with the Draw.io Integration extension

Important Notes

• Always run the 4 steps in sequence — each step depends on the previous step's outputs.
• SAM3 requires a FAL API key — set fal_api_key in config.json under tools.autofigure, or set the FAL_KEY environment variable.
• RMBG-2.0 requires PyTorch — torch, torchvision, and transformers must be installed. The model is auto-downloaded from HuggingFace on first use.
• DrawIO generation quality depends on the VLM model — better models (e.g., google/gemini-2.5-pro-preview) produce more accurate reconstructions.
• For complex figures, consider setting optimize_iterations=1 or 2 to refine the drawio layout.