general-plot-digitizer

General Plot Digitizer

Goal

Extract calibrated numeric X-Y data from images of experimental spectra (Raman, XRD, UV-Vis, IR, NMR, etc.) using a deterministic "Agent-in-the-Loop" workflow.

The labor is divided between two models:

1. Vision-Language Model (Visual Sensor): Reads the image and returns a rich, unstructured narrative description of axes, colors, and visual obstacles. It does not produce JSON.
2. Coding Agent (Translator & Executor): Translates the VLM narrative into a precise metadata.json, runs the CV pipeline, inspects the overlay, and iterates until the curve is correctly isolated.

Instructions

Phase 1: Visual Inspection (VLM)

Do not attempt to generate JSON with the VLM. It acts only as a visual sensor.

1. Generate grid overlay:

# Env: base-agent
python .agents/skills/general-plot-digitizer/scripts/plot_utils.py plot.png --draw-grid

This produces plot_grid.png with a labeled pixel grid for precise coordinate reading.

2. Prompt the VLM to analyze plot_grid.png (not the raw image). Use the built-in vision capabilities or the notify_user VLM inspection tool. Provide the prompt guidelines from scripts/vlm_prompt_template.txt.

3. Expected VLM output — a natural-language report covering:

   • Axis labels, numeric ranges, and directions (is X reversed?).
   • Bounding box of the data region in pixels (read from the grid).
   • Color and style of each target curve (hex guess from the pixels on the line itself).
   • Pixel bounding boxes of all visual obstacles (legends, text annotations, gridlines, tick marks) that overlap the data curves.
   • Trace quality hints: thin/needle-like, thick/noisy, anti-aliased, JPEG artifacts.

Phase 2: Metadata Construction (Coding Agent)

Read the VLM narrative and construct metadata.json. Schema: resources/metadata_schema.json.

Required fields:

{
  "plot_title": "",
  "x_axis_label": "Wavelength (nm)",
  "y_axis_label": "Absorbance",
  "x_tick_min": 400, "x_tick_max": 800,
  "y_tick_min": 0, "y_tick_max": 1,
  "x_calibration_points": [
    { "pixel": 70, "value": 400 },
    { "pixel": 450, "value": 800 }
  ],
  "x_scale": "linear", "y_scale": "linear",
  "bounding_box": {"x_min": 72, "y_min": 28, "x_max": 452, "y_max": 318},
  "x_reversed": false, "y_reversed": false,
  "spectrum_type": "UV-Vis",
  "curves": [{"label": "sample", "color_hint": "#1f77b4"}],
  "text_regions": [{"x_min": 300, "y_min": 50, "x_max": 400, "y_max": 80, "label": "legend"}]
}

x_calibration_points (strongly recommended): anchor the X-axis transform to exact pixel→value pairs read from the grid, rather than assuming axis ticks align perfectly with bbox edges. Pick two well-separated ticks visible on the grid. If provided, these override x_tick_min/max for pixel-to-data mapping.

Translation rules (VLM narrative → metadata fields):

• Obstacles → text_regions[] and/or mask_regions[] with pixel bounding boxes.
• Curve colors → curves[].color_hint (from pixels on the plotted line, not from legend swatches).
• "Curve is black / same as axes" → "cli_hints": {"curve_is_black": true}.
• "Noisy / jagged trace" → "cli_hints": {"smooth": true}.
• "Thin, needle-like peaks" → do not set smooth; set upscale in Phase 3 instead.

If the VLM color guess is uncertain, run:

# Env: base-agent
python .agents/skills/general-plot-digitizer/scripts/suggest_colors.py plot.png \
  --bounding-box x_min,y_min,x_max,y_max

This reports dominant non-background colors in the cropped region. Use the top result as color_hint.

Phase 3: Pipeline Execution

Select CLI flags based on VLM visual cues:

Run the pipeline:

# Env: base-agent
python .agents/skills/general-plot-digitizer/scripts/digitize_pipeline.py \
  plot.png \
  --full \
  --metadata metadata.json \
  --output-dir ./output \
  --overlay \
  --format both

Append the VLM-dictated flags from the table above. The pipeline also reads cli_hints from metadata and auto-applies safe flags (--allow-black, --smooth, --all-curves).

Phase 4: Overlay Inspection & Iteration

1. Inspect *_digitized.overlay.png visually.
2. If extraction is wrong, diagnose using this table and re-run Phase 3:

3. Prefer metadata edits (adjusting color_hint, text_regions, mask_regions, bounding_box) over adding CLI flags. Re-run the same pipeline command after editing metadata.json.

Outputs:

• *_digitized.csv — comma-separated with x,y header
• *_digitized.xy — space-separated, no header (if --format xy or both)
• *_digitized.overlay.png — visual QC
• *_digitized.md — summary

Agent Rules

• Do not write ad-hoc NumPy/OpenCV pixel-scanning code. The pipeline already handles HSV masking, spatial filtering, and smoothing internally.
• Do not skip the grid overlay step. The VLM is significantly more accurate on plot_grid.png than on raw images.
• Do not guess CLI flags without VLM visual evidence. The flag table above is the complete decision tree.
• Always inspect the overlay image after each run before declaring success.

Key Parameters

Full flag list: python .agents/skills/general-plot-digitizer/scripts/digitize_pipeline.py --help

Examples

Constraints

• Bounding box must enclose only the inner data region (inside axis lines, excluding labels/legend/title). VLMs often get this wrong — verify against the grid overlay.
• Multi-curve: Each curve in curves[] must have a color_hint. Auto-detect is unreliable with multiple traces.
• Per-curve regions: For stacked/offset plots, curves[].region.y_min/y_max must include generous padding (10-20px) above tallest peaks and below baseline.
• Environments: All scripts require base-agent conda env.

Resources

• resources/metadata_schema.json — full metadata schema
• scripts/vlm_prompt_template.txt — VLM prompt guidelines
• scripts/suggest_colors.py — dominant color detection for unknown curve colors

Related Skills

• mat-xrd-digitizer: Peak-based XRD digitization for phase matching.

References

• Gonzalez & Woods, Digital Image Processing, Pearson, 4th ed., 2018. Standard reference for HSV color-space segmentation, morphological operations, and connected-component analysis used in the extraction pipeline.
• Bradski, G., "The OpenCV Library", Dr. Dobb's Journal of Software Tools, 2000. Core CV library underlying all image manipulation in this skill.

---

Author: Jesus Diaz Sanchez Contact: GitHub @jdsanc