analyses-snapshot-testing

Analyses Snapshot Testing Instructions

Overview

The analyses-snapshot-testing directory validates that protocol analysis output remains consistent across code changes by comparing JSON results against committed snapshots.

• Regression testing: Detect unintended changes in protocol analysis behavior
• Protocol validation: Ensure Flex protocols analyze correctly
• CI/CD: Automated testing in GitHub Actions with matrix-based parallel execution
• Snapshot management: Track expected output and flag deviations

This repository is Flex-only. All protocols must target Flex (or OT-3) robots.

A standalone OT-2 compatibility test (tests/test_ot2_compatibility.py) runs via make ot2-compatibility-test. It uses a fixture protocol in tests/fixtures/ and is not part of chunk generation, protocols.py, or syrupy snapshots. CI runs it in analyses-snapshot-test.yaml (not the lint workflow).

Architecture

Core Components

1. Protocol Files (files/protocols/) — testing protocols organized by source (standard, Protocol Designer exports, Protocol Library imports, generators)
2. Analysis Engine (automation/analyze.py) — runs protocol analysis in subprocess, 120-second timeout, JSON output
3. Snapshot Storage (tests/__snapshots__/) — committed JSON snapshots managed by syrupy with custom JSON extension
4. Test Suite (tests/) — analyses_snapshot_test.py (main), audit_snapshot_test.py (audit), custom_json_snapshot_extension.py (serialization)
5. Audit (automation/audit_snapshots.py) — validates snapshots: Flex_S protocols must have zero errors; Flex_X and other failure-case protocols are expected to have errors
6. Protocol Registry (automation/data/) — protocols.py (auto-generated), protocols_with_overrides.py (manual), protocol_registry.py (combined)
7. CI/CD (citools/, .github/workflows/) — matrix-based parallel execution via Docker

Protocol Naming Convention

{Robot}_{Status}_{Version}_{Source}_{Pipettes}_{Modules}_{Overrides}\_{Description}

• Robot: Flex
• Status: S (Success) or X (Failure expected)
• Version: API version (e.g., v2_19) or PD (Protocol Designer)
• Source (optional): PL_ (Protocol Library) or MPL_ (Manual Protocol Library)
• Pipettes: e.g., P1000M_P50M
• Modules: GRIP, HS, MM, MB, TC, TM
• Overrides: Overrides if protocol has parameter overrides
• Description: Max 25 characters

Example: Flex_S_v2_19_P1000M_GRIP_HS_TM_MB_KAPALibraryQuant.py

Development Workflow

Adding New Protocols

1. Create protocol file in files/protocols/ following naming convention
2. Run make prep (auto-updates automation/data/protocols.py, displays snapshot command)
3. Generate snapshots: make snapshot-test-update PROTOCOL_NAMES=YourProtocolName OVERRIDE_PROTOCOL_NAMES=none
4. Commit protocol file + snapshot + updated protocols.py

Adding Override Protocols

1. Create generator in files/protocols/generators/
2. Add entry to automation/data/protocols_with_overrides.py
3. Generate snapshots: make snapshot-test-update PROTOCOL_NAMES=none OVERRIDE_PROTOCOL_NAMES=YourOverrideProtocol
4. Commit generator + snapshot + updated registry

Updating Snapshots

make snapshot-test-update                                          # All
make snapshot-test-update PROTOCOL_NAMES="P1,P2" OVERRIDE_PROTOCOL_NAMES=none  # Specific

Running Tests Locally

make snapshot-test                                                   # All (reduced verbosity)
make snapshot-test PROTOCOL_NAMES=Flex_S_v2_19_Example OVERRIDE_PROTOCOL_NAMES=none  # Specific
uv run python -m pytest -k analyses_snapshot_test -vv --tb=short    # Verbose debugging

Code Modification Guidelines

When Modifying Analysis Logic (api/, shared-data/)

1. Run snapshot tests to detect changes
2. Review diffs carefully
3. Update snapshots only if changes are intentional
4. Document breaking changes in commit messages

Protocol Registry (automation/data/protocols.py)

• DO NOT manually edit — auto-generated by make prep
• Regenerate after adding/removing protocols

Snapshot Extension (tests/custom_json_snapshot_extension.py)

• Normalizes timestamps, IDs, file paths for stable comparisons
• Add new normalization rules to replacement_patterns
• Add new ID-like fields to id_keys_to_replace

Environment Setup

Prerequisites: Python 3.12, uv, Node.js/pnpm (for prettier)

cd analyses-snapshot-testing
make setup    # Creates uv venv and installs dependencies

The setup script (bootstrap_uv_env.py) installs api and shared-data as editable packages so analysis uses local code. It creates a Python 3.12 venv by default (UV_PY=3.12).

Common Commands

# Setup
make setup                    # Initial environment setup
make teardown                 # Remove virtual environment

# Testing
make snapshot-test            # Run all snapshot tests
make snapshot-test-update     # Update all snapshots
make snapshot-audit-test      # Validate snapshot audit metadata
make ot2-compatibility-test # Standalone OT-2 rejection test (not in snapshot battery)
make snapshot-validation    # snapshot-audit-test + ot2-compatibility-test

# Protocol Management
make prep                     # Regenerate protocol registry
make generate-protocols       # Generate override protocols

# Formatting
make format                   # Format Python + Markdown
make ruff                     # Format and lint Python

# CI Simulation
make gen-chunks               # Generate protocol chunks
make analyze-chunk CHUNK=chunk_0.json  # Analyze specific chunk

CI/CD Integration

Workflow (analyses-snapshot-test.yaml) triggers on:

• PRs affecting api/, shared-data/, or this directory
• Scheduled daily at 7:26 AM UTC
• Manual dispatch

PR label gen-analyses-snapshot-pr auto-opens a PR with updated snapshots on failure.

Troubleshooting

• Snapshot mismatch: Review diff, update if intentional (make snapshot-test-update PROTOCOL_NAMES=...), fix code if not
• Protocol not found: Check registry, run make prep, verify file exists and follows naming convention
• Analysis timeout (120s): Simplify protocol or increase timeout in automation/analyze.py
• Environment issues: Run make setup, ensure uv is installed and Python 3.12 is available