elodin-monte-carlo

Truth-Data-Driven Simulation with Elodin Monte Carlo

The most effective way to build a credible simulation is to anchor it to experimental truth data and use elodin monte-carlo as the test harness: every model change is judged by a 30-run campaign against recorded reality, in under a minute. This skill codifies that workflow. The canonical worked example is examples/apollo-lander/ (with the full methodology in its WHITEPAPER.md).

The development loop

vendor truth data -> build reference module -> sim + truth ghost + graphs
      ^                                                    |
      |                                                    v
 narrow spec.toml <- read report <- run campaign <- score runs vs truth
                         |
                         v
              export worst run CSV -> diagnose time series -> fix model

Run the campaign after every meaningful change. Distribution deltas (RMSE, success rate, margins, dispersion) tell you immediately whether a change helped, broke something, or just moved noise.

1. Vendor truth data with provenance

• Keep the verbatim source file untouched in data/ next to a derived, SI-unit version the sim actually loads. Cite the source URL in README/docs.
• Write a sanity_check() in the reference module that re-derives the cleaned data from the raw file and asserts agreement, plus checks documented anchor values. Make python reference.py print the profile and the check.
• Distrust every column. Real-world traps hit in practice: a "meters" column converted at x0.3405 instead of x0.3048 (recompute conversions from the original-unit column); digitized charts with out-of-order and duplicate timestamps (sort, average duplicates); head samples from a different measurement frame (a state-vector update step) that must be trimmed and back-extrapolated; a sensor channel that is not the state you want (radar slant range is not altitude).
• Cross-validate datasets against each other and against documented events: mission reports, transcripts, and timelines give anchor points (event times, velocity callouts, fuel consumed) that catch unit and frame errors no amount of smoothing will.

2. Build a dependency-free reference module

Put all truth handling in one stdlib-only Python module (see examples/apollo-lander/reference.py) shared by the sim and any external controller:

• Resample onto a uniform grid, despike with a median filter, smooth with a moving average. Only enforce monotonicity if the physics demands it — real profiles wiggle, and the wiggles are history worth keeping.
• Expose interpolators (altitude(t), descent_rate(t), ...) plus the raw arrays. In sim.py, convert once to jnp.asarray and close over them in systems: they become JIT-time constants (large baked constants are interned and shared across campaign workers).

3. Reconstruct missing channels from physics

Truth datasets are rarely complete. When a needed channel is missing (e.g. horizontal velocity existed only as a chart image), reconstruct it:

• Integrate the vehicle dynamics along the channels you do have, using documented schedules (throttle history, event times) for the rest.
• Allocate consistently: if total thrust is known and the vertical share is implied by the recorded altitude, the horizontal share is the remainder — this makes the reconstructed profile flyable by construction, which matters when a controller later tracks it (see section 7).
• Calibrate segment-by-segment through documented anchors so the profile passes through known values exactly; integrate the best-documented segment unscaled and let it yield the unknown initial condition.
• Iterate fixed-point if channels couple (two passes usually converge).
• Document the uncertainty (~±10% is typical) and assert anchors in sanity_check().

4. Truth ghost: replay reality inside the sim

Render the recorded vehicle next to the simulated one. The robust pattern:

# Kinematic ghost: NO el.Body — gravity/integrator/telemetry systems never match it.
world.spawn([
    StaticSceneObject(el.WorldPos(...)),           # world_pos only
    el.C(TruthMarker, jnp.array([1.0])),           # scopes the playback query
    el.C(Altitude, ...), el.C(VerticalSpeed, ...), # graph channels
], name="lander_truth")

@el.system
def truth_playback(
    tick: el.Query[el.SimulationTick], truth: el.Query[TruthMarker]
) -> el.Query[el.WorldPos, Altitude, VerticalSpeed]:
    t_s = tick[0] * SIM_TIME_STEP
    alt = jnp.interp(t_s, ref_time, ref_altitude)
    ...
    return truth.map((el.WorldPos, Altitude, VerticalSpeed), lambda _: (...))

Hard-won rules:

• Never give the ghost an el.Body. If physics systems match it, gravity integrates its velocity while replay snaps its position — sawtooth motion, runaway velocities in the graphs, and corrupted exports.
• Do not drive the ghost from pre_step DB writes: explicit writes land on their own clock and interleave with telemetry commits, producing CSV exports with holes. The in-sim el.SimulationTick playback keeps truth on exactly the simulated vehicle's commit clock, so exports line up row-for-row.
• A marker component is the cheapest way to scope a playback query to one entity.

Put sim-vs-truth pairs on every KDL graph, and include raw measurement curves (e.g. slant range next to true altitude) — visible divergence between a sensor and the state teaches more than hiding it.

5. Campaign as the test harness

• el.monte_carlo.params_spec(...) defaults drive single runs; spec.toml drives campaigns. Keep ranges mirrored, and anchor ranges in data (a fuel chart pins the propellant load; navigation accuracy pins IC dispersions). When control authority is saturated (e.g. a full-throttle braking burn), keep IC ranges tight — the real system could not recover big errors either, and neither can yours.
• Score every run against truth in post_step and emit el.monte_carlo.result(traj_rmse=..., pitch_rmse=..., miss_distance=..., soft_landing=...). Fit metrics (RMSE vs truth) are what turn a campaign from a stress test into a calibration engine: report the best-fit run's params, narrow spec.toml around them, repeat (or automate the loop, see examples/apollo-lander/calibrate.py).
• Latch event metrics in-sim, at the event. A touchdown-speed metric read one tick after contact reads the zeroed post-contact state; capture it in the same system that detects the event, before state is clobbered.
• Keep the LHS seed fixed while iterating so campaign-to-campaign deltas reflect your changes, not resampling. If you edit spec.toml, re-run elodin monte-carlo sample (the runner warns when a --plan CSV is older than its sibling spec).
• Control concurrency with --workers N / workers = N in campaign.toml (exactly N runs at once). S10_MAX_INFLIGHT is only the low-level process-count escape hatch.
• Use [[build]] steps in campaign.toml to compile external FSW (and generate configs) once before workers start; [env] sets campaign-wide process environment. Put per-worker resources under [resources.ports] — numeric bases give a validated static plan, "auto" allocates per run — and read them with el.monte_carlo.port("name", default) or ELODIN_MC_PORT_<NAME>.
• At scale: scratch_dir = "auto" moves per-run DB IO to tmpfs when the artifact disk can't sustain workers x write IOPS; [retention] prune_on_pass/prune_on_fail globs bound disk without hook-side cleanup; export_db(..., pattern="rocket.*") exports only scored components; and a [quality] max_behind_deadline_frac = 0.05 gate marks load-degraded real-time runs degraded instead of silently ingesting skewed samples. Failed runs carry a one-line failure_reason in results.csv.

6. Diagnose with the data, not by staring at code

• Export any run and interrogate it with quick Python:

  elodin-db export dbs/<campaign>/runs/run_0000012/db \
      --format csv --join --flatten -o /tmp/run12
  

  Print a time-series table of the suspect channels (state, reference, command, actuator) every N seconds — most control/physics bugs are obvious in 20 rows.

• Rank runs by the failing metric (post_run_result.json per run) and export the worst one, not the best.

• Treat "too perfect" as a bug. Metrics that are identically zero or pegged to a clamp value usually mean the measurement is wrong (read after state-clobber, trivially satisfied criterion), not that the system is great.

• Verify exports are clean: row counts equal across entities, no empty cells, no physically impossible values. Data quality bugs masquerade as physics.

• Use the editor for visual regressions (attitude behavior, ghost overlap, terrain seating); confirm emergent event times against the historical record.

7. Tracking recorded profiles: control lessons

When a controller tracks the truth profile, the campaign will find every weakness. Patterns that survived:

• Feed-forward the reconstructed profile; bound the feedback. Clamp each feedback channel's authority (e.g. ±0.8 m/s²) around the flyable feed-forward. Unbounded feedback lets channels fight over a saturated actuator and limit-cycle.
• If the feed-forward is not flyable (demands more than the actuator), the tracker oscillates no matter the gains — fix the reference (section 3), not the gains.
• Prefer one unified tracking law with state-dependent limits over discrete phase switches; let phases emerge (a speed-dependent tilt budget reproduces a pitchover; a demand-dependent throttle latch reproduces throttle-down).
• Add terminal logic: near the end, stop chasing the time-indexed reference and null residual errors — event timing shifts with small tracking errors, and arriving early must not mean arriving with the reference's residual velocity.
• Capped, slowly-fading position trim erodes IC offsets without destabilizing the velocity loop.

8. Operational pitfalls

• Stale processes hold UDP ports. Prefer direct world.recipe() sidecars plus s10 readiness probes. Linux campaigns use cgroup teardown to reap reparented daemons; --keep-existing opts out of scoped pre-reaping.
• Return valid = False from post_run hooks for infrastructure failures so they are reported separately from scored misses.
• CI hygiene per repo rules: ruff format && ruff check --fix for the Python, cargo fmt/cargo clippy -- -Dwarnings for external controllers.

Reference example

examples/apollo-lander/ exercises every pattern above: vendored NASA telemetry with unit-bug handling and anchor checks, a physics-based horizontal-profile reconstruction, a kinematic truth ghost on el.SimulationTick, an external Rust LGC tracking the profile through a SITL UDP bridge, campaign hooks producing fit metrics and landing-ellipse statistics, and a calibration loop. Its WHITEPAPER.md documents the full methodology and the honest caveats.