OpenParable
Skip to content
← Back to skills

Mobisys Experiments

Testing & Quality
mobisys-experiments

Mobisys Experiments is an Agent Skill for testing & quality published in the brycewang-stanford/Awesome-Journal-Skills repository on GitHub: a SKILL.md instruction file an AI agent can load to do this task.

Use when designing or auditing the evaluation of a MobiSys submission — building real-device testbeds, instrumenting energy and thermal behavior, measuring latency and frame-rate tails, bounding memory footprint, choosing tuned system baselines, and running deployments or user studies, so systems reviewers see where the system wins and breaks on the device.

View on GitHub ↗
QUICK START

How to use this skill

Bring this guide into your coding agent with a prompt tailored to the tool you use.

  1. Open your project in Codex.
  2. Copy and paste the prompt below.
  3. Review files and risks before approving any action.
Prompt to paste
Inspect the Agent Skill at https://github.com/brycewang-stanford/Awesome-Journal-Skills/blob/HEAD/MobiSys-Skills/skills/mobisys-experiments/SKILL.md and all supporting files. Report provenance, permissions, scripts, dependencies, and security risks. Propose an exact installation into .agents/skills/mobisys-experiments, but wait for my approval before writing files or running scripts.

Copying this prompt does not install or run the skill. Review third-party files before use.

MobiSys Experiments

A MobiSys result becomes believable only when it is measured on real hardware, driven to steady state, and reported in latency, energy, memory, and thermal terms. Simulation-only or single-run numbers read as under-done. The evaluation axis here is the device and platform — compute, energy, latency, memory, heat — not the radio or channel a wireless venue grades, and not the traffic-and-topology an infrastructure venue grades. Build the evaluation as one device-behavior claim per experiment.

Turn each claim into one device experiment

The clean structure gives each subsection a claim about the running system, then the one experiment that settles it. The opposite habit — run a benchmark suite and describe whatever survives — is the unfocused evaluation MobiSys reviewers push back on. The claim-to-experiment map for a running mobile system:

Claim about the systemExperiment that settles itMetrics
It meets the target in the motivating scenarioon-device run under the real workloadp50/p95 latency, throughput, task success
Its energy cost is acceptableinstrumented run on a power monitorenergy-per-operation, average and peak power
It survives sustained usemulti-minute run to thermal steady stateframe-rate stability, temperature trace, throttle onset
Each mechanism earns its placecomponent breakdown / ablationper-mechanism contribution
It fits within device memorypeak-memory and storage accountingpeak RSS, model and cache size
Its failure mode is understoodhot, low-memory, low-battery device statesthe state where a baseline wins

What the setup must state

  • The full device context: phone/board model, SoC, OS build, framework/runtime version, battery vs. wall power, and ambient temperature. A number without its device is not interpretable.
  • Define the energy boundary: the power instrument (on-device rail, external monitor, shunt + DAQ), the sampling rate, and what is inside the measured envelope (SoC only? whole device?). Do not let "efficient" stand in for a measured joules figure.
  • Make load explicit: the workload trace, its duration, and whether the run reached thermal steady state; a 30-second benchmark hides the throttling that a 20-minute run shows.
  • Report ground truth honestly: for a service or inference claim, how correctness was established and its own error.

Making energy and heat auditable

Battery, energy, and thermal claims are common at MobiSys and are held to instrument-level scrutiny:

Energy/thermal report checklist:
  instrument: power monitor / on-device rail / shunt + DAQ, sampling rate
  quantity: energy-per-operation (mJ), not just average power (mW)
  boundary: what is inside the measured envelope (SoC? whole device?)
  thermal: skin/SoC temperature trace and whether steady state was reached
  battery: state-of-charge span, or wall power stated explicitly

An energy or thermal claim that cannot be re-derived from a described setup should not survive your own audit.

Comparators a systems reviewer will trust

  • Compare against the system people actually run, configured the way its own documentation prescribes — an untuned baseline is the most common credibility wound in systems reviewing.
  • Include the do-less baseline: the trivial fix (a smaller model, a fixed low frame rate, more aggressive quantization). If the system cannot beat it at equal quality, that is the finding.
  • When a competitor cannot be run (proprietary runtime, unavailable device), reimplement and label it a reimplementation, or compare on published numbers with the configuration deltas stated.

Report the spread, never a lone peak

  • Report percentiles and confidence intervals for latency, energy, and throughput; a single "up to N×" without the distribution is a review risk (mobisys-writing-style).
  • Show tails and CDFs for headline latency results, and repeat runs across enough devices, thermal states, or battery levels to expose run-to-run spread.
  • On-device results are device- and state-dependent — a number from one phone at full charge in a cool room is not a claim about the system until the spread is characterized.

Pre-submission evaluation pass

  • Every evaluation subsection answers a named question.
  • Device, OS, runtime version, power source, and ambient fully specified per experiment.
  • Energy measured with a described instrument and boundary, not estimated.
  • Sustained-load run reaches steady state; thermal behavior reported.
  • Incumbent-grade baseline present and tuned; do-less baseline present.
  • Percentiles + tails for headline latency; multi-device or multi-state spread reported.
  • At least one device state the system does not win, discussed rather than buried.
  • Numbers in abstract/intro regenerate from the recorded runs.
  • A hardware-optional or downscaled variant exists for artifact evaluators (mobisys-artifact-evaluation).

Output format

[Evidence form] on-device measurement / deployment / user study / trace / simulation (claimed vs actual)
[Question map] question -> experiment -> metric (gaps flagged)
[Device+energy] setup and boundary specified? y/n per headline experiment
[Sustained load] steady state reached and thermal reported? y/n
[Baseline audit] incumbent tuned? do-less present?
[Break condition] device state where the system loses: <named or MISSING>
[Priority additions] ordered by review-risk reduction per bench-day