agentsociety-run-experiment

Run Experiment

Manage experiment execution using the AgentSociety2 CLI: start, monitor, and stop simulation runs.

When to Use

• init_config.json and steps.yaml exist and the user wants to start a simulation
• User wants to check experiment status or view logs
• User wants to stop a running experiment

Do NOT use when:

• Configuration files do not exist yet (use experiment-config skill)
• User wants to analyze results (use analysis skill)

Quick Reference

Run commands from the workspace root through .agentsociety/bin/ags.py.

Use the Python interpreter from .env. See CLAUDE.md for setup.

Workflow

digraph run_experiment_flow {
    rankdir=LR;
    node [shape=box, style=filled, fillcolor="#E8F4FD"];
    check [label="confirm init_config.json\nand steps.yaml"];
    start [label="start"];
    status [label="status / logs"];
    done [label="run/replay/_schema.json\nand artifacts present"];
    stop [label="stop"];
    fix [label="inspect logs\nand revise config"];

    check -> start -> status;
    status -> done [label="completed"];
    status -> stop [label="user stops"];
    status -> fix [label="failed"];
    fix -> check;
}

CLI Arguments (.agentsociety/bin/ags.py run-experiment ...)

Start Experiment

$PYTHON_PATH .agentsociety/bin/ags.py run-experiment start --hypothesis-id 1 --experiment-id 1

The script auto-resolves paths from hypothesis_{id}/experiment_{id}/ structure. By default, start launches the experiment in the background, writes logs to run/stdout.log and run/stderr.log, and keeps run/pid.json after completion so the extension can show final status.

Foreground Mode

Use --foreground only when you explicitly want to block the current terminal session:

$PYTHON_PATH .agentsociety/bin/ags.py run-experiment start --hypothesis-id 1 --experiment-id 1 --foreground

Advanced: Direct CLI

For cases requiring full control (custom log file, manual background execution):

$PYTHON_PATH -m agentsociety2.society.cli \
    --config hypothesis_1/experiment_1/init/init_config.json \
    --steps hypothesis_1/experiment_1/init/steps.yaml \
    --run-dir hypothesis_1/experiment_1/run \
    --experiment-id "1_1" \
    --log-file hypothesis_1/experiment_1/run/stdout.log &

Batch Size & Concurrency (large runs)

• --batch-size (env AGENTSOCIETY_BATCH_SIZE, default 256): agents per step_agent_batch Ray Task.
• Tasks per tick = ⌈N / batch_size⌉, capped by AGENTSOCIETY_LLM_RAY_MAX_WORKERS (default = machine CPU count).
• Pick --batch-size so ⌈N / batch_size⌉ ≈ worker count to saturate CPUs; otherwise workers sit idle.
• Per-process LLM concurrency self-tunes via AIMD (AGENTSOCIETY_LLM_RAY_CONCURRENCY, default 16, no hard cap).

Common Mistakes

Pipeline Position

Predecessors: experiment-config (completed check action) Successors: analysis Required Sub-Skills: None

Output Files

Resume Interrupted Run

If an experiment is interrupted (crash, stop, OOM), append --resume to continue from the last completed step:

$PYTHON_PATH .agentsociety/bin/ags.py run-experiment start --hypothesis-id 1 --experiment-id 1 --resume

Resume reads SOCIETY.json + SOCIETY_STEP.json to restore the society clock/step-count and each env module's ENV_STATE.json (via EnvBase.restore), then skips already-completed run steps. Only env modules that implement to_workspace/restore survive a restart (see the agentsociety-create-env-module skill). If steps.yaml changed since the checkpoint, a drift warning is logged. Works with both --foreground and background modes.

Monitoring

# Check status (recommended)
$PYTHON .agentsociety/bin/ags.py run-experiment status --hypothesis-id 1 --experiment-id 1

# Follow logs in real time
tail -f hypothesis_1/experiment_1/run/stdout.log

# Stop a running experiment (recommended)
$PYTHON .agentsociety/bin/ags.py run-experiment stop --hypothesis-id 1 --experiment-id 1

Troubleshooting

Environment Validation

The CLI validates required environment variables before execution. Missing variables cause early exit with clear error messages.

Connection Errors

litellm.InternalServerError: Connection error -- verify API endpoint is reachable, API key is valid, and model name is correct.

Hard Constraints

• Default start mode is background execution with persisted logs and pid.json.
• When using the direct CLI in background, --log-file is NOT optional. Without it, verbose logs are lost and there is no way to diagnose failures.
• Environment variables (AGENTSOCIETY_LLM_API_KEY, AGENTSOCIETY_LLM_API_BASE, AGENTSOCIETY_LLM_MODEL) must be set. See CLAUDE.md for full configuration reference.

Programmatic API

See references/programmatic-api.md for the async Python API for starting, stopping, and querying experiments programmatically.

Progress Tracking

After starting an experiment:

$PYTHON .agentsociety/bin/ags.py research-pipeline update-stage run_experiment in_progress

After experiment completes (run/replay/_schema.json exists):

$PYTHON .agentsociety/bin/ags.py research-pipeline update-stage run_experiment completed

If the experiment fails:

$PYTHON .agentsociety/bin/ags.py research-pipeline update-stage run_experiment failed --error "error message"