analyze-datagen-campaign-summary

analyze-datagen-campaign-summary

Turn a datagen campaign's prose tracker (e.g. ~/Documents/experiments/{active,complete}/<campaign>/tracker.md, whose per-dataset status lives in sentences, not columns) into a clean, auditable per-dataset table/CSV with computed metrics. Built for the qwen3.5-122b-tt 32k campaign but campaign-agnostic — swap the dataset list.

Target columns: Datagen Model | Task Source | Status | N Trials Completed | Mean Turns / Trace | Mean Tok / Trace | Mean Reward | HF Repo Link.

Why this skill exists (the two traps)

1. The tracker rarely has reward / turns / tokens. Trackers record throughput (gen tok/s) + row counts in prose; mean reward, mean turns, and mean tokens are almost never written down. They must be COMPUTED from the uploaded HF trace datasets.
2. Status in prose is stale/ambiguous (a "RUNNING" row that actually finished; a "rescued" row with no clean repo name). Ground-truth Status against HF: if the trace repo exists with rows → COMPLETED (and its row count IS N Trials Completed); otherwise fall back to the tracker's status hint.

The data model (uploaded OT-Agent trace dataset)

Each row of penfever/<slug>-<model>-traces is one trial/trace: {conversations: [{role,content},…], agent, model, model_provider, date, task, episode, run_id, trial_name, result, verifier_output}.

• N Trials Completed = row count of the dataset (one row = one completed trace).
• Mean Turns / Trace = mean count_turns(row) = mean number of conversation messages (canonical definition in scripts/analysis/utils.py; total messages, not just assistant turns — state it in the notes).
• Mean Tok / Trace = mean Qwen3-8B token length of the whole conversation — plain method from the analyze-dataset-token-length skill: tokenizer(extract_conversation_text(row), add_special_tokens=False). Tokenizer is always Qwen/Qwen3-8B for these datasets (their trace-dataset convention), regardless of the served model name (model field is hosted_vllm/<numeric-id>, not a usable tokenizer).
• Mean Reward = Harbor-flat mean of result via mean_reward_per_trial semantics: extract_reward each row (parses the result string, e.g. "0.0" → 0.0), missing/non-numeric counts as 0.0. This matches harbor's <done>/<total> Mean: accuracy exactly — do NOT drop nulls or the number won't reconcile.

Reuse the canonical tools (do NOT reinvent)

/Users/benjaminfeuer/Documents/OpenThoughts-Agent/scripts/analysis/utils.py:

• extract_conversation_text(record) — conversation → full text to tokenize (handles messages/conversations).
• count_turns(record) — turns.
• extract_reward(record) — parses result → float|None. mean_reward_per_trial(rows) — Harbor-flat mean.
• load_hf_trace_dataset(repo_id) — non-streaming loader (fine for small repos; see disk note for large ones).

Token-length details (methods, tokenizer, the metadata-confound trap) → the analyze-dataset-token-length skill. If you'd rather source Mean Reward + trials from the job logs instead of the HF dataset (e.g. the repo was never uploaded), the analyze-job-history-iris skill's analyze_job_history.py sidecar carries the harbor Mean: + non_empty_trials per job — but the uploaded dataset is the more reliable ground truth for a COMPLETED row.

⚠ Handling the LARGE trace datasets (disk + bandwidth)

Some campaign datasets are big (tens of thousands of rows / hundreds of MB / dozens of shards). Full load_dataset caches the whole parquet to ~/.cache/huggingface → can blow local disk (a full disk bricks the supervisor — see the disk-health rule in supervisor-init). So:

• STREAM (load_dataset(repo, split="train", streaming=True)) and accumulate in ONE pass — disk stays bounded (shards read on the fly, not cached whole).
• Point HF_HOME / HF_DATASETS_CACHE at the scratchpad and df -h / before launching; bandwidth is unavoidable (the conversations column is the bulk, needed for both turns and tokens) but streaming avoids the disk blowup.
• Batch the tokenizer (e.g. 128 texts) rather than per-row; TOKENIZERS_PARALLELISM=false to avoid the fork-after-tokenizer deadlock when parallelizing.
• Parallelize across datasets with a ProcessPoolExecutor (≈5 workers) — CPU-bound tokenization scales well; each worker streams its own datasets. Checkpoint per-dataset to JSONL so a crash/interrupt resumes instead of recomputing the expensive large ones.

Procedure

1. Build the dataset list from the campaign tracker. One entry per task source: (idx, task_source, candidate_hf_repo_or_None, status_hint, note). candidate_hf_repo = the exact penfever/<slug>-…-traces slug the tracker names (the slug transform is IRREGULAR — copy the stated repo, don't derive it). status_hint ∈ {COMPLETED, FAILED, RUNNING, NOT STARTED} (pending → NOT STARTED & repo=None; killed-not-rescued / blocked-skipped → FAILED & repo=None).
2. Per dataset: probe HF (HfApi().dataset_info(repo)); on 404 keep the hint + NULL metrics. Else stream, compute n_trials, mean_turns, mean_tok (Qwen3-8B), mean_reward (Harbor-flat), set Status=COMPLETED and HF Repo Link = https://huggingface.co/datasets/<repo>.
3. Write the CSV sorted by idx; NULL metrics render as empty cells, missing repo as NULL.
4. VERIFY before delivering (the user asked for it to be correct): spot-check that computed n_trials matches the tracker's stated row counts on a few datasets, and that a KNOWN-degenerate dataset reconciles (e.g. qwen3.5-122b-tt codenet-python-v2 mean reward ≈ 0.017 ↔ the tracker's "~2% pass-rate"). Mean tokens should sit under the campaign's context window (32k here) for the vast majority.

Run (reference implementation)

scripts/analysis/build_campaign_summary_csv.py carries the full qwen3.5-122b-tt list + logic. Adapt the DATASETS/PENDING lists + OUT_CSV for another campaign, then:

source "${DC_AGENT_SECRET_ENV:?}"                                   # HF_TOKEN for the reads
df -h /                                                             # confirm headroom first
nohup /Users/benjaminfeuer/miniconda3/envs/otagent/bin/python \
  /Users/benjaminfeuer/Documents/OpenThoughts-Agent/scripts/analysis/build_campaign_summary_csv.py \
  > build_csv.log 2>&1 &                                            # background; ~30–90 min for a ~40-repo campaign

Watch build_csv.log (one line per dataset as it finishes) and the JSONL checkpoint; re-running resumes from the checkpoint. Output CSV lands next to the campaign tracker (e.g. ~/Documents/experiments/complete/<campaign>/completed_datasets_summary.csv).

Definitions to state alongside the table (so it's auditable)

• Datagen Model = the trajectory-generation model (constant per campaign; e.g. Qwen3.5-122B-A10B-FP8), NOT the row's model field.
• Mean Turns/Trace = mean total conversation messages (count_turns).
• Mean Tok/Trace = mean Qwen3-8B plain token count of the full conversation.
• Mean Reward = Harbor-flat trial mean (missing/error = 0.0) — reconciles with the harbor Mean: line.
• N Trials Completed = uploaded productive rows (may be < tasks for partial/rescued jobs; note it).

Cross-reference

• analyze-dataset-token-length — token-length method, Qwen3-8B convention, the metadata-confound trap.
• analyze-job-history-iris — harbor Mean: + productive-trial counts from job logs (alt metric source).
• datagen-launch-iris — how the trace datasets are produced/rescued/uploaded (upstream of this table).
• scripts/analysis/utils.py — the canonical extract/count/reward helpers this skill reuses.