OpenParable
Skip to content
← Back to skills

Run Rlhf Code Experiment

Testing & Quality
run-rlhf-code-experiment

Run Rlhf Code Experiment is an Agent Skill for testing & quality published in the natolambert/rlhf-book repository on GitHub: a SKILL.md instruction file an AI agent can load to do this task.

Plan, run, and report a small RLHF Book code experiment.

View on GitHub ↗
QUICK START

How to use this skill

Bring this guide into your coding agent with a prompt tailored to the tool you use.

  1. Open your project in Codex.
  2. Copy and paste the prompt below.
  3. Review files and risks before approving any action.
Prompt to paste
Inspect the Agent Skill at https://github.com/natolambert/rlhf-book/blob/HEAD/.claude/skills/run-rlhf-code-experiment/SKILL.md and all supporting files. Report provenance, permissions, scripts, dependencies, and security risks. Propose an exact installation into .agents/skills/run-rlhf-code-experiment, but wait for my approval before writing files or running scripts.

Copying this prompt does not install or run the skill. Review third-party files before use.

Run RLHF Code Experiment

Use this skill when the user wants to run, adapt, compare, or document an experiment from code/.

Pick The Starting Point

  • Policy gradients / RL / GRPO / PPO: read code/policy_gradients/README.md.
  • Reward models / ORM / PRM / Bradley-Terry RM: read code/reward_models/README.md.
  • DPO / IPO / SimPO / ORPO / KTO / APO: read code/direct_alignment/README.md.
  • Rejection sampling / best-of-N / GSM8K filtering: read code/rejection_sampling/README.md.

Run Protocol

  1. Work from the repository root unless a command explicitly says cd code/.
  2. Install or refresh dependencies with cd code/ && uv sync only when needed.
  3. Use uv run python, never bare python.
  4. Start with a short run:
    • Reward models: lower --samples and --epochs.
    • Direct alignment: use --max_samples or copy a YAML with a smaller sample count.
    • Policy gradients: copy a YAML and reduce data.size before changing algorithm logic.
    • Rejection sampling: reduce max_train_samples, max_test_samples, or num_completions_per_prompt in a copied YAML.
  5. For any long training, preprocessing, evaluation, or sweep command, launch the command in the background rather than the foreground. In Claude Code, use the background-run option for the shell command, then start a monitor for it.
  6. Watch the monitor until the run has produced initial logs or failed. The Claude Code status bar should show a background task and monitor (for example, [1 background task] [1 monitor]). Keep checking the monitor periodically for loss, metrics, W&B URLs, OOMs, dataset download errors, and stalled output.
  7. Run one training job at a time unless GPU memory has been checked.
  8. If W&B is not desired, set WANDB_MODE=disabled or use the module's no-W&B flag when available.

What To Report

Report enough detail for another reader to reproduce the result:

  • Exact command.
  • Model, dataset, seed, and config file.
  • Config values changed from the checked-in defaults.
  • Final metrics and any observed failure mode.
  • W&B run URL if logging was enabled.
  • Follow-up sweep worth trying next.

Comparison Rules

  • For policy gradients, compare avg_correctness, avg_format, avg_binary, loss, and whether sampled groups contain reward contrast.
  • For reward models, compare reward margins or correctness scores on held-out examples, not just training loss.
  • For direct alignment, compare accuracy, margins, chosen_rewards, rejected_rewards, and sample generations. IPO loss scale is not directly comparable to DPO loss scale.
  • For rejection sampling, always compare each reward-selected run to its matched random baseline.

Documentation Rule

If the run exposes a new setup requirement, failure mode, or useful workflow shortcut, update the relevant README, code/CLAUDE.md, or this skill before finishing.