llm4ad-task-builder

Building an LLM4AD_Next Task Package

What LLM4AD_Next is

LLM4AD_Next is an automated algorithm-design platform: it combines an LLM with evolutionary optimization to discover and improve algorithms automatically. The user describes a problem (e.g. "evolve a solver for the Traveling Salesman Problem"), marks the code region to evolve with # EVOLVE_START / # EVOLVE_END, and the platform repeatedly asks an LLM to propose better code inside that region, scores each candidate with a custom evaluator, and keeps the best across generations. Better algorithms emerge through guided search rather than manual trial and error.

Your job with this skill: turn a user's problem into a complete, runnable task package, then verify it actually runs before handing it over.

What a task package is

A task package is a self-contained directory with everything the platform needs to evolve algorithms for one problem:

The contracts each file must satisfy

Algorithm file (<algo_dir>/<algo>.py):

• The algorithm directory is the one named in version_control.local_path; the platform copies it into a git worktree for each candidate and edits only the code between the markers.
• The function to evolve is wrapped exactly in # EVOLVE_START and # EVOLVE_END comment markers (the platform only edits code between them).
• It reads its input as a single JSON string from sys.argv[1], and prints a JSON result to stdout — so the evaluator can run it as a subprocess.
• The file name must match the name the evaluator looks for (see below). In the TSP example both sides use solve.py.

Evaluator (<name>_evaluator.py):

• Subclasses BaseEvaluator, decorated @BaseEvaluator.register("<name>"), with a no-argument __init__(self) (the base __init__ takes no config).
• Implements metrics (list of Metric(name, type=MINIMIZE|MAXIMIZE, weight, description)), name, and async def evaluate(self, cfg: EvalContext) -> EvaluationResult.
• cfg: EvalContext carries cfg.project_root (the candidate's worktree dir), cfg.data_path (the current data file), and cfg.timeout (seconds per instance).
• evaluate locates the algorithm file by hardcoded name under cfg.project_root (e.g. Path(cfg.project_root) / "solve.py"), runs it as a subprocess on cfg.data_path, parses its JSON output, validates it, and returns EvaluationResult(score, metrics, success, ...) (negative cost for a minimization objective). That hardcoded name must equal the algorithm file's actual name — this is the most common wiring mistake.

config.yaml (10 sections — generated programmatically, not hand-written):

• providers use ${LLM_BASE_URL} / ${LLM_API_KEY} / ${LLM_MODEL} env placeholders (the platform fills them in).
• The evaluator module reference (<file>.py:<ClassName>), the metrics list, the dataset path (data/sample), and the coder prompt_template's EVOLVE block must all be consistent with the other files.

debug_run.py / test_evaluator.py: standard boilerplate that runs config.yaml end-to-end and loads the evaluator, respectively.

See reference/config-template.yaml for the config shape and reference/example-tsp.md for a pointer to a complete, known-good package in this repo to adapt.

The information to gather before building

A correct package needs these decided (ask the user; let the agent fill sensible defaults where the user has no preference):

1. Problem description — what is being optimized. (Required — ask.)
2. Function/algorithm to evolve + input/output format — ask if the user has any constraints; if not, the agent designs it.
3. Evaluation metric(s) — what to minimize/maximize. (Required — ask clearly.)
4. Reuse existing code/evaluator? — user provides existing code, or generate from scratch. (Ask.)
5. Data source — user provides data, or generate sample instances. (Ask.)
6. Programming language — default Python. (Offer as a choice.)
7. Project name — the agent proposes one from the description; confirm with user.

Build workflow

1. Understand the problem and the 7 items above (inspect any user-provided code/data).
2. Write the algorithm file first; verify it runs standalone and prints valid JSON.
3. Write 2-3 sample data files under data/sample/.
4. Write the evaluator; keep its metrics consistent with config.yaml.
5. Write config.yaml, debug_run.py, test_evaluator.py.
6. Self-test (required): run test_evaluator.py (evaluator loads — no LLM, no network, cheap and always runnable). Then run debug_run.py, which executes the full pipeline and does call the LLM: it needs the LLM_BASE_URL / LLM_API_KEY / LLM_MODEL env vars set (the config uses these placeholders) and consumes tokens over the network. Read every error, fix the relevant file, re-run. Not done until both run cleanly. If no provider credentials are available, say so — a missing-credential failure of debug_run.py is not a package defect.

How the package is used on the LLM4AD platform

Once the package exists, the user runs it to evolve their algorithm. Two paths:

CLI:

llm4ad run path/to/config.yaml
# resume an interrupted run: pass a real checkpoint file written under
# ./runs/<project>/<run_id>/checkpoints/ (Island GA names them
# iga_checkpoint_gen_<N>_<hash>.json; MEoH names them meoh_<N>.json).
llm4ad run config.yaml --resume ./runs/<project>/<run_id>/checkpoints/iga_checkpoint_gen_5_a1b2c3d4.json

It launches the evolution pipeline and prints per-generation progress (best score each generation).

Web platform (this project's Web UI):

1. Create or open a task (the package's config.yaml + files back it).
2. Configure an LLM provider (the user's own key/model — OpenAI-compatible or Anthropic). The package's config.yaml references providers by the platform's env placeholders; the platform injects the real credentials at run time.
3. Start the evolution run; monitor progress in the browser — current generation, best individual's score, metrics, and logs — in real time.
4. When it finishes, inspect the best evolved algorithm and its score.

Tuning after generation (optional): users often adjust config.yaml evolution parameters to trade cost vs. quality — e.g. max_generations, num_islands, island_population_size, mutation_rate / crossover_rate, early_stop_patience, the provider model, and evaluator.timeout.

Where results go: each run writes to ./runs/{project}/{run_id}/:

• best/code/ — the evolved algorithm source (the main deliverable)
• best/metadata.json, best/summary.txt — score, generation, metrics
• state/evolution_state.json — full generation history (drives the Web UI dashboards)
• logs/llm4ad.log — full execution log for troubleshooting
• checkpoints/ — snapshots to resume from

Completion criteria

The package is done only when test_evaluator.py loads the evaluator AND debug_run.py runs without raising. Then summarize what was built (files + the 7 decisions) and the verification result.