Dataset Manager
Testing & Qualitydataset-managerDataset Manager is an Agent Skill for testing & quality published in the sirius-db/sirius repository on GitHub: a SKILL.md instruction file an AI agent can load to do this task.
Use this skill to generate benchmark datasets (TPC-H, TPC-DS, etc.). Trigger when the user needs test data at a specific scale factor for benchmarking or testing. Supports parquet and duckdb output formats.
How to use this skill
Bring this guide into your coding agent with a prompt tailored to the tool you use.
- Open your project in Codex.
- Copy and paste the prompt below.
- Review files and risks before approving any action.
Inspect the Agent Skill at https://github.com/sirius-db/sirius/blob/HEAD/.claude/skills/dataset-manager/SKILL.md and all supporting files. Report provenance, permissions, scripts, dependencies, and security risks. Propose an exact installation into .agents/skills/dataset-manager, but wait for my approval before writing files or running scripts.
Copying this prompt does not install or run the skill. Review third-party files before use.
Dataset Generator
Generate benchmark datasets by running the corresponding shell script under test/<benchmark>_performance/.
Gather Parameters
Parse $ARGUMENTS for:
- Benchmark: name from the registry below (first positional arg)
- Scale factor: integer (second positional arg)
- Format:
--format duckdb|parquet(optional — each benchmark has a default) - Cluster (TPC-H duckdb only):
--cluster/--cluster-keys <spec>(optional — physically sorts tables so row-group pruning works; errors with--format parquet) - Output path:
--output <path>(optional — scripts have sensible defaults)
If any required parameter (benchmark, scale factor) is missing, ask the user.
Benchmark Registry
Each entry follows the same structure: script location, command template, supported formats, defaults, and prerequisites.
TPC-H
| Field | Value |
|---|---|
| Script | test/tpch_performance/generate_tpch_data.sh |
| Default format | parquet |
| Formats | parquet (tpchgen-rs), duckdb (DuckDB dbgen()) |
| Default output (parquet) | test_datasets/tpch_parquet_sf<SF> |
| Default output (duckdb) | test_datasets/tpch_sf<SF>.duckdb |
Default output (duckdb + --cluster) | test_datasets/tpch_sf<SF>_sorted.duckdb |
| Prerequisites | Parquet: pixi env (rust, python, pyarrow). DuckDB: build/release/duckdb |
cd test/tpch_performance && pixi run bash generate_tpch_data.sh <SF> --format <FORMAT> [--cluster] [--cluster-keys <spec>] [--output <path>]
Notes:
- If the parquet output directory already exists, the script skips generation
--cluster(duckdb only) physically sorts tables at load so each row group's per-column min/max becomes selective — this is what makes Sirius's native-scan row-group pruning skip work on date-filtered queries. It writes a distincttpch_sf<SF>_sorted.duckdbso it can coexist with the unsorted dataset.- Default cluster keys:
lineitem:l_shipdate,orders:o_orderdate. Override with--cluster-keys "table:col,table:col,..."(implies--cluster). --clustererrors if combined with--format parquet(tpchgen-rs writes decimals as FIXED_LEN_BYTE_ARRAY, which disables Sirius row-group pruning for the whole file).
TPC-DS
| Field | Value |
|---|---|
| Script | test/tpcds_performance/generate_tpcds_data.sh |
| Default format | duckdb |
| Formats | duckdb, parquet |
| Default output (duckdb) | test_datasets/tpcds_sf<SF>.duckdb |
| Default output (parquet) | test_datasets/tpcds_parquet_sf<SF> |
| Prerequisites | build/release/duckdb |
cd test/tpcds_performance && bash generate_tpcds_data.sh <SF> --format <FORMAT> [--output <path>]
Notes:
- Also extracts TPC-DS query files to
test/tpcds_performance/queries/q{1..99}.sql
Prerequisites
For any benchmark that requires the DuckDB binary, check before running:
test -x build/release/duckdb
If missing, tell the user to build first: CMAKE_BUILD_PARALLEL_LEVEL=$(nproc) make
Report Results
- Benchmark name
- Output path
- Format used
- Whether generation was skipped (output already existed) or completed