Data Collection Sim
Developmentdata-collection-simData Collection Sim is an Agent Skill for development published in the isaac-sim/IsaacSim repository on GitHub: a SKILL.md instruction file an AI agent can load to do this task.
Build headless static-scene data collection pipelines using Isaac Sim 6 / Kit 110 Replicator. Produces annotated RGB, depth, segmentation, bounding boxes, and pose data via `BasicWriter`, `KittiWriter`, `CocoWriter`, `CosmosWriter`, `FPSWriter` (from `omni.replicator.core`) and `PoseWriter`, `DataVisualizationWriter` (from `isaacsim.replicator.writers`). Uses `isaacsim.core.experimental.utils.stage` to author the scene and either `rep.functional.modify.semantics` or `isaacsim.core.experimental.utils.semantics.add_labels` to label prims for annotators. Use when generating synthetic data, creating training datasets, or building SDG pipelines. For mobile-robot trajectory-driven SDG see `mobility-gen`; for grasp / teleop / episode-record workflows see `isaacsim.replicator.*` pointers below.
How to use this skill
Bring this guide into your coding agent with a prompt tailored to the tool you use.
- Open your project in Codex.
- Copy and paste the prompt below.
- Review files and risks before approving any action.
Inspect the Agent Skill at https://github.com/isaac-sim/IsaacSim/blob/HEAD/skills/data-collection-sim/SKILL.md and all supporting files. Report provenance, permissions, scripts, dependencies, and security risks. Propose an exact installation into .agents/skills/data-collection-sim, but wait for my approval before writing files or running scripts.
Copying this prompt does not install or run the skill. Review third-party files before use.
Data Collection Simulation
Build headless SDG pipelines using Isaac Sim 6.0 Replicator. Outputs annotated frames (RGB, depth, segmentation, bbox, pose) to disk via writers.
Related Skills
mobility-gen— mobile-robot trajectory recording then replay+render (two-phase, robot-mounted sensors)isaac-sim-sensor— sensor primitives (camera, LiDAR, IMU, contact)occupancy-map— produces occupancy maps for spawn placement
Architecture
Config (YAML/JSON) → SimulationApp (headless) → Scene Setup → Randomizers → Capture Loop → Writer → Disk
Required Imports
from isaacsim import SimulationApp
simulation_app = SimulationApp({"renderer": "RealTimePathTracing", "headless": True})
import carb.settings
import omni.replicator.core as rep
import omni.usd
import isaacsim.core.experimental.utils.stage as stage_utils
from isaacsim.core.experimental.utils.semantics import add_labels, remove_all_labels
from isaacsim.storage.native import get_assets_root_path
stage_utils.open_stage() / stage_utils.add_reference_to_stage() / stage_utils.define_prim() / stage_utils.get_current_stage() are the Kit 110 replacements for the legacy isaacsim.core.utils.stage.* and isaacsim.core.utils.prims.create_prim flow. Keep omni.replicator.core for SDG primitives.
Migration: for the full
omni.isaac.*→isaacsim.*mapping, see Renaming Extensions.
Writers
Resolve writers via rep.WriterRegistry.get(name) (or rep.writers.get(name), same registry).
omni.replicator.core built-in writers (omni.replicator.core/scripts/writers_default/):
| Writer | Use case |
|---|---|
BasicWriter | RGB, bbox_2d (tight/loose), bbox_3d, semantic/instance/instance_id segmentation, depth (image-plane / camera), normals, occlusion, motion vectors, camera params, pointcloud, skeleton |
KittiWriter | KITTI-format datasets |
CocoWriter | COCO-format datasets (instance/bbox) |
CosmosWriter | Cosmos warehouse video clips (PNG sequences + MP4 per modality: rgb, shaded_seg, segmentation, depth, edges). Requires /app/omni.graph.scriptnode/opt_in = True |
FPSWriter | Frame-rate / capture-time telemetry |
Custom Writer subclass | direct access to annotator tensors (subclass omni.replicator.core.Writer, register with rep.writers.register_writer) |
isaacsim.replicator.writers adds Isaac-specific writers:
| Writer | Use case |
|---|---|
PoseWriter | 6-DoF object pose estimation (optional write_debug_images=True) |
DataVisualizationWriter | debug / overlay visualization |
Deprecated and not for new work: DOPEWriter, YCBVideoWriter, PytorchWriter, PytorchListener (also OgnPose node). They will be removed in a future major release.
Writer initialization patterns
Two equivalent patterns are supported. Recent (Isaac Sim 6.0+) examples favor the explicit-backend form:
backend = rep.backends.get("DiskBackend")
backend.initialize(output_dir="/tmp/sdg_output")
writer = rep.writers.get("BasicWriter")
writer.initialize(backend=backend, rgb=True, bounding_box_2d_tight=True)
writer.attach(rp)
Legacy short form (still works; backend created implicitly from output_dir):
writer = rep.WriterRegistry.get("BasicWriter")
writer.initialize(output_dir="/tmp/sdg_output", rgb=True, bounding_box_2d_tight=True)
writer.attach(rp)
Annotators
Core annotators available via rep.annotators.get(name):
rgb— RGBA uint8distance_to_image_plane— depth float32semantic_segmentation— per-pixel class labelsinstance_segmentation— per-pixel instance IDsbounding_box_2d_tight— tight 2D bboxesbounding_box_2d_loose— loose 2D bboxesbounding_box_3d— 3D bboxes in world coordscamera_params— intrinsics, extrinsics, resolutionocclusion— visibility ratio per instancenormals— surface normalspointcloud— 3D point cloud from depth
Minimal Pipeline
# 1. Scene
rep.orchestrator.set_capture_on_play(False)
carb.settings.get_settings().set("rtx/post/dlss/execMode", 2) # DLSS Quality
assets_root = get_assets_root_path()
stage_utils.open_stage(assets_root + "/Isaac/Environments/Simple_Warehouse/full_warehouse.usd")
# 2. Semantic labels — add a referenced prop and tag it for annotators.
stage_utils.add_reference_to_stage(
usd_path=assets_root + "/Isaac/Props/YCB/Axis_Aligned/003_cracker_box.usd",
path="/World/MyObj",
)
prim = stage_utils.get_current_stage().GetPrimAtPath("/World/MyObj")
# Either API writes the UsdSemantics LabelsAPI schema on the prim:
add_labels(prim, labels=["cracker_box"], taxonomy="class")
# or, equivalently:
# rep.functional.modify.semantics(prim, {"class": "cracker_box"}, mode="add")
# 3. Camera + render product
cam = rep.functional.create.camera(position=(3, 3, 2), look_at=(0, 0, 0), name="DataCam")
rp = rep.create.render_product(cam, (1280, 720), name="main_view")
# 4. Writer (explicit backend form)
backend = rep.backends.get("DiskBackend")
backend.initialize(output_dir="/tmp/sdg_output")
writer = rep.writers.get("BasicWriter")
writer.initialize(
backend=backend,
rgb=True,
bounding_box_2d_tight=True,
semantic_segmentation=True,
distance_to_image_plane=True,
)
writer.attach(rp)
# 5. Capture loop with randomization (static scene: delta_time=0.0 freezes timeline)
for i in range(NUM_FRAMES):
# Randomize object pose, camera, lights here
rep.orchestrator.step(delta_time=0.0, rt_subframes=32)
# 6. Cleanup
rep.orchestrator.wait_until_complete()
writer.detach()
rp.destroy()
simulation_app.close()
Domain Randomization
Use rep.functional API for randomization each frame:
# Scatter objects on surface
rep.functional.randomizer.scatter_2d(prims=objects, surface_prims=plane, check_for_collisions=True, rng=rng)
# Randomize camera pose
rep.functional.modify.pose(cam, position_value=rng.uniform(pos_min, pos_max),
look_at_value=target_prim, look_at_up_axis=(0, 0, 1))
# Randomize lights via OmniGraph events
rep.utils.send_og_event(event_name="randomize_lights")
Headless Execution
The canonical Isaac Sim Python launcher is python.sh (use python.bat on Windows). isaac-sim.sh launches the full editor app and is not the right entry point for standalone SDG scripts.
# $ISAAC_SIM_DIR is either the install root or <repo>/_build/linux-x86_64/release
"$ISAAC_SIM_DIR/python.sh" "$WORKSPACE_DIR/data_collection.py" --config config.yaml
Headlessness is controlled by SimulationApp({"headless": True}) inside the script, not by a launcher flag.
Or via the Isaac Lab runner ($ISAAC_LAB_DIR is your Isaac Lab checkout):
"$ISAAC_LAB_DIR/isaaclab.sh" -p data_collection.py --config config.yaml
Configuration Pattern
Use YAML config to parameterize everything:
resolution: [1280, 720]
rt_subframes: 32
num_frames: 100
headless: true
env_url: "/Isaac/Environments/Simple_Warehouse/full_warehouse.usd"
writer: BasicWriter
output_dir: /tmp/sdg_output
annotations:
rgb: true
bounding_box_2d_tight: true
semantic_segmentation: true
distance_to_image_plane: true
bounding_box_3d: true
objects:
- url: "/Isaac/Props/YCB/Axis_Aligned/003_cracker_box.usd"
label: cracker_box
count: 5
- url: "/Isaac/Props/YCB/Axis_Aligned/008_pudding_box.usd"
label: pudding_box
count: 3
Validation Checklist
- Output directory contains expected number of frames
- RGB images are non-black (mean RGB > 30)
- Annotation files match frame count
- Semantic labels appear in segmentation maps
- Bounding boxes have non-zero area
- No NaN in depth maps
Key Rules
- Set
rep.orchestrator.set_capture_on_play(False)for manual step control. rt_subframes: render the same frame multiple times to reduce ghosting from large pose deltas and to let materials/textures converge. Tune for your renderer: small (4-8) is often enough for RTX Real-Time + DLSS Quality; 16-32 is typical for path tracing or scenes with heavy material streaming.- DLSS Quality:
carb.settings.get_settings().set("rtx/post/dlss/execMode", 2). Recommended for SDG; default Performance mode can produce edge artifacts below ~600x600. - Tag every prim you want annotated via
add_labels(...)(taxonomy-aware) orrep.functional.modify.semantics(...). Both write theUsdSemantics.LabelsAPIschema. - For static-scene SDG, pass
delta_time=0.0torep.orchestrator.stepso the timeline does not advance between captures. - Call
rep.orchestrator.wait_until_complete()before cleanup so the background backend has flushed everything to disk. - Use
rng = np.random.default_rng(seed)andrep.set_global_seed(seed)for reproducible randomization. - Performance knobs for high-throughput captures (see
sdg_getting_started_05.py):rep.orchestrator.step(wait_for_render=False)decouples capture from render completion. Data may correspond to a previous frame; only use when strict frame-to-data correspondence is not required.carb.settings.get_settings().set("/exts/omni.replicator.core/enableWriteToFabric", True)writes randomization deltas directly to Fabric instead of going through USD first. Faster, but transient — changes are not persisted in the USD stage.
Sibling SDG workflows
| Workflow | Module / example |
|---|---|
| Mobile-robot trajectory record + replay | mobility-gen skill + isaacsim.replicator.mobility_gen |
| Grasp dataset generation | isaacsim.replicator.grasping (GraspingManager, GraspPhase) — source/standalone_examples/api/isaacsim.replicator.grasping/grasping_workflow_sdg.py |
| Episode record + replay | isaacsim.replicator.episode_recorder |
| Teleop record + replay | isaacsim.replicator.teleop |
| Reusable randomization behavior scripts (attached to prims, USD-persistent) | isaacsim.replicator.behavior |
| Sensor primitives (LiDAR, IMU, camera) | isaac-sim-sensor, isaac-camera |
| Spawn placement from a map | occupancy-map |
Reference examples
Paths relative to $ISAAC_SIM_DIR (install root or <this-repo> for a source build) and $ISAAC_LAB_DIR:
- API examples:
$ISAAC_SIM_DIR/source/standalone_examples/api/isaacsim.replicator.examples/sdg_getting_started_0[1-5].py,sdg_workflow_0[12].py,multi_camera.py,motion_blur.py,cosmos_writer_simple.py,simready_assets_sdg.py,sdg_deformables.py,sdg_geomsubset.py,subscribers_and_events.py,custom_event_and_write.py,custom_fps_writer_annotator.py. - Scene-based SDG:
$ISAAC_SIM_DIR/source/standalone_examples/replicator/scene_based_sdg/ - Object-based SDG:
$ISAAC_SIM_DIR/source/standalone_examples/replicator/object_based_sdg/ - Augmentation pipelines:
$ISAAC_SIM_DIR/source/standalone_examples/replicator/augmentation/ - Cosmos warehouse writer:
$ISAAC_SIM_DIR/source/standalone_examples/replicator/cosmos_writer_warehouse.py - Infinigen SDG:
$ISAAC_SIM_DIR/source/standalone_examples/replicator/infinigen/ - Grasping SDG:
$ISAAC_SIM_DIR/source/standalone_examples/api/isaacsim.replicator.grasping/grasping_workflow_sdg.py - Isaac Lab imitation learning:
$ISAAC_LAB_DIR/scripts/imitation_learning/