OpenParable
Skip to content
← Back to skills

Isaac Camera

Development
isaac-camera

Isaac Camera is an Agent Skill for development published in the isaac-sim/IsaacSim repository on GitHub: a SKILL.md instruction file an AI agent can load to do this task.

Camera creation, configuration, calibration, render products, lens distortion, and runtime sensor wrapping in Isaac Sim 6 / Kit 110. Modern path: `isaacsim.sensors.experimental.rtx.RtxCamera` (authoring) + `CameraSensor` / `TiledCameraSensor` / `SingleViewDepthCameraSensor` (runtime) on top of `UsdGeomCamera`. Covers AOVs / annotators, OpenCV / fisheye / LUT lens distortion via `OmniLensDistortion*API` schemas, OpenCV intrinsics conversion, animation, and Replicator randomization. Use when creating/configuring camera prims, attaching annotators, applying distortion models, converting real-world calibration to Omniverse units, animating cameras, or generating synthetic data with Replicator.

View on GitHub ↗
QUICK START

How to use this skill

Bring this guide into your coding agent with a prompt tailored to the tool you use.

  1. Open your project in Codex.
  2. Copy and paste the prompt below.
  3. Review files and risks before approving any action.
Prompt to paste
Inspect the Agent Skill at https://github.com/isaac-sim/IsaacSim/blob/HEAD/skills/isaac-camera/SKILL.md and all supporting files. Report provenance, permissions, scripts, dependencies, and security risks. Propose an exact installation into .agents/skills/isaac-camera, but wait for my approval before writing files or running scripts.

Copying this prompt does not install or run the skill. Review third-party files before use.

Camera (Isaac Sim 6 / Kit 110)

Two layers stack:

  1. UsdGeomCamera — the USD camera prim (focal length, apertures, clipping, transform).
  2. isaacsim.sensors.experimental.rtx — the runtime sensor wrapper (authoring class + runtime sensor class). Use this for any new code that captures frames, attaches annotators, or applies lens distortion.

The legacy isaacsim.sensors.camera.Camera class still works but the extension itself is a stub (no Python). The implementation moved to isaacsim.sensors.experimental.rtx; new code should use that.

Migration: see the Migration from isaacsim.sensors.camera.Camera table below, plus the official camera migration guide for the full API change set.

When to use

  • Create or configure camera prims in USD stages.
  • Attach RGB / depth / segmentation / bbox annotators for capture.
  • Apply lens distortion (OpenCV pinhole, OpenCV fisheye, LUT).
  • Convert real-world calibration (OpenCV / ROS) to Omniverse units.
  • Build stereo, multi-sensor, or tiled multi-camera rigs.
  • Animate cameras (keyframes, motion paths, orbit, fly-through).
  • Generate synthetic data with Replicator randomization.
  • Set up Metropolis cameras for MEGA scenarios.

Fundamentals

Unit system

OpenUSD expresses optical properties in tenths of a scene unit.

PropertyUnitsNotes
Focal lengthtenths of scene unit35 = 35 mm when scene units are cm
Horizontal aperturetenths of scene unitsensor / film width
Vertical aperturetenths of scene unitsensor / film height
Focus distancescene unitsperfectly-sharp distance
Clipping rangescene unitsnear / far clip planes

Meter stage: a 25 mm focal length is 0.025.

Coordinate system

Camera looks down -Z; +Y is up, +X is right. Same regardless of stage up-axis.

Path A — RtxCamera + CameraSensor (recommended)

RtxCamera wraps a UsdGeom.Camera prim with OmniSensorAPI applied; CameraSensor builds a Replicator render product on top and exposes annotators. Both extend the isaacsim.core.experimental.prims.XformPrim family, so the standard pose / transform methods apply.

from isaacsim.sensors.experimental.rtx import RtxCamera, CameraSensor
import isaacsim.core.experimental.utils.app as app_utils

cam = RtxCamera(
    "/World/cam",
    tick_rate=30.0,                       # 0 = autotrigger
    aux_output_level="NONE",              # RtxCamera supports NONE only
)
cam.camera.set_focal_lengths(24.0)
cam.camera.set_clipping_ranges(0.01, 1000.0)

sensor = CameraSensor(
    cam,
    resolution=(480, 640),                # (height, width)
    annotators=["rgb", "distance_to_image_plane"],
)
app_utils.play(commit=True)
data = sensor.get_data("rgb")

TiledCameraSensor batches many cameras into one render call; useful for multi-view RL data collection. SingleViewDepthCameraSensor simulates stereo depth via OmniSensorDepthSensorSingleViewAPI.

Lens distortion (OpenCV fisheye / pinhole)

Set via API schemas + USD attributes at authoring time. The schema and attribute names are stable; use them through RtxCamera's schemas and attributes constructor args, or Apply them on the camera prim directly.

from isaacsim.sensors.experimental.rtx import RtxCamera
from pxr import Gf

cam = RtxCamera(
    "/World/fisheye",
    schemas=["OmniLensDistortionOpenCvFisheyeAPI"],
    attributes={
        "omni:lensdistortion:opencvFisheye:fx": 500.0,
        "omni:lensdistortion:opencvFisheye:fy": 500.0,
        "omni:lensdistortion:opencvFisheye:cx": 640.0,
        "omni:lensdistortion:opencvFisheye:cy": 360.0,
        "omni:lensdistortion:opencvFisheye:k1": 0.05,
        "omni:lensdistortion:opencvFisheye:imageSize": Gf.Vec2i(1280, 720),
    },
)

Equivalent OpenCV-pinhole schema: OmniLensDistortionOpenCvPinholeAPI (attributes omni:lensdistortion:opencvPinhole:{fx,fy,cx,cy,k1,k2,p1,p2,...}). LUT-based distortion uses OmniLensDistortionLutAPI.

Path B — UsdGeomCamera + omni.replicator.core (no RTX wrapper)

When you don't need RTX sensor features (lens distortion, ISP, tick-rate control), straight USD + Replicator works:

from pxr import UsdGeom, Gf
import omni.replicator.core as rep

cam = UsdGeom.Camera.Define(stage, "/World/Cam")
cam.GetFocalLengthAttr().Set(24.0)
cam.GetHorizontalApertureAttr().Set(36.0)            # full-frame 35 mm equiv
cam.GetVerticalApertureAttr().Set(20.25)
cam.GetClippingRangeAttr().Set(Gf.Vec2f(0.01, 1000.0))

rp = rep.create.render_product(cam.GetPath(), (1920, 1080))
rgb     = rep.AnnotatorRegistry.get_annotator("rgb")
depth   = rep.AnnotatorRegistry.get_annotator("distance_to_camera")
semseg  = rep.AnnotatorRegistry.get_annotator("semantic_segmentation",
                                              init_params={"colorize": True})
normals = rep.AnnotatorRegistry.get_annotator("normals")
for ann in (rgb, depth, semseg, normals):
    ann.attach([rp])

For pose authoring use isaacsim.core.experimental.objects.Camera (wraps UsdGeom.Camera with the standard pose / scale API), or the look-at helper:

from isaacsim.core.experimental.objects import Camera

cam = Camera(paths="/World/Cam", positions=[2.0, 2.0, 2.0])
cam.set_world_poses(positions=[[2.0, 2.0, 2.0]],
                    orientations=[[1.0, 0.0, 0.0, 0.0]])

OpenCV intrinsics -> USD

from pxr import UsdGeom

def opencv_to_usd_camera(stage, path, fx, fy, cx, cy, image_w, image_h,
                         sensor_w_m=0.036):
    """OpenCV intrinsics -> USD camera on a meter-stage."""
    focal_length_m = fx * sensor_w_m / image_w
    h_aperture     = sensor_w_m
    v_aperture     = fy * sensor_w_m / fx   # preserve aspect
    cam = UsdGeom.Camera.Define(stage, path)
    cam.GetFocalLengthAttr().Set(focal_length_m)
    cam.GetHorizontalApertureAttr().Set(h_aperture)
    cam.GetVerticalApertureAttr().Set(v_aperture)
    # Principal-point offset:
    cam.GetHorizontalApertureOffsetAttr().Set((cx - image_w / 2) * sensor_w_m / image_w)
    cam.GetVerticalApertureOffsetAttr().Set((cy - image_h / 2) * sensor_w_m / image_w)
    return cam

Animation

from pxr import Usd, Gf

for frame, pos in enumerate(camera_positions):
    cam.GetPrim().GetAttribute("xformOp:translate").Set(
        Gf.Vec3d(*pos), Usd.TimeCode(frame)
    )

Replicator randomization

import omni.replicator.core as rep

with rep.new_layer():
    cameras = rep.get.prims(path_pattern="/World/Camera.*")
    with rep.trigger.on_frame(num_frames=200):
        with cameras:
            rep.modify.pose(
                position=rep.distribution.uniform((-5, -5, 2), (5, 5, 8)),
                look_at="/World/Target",
            )

Sensor checker / supported configs

For asset-driven lidar / camera configs and validation helpers, see isaacsim.sensors.experimental.rtx.SUPPORTED_LIDAR_CONFIGS and the sensor_checker module bundled with the extension. The Replicator AOV annotator names match the standard registry: rgb, distance_to_image_plane, distance_to_camera, normals, semantic_segmentation, instance_segmentation, bounding_box_2d_tight, bounding_box_2d_loose, bounding_box_3d, camera_params, pointcloud, occlusion.

Migration from isaacsim.sensors.camera.Camera

Full guide: isaacsim.sensors.camera → isaacsim.sensors.experimental.rtx.

OldNew
isaacsim.sensors.camera.Camera("/World/cam", resolution=(1280, 720))RtxCamera("/World/cam") + CameraSensor(cam, resolution=(720, 1280), annotators=["rgb"])
camera.set_focal_length(24)cam.camera.set_focal_lengths(24.0)
camera.set_clipping_range(0.01, 1000)cam.camera.set_clipping_ranges(0.01, 1000)
camera.initialize() then camera.get_current_frame()app_utils.play(commit=True) then sensor.get_data(annotator)
Carb-settings distortion (/<path>/distortionModel)OmniLensDistortion*API schemas on the camera prim