OpenParable
Skip to content
← Back to skills

Habitat Gs Control

Apps & Automation
habitat-gs-control

Habitat Gs Control is an Agent Skill for apps & automation published in the zju3dv/habitat-gs repository on GitHub: a SKILL.md instruction file an AI agent can load to do this task.

Interactively pilot a robot in habitat-gs 3D Gaussian Splatting indoor scenes via MCP tools — load a scene, move, observe with RGB/depth, run autonomous nav loops, and export video. Use for hands-on control of a live sim session, NOT for training or evaluating a navigation policy (use the habitat-gs-train skill for that).

View on GitHub ↗
QUICK START

How to use this skill

Bring this guide into your coding agent with a prompt tailored to the tool you use.

  1. Open your project in Codex.
  2. Copy and paste the prompt below.
  3. Review files and risks before approving any action.
Prompt to paste
Inspect the Agent Skill at https://github.com/zju3dv/habitat-gs/blob/HEAD/skills/habitat-gs-control/SKILL.md and all supporting files. Report provenance, permissions, scripts, dependencies, and security risks. Propose an exact installation into .agents/skills/habitat-gs-control, but wait for my approval before writing files or running scripts.

Copying this prompt does not install or run the skill. Review third-party files before use.

habitat-gs-control

Control a robot navigating through photo-realistic 3D indoor environments. You have MCP tools for scene loading, movement, observation, and autonomous navigation.

IMPORTANT: Always use MCP tools (hab_init, hab_look, hab_forward, etc.) directly. Do NOT use exec or shell commands — the MCP tools are the only supported interface.

What you can do

  • Observe: hab_look (current view), hab_panorama (4-direction scan with depth), hab_depth_analyze (obstacle distances)
  • Move: hab_forward (walk forward), hab_turn (rotate left/right)
  • Navigate: hab_navigate (auto-walk to coordinates), hab_find_path (plan route), hab_topdown (overhead map)
  • Long tasks: hab_nav_loop_start (launch autonomous navigation sub-agent), hab_nav_loop_status (check progress), hab_nav_loop_stop
  • Session: hab_init (load scene), hab_close (end session), hab_export_video (replay video)

How to use

  1. Always init first: call hab_init to load a scene before anything else.
  2. Look before you move: use hab_look or hab_panorama to see the environment, then decide where to go.
  3. For simple tasks (move forward, look around): use tools directly.
  4. For complex navigation (find an object, reach a goal, follow instructions): use hab_nav_loop_start with a goal description. It launches an autonomous sub-agent that navigates independently. Check progress with hab_nav_loop_status only when asked.
  5. Export video when the task is done: hab_export_video encodes all captured frames into an mp4.

Navigation modes

The scene may run in two modes (auto-detected):

navmeshmapless
Map tools (hab_navigate, hab_find_path, hab_topdown)availablenot available
Coordinates visibleyesno — only euclidean_distance_to_goal and goal_direction_deg
How to navigateuse hab_navigate for efficient multi-step movementuse hab_forward/hab_turn with depth checks

Task types for nav_loop

TypeWhat it does
pointnavNavigate to a target coordinate
objectnavFind a specific object in the scene
imagenavFind the viewpoint matching a reference image
eqaExplore and answer a question about the environment
instruction_followingFollow step-by-step natural language instructions

Examples:

  • PointNav: hab_nav_loop_start(task_type="pointnav", goal_x=1.0, goal_y=0.2, goal_z=-3.0)
  • ObjectNav: hab_nav_loop_start(task_type="objectnav", goal_description="find the kitchen door")
  • EQA: hab_nav_loop_start(task_type="eqa", goal_description="Is there a microwave in this scene?")
  • ImageNav: requires reference_image — see below

ImageNav workflow

ImageNav needs a reference image to match. You must capture it first:

  1. Call hab_look — note the image path from the result (e.g. "/path/to/session_step000001_color_sensor.png")
  2. Move the robot away: hab_forward, hab_turn
  3. Start imagenav with the saved path: hab_nav_loop_start(task_type="imagenav", goal_description="match the reference view", reference_image="/path/to/session_step000001_color_sensor.png")

If you skip reference_image, the sub-agent will not know what view to look for.

Sending results to users

Use the media parameter in the message tool to send images or videos:

message(action="send", message="Here is the navigation replay", media="/path/to/video.mp4", filename="replay.mp4")

Do NOT paste file paths as plain text — always use the media parameter.

Troubleshooting

ProblemSolution
Connection refusedBridge is not running — ask the user to start it
Unknown session_idSession expired — call hab_init again
ESP_CHECK failedScene failed to load — try a different scene