# Habitat Gs Control

> Habitat Gs Control is an Agent Skill for apps & automation published in the zju3dv/habitat-gs repository on GitHub: a SKILL.md instruction file an AI agent can load to do this task.

Interactively pilot a robot in habitat-gs 3D Gaussian Splatting indoor scenes via MCP tools — load a scene, move, observe with RGB/depth, run autonomous nav loops, and export video. Use for hands-on control of a live sim session, NOT for training or evaluating a navigation policy (use the habitat-gs-train skill for that).

- Skill name: `habitat-gs-control`
- Category: [Apps & Automation](https://www.openparable.com/skill/apps-automation/1)
- Source: https://github.com/zju3dv/habitat-gs/blob/HEAD/skills/habitat-gs-control/SKILL.md
- Repository: zju3dv/habitat-gs (249 GitHub stars when collected)

## SKILL.md

# habitat-gs-control

Control a robot navigating through photo-realistic 3D indoor environments. You have MCP tools for scene loading, movement, observation, and autonomous navigation.

**IMPORTANT**: Always use MCP tools (`hab_init`, `hab_look`, `hab_forward`, etc.) directly. Do NOT use `exec` or shell commands — the MCP tools are the only supported interface.

## What you can do

- **Observe**: `hab_look` (current view), `hab_panorama` (4-direction scan with depth), `hab_depth_analyze` (obstacle distances)
- **Move**: `hab_forward` (walk forward), `hab_turn` (rotate left/right)
- **Navigate**: `hab_navigate` (auto-walk to coordinates), `hab_find_path` (plan route), `hab_topdown` (overhead map)
- **Long tasks**: `hab_nav_loop_start` (launch autonomous navigation sub-agent), `hab_nav_loop_status` (check progress), `hab_nav_loop_stop`
- **Session**: `hab_init` (load scene), `hab_close` (end session), `hab_export_video` (replay video)

## How to use

1. **Always init first**: call `hab_init` to load a scene before anything else.
2. **Look before you move**: use `hab_look` or `hab_panorama` to see the environment, then decide where to go.
3. **For simple tasks** (move forward, look around): use tools directly.
4. **For complex navigation** (find an object, reach a goal, follow instructions): use `hab_nav_loop_start` with a goal description. It launches an autonomous sub-agent that navigates independently. Check progress with `hab_nav_loop_status` only when asked.
5. **Export video** when the task is done: `hab_export_video` encodes all captured frames into an mp4.

## Navigation modes

The scene may run in two modes (auto-detected):

| | **navmesh** | **mapless** |
|---|---|---|
| Map tools (`hab_navigate`, `hab_find_path`, `hab_topdown`) | available | not available |
| Coordinates visible | yes | no — only `euclidean_distance_to_goal` and `goal_direction_deg` |
| How to navigate | use `hab_navigate` for efficient multi-step movement | use `hab_forward`/`hab_turn` with depth checks |

## Task types for nav_loop

| Type | What it does |
|---|---|
| `pointnav` | Navigate to a target coordinate |
| `objectnav` | Find a specific object in the scene |
| `imagenav` | Find the viewpoint matching a reference image |
| `eqa` | Explore and answer a question about the environment |
| `instruction_following` | Follow step-by-step natural language instructions |

Examples:
- PointNav: `hab_nav_loop_start(task_type="pointnav", goal_x=1.0, goal_y=0.2, goal_z=-3.0)`
- ObjectNav: `hab_nav_loop_start(task_type="objectnav", goal_description="find the kitchen door")`
- EQA: `hab_nav_loop_start(task_type="eqa", goal_description="Is there a microwave in this scene?")`
- ImageNav: **requires reference_image** — see below

### ImageNav workflow

ImageNav needs a reference image to match. You must capture it first:

1. Call `hab_look` — note the image **path** from the result (e.g. `"/path/to/session_step000001_color_sensor.png"`)
2. Move the robot away: `hab_forward`, `hab_turn`
3. Start imagenav with the saved path: `hab_nav_loop_start(task_type="imagenav", goal_description="match the reference view", reference_image="/path/to/session_step000001_color_sensor.png")`

If you skip `reference_image`, the sub-agent will not know what view to look for.

## Sending results to users

Use the `media` parameter in the `message` tool to send images or videos:

```
message(action="send", message="Here is the navigation replay", media="/path/to/video.mp4", filename="replay.mp4")
```

Do NOT paste file paths as plain text — always use the `media` parameter.

## Troubleshooting

| Problem | Solution |
|---|---|
| `Connection refused` | Bridge is not running — ask the user to start it |
| `Unknown session_id` | Session expired — call `hab_init` again |
| `ESP_CHECK failed` | Scene failed to load — try a different scene |

---

Source: https://www.openparable.com/skill/apps-automation/habitat-gs-control (OpenParable). Cite the canonical URL above.
