### Starting GRUtopia Docker Container (Bash)

Source: https://github.com/openrobotlab/grutopia/blob/main/docs/en/get_started/installation.md

Starts an interactive Docker container from the built GRUtopia image, mounting necessary volumes for display access, project files, and cache directories. It runs with GPU support and host networking. Requires Docker, NVIDIA Container Toolkit, and appropriate directory structures for caches.

```bash
xhost +local:root # Allow the container to access the display

cd PATH/TO/GRUTOPIA/ROOT

docker run --name grutopia -it --rm --gpus all --network host \
  -e "ACCEPT_EULA=Y" \
  -e "PRIVACY_CONSENT=Y" \
  -e "DISPLAY=${DISPLAY}" \
  -v /tmp/.X11-unix/:/tmp/.X11-unix \
  -v ${PWD}:/isaac-sim/GRUtopia \
  -v ${HOME}/docker/isaac-sim/cache/kit:/isaac-sim/kit/cache:rw \
  -v ${HOME}/docker/isaac-sim/cache/ov:/root/.cache/ov:rw \
  -v ${HOME}/docker/isaac-sim/cache/pip:/root/.cache/pip:rw \
  -v ${HOME}/docker/isaac-sim/cache/glcache:/root/.cache/nvidia/GLCache:rw \
  -v ${HOME}/docker/isaac-sim/cache/computecache:/root/.nv/ComputeCache:rw \
  -v ${HOME}/docker/isaac-sim/logs:/root/.nvidia-omniverse/logs:rw \
  -v ${HOME}/docker/isaac-sim/data:/root/.local/share/ov/data:rw \
  -v ${HOME}/docker/isaac-sim/documents:/root/Documents:rw \
  grutopia:<your tag>
```

--------------------------------

### Verifying GRUtopia Installation with Demo (Shell/Python)

Source: https://github.com/openrobotlab/grutopia/blob/main/docs/en/get_started/installation.md

Runs a specific GRUtopia demo module (`h1_locomotion`) to verify that the installation is working correctly. A successful execution should launch the Isaac Sim GUI and display a simulation. Requires GRUtopia to be installed and accessible via `python -m`.

```shell
python -m grutopia.demo.h1_locomotion  # start simulation
```

--------------------------------

### Example: Start Hamer Server with Local Video URL (Bash)

Source: https://github.com/openrobotlab/grutopia/blob/main/docs/en/get_started/teleoperating-with-mocap.md

An example command to start the Hamer server using a local loopback address and port 5000 for the video stream URL, as generated by the sample RGB camera server.

```bash
python hamer_re.py --video_url http://127.0.0.1:5000/video
```

--------------------------------

### Configuring Conda Environment from Source (Bash)

Source: https://github.com/openrobotlab/grutopia/blob/main/docs/en/get_started/installation.md

Executes the setup script included in the GRUtopia source code. This script automatically creates and configures the necessary Conda environment for running GRUtopia.

```bash
./setup_conda.sh
```

--------------------------------

### Install Tmux - Shell

Source: https://github.com/openrobotlab/grutopia/blob/main/grutopia_extension/agents/social_navigation_agent/README.md

Installs the tmux terminal multiplexer using apt. Tmux is required for running the agent entrypoint script. This command updates the package list and then installs tmux.

```shell
sudo apt update
sudo apt install tmux
```

--------------------------------

### Install Python Dependencies (Shell)

Source: https://github.com/openrobotlab/grutopia/blob/main/grutopia_extension/agents/mobile_manipulation_agent/README.md

Installs required Python packages listed in the requirements.txt file using pip. Ensure the CUDA toolkit version matches the PyTorch CUDA version.

```shell
pip install -r requirements.txt
```

--------------------------------

### Install Submodules (Shell)

Source: https://github.com/openrobotlab/grutopia/blob/main/grutopia_extension/agents/mobile_manipulation_agent/README.md

Creates an 'images' directory and clones the GroundingDINO and yolov7 repositories as submodules. Ensure the torch CUDA version matches the system CUDA version.

```shell
mkdir images
git clone https://github.com/IDEA-Research/GroundingDINO.git
git clone https://github.com/WongKinYiu/yolov7.git
```

--------------------------------

### Cloning GRUtopia Repository (Bash)

Source: https://github.com/openrobotlab/grutopia/blob/main/docs/en/get_started/installation.md

Clones the GRUtopia project repository from GitHub to the local machine. This is the first step in setting up the project. Requires Git to be installed.

```bash
git clone git@github.com:OpenRobotLab/GRUtopia.git
```

--------------------------------

### Install Python Dependencies (Agent) - Shell

Source: https://github.com/openrobotlab/grutopia/blob/main/grutopia_extension/agents/social_navigation_agent/README.md

Installs the necessary Python packages for the social navigation agent from the `requirements.txt` file using pip. Requires being in the `grutopia_extension/agents/social_navigation_agent` directory. Ensure CUDA toolkit version matches PyTorch CUDA version.

```shell
pip install -r requirements.txt
```

--------------------------------

### Install Submodules (GroundingDINO, YOLOv7) - Shell

Source: https://github.com/openrobotlab/grutopia/blob/main/grutopia_extension/agents/social_navigation_agent/README.md

Creates an `images` directory and clones the GroundingDINO and YOLOv7 repositories as submodules within the `grutopia_extension/agents/social_navigation_agent` directory. Requires being in this directory. Ensure PyTorch CUDA version matches CUDA version.

```shell
mkdir images
git clone https://github.com/IDEA-Research/GroundingDINO.git
git clone https://github.com/WongKinYiu/yolov7.git
```

--------------------------------

### Install Tmux (Shell)

Source: https://github.com/openrobotlab/grutopia/blob/main/grutopia_extension/agents/mobile_manipulation_agent/README.md

Updates the package list and installs the tmux terminal multiplexer using apt. Tmux is required for running the agent entrypoint script.

```shell
sudo apt update
sudo apt install tmux
```

--------------------------------

### Cloning GRUtopia Repository (Bash)

Source: https://github.com/openrobotlab/grutopia/blob/main/docs/en/get_started/installation.md

Clones the GRUtopia source code repository from GitHub using the git command. This is the initial step when installing GRUtopia from source.

```bash
git clone git@github.com:OpenRobotLab/GRUtopia.git
```

--------------------------------

### Launch VLM Servers (Agent Process) - Shell

Source: https://github.com/openrobotlab/grutopia/blob/main/grutopia_extension/agents/social_navigation_agent/README.md

Executes the `launch_vlm_servers.sh` script located in the `./scripts` directory relative to the current working directory (`grutopia_extension/agents/social_navigation_agent`). This script is responsible for starting the agent process.

```shell
./scripts/launch_vlm_servers.sh
```

--------------------------------

### Setup Conda Env and Run IK Solver (Bash)

Source: https://github.com/openrobotlab/grutopia/blob/main/docs/en/get_started/teleoperating-with-visionpro.md

Creates a new conda environment named `sim-teleop` with Python 3.10, activates it, installs necessary Python packages from a requirements file, and then executes the `gr1_teleop.py` script within this environment to start the IK solver process.

```bash
$ conda create -n sim-teleop python=3.10
$ conda activate sim-teleop
$ pip install -r GRUtopia/requirements/teleop.txt
$ cd GRUtopia/grutopia_extension/controllers && conda run --no-capture-output -n sim-teleop python gr1_teleop.py
```

--------------------------------

### Navigating to GRUtopia Root Directory (Bash)

Source: https://github.com/openrobotlab/grutopia/blob/main/docs/en/get_started/installation.md

Changes the current directory to the root directory of the cloned GRUtopia repository. Replace 'PATH/TO/GRUTOPIA/ROOT' with the actual path where the repository was cloned.

```bash
cd PATH/TO/GRUTOPIA/ROOT
```

--------------------------------

### Activating GRUtopia Conda Environment (Bash)

Source: https://github.com/openrobotlab/grutopia/blob/main/docs/en/get_started/installation.md

Changes back to the parent directory and then activates the Conda environment named 'grutopia' (or the name chosen during the setup script execution). This environment contains the required dependencies.

```bash
cd .. && conda activate grutopia  # or your conda env name
```

--------------------------------

### Installing GRUtopia via Pip (Bash)

Source: https://github.com/openrobotlab/grutopia/blob/main/docs/en/get_started/installation.md

Installs the GRUtopia package from the Python Package Index (PyPI) into the currently active Conda environment using the pip package manager. Requires git to be installed.

```bash
pip install grutopia
```

--------------------------------

### Downloading GRUtopia Assets Automatically (Shell/Python)

Source: https://github.com/openrobotlab/grutopia/blob/main/docs/en/get_started/installation.md

Executes a Python module within the GRUtopia installation to automatically download required assets. The script prompts the user to choose between full or minimum asset sets and specify a download path. Requires GRUtopia to be installed and accessible via `python -m`.

```shell
python -m grutopia.download_assets
```

--------------------------------

### Activating Conda Environment for PyPI Installation (Bash)

Source: https://github.com/openrobotlab/grutopia/blob/main/docs/en/get_started/installation.md

Activates the newly created Conda environment specified by '<env>'. Subsequent commands, like pip install, will operate within this environment.

```bash
conda activate <env>
```

--------------------------------

### Create and Enter Data Directory - Shell

Source: https://github.com/openrobotlab/grutopia/blob/main/grutopia_extension/agents/social_navigation_agent/README.md

Creates a `data` directory within the current working directory (`grutopia_extension/agents/social_navigation_agent`) and then changes the current directory to the newly created `data` directory. This directory is used to store downloaded model weights.

```shell
mkdir data
cd data
```

--------------------------------

### Building GRUtopia Docker Image (Bash)

Source: https://github.com/openrobotlab/grutopia/blob/main/docs/en/get_started/installation.md

Navigates to the GRUtopia root directory and builds a custom Docker image based on the project's Dockerfile. The image is tagged with a user-specified name. Requires Docker and being in the project's root directory.

```bash
cd PATH/TO/GRUTOPIA/ROOT

docker build -t grutopia:<your tag> .
```

--------------------------------

### Reactivating Conda Environment after PyPI Setup (Bash)

Source: https://github.com/openrobotlab/grutopia/blob/main/docs/en/get_started/installation.md

Deactivates the current Conda environment and then immediately reactivates it. This step ensures that any changes made by the setup script are fully applied to the active environment.

```bash
conda deactivate && conda activate <env>
```

--------------------------------

### Create Data Directory (Shell)

Source: https://github.com/openrobotlab/grutopia/blob/main/grutopia_extension/agents/mobile_manipulation_agent/README.md

Creates a 'data' directory and changes the current directory into it. This directory is used to store downloaded model weights.

```shell
mkdir data
cd data
```

--------------------------------

### Configuring Conda Environment for PyPI (Bash)

Source: https://github.com/openrobotlab/grutopia/blob/main/docs/en/get_started/installation.md

Runs a specific Python module provided by the GRUtopia package to perform post-installation configuration of the Conda environment when installed via PyPI.

```bash
python -m grutopia.setup_conda_pypi
```

--------------------------------

### Installing Infinigen with Git and Conda (Bash)

Source: https://github.com/openrobotlab/grutopia/blob/main/toolkits/indoor_scenes_generation/readme.md

Clones the Infinigen repository from GitHub, navigates into the directory, creates and activates a new conda environment named 'infinigen' with Python 3.11, and then runs the interactive Blender installation script provided by Infinigen.

```bash
git clone https://github.com/princeton-vl/infinigen.git
cd infinigen
conda create --name infinigen python=3.11
conda activate infinigen
bash scripts/install/interactive_blender.sh
```

--------------------------------

### Launch VLM Servers (Shell)

Source: https://github.com/openrobotlab/grutopia/blob/main/grutopia_extension/agents/mobile_manipulation_agent/README.md

Executes the script to launch the Vision-Language Model (VLM) servers. This is the entrypoint for running the agent process.

```shell
./scripts/launch_vlm_servers.sh
```

--------------------------------

### Pulling Isaac Sim Docker Image (Bash)

Source: https://github.com/openrobotlab/grutopia/blob/main/docs/en/get_started/installation.md

Downloads the specified version of the NVIDIA Isaac Sim Docker image from NGC. Requires Docker and authentication with NGC (`docker login`).

```bash
docker pull nvcr.io/nvidia/isaac-sim:4.2.0
```

--------------------------------

### Install Benchmark Dependencies (Shell)

Source: https://github.com/openrobotlab/grutopia/blob/main/docs/en/get_started/run-benchmark-baseline.md

Navigates to the GRUtopia root directory and installs Python dependencies required for running the benchmark using pip and the specified requirements file.

```shell
cd PATH/TO/GRUTOPIA/ROOT
pip install -r requirements/benchmark.txt
```

--------------------------------

### Creating Conda Environment for PyPI Installation (Bash)

Source: https://github.com/openrobotlab/grutopia/blob/main/docs/en/get_started/installation.md

Creates a new Conda environment with a specified Python version (3.10). This environment will be used to install GRUtopia via pip when following the PyPI installation method. Replace '<env>' with your desired environment name.

```bash
conda create -n <env> python=3.10
```

--------------------------------

### Install Dependencies for RGB Camera Server (Bash)

Source: https://github.com/openrobotlab/grutopia/blob/main/docs/en/get_started/teleoperating-with-mocap.md

Installs the required Python packages (opencv-python and flask) to set up the real-time RGB camera data stream server.

```bash
pip install opencv-python flask
```

--------------------------------

### Install Tmux

Source: https://github.com/openrobotlab/grutopia/blob/main/docs/en/get_started/run-benchmark-baseline.md

Updates the package list and installs the tmux utility using apt. Tmux is required for running the agent entrypoint script.

```shell
sudo apt update
sudo apt install tmux
```

--------------------------------

### Installing Benchmark Dependencies (Shell)

Source: https://github.com/openrobotlab/grutopia/blob/main/docs/en/get_started/run-benchmark-baseline.md

Navigates to the GRUtopia root directory and uses `pip` to install Python packages listed in the `requirements/benchmark.txt` file. These dependencies are required for running the benchmark episodes.

```shell
cd PATH/TO/GRUTOPIA/ROOT
pip install -r requirements/benchmark.txt
```

--------------------------------

### Install pre-commit tool (Bash)

Source: https://github.com/openrobotlab/grutopia/blob/main/CONTRIBUTING.md

Installs the `pre-commit` tool using pip, which is required to set up and run pre-commit hooks for code formatting and linting.

```bash
pip install pre-commit
```

--------------------------------

### Install Agent Submodules

Source: https://github.com/openrobotlab/grutopia/blob/main/docs/en/get_started/run-benchmark-baseline.md

Creates an 'images' directory and clones the GroundingDINO and YOLOv7 repositories as submodules within the social navigation agent directory. Ensures necessary external repositories are available.

```shell
mkdir images
git clone https://github.com/IDEA-Research/GroundingDINO.git
git clone https://github.com/WongKinYiu/yolov7.git
```

--------------------------------

### Install Tmux (Shell)

Source: https://github.com/openrobotlab/grutopia/blob/main/docs/en/get_started/run-benchmark-baseline.md

Updates the package list and installs the tmux utility using apt, which is required for running the mobile manipulation agent entrypoint script.

```shell
sudo apt update
sudo apt install tmux
```

--------------------------------

### Setting GRUtopia Assets Path Manually (Shell/Python)

Source: https://github.com/openrobotlab/grutopia/blob/main/docs/en/get_started/installation.md

Executes a Python module within the GRUtopia installation to configure the path to manually downloaded assets. This command is used if assets were obtained from sources like HuggingFace or ModelScope instead of using the automatic downloader. Requires GRUtopia to be installed and accessible via `python -m`.

```shell
python -m grutopia.set_assets_path
```

--------------------------------

### Install GRScenes Dependencies - Bash

Source: https://github.com/openrobotlab/grutopia/blob/main/toolkits/grscenes_scripts/README.md

Instructions to create a conda environment and install required dependencies (`usd-core`, `isaacsim` SDK components) using pip, necessary if the GRUtopia environment is not already configured.

```bash
conda create -n <env_name> python=3.10
conda activate <env_name>
pip install usd-core==24.11
pip install isaacsim==4.2.0.2 isaacsim-extscache-physics==4.2.0.2 isaacsim-extscache-kit==4.2.0.2 isaacsim-extscache-kit-sdk==4.2.0.2 --extra-index-url https://pypi.nvidia.com
```

--------------------------------

### Generating Mobile Manipulation Episodes Config (Shell)

Source: https://github.com/openrobotlab/grutopia/blob/main/docs/en/get_started/run-benchmark-baseline.md

Navigates to the GRUtopia root directory and then executes the Python script `generate_mm_episodes.py` to create configuration files for mobile manipulation episodes. The first command shows how to get help (`--help`), and the second runs the script with default options.

```shell
# generate episodes config for mobile manipulation task
cd PATH/TO/GRUTOPIA/ROOT
cd ..
python GRUtopia/grutopia_extension/agents/mobile_manipulation_agent/generate_mm_episodes.py --help  # See options when generating episodes config
python GRUtopia/grutopia_extension/agents/mobile_manipulation_agent/generate_mm_episodes.py
```

--------------------------------

### Download GRUtopia Assets - Shell

Source: https://github.com/openrobotlab/grutopia/blob/main/README.md

This command downloads the necessary assets for the GRUtopia platform. Users can choose to download all assets (~80GB) or a minimum set (~500MB). The default download location is `${PATH/TO/GRUTOPIA/ROOT}/grutopia/assets`. A custom path can be specified during execution.

```Shell
python -m grutopia.download_assets
```

--------------------------------

### Install pre-commit hooks (Bash)

Source: https://github.com/openrobotlab/grutopia/blob/main/CONTRIBUTING.md

Installs the git hooks configured in the `.pre-commit-config.yaml` file at the repository root. This command sets up automatic checks before each commit.

```bash
pre-commit install
```

--------------------------------

### Run GRUtopia Teleoperation Demo (Bash)

Source: https://github.com/openrobotlab/grutopia/blob/main/docs/en/get_started/teleoperating-with-visionpro.md

Executes the main Python script `gr1_teleop.py` located within the `grutopia.demo` module, which starts the simulation and the teleoperation server, allowing connection from VisionPro.

```bash
$ python -m grutopia.demo.gr1_teleop
```

--------------------------------

### Complete Depth Camera Sensor Implementation Example (Python)

Source: https://github.com/openrobotlab/grutopia/blob/main/docs/en/advanced_tutorials/how-to-add-sensor.md

This example shows a full implementation of the `DepthCamera` sensor class. It demonstrates initializing the sensor, handling post-reset logic (including creating an Isaac Sim camera), and implementing the `get_data` method to retrieve depth information.

```Python
from typing import Dict

from omni.isaac.sensor import Camera as i_Camera

from grutopia.core.robot.robot import BaseRobot, Scene
from grutopia.core.robot.sensor import BaseSensor
from grutopia.core.util import log
from grutopia_extension.configs.sensors import DepthCameraCfg


@BaseSensor.register('DepthCamera')
class DepthCamera(BaseSensor):
    """
    wrap of isaac sim's Camera class
    """
     def __init__(self, config: DepthCameraCfg, robot: BaseRobot, name: str = None, scene: Scene = None):
        super().__init__(config, robot, scene)
        self.name = name
        self._camera = self.create_camera()

    def __init__(self,
                 config: DepthCameraCfg,
                 robot: BaseRobot,
                 name: str = None,
                 scene: Scene = None):
        super().__init__(config, robot, scene)

    def post_reset(self):
        if self.config.enable:
            resolution = (1280, 720) if self.config.resolution is None else self.config.resolution
            prim_path = self._robot.config.prim_path + '/' + self.config.prim_path
            self._camera = i_Camera(prim_path=prim_path, resolution=resolution)
            self._camera.initialize()

    def get_data(self) -> Dict:
         if self.config.enable:
            depth = self._camera.get_depth()
            return {'depth': depth}
        return {}
```

--------------------------------

### Start Hamer Gesture Recognition Server (Bash)

Source: https://github.com/openrobotlab/grutopia/blob/main/docs/en/get_started/teleoperating-with-mocap.md

Starts the Hamer real-time hand gesture recognition server, taking the video stream URL from the RGB camera server as an argument. Replace {{video_url}} with the actual URL.

```bash
python hamer_re.py --video_url {{video_url}}
```

--------------------------------

### Running Mobile Manipulation Demo Episode (Shell)

Source: https://github.com/openrobotlab/grutopia/blob/main/docs/en/get_started/run-benchmark-baseline.md

Navigates to the GRUtopia root directory and then executes the Python script `grutopia/demo/mobile_manipulation.py`. This runs a single demo episode to verify the setup before running the full benchmark.

```shell
cd PATH/TO/GRUTOPIA/ROOT
cd ..
python GRUtopia/grutopia/demo/mobile_manipulation.py
```

--------------------------------

### Start RGB Camera Stream Server (Bash)

Source: https://github.com/openrobotlab/grutopia/blob/main/docs/en/get_started/teleoperating-with-mocap.md

Executes the Python script to start the real-time image data stream server from an RGB camera. Note the server URL after execution.

```bash
python rgb_camera_server.py
```

--------------------------------

### Launching VLM Agent Process (Shell)

Source: https://github.com/openrobotlab/grutopia/blob/main/docs/en/get_started/run-benchmark-baseline.md

Executes the shell script `launch_vlm_servers.sh` located in the `scripts` directory relative to the `mobile_manipulation_agent` directory. This script starts the necessary servers for the Vision-Language Model (VLM) agent.

```shell
# PWD: grutopia_extension/agents/mobile_manipulation_agent
./scripts/launch_vlm_servers.sh
```

--------------------------------

### Generate Social Navigation Episodes Config (Shell)

Source: https://github.com/openrobotlab/grutopia/blob/main/docs/en/get_started/run-benchmark-baseline.md

Executes a Python script to generate configuration files or entrypoint scripts for social navigation benchmark episodes. It shows how to get help options and run the script with default settings.

```shell
# generate episodes config for social navigation task
cd PATH/TO/GRUTOPIA/ROOT
cd ..
python GRUtopia/grutopia_extension/agents/social_navigation_agent/generate_sn_episodes.py --help  # See options when generating episodes config
python GRUtopia/grutopia_extension/agents/social_navigation_agent/generate_sn_episodes.py
```

--------------------------------

### Install Social Navigation Agent Dependencies

Source: https://github.com/openrobotlab/grutopia/blob/main/docs/en/get_started/run-benchmark-baseline.md

Navigates to the social navigation agent directory and installs Python dependencies listed in requirements.txt using pip. Requires a Python virtual environment (conda/venv) and matching CUDA toolkit/PyTorch versions.

```shell
cd grutopia_extension/agents/social_navigation_agent
pip install -r requirements.txt
```

--------------------------------

### Start GRUtopia Franka Teleoperation Task (Bash)

Source: https://github.com/openrobotlab/grutopia/blob/main/docs/en/get_started/teleoperating-with-mocap.md

Launches the main GRUtopia task script for Franka manipulation teleoperation using Mocap. Ensure the mocap_url parameter in the script is updated with the Hamer server URL obtained in the previous step before running.

```bash
python -m grutopia.demo.franka_manipulation_mocap_teleop
```

--------------------------------

### Install Mobile Manipulation Agent Dependencies (Shell)

Source: https://github.com/openrobotlab/grutopia/blob/main/docs/en/get_started/run-benchmark-baseline.md

Changes directory to the mobile manipulation agent's location and installs its specific Python dependencies using pip and the requirements file. It includes a comment about matching CUDA versions.

```shell
# make sure your cuda toolkit version matches your pytorch cuda version
cd grutopia_extension/agents/mobile_manipulation_agent
pip install -r requirements.txt
```

--------------------------------

### Run Social Navigation Demo Episode (Shell)

Source: https://github.com/openrobotlab/grutopia/blob/main/docs/en/get_started/run-benchmark-baseline.md

Navigates to the GRUtopia root directory and executes a specific Python script to run a single demo episode for the social navigation task, allowing verification of the setup before running all episodes.

```shell
cd PATH/TO/GRUTOPIA/ROOT
cd ..
python GRUtopia/grutopia/demo/h1_social_navigation.py
```

--------------------------------

### Launch VLM Servers

Source: https://github.com/openrobotlab/grutopia/blob/main/docs/en/get_started/run-benchmark-baseline.md

Executes the launch script for the Vision-Language Model (VLM) servers. This script starts the necessary processes for the social navigation agent to function.

```shell
./scripts/launch_vlm_servers.sh
```

--------------------------------

### Set GRUtopia Assets Path - Shell

Source: https://github.com/openrobotlab/grutopia/blob/main/README.md

This command allows users to configure the path where GRUtopia assets are stored after they have been downloaded. Running this command will prompt the user to enter the preferred directory path.

```Shell
python -m grutopia.set_assets_path
```

--------------------------------

### Example Metric Output File (JSON)

Source: https://github.com/openrobotlab/grutopia/blob/main/docs/en/advanced_tutorials/how-to-add-task.md

Shows an example of the JSON output file generated by the GRUtopia task runtime, containing the calculated value for the `SimpleMetric` and a flag indicating if the episode ended normally.

```JSON
{"SimpleMetric": 0.7508679775492055, "normally_end": true}
```

--------------------------------

### Clone GRUtopia Repository

Source: https://github.com/openrobotlab/grutopia/blob/main/docs/en/get_started/run-benchmark-baseline.md

Clones the GRUtopia source code repository from GitHub. This is required to run the benchmark baseline if GRUtopia was installed via pip.

```shell
git clone git@github.com:OpenRobotLab/GRUtopia.git
```

--------------------------------

### Serve CA Certificate for Transfer (Bash)

Source: https://github.com/openrobotlab/grutopia/blob/main/docs/en/get_started/teleoperating-with-visionpro.md

Uses Python's built-in HTTP server to serve the generated CA certificate from the mkcert root directory, allowing it to be downloaded by VisionPro for installation and trust.

```bash
$ cd $(mkcert -CAROOT) && python -m http.server
```

--------------------------------

### Starting GRUtopia Layout Edit Task

Source: https://github.com/openrobotlab/grutopia/blob/main/docs/en/get_started/layout_edit_with_mocap.md

This command executes the layout editing task script within the GRUtopia environment. Ensure the 'mocap_url' parameter in the 'layout_edit.py' file has been updated with the correct Hamer server URL before running.

```bash
python -m toolkits.layout_edit.layout_edit
```

--------------------------------

### Implement Grutopia Robot Class with Isaac Sim (Python)

Source: https://github.com/openrobotlab/grutopia/blob/main/docs/en/advanced_tutorials/how-to-add-robot.md

Provides a detailed implementation example of a Grutopia robot class (`DemoRobot`) that utilizes the Isaac Sim `Robot` class. It shows initialization, adding the robot USD to the stage, applying actions via registered controllers, and collecting observations from the robot's base, controllers, and sensors.

```python
from omni.isaac.core.robots.robot import Robot as IsaacRobot
from omni.isaac.core.scenes import Scene
from omni.isaac.core.utils.stage import add_reference_to_stage

from grutopia.core.config.robot import RobotUserConfig as Config
from grutopia.core.robot.robot import BaseRobot
from grutopia.core.util import log

@BaseRobot.register('DemoRobot')  # Register this robot to grutopia
class DemoRobot(BaseRobot):

    def __init__(self, config: DemoRobotCfg, scene: Scene):  # Use the config class for this robot
        super().__init__(config, scene)
        self._sensor_config = config.sensors
        self._start_position = np.array(config.position) if config.position is not None else None
        self._start_orientation = np.array(config.orientation) if config.orientation is not None else None

        usd_path = config.usd_path

        add_reference_to_stage(prim_path=prim_path, usd_path=os.path.abspath(usd_path))
        self.isaac_robot = IsaacRobot(
            prim_path=config.prim_path,
            name=config.name,
            position=self._start_position,
            orientation=self._start_orientation,
            usd_path=usd_path,
        )

        self._robot_scale = np.array([1.0, 1.0, 1.0])
        if config.scale is not None:
            self._robot_scale = np.array(config.scale)
            self.isaac_robot.set_local_scale(self._robot_scale)
        # More initialization here...

    def apply_action(self, action: dict):
        """
        Args:
            action (dict): inputs for controllers.
        """
        for controller_name, controller_action in action.items():
            if controller_name not in self.controllers:
                log.warning(f'unknown controller {controller_name} in action')
                continue
            controller = self.controllers[controller_name]
            control = controller.action_to_control(controller_action)
            self.isaac_robot.apply_action(control)

    def get_obs(self):
        """
        Set the observation you need here.
        """
        position, orientation = self._robot_base.get_world_pose()

        # custom
        obs = {
            'position': position,
            'orientation': orientation,
            'controllers': {},
            'sensors': {},
        }

        # common
        for c_obs_name, controller_obs in self.controllers.items():
            obs['controllers'][c_obs_name] = controller_obs.get_obs()
        for sensor_name, sensor_obs in self.sensors.items():
            obs['sensors'][sensor_name] = sensor_obs.get_data()
        return obs
```

--------------------------------

### Generate Self-Signed SSL Certificates (Bash)

Source: https://github.com/openrobotlab/grutopia/blob/main/docs/en/get_started/teleoperating-with-visionpro.md

Provides bash commands to install necessary tools (Homebrew, mkcert), generate a local CA, and create a self-signed SSL certificate and key for a specified IP address and localhost, required for HTTPS communication with VisionPro.

```bash
# Install Homebrew on Linux
$ /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
$ (echo; echo 'eval "$(/home/linuxbrew/.linuxbrew/bin/brew shellenv)"') >> ~/.bashrc
$ source ~/.bashrc

# Install mkcert
$ sudo apt-get install build-essential libnss3-tools
$ brew install mkcert

# Generate certs for local address.
$ ip addr | grep inet  # Find the IP address Vision Pro can access, assume 192.168.100.101
$ cd GRUtopia && mkdir mkcert && cd mkcert
$ mkcert -install  # Generate the local CA and install it in the system trust store.
$ mkcert -cert-file cert.pem -key-file key.pem 192.168.100.101 localhost 127.0.0.1  # Replace 192.168.100.101 with your host IP addr

# Get the location of local CA cert.
$ mkcert -CAROOT
$ ls $(mkcert -CAROOT)
```

--------------------------------

### Implement Custom Robot Controller Class (Python)

Source: https://github.com/openrobotlab/grutopia/blob/main/docs/en/advanced_tutorials/how-to-add-controller.md

Provides an example implementation of a custom controller class inheriting from `BaseController`. It includes the constructor, an optional `forward` method for calculation logic, and the required `action_to_control` method to convert actions to joint signals.

```Python
from typing import List

import numpy as np
from omni.isaac.core.scenes import Scene
from omni.isaac.core.utils.types import ArticulationAction

from grutopia.core.robot.controller import BaseController
from grutopia.core.robot.robot import BaseRobot
from grutopia_extension.configs.controllers import DemoControllerCfg


@BaseController.register('DemoController')
class DemoController(BaseController):
    def __init__(self, config: DemoControllerCfg, robot: BaseRobot, scene: Scene) -> None:
        super().__init__(config=config, robot=robot, scene=scene)

    def forward(
        self,
        forward_speed: float = 0,
        rotation_speed: float = 0,
        scaler: float = 1,
    ) -> ArticulationAction:
        if forward_speed == 0 and rotation_speed == 0:
            return ArticulationAction(joint_velocities=np.array([0, 0]))

        forward_basis = np.array([1.0, 1.0])
        spin_basis = np.array([-1.0, 1.0])

        wheel_vel_for = forward_basis * forward_speed
        wheel_vel_rot = spin_basis * rotation_speed
        wheel_vel = wheel_vel_for + wheel_vel_rot

        return ArticulationAction(joint_velocities=wheel_vel)

    def action_to_control(self, action: List | np.ndarray) -> ArticulationAction:
        """
        Args:
            action (List | np.ndarray): n-element 1d array containing:
              0. forward_speed (float)
              1. rotation_speed (float)
        """
        assert len(action) == 2, 'action must contain 2 elements'
        return self.forward(
            forward_speed=action[0],
            rotation_speed=action[1],
            scaler=1 / self.robot.get_robot_scale()[0],
        )
```

--------------------------------

### Define Metric Configuration Class (Python)

Source: https://github.com/openrobotlab/grutopia/blob/main/docs/en/advanced_tutorials/how-to-add-task.md

This snippet shows the simplest example of a configuration class for a custom metric. The `SimpleMetricCfg` class inherits from `MetricUserConfig` and specifies the metric type as 'SimpleMetric'. This config is necessary to use a custom metric within a task.

```python
# This is also the simplest configuration.
from typing import Optional

from grutopia.core.config.metric import MetricUserConfig


class SimpleMetricCfg(MetricUserConfig):
    type: Optional[str] = 'SimpleMetric'

```

--------------------------------

### Control G1 Robot with Keyboard Input (Python)

Source: https://github.com/openrobotlab/grutopia/blob/main/docs/en/get_started/wander-with-keyboard.md

This Python snippet demonstrates how to use the KeyboardInteraction class to get keyboard input during a simulation loop. It translates the input vector into robot speeds (x, y, z) and applies them as an environment action using a predefined configuration (move_by_speed_cfg). It shows the core logic for mapping keyboard presses to robot movement commands within the simulation step.

```python
from grutopia_extension.interactions.keyboard import KeyboardInteraction

keyboard = KeyboardInteraction()

while env.simulation_app.is_running():
    i += 1
    command = keyboard.get_input()
    x_speed = command[0] - command[1]
    y_speed = command[2] - command[3]
    z_speed = command[4] - command[5]
    env_action = {
        move_by_speed_cfg.name: (x_speed, y_speed, z_speed),
    }
    obs, _, terminated, _, _ = env.step(action=env_action)
    ...
```

--------------------------------

### Initialize VuerTeleop with Certificates (Python)

Source: https://github.com/openrobotlab/grutopia/blob/main/docs/en/get_started/teleoperating-with-visionpro.md

Demonstrates how to instantiate the `VuerTeleop` class, specifying the file paths for the generated SSL certificate (`cert.pem`) and key (`key.pem`) using the `cert_file` and `key_file` parameters, which are required for secure communication.

```python
...
teleop = VuerTeleop(
    cert_file='./GRUtopia/mkcert/cert.pem', key_file='./GRUtopia/mkcert/key.pem',   # Specify locations of your cert and key here
    resolution=(720, 1280)
)
...
```

--------------------------------

### Preparing GRScenes-100 Static Assets (Bash)

Source: https://github.com/openrobotlab/grutopia/blob/main/toolkits/indoor_scenes_generation/readme.md

Executes a Python script to process object models from the GRScenes-100 dataset and prepare them as static assets for use within Infinigen. Requires the GRUtopia environment to be active and specifies input and output folders.

```bash
python toolkits/indoor_scenes_generation/util/preparing_static_assets.py \
    --input_folder {{path_to_GRScenes-100}}/home_scenes/target_69_new/models/object \
    --output_folder {{path_to_infinigen}}/infinigen/assets/static_assets/source
```

--------------------------------

### Exporting Infinigen .blend to .usdc (Bash)

Source: https://github.com/openrobotlab/grutopia/blob/main/toolkits/indoor_scenes_generation/readme.md

Runs the Infinigen export tool module using Python to convert generated Blender (.blend) files located in a specified input folder into the Universal Scene Description (USD) format, specifically USDC. The output is saved to a designated folder with a specified resolution.

```bash
python -m infinigen.tools.export \
     --input_folder {PATH_TO_FOLDER_OF_BLENDFILES} \
     --output_folder outputs/my_export \
     -f usdc \
     -r 1024
```

--------------------------------

### Create Data Directory for Mobile Manipulation Agent (Shell)

Source: https://github.com/openrobotlab/grutopia/blob/main/docs/en/get_started/run-benchmark-baseline.md

Creates a 'data' directory within the mobile manipulation agent's location and changes into it, preparing the location for downloading model weights. Includes a comment about the current working directory.

```shell
# PWD: grutopia_extension/agents/mobile_manipulation_agent
mkdir data
cd data
```

--------------------------------

### Configure MDL Material Search Path - Bash

Source: https://github.com/openrobotlab/grutopia/blob/main/toolkits/grscenes_scripts/README.md

Steps to configure the `MDL_SYSTEM_PATH` environment variable for loading scenes normally, recommending saving the configuration to `~/.bashrc` for persistence.

```bash
# step 1. Find the Materials folder path in the downloaded GRScenes folder, such as `GRScenes-100/home_scenes/Materials`
# step 2. Configure the environment variable `MDL_SYSTEM_PATH` (saved to `~/.bashrc` is recommended).
echo 'export MDL_SYSTEM_PATH=$MDL_SYSTEM_PATH:</path/to/your_downloaded_materials_folder>' >> ~/.bashrc
source ~/.bashrc
```

--------------------------------

### Run preprocess.py Script - Bash

Source: https://github.com/openrobotlab/grutopia/blob/main/toolkits/grscenes_scripts/README.md

Commands to execute the `preprocess.py` script for binding physics properties to raw scenes. Options allow processing for interaction (`-i`), navigation (`-n`), specifying individual files (`-f`), or processing all scenes within directories (`-d`).

```bash
## use `-i/--interaction` option to preprocess scenes for interaction.
python preprocess.py -i/--interaction -f/--files [</path/to/raw_scene_usd_file>...]
## use `-n/--navigation` option to preprocess scenes for navigation
python preprocess.py -n/--navigation -f/--files [</path/to/raw_scene_usd_file>...]
## besides, use `-d/--dirs` option to preprocess all scenes under the scenes folder such as `/home/$USER/home_scenes/scenes`
python preprocess.py -i/--interaction -n/--navigation -d/--dirs [</path/to/scene_root_folder>...]
```

--------------------------------

### Clone Mobile Manipulation Agent Submodules (Shell)

Source: https://github.com/openrobotlab/grutopia/blob/main/docs/en/get_started/run-benchmark-baseline.md

Creates an 'images' directory and clones the GroundingDINO and YOLOv7 repositories as submodules within the mobile manipulation agent directory. Includes comments about the current working directory and CUDA version.

```shell
# PWD: grutopia_extension/agents/mobile_manipulation_agent
mkdir images
# Please make sure your torch cuda version is equal with your cuda version
# GroundingDINO
git clone https://github.com/IDEA-Research/GroundingDINO.git

# YOLOv7
git clone https://github.com/WongKinYiu/yolov7.git
```

--------------------------------

### API Key File Locations

Source: https://github.com/openrobotlab/grutopia/blob/main/docs/en/get_started/run-benchmark-baseline.md

Shows the expected directory structure for placing Azure OpenAI API key files (`azure_api_key.txt` for `gpt-4o` and `azure_api_key_e.txt` for `text-embedding-3-large`) within the `grutopia_extension/agents/mobile_manipulation_agent/modules/vlm/api_key` directory.

```text
grutopia_extension
├── agents
│   ├── mobile_manipulation_agent
│   │   ├── modules
│   │   │   └── vlm
│   │   │   │   └── api_key
│   │   │   │   │   ├── azure_api_key.txt
│   │   │   │   │   └── azure_api_key_e.txt
```

--------------------------------

### Run warmup.py Script - Bash

Source: https://github.com/openrobotlab/grutopia/blob/main/toolkits/grscenes_scripts/README.md

Commands to run the `warmup.py` script to warm up the simulation process for specified scenes in a headless manner, saving physics cooking cache locally for improved performance during subsequent plays. Options include specifying files (`-f`) or directories (`-d`).

```bash
## warm up the specific scenes
python warmup.py -f/--files [</path/to/scene_usd_file>...]
## warm up all scenes
python warmup.py -d/--dirs [</path/to/scene_root_folder>...]
```

--------------------------------

### Creating and Running H1 Robot Simulation in GRUtopia (Python)

Source: https://github.com/openrobotlab/grutopia/blob/main/docs/en/tutorials/how-to-use-robot.md

This snippet shows the process of setting up a simulation environment with an H1 robot. It involves importing configuration classes, creating a robot configuration instance with desired controllers and sensors, defining the overall simulation configuration, initializing the simulation runtime and environment, and executing a basic simulation loop where the robot is controlled using the 'move_by_speed' controller. Finally, it demonstrates how to close the simulation application.

```Python
from grutopia_extension.configs.robots.h1 import (
    H1RobotCfg,
    h1_camera_cfg,
    move_along_path_cfg,
    move_by_speed_cfg,
    rotate_cfg,
)

h1_1 = H1RobotCfg(
    controllers=[
        move_by_speed_cfg,
        move_along_path_cfg,
        rotate_cfg,
    ],
    sensors=[h1_camera_cfg.update(name='camera', size=(320, 240), enable=True)],
)

config = Config(
    simulator=SimConfig(physics_dt=1 / 240, rendering_dt=1 / 240, use_fabric=False),
    task_config=SingleInferenceTaskCfg(
        episodes=[
            SingleInferenceEpisodeCfg(
                scene_asset_path=gm.ASSET_PATH + '/scenes/empty.usd',
                scene_scale=[0.01, 0.01, 0.01],
                robots=[h1_1],
            ),
        ],
    ),
)

sim_runtime = SimulatorRuntime(config_class=config, headless=headless, native=headless)

import_extensions()  # The robot class is registered here.

# Create the environment.
env = Env(sim_runtime)
obs, _ = env.reset()

...

while env.simulation_app.is_running():
    i += 1
    env_action = {move_by_speed_cfg.name: [1.0, 0.0, 0.0]}  # Use move_by_speed controller
    obs, _, terminated, _, _ = env.step(action=env_action)

env.simulation_app.close()
```

--------------------------------

### Generate GRUTopia Episodes - Shell

Source: https://github.com/openrobotlab/grutopia/blob/main/docs/en/tutorials/how-to-run-benchmark-with-custom-agent.md

Provides shell commands to execute Python scripts for generating episodes for specific GRUTopia agents (social navigation or mobile manipulation), recommending execution from the project root and suggesting checking the --help output for usage details.

```shell
# It is recommended to execute this script at the root directory of the project.
python grutopia_extension.agents.social_navigation_agent.generate_sn_episodes.py --help
# or
python grutopia_extension.agents.mobile_manipulation_agent.generate_mm_episodes.py --help
```

--------------------------------

### Create Data Directory for Models

Source: https://github.com/openrobotlab/grutopia/blob/main/docs/en/get_started/run-benchmark-baseline.md

Creates a 'data' directory within the social navigation agent directory and changes the current directory to 'data'. This directory is used to store downloaded model weights.

```shell
mkdir data
cd data
```

--------------------------------

### Run G1 Locomotion Demo (Bash)

Source: https://github.com/openrobotlab/grutopia/blob/main/docs/en/get_started/wander-with-keyboard.md

This command executes the g1 locomotion demonstration script from the grutopia.demo module using the Python interpreter. It launches the simulation environment where the robot can be controlled via keyboard.

```bash
$ python -m grutopia.demo.g1_locomotion
```

--------------------------------

### Run get_metadata.py Script - Bash

Source: https://github.com/openrobotlab/grutopia/blob/main/toolkits/grscenes_scripts/README.md

Commands to execute the `get_metadata.py` script to record metadata information for models and materials referenced within given model instance or scene USD files or directories. Options allow specifying files (`-f`) or directories (`-d`).

```bash
python get_metadata.py -f/--files [</path/to/single_instance_or_scene_usd>...]
python get_metadata.py -d/--dirs [</path/to/instance_or_scene_root_folder>...]
```

--------------------------------

### Configuring Robot with Controller in GRUtopia Python

Source: https://github.com/openrobotlab/grutopia/blob/main/docs/en/tutorials/how-to-use-controller.md

This snippet demonstrates how to associate a controller configuration (`move_by_speed_cfg`) with a specific robot instance (`H1RobotCfg`) by adding it to the `controllers` list within the robot's configuration object. This step is necessary before the controller can be used to control the robot in the simulation.

```Python
move_by_speed_cfg = MoveBySpeedCfg(...)

h1_1 = H1RobotCfg(
    position=(0.0, 0.0, 1.05),
    controllers=[
        move_by_speed_cfg,
    ],
）
```

--------------------------------

### Define Robot Configuration Class (Python)

Source: https://github.com/openrobotlab/grutopia/blob/main/docs/en/advanced_tutorials/how-to-add-robot.md

This Python snippet defines a configuration class `DemoRobotCfg` by inheriting from `RobotCfg`. It specifies essential metadata for a robot, such as its name, type, simulation path, creation flag, and the path to its USD asset file.

```Python
from grutopia.core.config import RobotCfg

class DemoRobotCfg(RobotCfg):
    # meta info
    name: str = 'demo'
    type: str = 'DemoRobot'
    prim_path: str = '/World/demo'
    create_robot: bool = True
    usd_path: Optional[str] = './assets/demo_robot.usd'
```

--------------------------------

### Listing Available GRUtopia Tasks (Python)

Source: https://github.com/openrobotlab/grutopia/blob/main/docs/en/tutorials/how-to-use-task.md

Imports various task modules from the `grutopia_extension.tasks` package, showing the available task types within the GRUtopia framework. This snippet is typically found in the `__init__.py` file listing the public API of the tasks module.

```Python
from grutopia_extension.tasks import (
    manipulation_task,
    mobile_manipulation_task,
    finite_step_task,
    single_inference_task,
    social_navigation_task,
)
```

--------------------------------

### Postprocessing Exported Scene Data (Bash)

Source: https://github.com/openrobotlab/grutopia/blob/main/toolkits/indoor_scenes_generation/readme.md

Executes a Python script for postprocessing the exported USDC scene file. This step is intended to refine the scene data, likely integrating or adjusting the GRScenes-100 assets for better compatibility or use in downstream applications like Isaac Sim. Requires the GRUtopia environment.

```bash
python toolkits/indoor_scenes_generation/util/data_post_processing.py \
    --input_path {{path_to_infinigen}}/outputs/my_export/export_scene.blend/export_scene.usdc \
    --source_assets {{path_to_infinigen}}/infinigen/assets/static_assets/source
```

--------------------------------

### Using Controller Action in GRUtopia Simulation Loop Python

Source: https://github.com/openrobotlab/grutopia/blob/main/docs/en/tutorials/how-to-use-controller.md

This Python snippet shows how to apply a controller action within the main simulation loop. An action dictionary is created where the key is the controller's name (`move_by_speed_cfg.name`) and the value is the desired action data (e.g., speed commands). This action dictionary is then passed to the environment's `step` method.

```Python
...
move_action = {move_by_speed_cfg.name: (1.0, 0.0, 0.0)}

while env.simulation_app.is_running():
    env_action = move_action
    obs, _, _, _, _ = env.step(actions=env_actions)
    ...
env.simulation_app.close()
```

--------------------------------

### Run play_scene.py Script - Bash

Source: https://github.com/openrobotlab/grutopia/blob/main/toolkits/grscenes_scripts/README.md

Command to execute the `play_scene.py` script to load and play a specified scene. This script is recommended for utilizing the local mesh cache generated by the `warmup.py` process, requiring a single scene file path (`-f`).

```bash
python play_scene.py -f/--file </path/to/scene_usd_file>
```

--------------------------------

### Implementing Required Custom Sensor Methods (Python)

Source: https://github.com/openrobotlab/grutopia/blob/main/docs/en/advanced_tutorials/how-to-add-sensor.md

This snippet illustrates the basic structure of a custom sensor class inheriting from `BaseSensor`. It shows the required `__init__` method for initialization and the `get_data` method for retrieving sensor data.

```Python
from grutopia.core.robot.robot import BaseRobot, Scene
from grutopia.core.robot.sensor import BaseSensor


@BaseSensor.register('DepthCamera')
class DepthCamera(BaseSensor):
    def __init__(self, config: DepthCameraCfg, robot: BaseRobot, scene: Scene):
        """Initialize the sensor with the given config.

        Args:
            config (DepthCameraCfg): sensor configuration.
            robot (BaseRobot): robot owning the sensor.
            scene (Scene): scene from isaac sim.
        """

    def get_data(self) -> Dict:
        """Get data from sensor.

        Returns:
            Dict: data dict of sensor.
        """
```

--------------------------------

### Define Minimal Grutopia Robot Class (Python)

Source: https://github.com/openrobotlab/grutopia/blob/main/docs/en/advanced_tutorials/how-to-add-robot.md

Defines the basic structure for a custom robot class in Grutopia, inheriting from `BaseRobot`. It includes the required `__init__`, `apply_action`, and `get_obs` methods and demonstrates class registration using the `@BaseRobot.register` decorator.

```python
from omni.isaac.core.scenes import Scene
from grutopia.core.robot.robot import BaseRobot

@BaseRobot.register('DemoRobot')  # Register this robot to grutopia
class DemoRobot(BaseRobot):
    def __init__(self, config: DemoRobotCfg, scene: Scene):
        """Initialize the robot with the given config.

        Args:
            config (DemoRobotCfg): config for the robot, should be a instance of corresponding config class.
            scene (Scene): current scene.
        """

    def apply_action(self, action: dict):
        """Apply actions of controllers to robot.

        Args:
            action (dict): action dict.
              key: controller name.
              value: corresponding action array.
        """

    def get_obs(self) -> dict:
        """Get observation of robot, including controllers, sensors, and world pose.
        """
```