RoboVerse

https://roboverse.wiki/
Admin
RoboVerse is a unified platform, dataset, and benchmark designed for scalable and generalizable...
Tokens:50,246
Snippets:16,130
Trust Score:4.7
Update:1 year ago
Show doc for...
Tokens:
Raw
### Install PyRep and Run Example

Source: https://roboverse.wiki/_sources/metasim/get_started/advanced_installation/pyrep

Clones the PyRep repository, sets up the environment using the `setup_pyrep` function, installs PyRep in editable mode, and runs a sample navigation example. This verifies the PyRep installation.

```bash
# Install PyRep
cd third_party
git clone https://github.com/stepjam/PyRep.git pyrep && cd pyrep
setup_pyrep
pip install -e .
python examples/example_youbot_navigation.py  # test
```

--------------------------------

### Install RLBench and Run Example

Source: https://roboverse.wiki/_sources/metasim/get_started/advanced_installation/pyrep

Clones the RLBench repository, installs it in editable mode, and installs the gymnasium library. It then executes an imitation learning example to test RLBench functionality.

```bash
# Install RLBench
cd ..
git clone https://github.com/stepjam/RLBench.git rlbench && cd rlbench
pip install -e .
pip install gymnasium
python rlbench/examples/imitation_learning.py  # test
```

--------------------------------

### Install PyRep and Dependencies

Source: https://roboverse.wiki/metasim/get_started/advanced_installation/pyrep

Clones the PyRep repository, sets environment variables for CoppeliaSim, installs PyRep in editable mode, and runs a test example. It also installs RLBench and its dependencies.

```bash
# utils function to setup pyrep
setup_pyrep(){
   export COPPELIASIM_ROOT=/path/to/CoppeliaSim_Edu_V4_1_0_Ubuntu20_04
   export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$COPPELIASIM_ROOT
   export QT_QPA_PLATFORM_PLUGIN_PATH=$COPPELIASIM_ROOT
}

# Install PyRep
cd third_party
git clone https://github.com/stepjam/PyRep.git pyrep && cd pyrep
setup_pyrep
pip install -e .
python examples/example_youbot_navigation.py  # test

# Install RLBench
cd ..
git clone https://github.com/stepjam/RLBench.git rlbench && cd rlbench
pip install -e .
pip install gymnasium
python rlbench/examples/imitation_learning.py  # test
```

--------------------------------

### Task: Create and Use Pre-built Environments

Source: https://roboverse.wiki/metasim/get_started/quick_start/guide

This example shows how to get a task class from the registry, update its default scenario with specific parameters, and then create a task environment. It demonstrates the Gym-style API with `reset` and `step` methods for interacting with the environment.

```python
from metasim.task.registry import get_task_class
import torch

# Get task class by name
task_cls = get_task_class(args.task)  # e.g., "example.reaching"

# Get default scenario and update with specific parameters
scenario = task_cls.scenario.update(
    robots=[args.robot], 
    simulator=args.sim, 
    num_envs=args.num_envs, 
    headless=args.headless, 
    cameras=[]
)

# Create task environment
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
env = task_cls(scenario=scenario, device=device)
```

```python
obs, info = env.reset()
action = ...
obs, reward, terminated, truncated, info = env.step(action)
```

--------------------------------

### Install Multiple Simulators

Source: https://roboverse.wiki/metasim/get_started/installation

Demonstrates how to install multiple RoboVerse simulators simultaneously by combining their installation commands in a single 'uv pip install -e "' command. This is crucial for correct dependency management.

```bash
uv pip install -e ".[mujoco,isaaclab221]"

```

```bash
cd third_party

```

```bash
git clone --depth 1 --branch v2.2.1 git@github.com:isaac-sim/IsaacLab.git IsaacLab221 && cd IsaacLab221

```

```bash
./isaaclab.sh -i none

```

```bash
uv pip install -e ".[mujoco,sapien3,genesis]"

```

--------------------------------

### Install IsaacLab and Modify Setup

Source: https://roboverse.wiki/metasim/get_started/docker

Downloads, unzips, and installs IsaacLab, including modifying its setup script to resolve dependency conflicts for newer GPU setups.

```shell
cd third_party

wget https://codeload.github.com/isaac-sim/IsaacLab/zip/refs/tags/v1.4.1 -O IsaacLab-1.4.1.zip && unzip IsaacLab-1.4.1.zip

cd IsaacLab-1.4.1

sed -i '/^EXTRAS_REQUIRE = {$/,/^}$/c\EXTRAS_REQUIRE = {
    "sb3": [],
    "skrl": [],
    "rl-games": [],
    "rsl-rl": [],
    "robomimic": [],
}' source/extensions/omni.isaac.lab_tasks/setup.py

./isaaclab.sh -i
```

--------------------------------

### Install Multiple Simulators (MuJoCo and IsaacSim)

Source: https://roboverse.wiki/_sources/metasim/get_started/installation

Demonstrates how to install multiple simulators, MuJoCo and IsaacSim (v5.0.0 via IsaacLab v2.2.1), in a single environment by combining their installation commands.

```bash
uv pip install -e ".[mujoco,isaaclab221]"
cd third_party
git clone --depth 1 --branch v2.2.1 git@github.com:isaac-sim/IsaacLab.git IsaacLab221 && cd IsaacLab221
./isaaclab.sh -i none
```

--------------------------------

### Install knn Package

Source: https://roboverse.wiki/_sources/metasim/get_started/advanced_installation/gsnet

Installs the knn package, which is adapted from pytorch_knn_cuda. This involves navigating to the knn directory and running the setup script.

```bash
cd knn
python setup.py install
cd ..
```

--------------------------------

### Complete End-Effector Control Example (Python)

Source: https://roboverse.wiki/_sources/metasim/get_started/quick_start/4_motion_planning

A comprehensive example demonstrating the setup of an IK solver, solving IK for multiple environments, processing gripper commands, and composing final joint actions as dictionaries. Includes necessary imports and device setup.

```python
import torch
from metasim.utils.ik_solver import setup_ik_solver, process_gripper_command

# Setup
ik_solver = setup_ik_solver(robot_cfg, solver="pyroki")
num_envs = 4
device = torch.device("cuda")

# Target end-effector poses
target_positions = torch.randn(num_envs, 3, device=device)  # (B, 3)
target_quaternions = torch.randn(num_envs, 4, device=device)  # (B, 4) wxyz
gripper_commands = torch.randint(0, 2, (num_envs,), device=device)  # (B,) binary

# Step 1: Solve IK for arm joints
ik_solver = setup_ik_solver(robot_cfg, solver="pyroki")  # or "curobo"
# IK solver expects original joint order, but state uses alphabetical order
reorder_idx = handler.get_joint_reindex(scenario.robots[0].name)
inverse_reorder_idx = [reorder_idx.index(i) for i in range(len(reorder_idx))]
curr_robot_q = obs.robots[scenario.robots[0].name].joint_pos[:, inverse_reorder_idx]

q_arm, ik_success = ik_solver.solve_ik_batch(
    ee_pos_target=target_positions,
    ee_quat_target=target_quaternions,
    seed_q=curr_robot_q 
)

# Step 2: Process gripper commands
gripper_widths = process_gripper_command(
    gripper_binary=gripper_commands,
    robot_cfg=robot_cfg,
    device=device
)

# Step 3: Compose full joint command and create actions directly
actions = ik_solver.compose_joint_action(
    q_solution=q_arm,
    gripper_widths=gripper_widths,
    return_dict=True  # Return action dictionaries directly
)

```

--------------------------------

### Install IsaacGym

Source: https://roboverse.wiki/_sources/metasim/get_started/installation

Installs IsaacGym by downloading a preview release, extracting it, patching Python float types, and then installing it using pip with a local path. Includes troubleshooting steps for common errors.

```bash
cd third_party
wget https://developer.nvidia.com/isaac-gym-preview-4 \
    && tar -xf isaac-gym-preview-4 \
    && rm isaac-gym-preview-4
find isaacgym/python -type f -name "*.py" -exec sed -i 's/np.float/np.float32/g' {} +
cd ..
uv pip install -e ".[isaacgym]" 'isaacgym @ file:${PROJECT_ROOT}/third_party/isaacgym/python'
```

--------------------------------

### Install graspnetAPI for Evaluation

Source: https://roboverse.wiki/_sources/metasim/get_started/advanced_installation/gsnet

Installs the graspnetAPI, used for evaluation purposes. It involves cloning the repository if it doesn't exist and then installing it using pip.

```bash
git clone https://github.com/graspnet/graspnetAPI.git # if not exists
cd graspnetAPI
pip install .
cd ..
```

--------------------------------

### Install pointnet2 Operators

Source: https://roboverse.wiki/_sources/metasim/get_started/advanced_installation/gsnet

Installs the pointnet2 operators, adapted from the votenet repository. This requires navigating to the pointnet2 directory and executing the setup script.

```bash
cd pointnet2
python setup.py install
cd ..
```

--------------------------------

### Install RoboVerse with Simulator Support

Source: https://roboverse.wiki/_sources/metasim/get_started/installation

Installs RoboVerse with support for a specified simulator (e.g., MuJoCo, SAPIEN2, SAPIEN3, Genesis, PyBullet) using 'uv pip install -e'. The '[<simulator>]' placeholder should be replaced with the desired simulator's key.

```bash
uv pip install -e ".[<simulator>]"
```

--------------------------------

### PPO Training with Different Simulators

Source: https://roboverse.wiki/_sources/metasim/get_started/advanced/rl_example/quick_examples

Provides examples of initiating PPO training using different simulation environments available in RoboVerse. These commands highlight the `--sim` argument for selecting backends such as Isaac Gym, Isaac Lab, MuJoCo, Genesis, and MJX.

```bash
python get_started/rl/0_ppo.py --sim isaacgym --headless
```

```bash
python get_started/rl/0_ppo.py --sim isaaclab --headless
```

```bash
python get_started/rl/0_ppo.py --sim mujoco --headless
```

```bash
python get_started/rl/0_ppo.py --sim genesis --headless
```

```bash
python get_started/rl/0_ppo_gym.py --sim mjx --headless
```

--------------------------------

### Install RoboVerse Simulators with uv

Source: https://roboverse.wiki/metasim/get_started/installation

Installs various RoboVerse simulators (MuJoCo, SAPIEN2, SAPIEN3, Genesis, PyBullet) using 'uv pip install -e "'. The specific simulator is selected by appending its name in brackets.

```bash
uv pip install -e ".[mujoco]"

```

```bash
uv pip install -e ".[sapien2]"

```

```bash
uv pip install -e ".[sapien3]"

```

```bash
uv pip install -e ".[genesis]"

```

```bash
uv pip install -e ".[pybullet]"

```

--------------------------------

### Install IsaacSim v5.0.0 (IsaacLab v2.2.1)

Source: https://roboverse.wiki/_sources/metasim/get_started/installation

Installs RoboVerse with IsaacLab v2.2.1 support and then manually installs IsaacLab from source. This method is recommended for Ubuntu 22.04.

```bash
uv pip install -e ".[isaaclab221]"
cd third_party
git clone --depth 1 --branch v2.2.1 git@github.com:isaac-sim/IsaacLab.git IsaacLab221 && cd IsaacLab221
./isaaclab.sh -i none
```

--------------------------------

### Clone RoboVerse Repository

Source: https://roboverse.wiki/_sources/metasim/get_started/installation

Clones the RoboVerse project from GitHub and navigates into the project directory. This is the initial step for setting up the environment.

```bash
git clone git@github.com:RoboVerseOrg/RoboVerse.git && cd RoboVerse
```

--------------------------------

### Install IsaacSim v4.5.0 (IsaacLab v2.1.1)

Source: https://roboverse.wiki/_sources/metasim/get_started/installation

Installs RoboVerse with IsaacLab v2.1.1 support and then manually installs IsaacLab from source. This method is recommended for Ubuntu 22.04.

```bash
uv pip install -e ".[isaaclab211]"
cd third_party
git clone --depth 1 --branch v2.1.1 git@github.com:isaac-sim/IsaacLab.git IsaacLab211 && cd IsaacLab211
./isaaclab.sh -i none
```

--------------------------------

### Setup PyRep Environment Variables

Source: https://roboverse.wiki/_sources/metasim/get_started/advanced_installation/pyrep

Configures essential environment variables for PyRep to locate and interact with the CoppeliaSim installation. `COPPELIASIM_ROOT` should point to the CoppeliaSim directory, and `LD_LIBRARY_PATH` and `QT_QPA_PLATFORM_PLUGIN_PATH` are set for proper library loading.

```bash
setup_pyrep(){
   export COPPELIASIM_ROOT=/path/to/CoppeliaSim_Edu_V4_1_0_Ubuntu20_04
   export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$COPPELIASIM_ROOT
   export QT_QPA_PLATFORM_PLUGIN_PATH=$COPPELIASIM_ROOT
}
```

--------------------------------

### Install IsaacLab and Modify Setup

Source: https://roboverse.wiki/_sources/metasim/get_started/docker

Downloads and unzips the IsaacLab repository, modifies its setup file to adjust extra requirements, and then installs it. It also forces a reinstallation of PyTorch to version 2.7.1 for compatibility.

```bash
cd third_party

wget https://codeload.github.com/isaac-sim/IsaacLab/zip/refs/tags/v1.4.1 -O IsaacLab-1.4.1.zip && unzip IsaacLab-1.4.1.zip

cd IsaacLab-1.4.1

sed -i '/^EXTRAS_REQUIRE = {$/,/^}$/c\EXTRAS_REQUIRE = {\n    "sb3": [],\n    "skrl": [],\n    "rl-games": [],\n    "rsl-rl": [],\n    "robomimic": [],\n}' source/extensions/omni.isaac.lab_tasks/setup.py

./isaaclab.sh -i

pip install --force-reinstall torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu128
```

--------------------------------

### Install IsaacGym

Source: https://roboverse.wiki/metasim/get_started/installation

Installs IsaacGym, which involves downloading the preview version, extracting it, modifying Python files to use np.float32, and then installing it via uv pip. This method is recommended for Ubuntu 22.04.

```bash
cd third_party

```

```bash
wget https://developer.nvidia.com/isaac-gym-preview-4 \
    && tar -xf isaac-gym-preview-4 \
    && rm isaac-gym-preview-4

```

```bash
find isaacgym/python -type f -name "*.py" -exec sed -i 's/np.float/np.float32/g' {} +

```

```bash
cd ..

```

```bash
uv pip install -e ".[isaacgym]" 'isaacgym @ file:${PROJECT_ROOT}/third_party/isaacgym/python'

```

--------------------------------

### Install PyRoki using pip

Source: https://roboverse.wiki/_sources/metasim/get_started/advanced_installation/pyroki

Installs PyRoki by cloning the repository and using pip for an editable installation. Requires Python 3.10+.

```bash
git clone https://github.com/chungmin99/pyroki.git
cd pyroki
pip install -e .
```

--------------------------------

### Install IsaacSim v5.0.0 (IsaacLab v2.2.1)

Source: https://roboverse.wiki/metasim/get_started/installation

Installs IsaacLab v2.2.1, which requires IsaacSim v5.0.0. This involves installing the package with uv, cloning the IsaacLab repository, and running an installation script. This method is recommended for Ubuntu 22.04.

```bash
uv pip install -e ".[isaaclab221]"

```

```bash
cd third_party

```

```bash
git clone --depth 1 --branch v2.2.1 git@github.com:isaac-sim/IsaacLab.git IsaacLab221 && cd IsaacLab221

```

```bash
./isaaclab.sh -i none

```

--------------------------------

### Basic cuRobo Installation Commands

Source: https://roboverse.wiki/_sources/metasim/get_started/advanced_installation/curobo

This snippet outlines the essential shell commands for cloning cuRobo, updating submodules, navigating to the directory, and installing it using pip.

```bash
sudo apt install git-lfs
git submodule update --init --recursive
cd third_party/curobo
uv pip install -e . --no-build-isolation
```

--------------------------------

### Basic PPO Training with Franka Robot and Different Simulators

Source: https://roboverse.wiki/metasim/get_started/advanced/rl_example/quick_examples

Demonstrates how to train a Proximal Policy Optimization (PPO) agent on the reaching task using the Franka robot. It shows variations for different simulation backends like Isaac Gym, Isaac Lab, MuJoCo, Genesis, and MJX, including options for headless execution and using the Gym interface.

```python
python get_started/rl/0_ppo.py --task reach_origin --robot franka --sim isaacgym --num-envs 128
```

```python
python get_started/rl/0_ppo.py --sim mujoco --headless
```

```python
python get_started/rl/0_ppo_gym.py --sim mjx --num-envs 256
```

```python
python get_started/rl/0_ppo.py --sim isaacgym --headless
```

```python
python get_started/rl/0_ppo.py --sim isaaclab --headless
```

```python
python get_started/rl/0_ppo.py --sim mujoco --headless
```

```python
python get_started/rl/0_ppo.py --sim genesis --headless
```

```python
python get_started/rl/0_ppo_gym.py --sim mjx --headless
```

--------------------------------

### Install IsaacSim v4.5.0 (IsaacLab v2.1.1)

Source: https://roboverse.wiki/metasim/get_started/installation

Installs IsaacLab v2.1.1, which requires IsaacSim v4.5.0. This involves installing the package with uv, cloning the IsaacLab repository, and running an installation script. This method is recommended for Ubuntu 22.04.

```bash
uv pip install -e ".[isaaclab211]"

```

```bash
cd third_party

```

```bash
git clone --depth 1 --branch v2.1.1 git@github.com:isaac-sim/IsaacLab.git IsaacLab211 && cd IsaacLab211

```

```bash
./isaaclab.sh -i none

```

--------------------------------

### Basic PPO Training with Franka Robot

Source: https://roboverse.wiki/_sources/metasim/get_started/advanced/rl_example/quick_examples

Demonstrates how to train a Proximal Policy Optimization (PPO) agent on a reaching task using the Franka robot. Supports various simulation backends like Isaac Gym and MuJoCo, and offers a Gym interface for easier integration.

```bash
python get_started/rl/0_ppo.py --task reach_origin --robot franka --sim isaacgym --num-envs 128
```

```bash
python get_started/rl/0_ppo.py --sim mujoco --headless
```

```bash
python get_started/rl/0_ppo_gym.py --sim mjx --num-envs 256
```

--------------------------------

### Install Linux Packages and Clone Blender Source

Source: https://roboverse.wiki/_sources/metasim/get_started/advanced_installation/blender

Installs essential packages like Python 3, Git, and Git LFS, creates a directory for the Blender source code, clones the Blender repository, and checks out a specific version.

```bash
sudo apt install python3 git git-lfs
mkdir ~/Downloads/blender-git
cd blender-git
git clone https://projects.blender.org/blender/blender.git
git checkout v4.2.7
```

--------------------------------

### Finetune Octo Model

Source: https://roboverse.wiki/_sources/roboverse_learn/imitation_learning/octo

Launches the Octo model finetuning process using the 'finetune.sh' script. Sets necessary NCCL environment variables for distributed training and specifies the pretrained model path.

```bash
export NCCL_IB_DISABLE=0
export NCCL_SOCKET_IFNAME=lo
export NCCL_DEBUG=INFO
export NCCL_NVLS_ENABLE=1

python scripts/finetune.py --config.pretrained_path=hf://rail-berkeley/octo-base
```

--------------------------------

### Python Task Environment Initialization and Usage

Source: https://roboverse.wiki/_sources/metasim/get_started/quick_start/guide

Demonstrates how to get a task class, configure its scenario, and create a task environment in Python. It also shows the basic Gym-style API calls for resetting the environment and stepping through actions.

```python
from metasim.task.registry import get_task_class
import torch

# Get task class by name
task_cls = get_task_class(args.task)  # e.g., "example.reaching"

# Get default scenario and update with specific parameters
scenario = task_cls.scenario.update(
    robots=[args.robot], 
    simulator=args.sim, 
    num_envs=args.num_envs, 
    headless=args.headless, 
    cameras=[]
)

# Create task environment
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
env = task_cls(scenario=scenario, device=device)


obs, info = env.reset()
action = ...
obs, reward, terminated, truncated, info = env.step(action)
```

--------------------------------

### Clone RoboVerse Repository and Install uv

Source: https://roboverse.wiki/metasim/get_started/installation

Clones the RoboVerse project from GitHub and installs the 'uv' package manager using pip. 'uv' is used for managing project dependencies.

```bash
git clone git@github.com:RoboVerseOrg/RoboVerse.git && cd RoboVerse

```

```bash
pip install uv

```

--------------------------------

### Clone RoboVerse Octo Repository

Source: https://roboverse.wiki/_sources/roboverse_learn/imitation_learning/octo

Clones the specified RoboVerse Octo repository from GitHub to your local machine.

```bash
git clone https://github.com/wbjsamuel/RoboVerse-Octo.git
```

--------------------------------

### Install Multiple RoboVerse Simulators (Bash)

Source: https://roboverse.wiki/_sources/metasim/get_started/installation

Installs multiple RoboVerse simulators (e.g., MuJoCo, SAPIEN3, Genesis) in a single command using pip. This approach correctly handles interdependencies between simulators.

```bash
uv pip install -e ".[mujoco,sapien3,genesis]"
```

```bash
uv pip install -e ".[mujoco]"
uv pip install -e ".[sapien3]"
uv pip install -e ".[genesis]"
```

--------------------------------

### Install RoboVerse RDT and Dependencies

Source: https://roboverse.wiki/_sources/roboverse_learn/imitation_learning/rdt

Clones the RoboVerse RDT repository, creates and activates a Conda environment, and installs PyTorch, packaging, flash-attn, and other requirements.

```bash
# Clone this repo
git clone https://github.com/wbjsamuel/RoboVerse-RDT
cd RoboticsDiffusionTransformer

# Create a Conda environment
conda create -n rdt python=3.10.0
conda activate rdt

# Install pytorch
# Look up https://pytorch.org/get-started/previous-versions/ with your cuda version for a correct command
pip install torch==2.1.0 torchvision==0.16.0  --index-url https://download.pytorch.org/whl/cu121

# Install packaging
pip install packaging==24.0

# Install flash-attn
pip install flash-attn --no-build-isolation

# Install other prequisites
pip install -r requirements.txt
```

--------------------------------

### Install MJX Package

Source: https://roboverse.wiki/_sources/metasim/get_started/advanced_installation/mjx

Installs the MJX package using pip. This command installs the package in editable mode and includes MJX-specific dependencies.

```bash
pip install -e ".[mjx]"
```

--------------------------------

### Install Minkowski Engine with OpenBLAS

Source: https://roboverse.wiki/_sources/metasim/get_started/advanced_installation/gsnet

Installs the Minkowski Engine, a dependency that requires OpenBLAS. This involves conda installation, cloning the repository if it doesn't exist, and then running the setup script with specificblas include directories.

```bash
conda install openblas-devel -c anaconda
git clone https://github.com/NVIDIA/MinkowskiEngine.git # if not exist
cd MinkowskiEngine
python setup.py install --blas_include_dirs=${CONDA_PREFIX}/include --blas=openblas
cd ..
```

--------------------------------

### Troubleshoot IsaacGym FileNotFoundError

Source: https://roboverse.wiki/_sources/metasim/get_started/installation

Provides a command to create a missing directory structure that can resolve FileNotFoundError issues when installing IsaacGym.

```bash
mkdir -p $CONDA_PREFIX/lib/python3.8/site-packages/isaacgym/_bindings/src
cp -r third_party/isaacgym/python/isaacgym/_bindings/src/gymtorch $CONDA_PREFIX/lib/python3.8/site-packages/isaacgym/_bindings/src/gymtorch
```

--------------------------------

### Install uv Dependency Manager

Source: https://roboverse.wiki/_sources/metasim/get_started/installation

Installs the 'uv' package manager using pip. 'uv' is used by RoboVerse for efficient dependency management.

```bash
pip install uv
```

--------------------------------

### Verify Docker Installation

Source: https://roboverse.wiki/_sources/metasim/get_started/docker

Runs the 'hello-world' container to confirm that Docker is installed and functioning correctly after adding the user to the docker group.

```bash
docker run hello-world
```

--------------------------------

### Troubleshooting cuRobo Build: Wstrict-prototypes Warning

Source: https://roboverse.wiki/_sources/metasim/get_started/advanced_installation/curobo

This Python code snippet addresses a common C++ compilation warning by modifying the cuRobo setup file to remove the problematic flag.

```python
import distutils.sysconfig
cfg_vars = distutils.sysconfig.get_config_vars()
for key, value in cfg_vars.items():
    if type(value) == str:
        cfg_vars[key] = value.replace("-Wstrict-prototypes", "")
```

--------------------------------

### Install Core RoboVerse and RL Dependencies

Source: https://roboverse.wiki/_sources/metasim/get_started/advanced/rl_example/infrastructure

Commands to install the core metasim framework and essential Reinforcement Learning libraries including stable-baselines3, PyTorch, tensordict, loguru, and tyro.

```bash
pip install -e .

pip install stable-baselines3
pip install torch torchvision
pip install tensordict
pip install loguru
pip install tyro
```

--------------------------------

### Clone and Install RoboVerse RDT

Source: https://roboverse.wiki/roboverse_learn/imitation_learning/rdt

Installs the RoboVerse RDT repository and sets up the necessary Conda environment, PyTorch, flash-attn, and other dependencies.

```shell
git clone https://github.com/wbjsamuel/RoboVerse-RDT
cd RoboticsDiffusionTransformer

conda create -n rdt python=3.10.0
conda activate rdt

pip install torch==2.1.0 torchvision==0.16.0  --index-url https://download.pytorch.org/whl/cu121

pip install packaging==24.0

pip install flash-attn --no-build-isolation

pip install -r requirements.txt
```

--------------------------------

### Run GSNet Test Case

Source: https://roboverse.wiki/_sources/metasim/get_started/advanced_installation/gsnet

Executes the GSNet test case using a Python script. This command verifies the installation and functionality of GSNet.

```bash
python third_party/gsnet/gsnet.py
```

--------------------------------

### Install OpenVLA and Flash Attention

Source: https://roboverse.wiki/_sources/roboverse_learn/imitation_learning/openvla

Clones the OpenVLA repository, installs it as an editable package, and then installs Flash Attention 2, which is a crucial dependency for performance. Includes troubleshooting steps for Flash Attention installation.

```bash
git clone https://github.com/openvla/openvla.git
cd openvla
pip install -e .

# Install Flash Attention 2
pip install packaging ninja
ninja --version   # should print a version number
pip install "flash-attn==2.5.5" --no-build-isolation
```

--------------------------------

### Handler Lifecycle and Initialization Example (Python)

Source: https://roboverse.wiki/_sources/metasim/concept/handler

Illustrates the basic structure of a handler's __init__ method, receiving a ScenarioCfg and extracting robot and camera references. This is a conceptual example showing the setup phase.

```python
class MyHandler:
    def __init__(self, config):
        self.config = config
        self.robots = config.robots
        self.cameras = config.cameras
        self.object_dict = config.objects

    def launch(self):
        # ... build simulator model ...
        pass

    def close(self):
        # ... release resources ...
        pass
```

--------------------------------

### Create Conda Environment and Install bpy Wheel

Source: https://roboverse.wiki/_sources/metasim/get_started/advanced_installation/blender

Creates a new conda environment named 'blender' with Python 3.11, activates it, navigates to the directory containing the bpy wheel file, and installs the wheel using pip.

```bash
conda create -n blender python=3.11
conda activate blender
cd ~/Downloads/blender-git/build_linux_bpy/bin/
pip install bpy-4.2.7-*.whl
```

--------------------------------

### Prepare RLDS Data Directory

Source: https://roboverse.wiki/_sources/roboverse_learn/imitation_learning/octo

Creates a 'data' directory and a symbolic link to your RLDS formatted data for Octo fine-tuning. Replace 'PATH_TO_RLDS_DATA' with the actual path to your data.

```bash
mkdir data
ln -s PATH_TO_RLDS_DATA data/YOUR_TASK_NAME
```

--------------------------------

### Fix: isaacgym/_bindings/src/gymtorch/gymtorch.cpp Not Found

Source: https://roboverse.wiki/metasim/get_started/installation

Resolves a FileNotFoundError for gymtorch.cpp by creating the necessary directory structure and copying the missing files. This is a workaround for a common installation issue.

```bash
mkdir -p $CONDA_PREFIX/lib/python3.8/site-packages/isaacgym/_bindings/src

```

```bash
cp -r third_party/isaacgym/python/isaacgym/_bindings/src/gymtorch $CONDA_PREFIX/lib/python3.8/site-packages/isaacgym/_bindings/src/gymtorch

```

--------------------------------

### Install JAX with CUDA Support

Source: https://roboverse.wiki/_sources/roboverse_learn/imitation_learning/octo

Installs the JAX library with CUDA support for GPU acceleration, specifying version 0.4.20 and CUDA 12.4 compatibility. Also installs cuDNN for NVIDIA GPUs.

```bash
pip install --upgrade "jax[cuda12_pip]==0.4.20" -f https://storage.googleapis.com/jax-releases/jax_cuda_releases.html
conda install -c nvidia cudnn=8.9
```

--------------------------------

### Install Isaacsim Dependencies for PyRoki

Source: https://roboverse.wiki/_sources/metasim/get_started/advanced_installation/pyroki

Installs specific versions of numpy and jax required for PyRoki when used with Isaacsim. These versions ensure compatibility.

```bash
pip install numpy==1.26.0 # For Isaacsim
pip install jax==0.6.0 # For Isaacsim
```

--------------------------------

### Finetune RDT Model

Source: https://roboverse.wiki/_sources/roboverse_learn/imitation_learning/rdt

Sets up environment variables and launches the finetuning process using 'accelerate launch' with DeepSpeed and specified model configurations.

```bash
export NCCL_IB_HCA=mlx5_0:1,mlx5_1:1,mlx5_2:1,mlx5_3:1,mlx5_4:1,mlx5_7:1,mlx5_8:1,mlx5_9:1
export NCCL_IB_DISABLE=0
export NCCL_SOCKET_IFNAME=lo
export NCCL_DEBUG=INFO
export NCCL_NVLS_ENABLE=1

export TEXT_ENCODER_NAME="google/t5-v1_1-xxl"
export VISION_ENCODER_NAME="google/siglip-so400m-patch14-384"
export OUTPUT_DIR="./checkpoints/rdt-finetune-1b-pickcube"
export CFLAGS="-I/usr/include"
export LDFLAGS="-L/usr/lib/x86_64-linux-gnu"
# export CUTLASS_PATH="/path/to/cutlass"

export WANDB_PROJECT="rdt-finetune-1b-pickcube"

if [ ! -d "$OUTPUT_DIR" ]; then
    mkdir "$OUTPUT_DIR"
    echo "Folder '$OUTPUT_DIR' created"
else
    echo "Folder '$OUTPUT_DIR' already exists"
fi

# For run in multi nodes/machines
# deepspeed --hostfile=hostfile.txt main.py \
#    --deepspeed="./configs/zero2.json" \
#     ...

accelerate launch main.py \
    --deepspeed="./configs/zero2.json" \
    --pretrained_model_name_or_path="robotics-diffusion-transformer/rdt-1b" \
    --pretrained_text_encoder_name_or_path=$TEXT_ENCODER_NAME \
    --pretrained_vision_encoder_name_or_path=$VISION_ENCODER_NAME \
    --output_dir=$OUTPUT_DIR \
    --train_batch_size=16 \
    --sample_batch_size=32 \
    --max_train_steps=200000 \
    --checkpointing_period=1000 \
    --sample_period=500 \
    --checkpoints_total_limit=40 \
    --lr_scheduler="constant" \
    --learning_rate=1e-4 \
    --mixed_precision="bf16" \
    --dataloader_num_workers=8 \
    --image_aug \
    --dataset_type="pretrain" \
    --state_noise_snr=40 \
    --report_to=wandb

    # Use this to resume training from some previous checkpoint
    # --resume_from_checkpoint="checkpoint-36000" \
    # Use this to load from saved lanuage instruction embeddings,
    # instead of calculating it during training
    # --precomp_lang_embed \

```

--------------------------------

### Set Up Conda Environment for Octo

Source: https://roboverse.wiki/_sources/roboverse_learn/imitation_learning/octo

Creates and activates a Conda environment named 'octo' with Python 3.10, installs the package in editable mode, and installs required dependencies.

```bash
conda create -n octo python=3.10
conda activate octo
pip install -e .
pip install -r requirements.txt
```

--------------------------------

### Install CoppeliaSim v4.1.0

Source: https://roboverse.wiki/metasim/get_started/advanced_installation/pyrep

Downloads and extracts the CoppeliaSim v4.1.0 educational version for Ubuntu 20.04. It assumes a 'third_party' directory exists.

```bash
cd third_party
wget https://www.coppeliarobotics.com/files/V4_1_0/CoppeliaSim_Edu_V4_1_0_Ubuntu20_04.tar.xz
tar xvf CoppeliaSim_Edu_V4_1_0_Ubuntu20_04.tar.xz
rm CoppeliaSim_Edu_V4_1_0_Ubuntu20_04.tar.xz
```

--------------------------------

### Install Roboverse Python Dependencies

Source: https://roboverse.wiki/_sources/metasim/get_started/advanced_installation/blender

Installs key Python libraries required for Roboverse, including specific versions of PyTorch with CUDA support, OpenCV, NumPy, Imageio, Gymnasium, and various utility libraries.

```bash
pip install torch==2.4.0 torchvision==0.19.0 --index-url https://download.pytorch.org/whl/cu118
pip install opencv-python "numpy<2" "imageio[ffmpeg]" gymnasium==0.29.0
pip install rootutils loguru rich tyro tqdm huggingface-hub
```

--------------------------------

### Build Blender Dependencies and Update Makefiles

Source: https://roboverse.wiki/_sources/metasim/get_started/advanced_installation/blender

Navigates to the Blender source directory, installs Linux-specific build dependencies, updates the build environment, and compiles the Blender project, including the bpy module.

```bash
cd ~/Downloads/blender-git/blender/
./build_files/build_environment/install_linux_packages.py
./build_files/utils/make_update.py --use-linux-libraries
make update
make bpy
```

--------------------------------

### Navigate to GSNet Directory

Source: https://roboverse.wiki/metasim/get_started/advanced_installation/gsnet

Changes the current working directory to the 'gsnet' folder located within the 'third_party' directory. This is typically the first step before installing GSNet.

```shell
cd third_party/gsnet
```

--------------------------------

### Install Core Metasim Framework and RL Libraries (Bash)

Source: https://roboverse.wiki/metasim/get_started/advanced/rl_example/infrastructure

Commands to install the core metasim framework and essential Reinforcement Learning libraries using pip. Dependencies include stable-baselines3, PyTorch, tensordict, loguru, and tyro.

```bash
# Core metasim framework
pip install -e .

# RL libraries
pip install stable-baselines3
pip install torch torchvision
pip install tensordict
pip install loguru
pip install tyro

```

--------------------------------

### Python Unified Simulator Integration with Handler

Source: https://roboverse.wiki/_sources/metasim/get_started/quick_start/guide

Illustrates integrating RoboVerse as a unified simulator by defining a scenario and creating a handler. It includes a basic control loop for setting state, applying actions, simulating, and getting updated states.

```python
from metasim.scenario.scenario import ScenarioCfg
from metasim.utils.setup_util import get_handler

# Define your scenario
scenario = ScenarioCfg(
    robots=["franka"],
    simulator="mujoco",
    num_envs=1
)

# Create handler directly - one step
handler = get_handler(scenario)

# Control loop
states = ...
handler.set_state(states)     # Set state (for resets)

actions = ...  # Your control actions
handler.set_dof_targets(actions)     # Apply actions to robot

handler.simulate()                  # Step physics
obs = handler.get_state(mode="tensor")  # Get updated state
```

--------------------------------

### Clone graspness_implementation Repository

Source: https://roboverse.wiki/_sources/metasim/get_started/advanced_installation/gsnet

Clones the graspness_implementation repository from GitHub, which is a dependency for GSNet. This command should be run from the third_party/gsnet directory.

```bash
cd third_party/gsnet
git clone https://github.com/rhett-chen/graspness_implementation.git
```

--------------------------------

### Install RoboVerse Reinforcement Learning Dependencies (PPO)

Source: https://roboverse.wiki/metasim/get_started/docker

Installs the Stable-Baselines3 library for running PPO reinforcement learning demos within RoboVerse.

```shell
pip install stable-baselines3
```

--------------------------------

### Fast TD3 Humanoid Training

Source: https://roboverse.wiki/_sources/metasim/get_started/advanced/rl_example/quick_examples

Initiates training for a humanoid robot using the Fast TD3 algorithm on a running task. The script allows for customization of robot types, tasks, and simulation backends by modifying the CONFIG variable within the script.

```bash
# Train H1 humanoid on running task
python get_started/rl/1_fttd3.py

# Modify CONFIG in the script for different settings:
# - Change "robots": ["franka"] for different robot
# - Change "task": "reach_origin" for different task
# - Change "sim": "isaacgym" for different simulator
```

--------------------------------

### Install Additional IsaacLab Libraries

Source: https://roboverse.wiki/_sources/metasim/get_started/docker

Installs essential Python libraries required for IsaacLab functionality, including rootutils, tyro, loguru, and open3d.

```bash
pip install rootutils
pip install tyro
pip install loguru
pip install open3d
```

--------------------------------

### Install libGLU for Headless Servers

Source: https://roboverse.wiki/_sources/metasim/troubleshooting/isaaclab

Installs the necessary libGLU library for servers running in headless mode that encounter 'libGLU.so.1: cannot open shared object file' errors. This is crucial for applications relying on OpenGL utility functions.

```bash
sudo apt-get update
sudo apt-get install libglu1-mesa
```

--------------------------------

### Install Additional Python Utility Libraries

Source: https://roboverse.wiki/metasim/get_started/advanced_installation/blender

Installs supplementary Python packages that provide utilities for data handling, logging, rich output, argument parsing, progress tracking, and model hub access.

```shell
pip install rootutils loguru rich tyro tqdm huggingface-hub
```

--------------------------------

### Static Scene Simulation Examples by Simulator

Source: https://roboverse.wiki/metasim/get_started/quick_start/0_static_scene

These commands demonstrate how to run the static scene simulation with different backend simulators. Each command specifies the simulator using the '--sim' argument. Some simulators have specific recommendations for headless mode usage.

```python
python get_started/0_static_scene.py  --sim isaacsim
```

```python
python get_started/0_static_scene.py  --sim isaacgym
```

```python
python get_started/0_static_scene.py  --sim mujoco --headless
```

```python
python get_started/0_static_scene.py  --sim genesis
```

```python
python get_started/0_static_scene.py  --sim sapien3
```

```python
python get_started/0_static_scene.py  --sim pybullet
```

--------------------------------

### Create and Activate Conda Environment for PyRep

Source: https://roboverse.wiki/_sources/metasim/get_started/advanced_installation/pyrep

Sets up a dedicated Conda environment with Python 3.10 for PyRep. This isolates dependencies and ensures a clean installation.

```bash
conda create -n pyrep python=3.10
conda activate pyrep
```

--------------------------------

### Troubleshoot IsaacGym ImportError (libpython)

Source: https://roboverse.wiki/_sources/metasim/get_started/installation

Provides an environment variable export command to resolve ImportError related to missing libpython shared objects, typically encountered with IsaacGym.

```bash
export LD_LIBRARY_PATH=$CONDA_HOME/envs/metasim_isaacgym/lib:$LD_LIBRARY_PATH
```

--------------------------------

### Download and Link Multi-modal Encoders

Source: https://roboverse.wiki/_sources/roboverse_learn/imitation_learning/rdt

Downloads T5 and Siglip encoders from Hugging Face and creates symbolic links to them in the repository's google directory.

```bash
huggingface-cli download google/t5-v1_1-xxl --cache-dir YOUR_CACHE_DIR
huggingface-cli download google/siglip-so400m-patch14-384 --cache-dir YOUR_CACHE_DIR

# Under the root directory of this repo
mkdir -p google

# Link the downloaded encoders to this repo
ln -s /YOUR_CACHE_DIR/t5-v1_1-xxl google/t5-v1_1-xxl
ln -s /YOUR_CACHE_DIR/siglip-so400m-patch14-384 google/siglip-so400m-patch14-384
```

--------------------------------

### Clone graspness_implementation Repository

Source: https://roboverse.wiki/metasim/get_started/advanced_installation/gsnet

Clones the 'graspness_implementation' repository from GitHub. This repository likely contains essential code for the graspness functionality within GSNet.

```shell
git clone https://github.com/rhett-chen/graspness_implementation.git
```

--------------------------------

### Build Blender Python Wheel

Source: https://roboverse.wiki/_sources/metasim/get_started/advanced_installation/blender

Compiles the Blender Python module (bpy) into a wheel file, which is a package format for Python. The output wheel file is placed in a specified directory.

```bash
python3 ./build_files/utils/make_bpy_wheel.py ../build_linux_bpy/bin/
```

--------------------------------

### Install RoboVerse Reinforcement Learning Dependencies

Source: https://roboverse.wiki/_sources/metasim/get_started/docker

Installs libraries for running RoboVerse's reinforcement learning demos. This includes stable-baselines3 for PPO and specific packages like mujoco-mjx, jax[cuda12], wandb, and tensordict for FastTD3.

```bash
pip install stable-baselines3

pip install mujoco-mjx
pip install dm-control
pip install jax[cuda12]
pip install wandb
pip install tensordict
```

--------------------------------

### Setup and Use IK Solver

Source: https://roboverse.wiki/_sources/metasim/get_started/quick_start/4_motion_planning

Demonstrates initializing the MetaSim IK solver with a specified backend ('pyroki' or 'curobo') and preparing current robot joint positions for inverse kinematics calculations. It handles reordering joint indices to match the solver's expected format.

```python
from metasim.utils.ik_solver import setup_ik_solver

# Initialize IK solver
ik_solver = setup_ik_solver(robot_cfg, solver="pyroki")  # or "curobo"
# IK solver expects original joint order, but state uses alphabetical order
reorder_idx = handler.get_joint_reindex(scenario.robots[0].name)
inverse_reorder_idx = [reorder_idx.index(i) for i in range(len(reorder_idx))]
curr_robot_q = obs.robots[scenario.robots[0].name].joint_pos[:, inverse_reorder_idx]
```

--------------------------------

### Example: Run Task with Isaac Gym

Source: https://roboverse.wiki/metasim/get_started/quick_start/9_cfg_task

This example demonstrates how to run the config-based task script specifically for the Isaac Gym simulator. It sets the simulator to 'isaacgym', uses 4 environments, and specifies the 'close_box' task.

```bash
python get_started/9_cfg_task.py --sim isaacgym --num_envs 4 --task close_box
```

--------------------------------

### Example: Run Task with IsaacSim

Source: https://roboverse.wiki/metasim/get_started/quick_start/9_cfg_task

This example demonstrates how to run the config-based task script specifically for the IsaacSim simulator. It sets the simulator to 'isaacsim', uses 4 environments, and specifies the 'close_box' task.

```bash
python get_started/9_cfg_task.py --sim isaacsim --num_envs 4 --task close_box
```
Tokens:
Raw