RoboTwin 2.0

https://robotwin-platform.github.io/doc/
Admin
RoboTwin 2.0 is a scalable data generator and benchmark platform for robust bimanual robotic...
Tokens:50,872
Snippets:362
Trust Score:8.3
Update:1 week ago
Show doc for...
Tokens:
Raw
### Install Vulkan on Linux

Source: https://robotwin-platform.github.io/doc/usage/robotwin-install.html

Install Vulkan libraries and tools using apt. Verify the installation by running 'vulkaninfo'.

```bash
sudo apt install libvulkan1 mesa-vulkan-drivers vulkan-tools
```

--------------------------------

### Start environment client on simulator or robot machine

Source: https://robotwin-platform.github.io/doc/usage/xpolicylab.html

Starts the environment client on the simulator or robot machine, connecting to the policy server's real IP. The <additional_info> is a comma-separated key=value string forwarded to the client.

```bash
cd policy/<POLICY>
bash setup_eval_env_client.sh \
  <bench_name> <task_name> <ckpt_name> <env_cfg_type> <action_type> <seed> \
  <env_gpu_id> <eval_env_conda_env> <additional_info> \
  <policy_server_port> <policy_server_ip>

```

--------------------------------

### Install requirements

Source: https://robotwin-platform.github.io/doc/usage/robotwin-install.html

Install Python requirements from scripts/requirements.txt.

```bash
pip install -r scripts/requirements.txt
```

--------------------------------

### Example Advanced Code Generation Command (pick_dual_bottles_easy)

Source: https://robotwin-platform.github.io/doc/usage/expert-code-gen.html

Example invocation of the advanced generation script for the task pick_dual_bottles_easy, which creates ./envs_gen/gpt_pick_dual_bottles_easy.py.

```bash
python code_gen/task_generation_mm.py pick_dual_bottles_easy
```

--------------------------------

### Clone XPolicyLab and install dependencies

Source: https://robotwin-platform.github.io/doc/usage/xpolicylab.html

Clones the XPolicyLab repository and installs it in editable mode. Run these commands from a new directory to set up the project for adapter development.

```bash
mkdir demo_env
cd demo_env
git clone https://github.com/XPolicyLab/XPolicyLab.git
cd XPolicyLab
pip install -e .

```

--------------------------------

### Install CuRobo

Source: https://robotwin-platform.github.io/doc/usage/robotwin-install.html

Clone and install CuRobo from the NVlabs repository.

```bash
cd envs
git clone https://github.com/NVlabs/curobo.git
cd curobo
pip install -e . --no-build-isolation
cd ../..
```

--------------------------------

### Common adapter workflow: install, process, train, eval

Source: https://robotwin-platform.github.io/doc/usage/xpolicylab.html

Standard sequence of bash commands for any policy adapter. Install the runtime, optionally prepare data, optionally train, then evaluate. Arguments like bench_name, ckpt_name, env_cfg_type, action_type, and seed are shared across process_data.sh, train.sh, and eval.sh.

```bash
cd policy/<POLICY>

# Install the policy runtime.
bash install.sh

# Optional: convert or prepare policy-specific data.
bash process_data.sh <bench_name> <ckpt_name> <env_cfg_type> <action_type> [extra_args...]

# Optional: train.
bash train.sh <bench_name> <ckpt_name> <env_cfg_type> <action_type> <seed> <gpu_id> [extra_args...]

# Evaluate on one machine.
bash eval.sh <bench_name> <task_name> <ckpt_name> <env_cfg_type> <action_type> <seed> \
  <policy_gpu_id> <env_gpu_id> <policy_env_or_uv_path> <eval_env_conda_env>

```

--------------------------------

### Install pytorch3d

Source: https://robotwin-platform.github.io/doc/usage/robotwin-install.html

Install pytorch3d from the stable branch of the Facebook Research repository.

```bash
pip install "git+https://github.com/facebookresearch/pytorch3d.git@stable"
```

--------------------------------

### Start policy server on GPU machine

Source: https://robotwin-platform.github.io/doc/usage/xpolicylab.html

Starts the policy server on the GPU machine, binding to 0.0.0.0 so other machines can reach it. Used for split-machine deployment.

```bash
cd policy/<POLICY>
bash setup_eval_policy_server.sh \
  <bench_name> <task_name> <ckpt_name> <env_cfg_type> <action_type> <seed> \
  <policy_gpu_id> <policy_env_or_uv_path> <policy_server_port> 0.0.0.0

```

--------------------------------

### Calibration Commands: resize, create, clone, rotate, align, remove, save, exit

Source: https://robotwin-platform.github.io/doc/usage/object-annotation.html

List of commands available in the calibration window. Each command has usage and examples. Remember to save before exiting.

```text
resize:
    Usage:
        resize <x_size> <y_size> <z_size>: Set scaling along x, y, z axes
        resize <size>: Uniformly scale all three axes
    Example:
        resize 0.1
create:
    Usage:
        create <type>: Create (t)arget, (c)ontact, (f)unctional, or (o)rientation point
        create: Waits for input of point name
    Examples:
        create t
        create f
clone:
    Usage:
        clone <type> <id>: Clone a specified type and ID point in place
        clone: Waits for input of point type and ID
    Examples:
        clone t 1: Clones contact point target_1 to create a new target point (e.g., target_2)
rotate:
    Usage:
        rotate <id> <axis> <interval>: Rotate a specified contact point around its own axis by a given interval, generating points belonging to the same group
    Example:
        rotate 1 x 90: Rotates contact_1 around its x-axis every 90 degrees, creating three additional contact points, and writes the group into concat_points_group
align:
    Usage:
        align: Aligns all group points' positions to the first point in the group
remove:
    Usage:
        remove <type> <id>: Removes a point with the specified name
        remove: Waits for input of point name
    Examples:
        remove t 0
save:
    Saves current calibration data — always remember to save!
exit:
    Exits the calibration window
```

--------------------------------

### Run basic environment installation script

Source: https://robotwin-platform.github.io/doc/usage/robotwin-install.html

Run scripts/_install.sh to install basic environments and CuRobo. If you encounter a curobo config path issue, run 'python scripts/update_embodiment_config_path.py'.

```bash
bash scripts/_install.sh
```

--------------------------------

### Concrete eval.sh example for AHA_WAM

Source: https://robotwin-platform.github.io/doc/usage/xpolicylab.html

Concrete example of running eval.sh for the AHA_WAM policy. It loads checkpoints/RoboDojo-cotrain-arx_x5-joint-0/ and evaluates on the stack_bowls task.

```bash
cd policy/AHA_WAM
bash eval.sh RoboDojo stack_bowls cotrain arx_x5 joint 0 0 0 aha_wam robodojo
# loads checkpoints/RoboDojo-cotrain-arx_x5-joint-0/ and evaluates on stack_bowls

```

--------------------------------

### Franka configuration example for config.yml

Source: https://robotwin-platform.github.io/doc/usage/new-embodiment.html

Complete YAML configuration for a Franka robot, including URDF/SRDF paths, joint stiffness, gripper settings, home state, and camera parameters. Adjust values for new embodiments as needed.

```yaml
urdf_path: "./panda.urdf"
srdf_path: "./panda.srdf"
joint_stiffness: 1000
joint_damping: 200
gripper_stiffness: 1000
gripper_damping: 200
move_group: ["panda_hand","panda_hand"]
ee_joints: ["panda_hand_joint","panda_hand_joint"]
arm_joints_name: [['panda_joint1', 'panda_joint2', 'panda_joint3', 'panda_joint4', 'panda_joint5', 'panda_joint6', 'panda_joint7'],
                  ['panda_joint1', 'panda_joint2', 'panda_joint3', 'panda_joint4', 'panda_joint5', 'panda_joint6', 'panda_joint7']]
gripper_name:
  - base: "panda_finger_joint1"
    mimic: [["panda_finger_joint2", 1., 0.]]
  - base: "panda_finger_joint1"
    mimic: [["panda_finger_joint2", 1., 0.]]
gripper_bias: 0.08
gripper_scale: [0.0, 0.04]
homestate: [[0, 0.19634954084936207, 0.0, -2.617993877991494, 0.0, 2.941592653589793, 0.7853981633974483],
            [0, 0.19634954084936207, 0.0, -2.617993877991494, 0.0, 2.941592653589793, 0.7853981633974483]]
delta_matrix: [[0,0,1],[0,-1,0],[1,0,0]]
global_trans_matrix: [[1,0,0],[0,-1,0],[0,0,-1]]
robot_pose: [[0, -0.65, 0.75, 0.707, 0, 0, 0.707],
             [0, -0.65, 0.75, 0.707, 0, 0, 0.707]]
planner: "curobo"
dual_arm: False
rotate_lim: [0.1, 0.8]
grasp_perfect_direction: ['right', 'left']
static_camera_list: 
- name: head_camera
  position: [0.0, 0.8, 0.9]
  forward: [0, -1, 0]
  left: [1, 0, 0]

```

--------------------------------

### Create Calibration Window: python scripts/create_object_data.py

Source: https://robotwin-platform.github.io/doc/usage/object-annotation.html

Run this command to create a calibration window for a model. The model_name is a subdirectory under assets/objects/. For example, use '020_hammer' for the hammer model.

```bash
python scripts/create_object_data.py [-s START] model_name

positional arguments:
    model_name            Model name

options:
    -s START, --start START Start id
```

--------------------------------

### Download selected tasks

Source: https://robotwin-platform.github.io/doc/usage/collect-data.html

Pass task names as arguments to download only those tasks. Example downloads adjust_bottle and beat_block_hammer.

```bash
bash scripts/download_xpolicylab_data.sh adjust_bottle beat_block_hammer

```

--------------------------------

### Migrate legacy data with process_data_xpolicylab.py

Source: https://robotwin-platform.github.io/doc/usage/collect-data.html

Converts legacy raw layout data to XPolicyLab format. Only works on legacy data, not freshly collected data. Includes an example with --overwrite.

```bash
python scripts/process_data_xpolicylab.py \
  <task_name> <task_config> [episode_num] --overwrite

# Example
python scripts/process_data_xpolicylab.py \
  adjust_bottle demo_clean 1 --overwrite

```

--------------------------------

### Split deployment: remote policy server and local simulator

Source: https://robotwin-platform.github.io/doc/usage/xpolicylab.html

Starts a policy server pool on the policy host, then runs the simulator locally. The server command uses `serve` with `remote_server.yml`. The simulator command uses `multitask` with `--enable-remote` and `--policy-server-ip`/`--policy-server-port`. These can be repeated for a server pool or configured via `enable_remote`/`policy_server_ip`/`policy_server_port` in the eval config.

```bash
# On the policy-server host (fill in the placeholders first):
bash scripts/eval_policy.sh serve --config env_cfg/eval/remote_server.yml

```

```bash
# On the simulator host:
bash scripts/eval_policy.sh multitask \
  --config env_cfg/eval/all_tasks.yml \
  --policy-name <policy_name> \
  --env-cfg-type arx_x5 \
  --eval-env-conda-env <robotwin_env> \
  --enable-remote \
  --policy-server-ip <server_ip> --policy-server-port <port>

```

--------------------------------

### Calibration Commands for URDF Articulation

Source: https://robotwin-platform.github.io/doc/usage/object-annotation.html

Lists all calibration commands available in the articulation calibration window. Each command has a usage syntax and some include examples. The commands are: run, qpos, mass, resize, create, rebase, clone, rotate, align, remove, save, and exit. Note that 'run' requires manual stop with Ctrl+C, and 'save' must be executed to save calibration data.

```text
run:
    Usage:
        run
        Press <Ctrl + C> to stop and save information
    Used to obtain stable points through steps, generally selected at the beginning of calibration to determine if running is necessary.
    Since this command does not limit the step upper limit, you need to manually stop running (press Ctrl+C) based on whether the asset in the UI interface is stable.
qpos:
    Usage:
        qpos
    Get the current joint state as the initial pose when loading the asset into the task.
mass:
    Usage:
        mass <m1> <m2> ...: Set the mass of the articulation joint, ensuring that the input matches the displayed link count (excluding base) in order.
    Example:
        mass 0.5 0.05
resize:
    Usage:
        resize <size>: Synchronize the scaling of all three axes of the object
    Example:
        resize 0.1
create:
    Usage:
        create <type> <base_link>: Create (t)arget, (c)ontact, (f)unctional, (o)rientation points
    Example:
        create c link1
rebase:
    Usage:
        rebase <type> <id> <base_link>: Modify the base link of the specified point
    Example:
        rebase c 0 link1
clone:
    Usage:
        clone <type> <id>: Create an in-place copy of the specified point (without base)
    Example:
        clone t 1: Create a new target point (e.g., target_2<link1>) by copying target_1<link1>
rotate:
    Usage:
        rotate <id> <axis> <interval>: Rotate the specified contact point around its own specified axis by the specified interval, generating points belonging to the same group
    Example:
        rotate 1 x 90: Rotate contact_1 around its own x-axis by 90 degrees, generating three contact points, and write the grouping of the four points into concat_points_group
align:
    Usage:
        align: Align the positions of all group points to the first point in the group
remove:
    Usage:
        remove <type> <id>: Remove the specified point (without base)
    Example:
        remove t 0
save:
    Usage:
        save: Save the current calibration data, and make sure to save!
exit:
    Usage:
        exit: Exit the calibration window
```

--------------------------------

### move_to_pose - Move arm to a specific pose

Source: https://robotwin-platform.github.io/doc/usage/API.html

Shows how to call move_to_pose to move the arm to a specific absolute pose. The example obtains the current pose, adjusts it for left or right arm, then moves the skillet to that target pose.

```python
target_pose = self.get_arm_pose(arm_tag=arm_tag)
if arm_tag == 'left':
    # Set specific position and orientation for left arm
    target_pose[:2] = [-0.1, -0.05]
    target_pose[2] -= 0.05
    target_pose[3:] = [-0.707, 0, -0.707, 0]
else:
    # Set specific position and orientation for right arm
    target_pose[:2] = [0.1, -0.05]
    target_pose[2] -= 0.05
    target_pose[3:] = [0, 0.707, 0, -0.707]

# Move the skillet to the defined target pose
self.move(
    self.move_to_pose(arm_tag=arm_tag, target_pose=target_pose)
)

```

--------------------------------

### Create embodiment directory and files

Source: https://robotwin-platform.github.io/doc/usage/new-embodiment.html

Commands to create the embodiment directory and initialize the required configuration files (curobo_tmp.yml and collision.yml) under assets/embodiments/new_robot.

```bash
cd ${ROBOTWIN_ROOT_PATH}
mkdir -p assets/embodiments/new_robot
cd assets/embodiments/new_robot
touch curobo_tmp.yml
touch collision.yml

```

--------------------------------

### Modify URDF: comment out collisions and remove joint limits

Source: https://robotwin-platform.github.io/doc/usage/new-embodiment.html

Example modifications to panda.urdf: comment out collision tags for each link and remove joint limits by changing revolute joints to continuous. The example shows commenting out a collision block and changing a joint type, with the limit line commented out.

```xml
<!-- Comment out collision -->
<link name="panda_link1">
    <visual>
      <geometry>
        <mesh filename="franka_description/meshes/visual/link1.glb"/>
      </geometry>
    </visual> 
    <!-- <collision>
      <geometry>
        <mesh filename="franka_description/meshes/collision/link1.stl"/>
      </geometry>
    </collision> -->
</link>

<!-- Remove joint limits, change revolute to continuous -->
<!-- <joint name="panda_joint3" type="revolute"> -->
<joint name="panda_joint3" type="continuous">
    <origin rpy="1.57079632679 0 0" xyz="0 -0.316 0"/>
    <parent link="panda_link2"/>
    <child link="panda_link3"/>
    <axis xyz="0 0 1"/>
    <!-- Remove this line: <limit effort="87" lower="-2.8973" upper="2.8973" velocity="2.1750"/> -->
</joint>

```

--------------------------------

### Initialize submodules for existing checkout

Source: https://robotwin-platform.github.io/doc/usage/robotwin-install.html

Initialize and update the XPolicyLab submodule to the version pinned by RoboTwin.

```bash
git submodule update --init --recursive XPolicyLab
```

--------------------------------

### Generate episode instructions with generate_episode_instructions.py

Source: https://robotwin-platform.github.io/doc/usage/description.html

Generates a specified number of episode instructions for a task and setting. Parameters include task_name, setting, and max_num (maximum number of descriptions per episode).

```bash
# Generate 60 task descriptions for a task
python3 utils/generate_episode_instructions.py place_shoe franka-panda-D435 1000

```

--------------------------------

### Set HF_LEROBOT_HOME and convert with LeRobot v2.1 and v3.0

Source: https://robotwin-platform.github.io/doc/usage/collect-data.html

Set the HF_LEROBOT_HOME environment variable to a directory with enough space (default is ~/.cache/huggingface/lerobot). Then run the conversion scripts for LeRobot v2.1 and v3.0 on all demo_clean tasks with the aloha_agilex embodiment, using --max_episode 50.

```bash
export HF_LEROBOT_HOME=/path/with/enough/space/lerobot

# LeRobot v2.1 — all demo_clean tasks
python XPolicyLab/scripts/transform_lerobot_v21_format.py \
  "demo_clean.*.aloha_agilex" \
  --repo_id robotwin_demo_clean_aloha_agilex \
  --max_episode 50

# LeRobot v3.0 — same selection
python XPolicyLab/scripts/transform_lerobot_v30_format.py \
  "demo_clean.*.aloha_agilex" \
  --repo_id robotwin_demo_clean_aloha_agilex_v30 \
  --max_episode 50

# Single task
python XPolicyLab/scripts/transform_lerobot_v21_format.py \
  "demo_clean.beat_block_hammer.aloha_agilex" \
  --repo_id beat_block_hammer_demo_clean

```

--------------------------------

### get_arm_pose(self, arm_tag: ArmTag) -> list[float]

Source: https://robotwin-platform.github.io/doc/usage/API.html

Gets the current pose of the end-effector of the specified arm. Returns a list of 7 floats representing position and orientation.

```APIDOC
## get_arm_pose(self, arm_tag: ArmTag) -> list[float]

### Description
Gets the current pose of the end-effector of the specified arm.

### Method
SDK Method

### Parameters
- **arm_tag** (ArmTag) - Required - Which arm to query

### Returns
A list of 7 floats: `[x, y, z, qw, qx, qy, qz]`, representing position and orientation.

### Example
```
pose = self.get_arm_pose(ArmTag("left"))
```
```

--------------------------------

### Create temporary URDF backup

Source: https://robotwin-platform.github.io/doc/usage/new-embodiment.html

Before calibrating delta_matrix and global_trans_matrix, create a temporary URDF. This example uses the Franka robot: navigate to its embodiment directory and copy panda.urdf to panda.urdf.save.

```bash
cd assets/embodiments/franka-panda
cp panda.urdf panda.urdf.save

```

--------------------------------

### move_by_displacement - Lift the arm after grasping

Source: https://robotwin-platform.github.io/doc/usage/API.html

Shows how to call move_by_displacement to lift the arm. Use this after grasping an object to avoid collision. The example moves the end-effector 7 cm upward in world coordinates.

```python
self.move(
    self.move_by_displacement(
        arm_tag=arm_tag,
        z=0.07,  # Move 7cm upward
        move_axis='world'
    )
)

```

--------------------------------

### Adapter evaluation template

Source: https://robotwin-platform.github.io/doc/usage/xpolicylab.html

Template for evaluating any adapter. Replace <POLICY> and the argument values with your own. The command runs the evaluation script with the specified benchmark, task, checkpoint, environment config, action type, seeds, GPU IDs, and paths.

```bash
export EVAL_ENV_TYPE=debug
cd policy/<POLICY>
bash eval.sh <bench_name> <task_name> <ckpt_name> <env_cfg_type> <action_type> \
  <seed> <policy_gpu_id> <env_gpu_id> <policy_env_or_uv_path> <eval_env_conda_env>

```

--------------------------------

### Expected debug output for target_pose_q

Source: https://robotwin-platform.github.io/doc/usage/new-embodiment.html

Expected output from the planner.py modification. Use your actual output quaternion values, not the example above. Each robot arm will produce different quaternion values based on its specific configuration.

```text
[debug]: target_pose_q:  [ 1.68244557e-03 -9.98540531e-01 -3.19133105e-04 -5.39803316e-02]
```

--------------------------------

### Directory structure after demo data download

Source: https://robotwin-platform.github.io/doc/usage/xpolicylab.html

Shows the resulting directory layout after downloading the demo data. The data folder contains a small 10-episode HuggingFace bundle.

```text
demo_env/
├── data/        # demo data, including a small 10-episode HuggingFace bundle
└── XPolicyLab/

```

--------------------------------

### Temporarily add successful xyz coordinates in planner.py

Source: https://robotwin-platform.github.io/doc/usage/new-embodiment.html

Modify envs/robot/planner.py around line 126 to temporarily set target_pose_p and target_pose_q with a successful xyz coordinate. The example uses 0.35, 0.23, 0.09.

```python
## Temporarily add the successful xyz coordinates ##
target_pose_p = [0.35, 0.23, 0.09]  # Example: using 0.35, 0.23, 0.09
target_pose_q = [1., 0., 0., 0.]
## End temporary addition ## 
goal_pose_of_gripper = CuroboPose.from_list(list(target_pose_p) + list(target_pose_q))

```

--------------------------------

### Download assets script

Source: https://robotwin-platform.github.io/doc/usage/robotwin-install.html

Run scripts/_download_assets.sh to download RoboTwin-OD, texture library, and embodiments. If rate-limited, log in to Hugging Face with 'huggingface-cli login'.

```bash
bash scripts/_download_assets.sh
```

--------------------------------

### Set camera type in task config

Source: https://robotwin-platform.github.io/doc/usage/new-camera.html

Update the `camera` section in the task config file to use the newly defined camera type. The example assigns `Demo_Camera` to `head_camera_type` and keeps `D435` for `wrist_camera_type`.

```yaml
camera:
  head_camera_type: Demo_Camera
  wrist_camera_type: D435

```

--------------------------------

### Basic Task Code Template (${task_name}.py)

Source: https://robotwin-platform.github.io/doc/usage/expert-code-gen.html

Template for the basic task file placed in ./envs/. It must define setup_demo, load_actors, play_once, and check_success methods. The {task_name} placeholder must match the filename, and check_success determines task success.

```python
from .base_task import Base_task
from .utils import *
import sapien

class ${task_name}(Base_task):
    def setup_demo(self, **kwargs):
        # Initializes the simulation environment for the task
        # Sets up the table, robot, planner, camera, and initial positions
        # This function is called once at the beginning of each episode
        pass

    def load_actors(self):
        # Loads all the necessary objects for the task into the environment
        # Typically called from setup_demo to initialize scene objects
        # Can also be used to set initial poses for objects
        pass

    def play_once(self):
        # Contains the robot control code to complete the task
        # This is the main function that will be generated by the LLM
        # Implements the sequence of actions for the robot to achieve the task
        pass

    # Check success
    def check_success(self):
        # Defines criteria to determine if the task was completed successfully
        # Returns a boolean indicating success or failure
        # Used for evaluation and feedback during code generation
        pass
```
Tokens:
Raw