InternDataEngine

https://internrobotics.github.io/InternDataEngine-Docs/
Admin
InternDataEngine is a high-fidelity synthetic data generator for robotic manipulation that...
Tokens:48,949
Snippets:412
Trust Score:9.7
Update:1 month ago
Show doc for...
Tokens:
Raw
### Configure a GeometryObject

Source: https://internrobotics.github.io/InternDataEngine-Docs/concepts/objects.html

Example YAML configuration for a static geometry object used for environmental fixtures.

```yaml
objects:
  -
    name: table
    path: table0/instance.usd
    target_class: GeometryObject
    translation: [0.0, 0.0, 0.375]
    scale: [0.001, 0.001053, 0.001056]
```

--------------------------------

### Minimal Task Configuration

Source: https://internrobotics.github.io/InternDataEngine-Docs/config/yaml.html

A complete YAML configuration example defining a task, including robot, object, camera, and skill parameters.

```yaml
tasks:
  -
    name: banana_base_task
    asset_root: workflows/simbox/assets
    task: BananaBaseTask
    task_id: 0
    offset: null
    render: True

    arena_file: workflows/simbox/core/configs/arenas/example.yaml

    env_map:
      envmap_lib: envmap_lib
      apply_randomization: False

    robots:
      -
        name: "fr3"
        robot_config_file: workflows/simbox/core/configs/robots/fr3.yaml
        euler: [0.0, 0.0, 0.0]

    objects:
      -
        name: bottle
        path: objects/bottle.usd
        target_class: RigidObject
        dataset: oo3d
        category: bottle
        prim_path_child: Aligned
        translation: [0.3, 0.0, 0.0]
        euler: [0.0, 0.0, 0.0]
        scale: [1, 1, 1]
        apply_randomization: False

    regions:
      -
        object: bottle
        target: table
        random_type: A_on_B_region_sampler
        random_config:
          pos_range: [[0.2, -0.1, 0.0], [0.4, 0.1, 0.0]]
          yaw_rotation: [-30.0, 30.0]

    cameras:
      -
        name: head_camera
        translation: [0.0, 0.0, 0.5]
        orientation: [0.707, 0.0, 0.0, 0.707]
        camera_file: workflows/simbox/core/configs/cameras/realsense_d455.yaml
        parent: ""

    data:
      task_dir: "pick_bottle"
      language_instruction: "Pick the bottle."
      detailed_language_instruction: "Pick the bottle."
      collect_info: ""
      version: "v1.0"
      update: True
      max_episode_length: 500

    skills:
      -
        fr3:
          -
            name: pick
            objects: [bottle]
            pre_grasp_offset: 0.05
```

--------------------------------

### Camera Mounting Configurations

Source: https://internrobotics.github.io/InternDataEngine-Docs/concepts/cameras.html

Examples for attaching cameras to robot links or fixing them in the world frame.

```yaml
parent: "lift2/lift2/lift2/fl/link6"  # End-effector link
translation: [0.07, 0.01, 0.08]       # Offset from link
```

```yaml
parent: ""
translation: [0.5, 0.5, 1.2]
orientation: [0.707, 0.707, 0, 0]  # Looking down
```

--------------------------------

### Install CuRobo for Isaac Sim

Source: https://internrobotics.github.io/InternDataEngine-Docs/guides/installation.html

Install the CuRobo package in editable mode specifically for the Isaac Sim environment.

```bash
cd InternDataEngine/workflows/simbox/curobo
YOUR_PATH_TO_ISAACSIM/python.sh -m pip install -e .[isaacsim] --no-build-isolation
```

--------------------------------

### Implement Gripper State Logic

Source: https://internrobotics.github.io/InternDataEngine-Docs/custom/robot.html

Examples for determining gripper state based on joint positions for different gripper types.

```python
# Example 1: ARX Lift-2 (single-joint parallel gripper)
# Gripper is open when joint value >= 0.044 (44mm)
def _get_gripper_state(self, gripper_home):
    return 1.0 if gripper_home and gripper_home[0] >= 0.044 else -1.0

# Example 2: Franka with Robotiq 2F-85 (two-joint gripper)
# Gripper is open when both joints are 0.0
def _get_gripper_state(self, gripper_home):
    if not gripper_home or len(gripper_home) < 2:
        return 1.0
    return 1.0 if (gripper_home[0] == 0.0 and gripper_home[1] == 0.0) else -1.0
```

--------------------------------

### Configure a RigidObject

Source: https://internrobotics.github.io/InternDataEngine-Docs/concepts/objects.html

Example YAML configuration for a rigid body object, including path, scale, and randomization settings.

```yaml
objects:
  -
    name: pick_object_left
    path: pick_and_place/pre-train-pick/assets/omniobject3d-banana/omniobject3d-banana_001/Aligned_obj.usd
    target_class: RigidObject
    dataset: oo3d
    category: "omniobject3d-banana"
    prim_path_child: Aligned
    translation: [0.0, 0.0, 0.0]
    euler: [0.0, 0.0, 0.0]
    scale: [0.001, 0.001, 0.001]
    apply_randomization: True
    orientation_mode: "suggested"
```

--------------------------------

### Define Execution Pipeline Configuration

Source: https://internrobotics.github.io/InternDataEngine-Docs/guides/quickstart.html

Example YAML structure for configuring the Load, Plan, Render, and Store stages of the data pipeline.

```yaml
load_stage:
  scene_loader:                          # Scene loader
    type: env_loader
    args:
      workflow_type: SimBoxDualWorkFlow
      cfg_path: workflows/simbox/core/configs/tasks/...  # Task config path
      simulator:
        physics_dt: 1/30                 # Physics update rate
        rendering_dt: 1/30              # Render update rate
        headless: True                   # Headless mode (no GUI)
        anti_aliasing: 0                 # Anti-aliasing level
  layout_random_generator:               # Scene randomization
    type: env_randomizer
    args:
      random_num: 5                      # Number of random samples
      strict_mode: true                  # true: output count must equal random_num

plan_stage:
  seq_planner:
    type: env_planner                    # Trajectory planner

render_stage:
  renderer:
    type: env_renderer                   # Renderer

store_stage:
  writer:
    type: env_writer
    args:
      batch_async: true                  # Async writes (better perf, more memory)
      output_dir: output/${name}/        # Output directory
```

--------------------------------

### Install Python Dependencies via Isaac Sim

Source: https://internrobotics.github.io/InternDataEngine-Docs/guides/installation.html

Use the built-in Python interpreter provided by Isaac Sim to install project requirements.

```bash
cd InternDataEngine
YOUR_PATH_TO_ISAACSIM/python.sh -m pip install --upgrade pip
YOUR_PATH_TO_ISAACSIM/python.sh -m pip install -r requirements.txt
```

--------------------------------

### Configure Pick Skill for Split Aloha

Source: https://internrobotics.github.io/InternDataEngine-Docs/concepts/skills/pick

Example configuration for the Split Aloha robot using Y-axis for forward direction and Z-axis for downward approach.

```yaml
skills:
  - split_aloha:
      - left:
          - name: pick
            objects: [pick_object_left]
            filter_y_dir: ["forward", 90]
            filter_z_dir: ["downward", 140]
```

--------------------------------

### Franka Research 3 Pick Configuration

Source: https://internrobotics.github.io/InternDataEngine-Docs/concepts/skills/pick

Example configuration for a Franka robot performing a pick task with specific end-effector orientation constraints.

```yaml
# Source: workflows/simbox/core/configs/tasks/pick_and_place/franka/single_pick/omniobject3d-banana.yaml
skills:
  - franka:
      - left:
          - name: pick
            objects: [pick_object_left]
            filter_x_dir: ["forward", 90]
            filter_z_dir: ["downward", 140]
```

--------------------------------

### Configure Dual-Arm Robot YAML

Source: https://internrobotics.github.io/InternDataEngine-Docs/custom/controller.html

Reference multiple kinematics configuration files for dual-arm robot setups.

```yaml
# CuRobo kinematics config files (one per arm)
robot_file:
  - workflows/simbox/curobo/src/curobo/content/configs/robot/your_robot_left_arm.yml
  - workflows/simbox/curobo/src/curobo/content/configs/robot/your_robot_right_arm.yml
```

--------------------------------

### Rigid Object Directory Organization

Source: https://internrobotics.github.io/InternDataEngine-Docs/config/assets.html

Example directory structure for organizing rigid objects by category to support randomization.

```text
workflows/simbox/example_assets/task/sort_the_rubbish/
├── garbage_can/             # Container category
│   ├── recyclable_can/
│   └── nonrecyclable_can/
├── non_recyclable_garbage/  # Non-recyclable items category
│   ├── obj_1/
│   ├── obj_2/
│   └── ...
└── recyclable_garbage/      # Recyclable items category
    ├── bottle_0/
    ├── bottle_1/
    └── ...
```

--------------------------------

### Configure Pick Skill for ARX Lift-2

Source: https://internrobotics.github.io/InternDataEngine-Docs/concepts/skills/pick

Example configuration for the ARX Lift-2 robot using X-axis for downward approach and Z-axis for forward direction.

```yaml
skills:
  - lift2:
      - left:
          - name: pick
            objects: [pick_object_left]
            filter_z_dir: ["forward", 90]
            filter_x_dir: ["downward", 140]
```

--------------------------------

### Set Isaac Sim Path Environment Variable

Source: https://internrobotics.github.io/InternDataEngine-Docs/guides/installation.html

Configure the ISAAC_SIM_PATH environment variable to point to the local Isaac Sim installation.

```bash
export ISAAC_SIM_PATH=/path/to/isaac-sim
```

--------------------------------

### Configure Dual-Arm Mode

Source: https://internrobotics.github.io/InternDataEngine-Docs/custom/task.html

Defines a dual-arm setup where the left arm operates on obj0 and the right arm operates on obj1 and obj2.

```yaml
skills:
  - new_robot:
      - left:
          - name: skill0
            objects: [obj0]
      - right:
          - name: skill1
            objects: [obj1, obj2]
```

--------------------------------

### __init__(name, robot_file, task, world, constrain_grasp_approach, collision_activation_distance, ignore_substring, use_batch, **kwargs)

Source: https://internrobotics.github.io/InternDataEngine-Docs/concepts/controllers

Initializes the controller, loads the robot configuration, and sets up the CuRobo motion generator.

```APIDOC
## __init__(name, robot_file, task, world, constrain_grasp_approach, collision_activation_distance, ignore_substring, use_batch, **kwargs)

### Description
Initialize the controller, load robot configuration, and set up CuRobo motion generator.

### Parameters
- **name** (str) - Required - Controller name, matching the robot name in task.
- **robot_file** (str) - Required - Path to CuRobo robot YAML configuration file.
- **task** (BaseTask) - Required - The task instance containing robots and scene.
- **world** (World) - Required - Isaac Sim world instance.
- **constrain_grasp_approach** (bool) - Optional - Whether to add grasp approach constraint. Default is False.
- **collision_activation_distance** (float) - Optional - Distance threshold for collision activation. Default is 0.03.
- **ignore_substring** (List[str]) - Optional - Substrings for objects to ignore in collision checking.
- **use_batch** (bool) - Optional - Enable batch planning mode for multiple goals. Default is False.
- **kwargs** (dict) - Optional - Additional keyword arguments.
```

--------------------------------

### Install Python Dependencies via Conda

Source: https://internrobotics.github.io/InternDataEngine-Docs/guides/installation.html

Create and activate a dedicated Conda environment before installing project requirements.

```bash
conda create -n banana python=3.10
conda activate banana
source YOUR_PATH_TO_ISAACSIM/setup_conda_env.sh
cd InternDataEngine
pip install --upgrade pip
pip install -r requirements.txt
```

--------------------------------

### Create Project Symlinks

Source: https://internrobotics.github.io/InternDataEngine-Docs/guides/installation.html

Establish symbolic links for assets, curobo, and panda_drake directories within the simbox workflow.

```bash
cd InternDataEngine/workflows/simbox
ln -s YOUR_PATH_TO_ASSETS assets
ln -s YOUR_PATH_TO_CUROBO curobo
ln -s YOUR_PATH_TO_PANDA_DRAKE panda_drake
```

--------------------------------

### Execute in different modes

Source: https://internrobotics.github.io/InternDataEngine-Docs/guides/quickstart.html

Run the engine in debug or production mode by selecting the appropriate configuration file.

```bash
# Debug mode
/isaac-sim/python.sh launcher.py --config configs/simbox/de_plan_with_render_template.yaml

# Production mode
/isaac-sim/python.sh launcher.py --config configs/simbox/de_pipe_template.yaml
```

--------------------------------

### Run a task with the launcher script

Source: https://internrobotics.github.io/InternDataEngine-Docs/guides/quickstart.html

Execute a data generation task using the primary launcher script and a configuration file.

```bash
/isaac-sim/python.sh launcher.py \
  --config configs/simbox/de_plan_with_render_template.yaml
```

--------------------------------

### Run a task with the wrapper script

Source: https://internrobotics.github.io/InternDataEngine-Docs/guides/quickstart.html

Execute a task using the simplified wrapper script with optional sample count and random seed arguments.

```bash
bash scripts/simbox/simbox_plan_with_render.sh <task_config_path> [num_samples] [random_seed]

# Example
bash scripts/simbox/simbox_plan_with_render.sh \
  workflows/simbox/core/configs/tasks/basic/split_aloha/track_the_targets/track_the_targets.yaml 10
```

--------------------------------

### Execute Simbox planning and generation scripts

Source: https://internrobotics.github.io/InternDataEngine-Docs/custom/task.html

Run these scripts from the command line to initiate task planning or data generation pipelines.

```bash
bash scripts/simbox/simbox_plan_with_render.sh new_task [num_samples] [random_seed]
```

```bash
bash scripts/simbox/simbox_plan_and_render.sh new_task [num_samples] [random_seed]
```

```bash
bash scripts/simbox/simbox_pipe.sh new_task [num_samples] [random_seed]
```

--------------------------------

### Implement Controller Wrapper Methods

Source: https://internrobotics.github.io/InternDataEngine-Docs/concepts/controllers

A collection of methods for configuring the world, initializing motion generation, and updating controller states.

```python
    def _load_world(self, use_default: bool = True) -> None:
        if use_default:
            self.world_cfg = WorldConfig()
        else:
            world_cfg_table = WorldConfig.from_dict(
                load_yaml(join_path(get_world_configs_path(), "collision_table.yml"))
            )
            self._world_cfg_table = world_cfg_table
            self._world_cfg_table.cuboid[0].pose[2] -= 10.5
            world_cfg1 = WorldConfig.from_dict(
                load_yaml(join_path(get_world_configs_path(), "collision_table.yml"))
            ).get_mesh_world()
            world_cfg1.mesh[0].name += "_mesh"
            world_cfg1.mesh[0].pose[2] = -10.5
            self.world_cfg = WorldConfig(cuboid=world_cfg_table.cuboid, mesh=world_cfg1.mesh)

    def _get_motion_gen_collision_cache(self):
        return {"obb": 700, "mesh": 700}

    def _get_grasp_approach_linear_axis(self) -> int:
        return 2

    def _get_sort_path_weights(self) -> Optional[List[float]]:
        return None

    def _init_motion_gen(self) -> None:
        pose_metric = None
        if self.constrain_grasp_approach:
            pose_metric = PoseCostMetric.create_grasp_approach_metric(
                offset_position=0.1,
                linear_axis=self._get_grasp_approach_linear_axis(),
            )
        if self.use_batch:
            self.plan_config = MotionGenPlanConfig(
                enable_graph=True,
                enable_opt=True,
                need_graph_success=True,
                enable_graph_attempt=4,
                max_attempts=4,
                enable_finetune_trajopt=True,
                parallel_finetune=True,
                time_dilation_factor=1.0,
            )
        else:
            self.plan_config = MotionGenPlanConfig(
                enable_graph=False,
                enable_graph_attempt=7,
                max_attempts=10,
                pose_cost_metric=pose_metric,
                enable_finetune_trajopt=True,
                time_dilation_factor=1.0,
            )
        motion_gen_config = MotionGenConfig.load_from_robot_config(
            self.robot_cfg,
            self.world_cfg,
            self.tensor_args,
            interpolation_dt=0.01,
            collision_activation_distance=self.collision_activation_distance,
            trajopt_tsteps=32,
            collision_checker_type=CollisionCheckerType.MESH,
            use_cuda_graph=True,
            self_collision_check=True,
            collision_cache=self._get_motion_gen_collision_cache(),
            num_trajopt_seeds=12,
            num_graph_seeds=12,
            optimize_dt=True,
            trajopt_dt=None,
            trim_steps=None,
            project_pose_to_goal_frame=False,
        )
        ik_config = IKSolverConfig.load_from_robot_config(
            self.robot_cfg,
            self.world_cfg,
            rotation_threshold=0.05,
            position_threshold=0.005,
            num_seeds=20,
            self_collision_check=True,
            self_collision_opt=True,
            tensor_args=self.tensor_args,
            use_cuda_graph=True,
            collision_checker_type=CollisionCheckerType.MESH,
            collision_cache={"obb": 700, "mesh": 700},
        )
        self.ik_solver = IKSolver(ik_config)
        self.motion_gen = MotionGen(motion_gen_config)
        print("warming up..")
        if self.use_batch:
            self.motion_gen.warmup(parallel_finetune=True, batch=CUROBO_BATCH_SIZE)
        else:
            self.motion_gen.warmup(enable_graph=True, warmup_js_trajopt=False)
        self.world_model = self.motion_gen.world_collision
        self.motion_gen.clear_world_cache()
        self.motion_gen.reset(reset_seed=False)
        self.motion_gen.update_world(self.world_cfg)

    def update_pose_cost_metric(self, hold_vec_weight: Optional[List[float]] = None) -> None:
        if hold_vec_weight:
            pose_cost_metric = PoseCostMetric(
                hold_partial_pose=True,
                hold_vec_weight=self.motion_gen.tensor_args.to_device(hold_vec_weight),
            )
        else:
            pose_cost_metric = None
        self.plan_config.pose_cost_metric = pose_cost_metric

    def update(self) -> None:
        obstacles = self.usd_help.get_obstacles_from_stage(
            ignore_substring=self.ignore_substring, reference_prim_path=self.reference_prim_path
        ).get_collision_check_world()
        if self.motion_gen is not None:
            self.motion_gen.update_world(obstacles)
        self.world_cfg = obstacles
```

--------------------------------

### simple_generate_manip_cmds()

Source: https://internrobotics.github.io/InternDataEngine-Docs/concepts/skills/pick

Generates the full pick motion sequence, including updating planning settings, sampling poses, testing feasibility, and building the command list.

```APIDOC
## simple_generate_manip_cmds()

### Description
Generates the full pick motion sequence. This is the core method that defines the pick behavior.

### Steps
1. Update planning settings (reset cost metrics and collision settings).
2. Sample EE poses (filter valid grasp candidates).
3. Generate pre-grasp poses (offset grasp poses along approach direction).
4. Test motion feasibility (check reachability).
5. Build manip_list (construct command sequence: move to pre-grasp, move to grasp, close gripper, attach object, lift object).
```

--------------------------------

### Implement FrankaRobotiq85Controller

Source: https://internrobotics.github.io/InternDataEngine-Docs/api/controllers.html

Controller implementation for the Franka robot with a Robotiq 2F-85 gripper, featuring inverted gripper mapping.

```python
@register_controller
class FrankaRobotiq85Controller(TemplateController):
    def get_gripper_action(self):
        return np.clip(
            -self._gripper_state * self._gripper_joint_position,
            0.0, 5.0
        )
```

--------------------------------

### Clone the InternDataEngine Repository

Source: https://internrobotics.github.io/InternDataEngine-Docs/guides/installation.html

Initial step to download the source code from GitHub and navigate into the project directory.

```bash
git clone https://github.com/InternRobotics/InternDataEngine.git
cd InternDataEngine
```

--------------------------------

### Download Assets via Script

Source: https://internrobotics.github.io/InternDataEngine-Docs/guides/installation.html

Use the provided download script to fetch scene assets, robot models, and task files. Options allow for selective downloading to manage disk space and bandwidth.

```bash
bash scripts/download_assets.sh [OPTIONS]
```

```bash
# Minimum assets + CuRobo + panda_drake
bash scripts/download_assets.sh --min --with-curobo --with-drake

# Full download (~200GB)
bash scripts/download_assets.sh --full --with-curobo --with-drake
```

--------------------------------

### Launch Data Generation Tasks

Source: https://internrobotics.github.io/InternDataEngine-Docs/guides/quickstart.html

Commands to execute specific data generation tasks using the launcher script and task-specific configuration files.

```bash
/isaac-sim/python.sh launcher.py \
  --config configs/simbox/de_plan_with_render_template.yaml \
  --load_stage.scene_loader.args.cfg_path=workflows/simbox/core/configs/tasks/basic/lift2/insert_the_markpen_in_penholder/left/insert_the_markpen_in_penholder_part0.yaml
```

```bash
/isaac-sim/python.sh launcher.py \
  --config configs/simbox/de_plan_with_render_template.yaml \
  --load_stage.scene_loader.args.cfg_path=workflows/simbox/core/configs/tasks/basic/lift2/pour_redwine_left.yaml
```

```bash
/isaac-sim/python.sh launcher.py \
  --config configs/simbox/de_plan_with_render_template.yaml \
  --load_stage.scene_loader.args.cfg_path=workflows/simbox/core/configs/tasks/long_horizon/lift2/dexpnp/sort_parts_0.yaml
```

--------------------------------

### Configure Environment Map

Source: https://internrobotics.github.io/InternDataEngine-Docs/config/yaml.html

Sets up HDR background lighting with options for intensity and rotation randomization.

```yaml
env_map:
  envmap_lib: envmap_lib              # Path to HDR environment maps
  apply_randomization: True           # Enable random lighting
  intensity_range: [4000, 7000]       # Light intensity range
  rotation_range: [0, 180]            # Environment rotation range (degrees)
```

--------------------------------

### Enable Simulation Visualization

Source: https://internrobotics.github.io/InternDataEngine-Docs/guides/quickstart.html

Disable headless mode to display the simulation GUI during execution.

```bash
--load_stage.scene_loader.args.simulator.headless=false
```

--------------------------------

### Define keypoints_config.json

Source: https://internrobotics.github.io/InternDataEngine-Docs/custom/assets

Configuration file defining the USD asset paths, prim paths, and axis orientations for the manipulation skill.

```json
{
    "DIR": "/path/to/your/usd/directory",
    "USD_NAME": "microwave_0.usd",
    "INSTANCE_NAME": "microwave7265",
    "link0_initial_prim_path": "/root/group_18",
    "base_initial_prim_path": "/root/group_0",
    "revolute_joint_initial_prim_path": "/root/group_18/RevoluteJoint",
    "joint_index": 0,
    "LINK0_ROT_AXIS": "y",
    "BASE_FRONT_AXIS": "z",
    "LINK0_CONTACT_AXIS": "-y",
    "SCALED_VOLUME": 0.02
}
```

--------------------------------

### Define basic task metadata

Source: https://internrobotics.github.io/InternDataEngine-Docs/config/yaml.html

Sets up core task identifiers, asset paths, and rendering preferences for a specific task instance.

```yaml
tasks:
  -
    name: banana_base_task      # Task identifier
    asset_root: workflows/simbox/example_assets  # Root path for all assets
    task: BananaBaseTask        # Task class name
    task_id: 0                  # Task instance ID

    offset: null                # Optional position offset
    render: True                # Enable rendering

    neglect_collision_names: ["table"]  # Collision filter names
```

--------------------------------

### Define Robot Configuration

Source: https://internrobotics.github.io/InternDataEngine-Docs/custom/robot.html

Use this YAML template to specify robot class, USD path, kinematics settings, and gripper constraints.

```yaml
target_class: NewRobot                    # Python class name for robot wrapper
path: "YOUR_PATH_TO_ROBOT_USD_PATH (relative to asset root)"                                  # USD file path relative to asset root

# -----------------------------------------------------------------------------
# CuRobo Kinematics Config (one per arm for dual-arm robots)
# -----------------------------------------------------------------------------
robot_file:
  - ""                                    # Left arm CuRobo config path
  - ""                                    # Right arm CuRobo config path (dual-arm only)

# -----------------------------------------------------------------------------
# Gripper Parameters
# -----------------------------------------------------------------------------
gripper_max_width: 0.0                    # Maximum gripper opening (meters)
gripper_min_width: 0.0                    # Minimum gripper closing (meters)
tcp_offset: 0.0                           # Distance from EE origin to TCP (meters, MUST be positive)
```

--------------------------------

### Register New Skill in __init__.py

Source: https://internrobotics.github.io/InternDataEngine-Docs/custom/skill

Add the new skill class to the __all__ list in the skills initialization file to make it available for use.

```python
from core.skills.new_skill import NewSkill

__all__ = [
    # ... existing skills
    "NewSkill",
]
```

--------------------------------

### Object Instance File Structure

Source: https://internrobotics.github.io/InternDataEngine-Docs/config/assets.html

Required files and subdirectories within an individual object instance folder.

```text
workflows/simbox/example_assets/task/sort_the_rubbish/non_recyclable_garbage/obj_2/
├── Aligned_obj.usd              # Object USD with physics properties (mass, collision, etc.)
├── Aligned_grasp_sparse.npy     # Grasp pose annotations (for pickable objects)
└── textures/                    # Texture files
    └── baked_mesh_*.png
```

--------------------------------

### Initialize TemplateController

Source: https://internrobotics.github.io/InternDataEngine-Docs/api/controllers.html

Constructor signature for the base controller class.

```python
TemplateController(cfg, task, robot_file: str, **kwargs)
```

--------------------------------

### Implement Genie1Controller

Source: https://internrobotics.github.io/InternDataEngine-Docs/api/controllers.html

Controller implementation for the Genie1 dual-arm robot with custom path selection weights.

```python
@register_controller
class Genie1Controller(TemplateController):
    def _get_sort_path_weights(self) -> List[float]:
        return [1, 1, 1, 1, 3, 3, 1]
```

--------------------------------

### RigidObject.__init__

Source: https://internrobotics.github.io/InternDataEngine-Docs/concepts/objects.html

Initializes a rigid body object with physics properties for simulation.

```APIDOC
## RigidObject.__init__(asset_root, root_prim_path, cfg, *args, **kwargs)

### Description
Initializes a rigid object in the simulation scene that supports collision detection and mass properties.

### Parameters
- **asset_root** (str) - Required - Root path for asset files.
- **root_prim_path** (str) - Required - Root prim path in USD stage.
- **cfg** (dict) - Required - Configuration dictionary containing:
  - **name** (str) - Required - Object name.
  - **path** (str) - Required - USD file path relative to asset_root.
  - **prim_path_child** (str) - Required - Child prim path for rigid body.
  - **translation** (list) - Optional - Initial translation [x, y, z].
  - **euler** (list) - Optional - Initial euler rotation [rx, ry, rz] in degrees.
  - **scale** (list) - Optional - Scale factor [sx, sy, sz].
  - **mass** (float) - Optional - Object mass.
- **kwargs** - Optional - Additional keyword arguments passed to RigidPrim.
```

--------------------------------

### Plan and Render Mode Lifecycle

Source: https://internrobotics.github.io/InternDataEngine-Docs/concepts/workflows.html

Lifecycle flow for executing planning and rendering stages sequentially.

```text
EnvLoader <-> get_task_name() → init_task() → reset()
EnvRandomizer <-> randomization()
EnvPlanner <-> generate_seq()
EnvRenderer <-> seq_replay(sequence)
EnvWriter <-> save()
```

--------------------------------

### Check feasibility and task completion

Source: https://internrobotics.github.io/InternDataEngine-Docs/concepts/skills/place.html

Methods to verify controller feasibility and determine if the current subtask or the entire placement task is finished.

```python
def is_feasible(self, th=5):
        return self.controller.num_plan_failed <= th

    def is_subtask_done(self, t_eps=1e-3, o_eps=5e-3):
        assert len(self.manip_list) != 0
        p_base_ee_cur, q_base_ee_cur = self.controller.get_ee_pose()
        p_base_ee, q_base_ee, *_ = self.manip_list[0]
        diff_trans = np.linalg.norm(p_base_ee_cur - p_base_ee)
        diff_ori = 2 * np.arccos(min(abs(np.dot(q_base_ee_cur, q_base_ee)), 1.0))
        pose_flag = np.logical_and(diff_trans < t_eps, diff_ori < o_eps)
        self.plan_flag = self.controller.num_last_cmd > 10
        return np.logical_or(pose_flag, self.plan_flag)

    def is_done(self):
        if len(self.manip_list) == 0:
            return True
        if self.is_subtask_done(t_eps=self.skill_cfg.get("t_eps", 1e-3), o_eps=self.skill_cfg.get("o_eps", 5e-3)):
            self.manip_list.pop(0)
        return len(self.manip_list) == 0
```

--------------------------------

### Run Keypoint Selection Script

Source: https://internrobotics.github.io/InternDataEngine-Docs/custom/assets

Opens an Open3D visualization window for interactive selection of contact points on the articulated object.

```bash
python select_keypoint.py --config $CONFIG_PATH
```

--------------------------------

### Registering Workflow Extensions

Source: https://internrobotics.github.io/InternDataEngine-Docs/concepts/workflows.html

Use this function in workflows/__init__.py to dynamically import specific workflow implementations based on the provided type.

```python
def import_extensions(workflow_type):
    if workflow_type == "SimBoxDualWorkFlow":
        import workflows.simbox_dual_workflow
    else:
        raise ValueError(f"Unsupported workflow type: {workflow_type}")
```

--------------------------------

### Verify Symlink Configuration

Source: https://internrobotics.github.io/InternDataEngine-Docs/guides/installation.html

List directory contents to confirm that symlinks for assets, curobo, and panda_drake are correctly established.

```bash
ls -la workflows/simbox/assets
ls -la workflows/simbox/curobo
ls -la workflows/simbox/panda_drake
```

--------------------------------

### Generate Grasp Poses

Source: https://internrobotics.github.io/InternDataEngine-Docs/custom/assets

Runs the grasp pose generation tool for objects intended for robotic manipulation.

```bash
cd workflows/simbox/tools/grasp
# See README.md for detailed usage
python gen_sparse_label.py --obj_path /path/to/Aligned_obj.obj --unit m
```

--------------------------------

### Implement SplitAlohaController

Source: https://internrobotics.github.io/InternDataEngine-Docs/api/controllers.html

Controller implementation for the Agilex Split Aloha dual-arm robot with z-axis grasp approach.

```python
@register_controller
class SplitAlohaController(TemplateController):
    def _get_grasp_approach_linear_axis(self) -> int:
        return 2  # z-axis
```

--------------------------------

### Plan with Render Mode Lifecycle

Source: https://internrobotics.github.io/InternDataEngine-Docs/concepts/workflows.html

Lifecycle flow for executing planning and rendering simultaneously.

```text
EnvLoader <-> get_task_name() → init_task() → reset()
EnvRandomizer <-> randomization()
EnvPlanWithRender <-> generate_seq_with_obs()
EnvWriter <-> save()
```

--------------------------------

### forward(manip_cmd, eps=5e-3)

Source: https://internrobotics.github.io/InternDataEngine-Docs/concepts/controllers

Executes a manipulation command, serving as the main entry point for skills to execute motions.

```APIDOC
## forward(manip_cmd, eps=5e-3)

### Description
Execute a manipulation command. This is the main entry point called by skills to execute motions.

### Parameters
- **manip_cmd** (tuple) - Required - Command tuple (ee_trans, ee_ori, gripper_fn, params).
- **eps** (float) - Optional - Position/orientation threshold to trigger new planning. Default is 5e-3.

### Response
- **dict** - Action dictionary containing joint_positions, joint_indices, lr_name, arm_action, and gripper_action.
```

--------------------------------

### Implement FR3Controller

Source: https://internrobotics.github.io/InternDataEngine-Docs/api/controllers.html

Controller implementation for the Franka FR3 single-arm robot with increased collision cache.

```python
@register_controller
class FR3Controller(TemplateController):
    def _get_motion_gen_collision_cache(self):
        return {"obb": 1000, "mesh": 1000}
```

--------------------------------

### PlaneObject.__init__

Source: https://internrobotics.github.io/InternDataEngine-Docs/concepts/objects.html

Initializes a new plane object within the simulation environment using the provided configuration.

```APIDOC
## __init__(self, asset_root, root_prim_path, cfg, *args, **kwargs)

### Description
Initialize a plane object in the simulation scene. This is used to create floors, backgrounds, and other flat surfaces.

### Parameters
- **asset_root** (str) - Required - Root path for asset files.
- **root_prim_path** (str) - Required - Root prim path in USD stage.
- **cfg** (dict) - Required - Configuration dictionary containing:
  - **name** (str) - Required - Object name.
  - **size** (list) - Required - Plane dimensions [width, height].
  - **translation** (list) - Optional - Position [x, y, z].
  - **euler** (list) - Optional - Rotation in degrees [roll, pitch, yaw].
  - **texture** (dict) - Optional - Texture configuration.
- **kwargs** (dict) - Optional - Additional keyword arguments.
```

--------------------------------

### Implement Lift2Controller

Source: https://internrobotics.github.io/InternDataEngine-Docs/api/controllers.html

Controller implementation for the ARX-Lift2 dual-arm robot with x-axis grasp approach.

```python
@register_controller
class Lift2Controller(TemplateController):
    def _get_grasp_approach_linear_axis(self) -> int:
        return 0  # x-axis
```

--------------------------------

### GeometryObject.__init__

Source: https://internrobotics.github.io/InternDataEngine-Docs/concepts/objects.html

Initializes a static geometry object without physics simulation.

```APIDOC
## GeometryObject.__init__(asset_root, root_prim_path, cfg, *args, **kwargs)

### Description
Initializes a static geometry object for environmental elements like tables or fixtures that do not require physics interaction.

### Parameters
- **asset_root** (str) - Required - Root path for asset files.
- **root_prim_path** (str) - Required - Root prim path in USD stage.
- **cfg** (dict) - Required - Configuration dictionary containing:
  - **name** (str) - Required - Object name.
  - **path** (str) - Required - USD file path relative to asset_root.
  - **prim_path_child** (str) - Optional - Child prim path suffix.
- **kwargs** - Optional - Additional keyword arguments passed to GeometryPrim.
```

--------------------------------

### Run Keypoint Transfer Script

Source: https://internrobotics.github.io/InternDataEngine-Docs/custom/assets

Transfers the selected keypoints to the configured asset.

```bash
python transfer_keypoints.py --config $CONFIG_PATH
```

--------------------------------

### USD Prim Hierarchy

Source: https://internrobotics.github.io/InternDataEngine-Docs/custom/assets

Represents the expected USD file structure for objects with physics properties.

```text
World (defaultprim)
├── Looks
│   └── materials
└── Aligned
    └── mesh
```

--------------------------------

### Define global simulation settings

Source: https://internrobotics.github.io/InternDataEngine-Docs/config/yaml.html

Configures physics, rendering, and display settings for the simulation environment.

```yaml
simulator:
  physics_dt: 1/30                   # Physics update rate
  rendering_dt: 1/30                 # Render update rate
  stage_units_in_meters: 1.0         # Stage unit scale
  headless: True                     # Run without GUI; set to False for visual debugging
  anti_aliasing: 0                   # Anti-aliasing level
```

--------------------------------

### Implement Close Skill with KPAMPlanner

Source: https://internrobotics.github.io/InternDataEngine-Docs/concepts/skills/articulation.html

This class defines the Close skill, inheriting from BaseSkill and utilizing KPAMPlanner to generate manipulation trajectories for closing articulated objects.

```python
# Source: workflows/simbox/core/skills/close.py
from copy import deepcopy

import numpy as np
from core.skills.base_skill import BaseSkill, register_skill
from omegaconf import DictConfig
from omni.isaac.core.controllers import BaseController
from omni.isaac.core.robots.robot import Robot
from omni.isaac.core.tasks import BaseTask
from omni.isaac.core.utils.prims import get_prim_at_path
from omni.isaac.core.utils.transformations import get_relative_transform
from scipy.spatial.transform import Rotation as R
from solver.planner import KPAMPlanner


@register_skill
class Close(BaseSkill):
    def __init__(self, robot: Robot, controller: BaseController, task: BaseTask, cfg: DictConfig, *args, **kwargs):
        super().__init__()
        self.robot = robot
        self.controller = controller
        self.task = task
        self.stage = task.stage
        self.name = cfg["name"]
        art_obj_name = cfg["objects"][0]
        self.skill_cfg = cfg
        self.art_obj = task.objects[art_obj_name]
        self.planner_setting = cfg["planner_setting"]
        self.contact_pose_index = self.planner_setting["contact_pose_index"]
        self.success_threshold = self.planner_setting["success_threshold"]
        self.update_art_joint = self.planner_setting.get("update_art_joint", False)
        if kwargs:
            self.world = kwargs["world"]
            self.draw = kwargs["draw"]
        self.manip_list = []

        if self.skill_cfg.get("obj_info_path", None):
            self.art_obj.update_articulated_info(self.skill_cfg["obj_info_path"])

        lr_arm = "left" if "left" in self.controller.robot_file else "right"
        self.fingers_link_contact_view = task.artcontact_views[robot.name][lr_arm][art_obj_name + "_fingers_link"]
        self.fingers_base_contact_view = task.artcontact_views[robot.name][lr_arm][art_obj_name + "_fingers_base"]
        self.forbid_collision_contact_view = task.artcontact_views[robot.name][lr_arm][
            art_obj_name + "_forbid_collision"
        ]
        self.collision_valid = True
        self.process_valid = True

    def setup_kpam(self):
        self.planner = KPAMPlanner(
            env=self.world,
            robot=self.robot,
            object=self.art_obj,
            cfg_path=self.planner_setting,
            controller=self.controller,
            draw_points=self.draw,
            stage=self.stage,
        )

    def simple_generate_manip_cmds(self):
        if self.skill_cfg.get("obj_info_path", None):
            self.art_obj.update_articulated_info(self.skill_cfg["obj_info_path"])

        self.setup_kpam()
        traj_keyframes, sample_times = self.planner.get_keypose()
        if len(traj_keyframes) == 0 and len(sample_times) == 0:
            print("No keyframes found, return empty manip_list")
            self.manip_list = []
            return
```

--------------------------------

### Configure Place with Retreat

Source: https://internrobotics.github.io/InternDataEngine-Docs/concepts/skills/place.html

Defines a placement operation that includes a post-place movement vector for retreating.

```yaml
skills:
  - split_aloha:
      - left:
          - name: place
            objects: [item, shelf]
            place_direction: "vertical"
            post_place_vector: [-0.05, 0.0, 0.1]
            gripper_change_steps: 15
```

--------------------------------

### Configure Camera Parameters in YAML

Source: https://internrobotics.github.io/InternDataEngine-Docs/config/yaml.html

Defines camera viewpoint, extrinsic parameters, and attachment points within the simulation environment.

```yaml
cameras:
  -
    name: split_aloha_hand_left
    translation: [0.0, 0.08, 0.05]
    orientation: [0.0, 0.0, 0.965, 0.259]
    camera_axes: usd
    camera_file: workflows/simbox/core/configs/cameras/astra.yaml
    parent: "split_aloha/split_aloha_mid_360_with_piper/split_aloha_mid_360_with_piper/fl/link6"
    apply_randomization: False
```

--------------------------------

### Define a Custom Task Configuration

Source: https://internrobotics.github.io/InternDataEngine-Docs/custom/task.html

A complete YAML configuration structure for setting up a new task, including environment maps, robot definitions, object placement, and camera parameters.

```yaml
tasks:
  - name: new_task
    asset_root: workflows/simbox/assets
    task: BananaBaseTask
    task_id: 0
    offset: null
    render: True

    # =========================================================================
    # Arena Configuration
    # =========================================================================
    arena_file: workflows/simbox/core/configs/arenas/example.yaml

    # =========================================================================
    # Environment Map (Lighting)
    # =========================================================================
    env_map:
      envmap_lib: envmap_lib
      apply_randomization: False
      intensity_range: [4000, 7000]
      rotation_range: [0, 180]

    # =========================================================================
    # Robots
    # =========================================================================
    robots:
      - name: "new_robot"
        robot_config_file: workflows/simbox/core/configs/robots/new_robot.yaml
        euler: [0.0, 0.0, 0.0]
        ignore_substring: ["material", "table", "floor"]

    # =========================================================================
    # Objects
    # =========================================================================
    objects:
      - name: obj0
        path: new_task/new_objs/new_obj0/Aligned_obj.usd
        target_class: RigidObject
        dataset: custom
        category: new_object
        prim_path_child: Aligned
        translation: [0.3, 0.0, 0.0]
        euler: [0.0, 0.0, 0.0]
        scale: [1, 1, 1]
        apply_randomization: False

      - name: obj1
        path: new_task/new_objs/new_obj1/Aligned_obj.usd
        target_class: RigidObject
        dataset: custom
        category: new_object
        prim_path_child: Aligned
        translation: [0.0, 0.2, 0.0]
        euler: [0.0, 0.0, 0.0]
        scale: [1, 1, 1]
        apply_randomization: False

      - name: obj2
        path: new_task/new_objs/new_obj2/Aligned_obj.usd
        target_class: RigidObject
        dataset: custom
        category: new_object
        prim_path_child: Aligned
        translation: [0.0, -0.2, 0.0]
        euler: [0.0, 0.0, 0.0]
        scale: [1, 1, 1]
        apply_randomization: False

    # =========================================================================
    # Regions (Object Placement)
    # =========================================================================
    regions:
      - object: obj0
        target: table
        random_type: A_on_B_region_sampler
        random_config:
          pos_range: [[0.2, -0.15, 0.0], [0.4, 0.15, 0.0]]
          yaw_rotation: [-30.0, 30.0]

      - object: obj1
        target: table
        random_type: A_on_B_region_sampler
        random_config:
          pos_range: [[-0.4, -0.15, 0.0], [-0.2, 0.15, 0.0]]
          yaw_rotation: [-180.0, 180.0]

      - object: obj2
        target: table
        random_type: A_on_B_region_sampler
        random_config:
          pos_range: [[-0.4, -0.15, 0.0], [-0.2, 0.15, 0.0]]
          yaw_rotation: [-180.0, 180.0]

    # =========================================================================
    # Cameras
    # =========================================================================
    cameras:
      - name: head_camera
        translation: [0.5, 0.0, 0.8]
        orientation: [0.924, 0.383, 0.0, 0.0]
        camera_axes: usd
        camera_file: workflows/simbox/core/configs/cameras/realsense_d455.yaml
        parent: ""
        apply_randomization: False

      - name: wrist_camera_left
        translation: [0.0, 0.05, 0.03]
        orientation: [0.0, 0.0, 0.0, 1.0]
        camera_axes: usd
        camera_file: workflows/simbox/core/configs/cameras/realsense_d435.yaml
        parent: "new_robot/path/to/fl_ee_link"
        apply_randomization: False

      - name: wrist_camera_right
        translation: [0.0, -0.05, 0.03]
        orientation: [0.0, 0.0, 0.0, 1.0]
        camera_axes: usd
        camera_file: workflows/simbox/core/configs/cameras/realsense_d435.yaml
        parent: "new_robot/path/to/fr_ee_link"
        apply_randomization: False

    # =========================================================================
    # Data Settings
    # =========================================================================
    data:
      task_dir: "new_task_demo"
      language_instruction: "New task."
      detailed_language_instruction: "New task."
      collect_info: "Custom task with new robot and skill"
      version: "v1.0"
      update: True
      max_episode_length: 500
```

--------------------------------

### Implement Environment Map Randomization

Source: https://internrobotics.github.io/InternDataEngine-Docs/config/dr.html

Python method to apply HDR dome light randomization based on the provided configuration.

```python
# Source: workflows/simbox/core/tasks/banana.py
def _set_envmap(self):
    """Randomize or reset the environment map (HDR dome light)."""
    cfg = self.cfg["env_map"]
    if cfg.get("light_type", "DomeLight") == "DomeLight":
        envmap_hdr_path_list = glob.glob(os.path.join(self.asset_root, cfg["envmap_lib"], "*.hdr"))
        envmap_hdr_path_list.sort()
        if cfg.get("apply_randomization", False):
            envmap_id = random.randint(0, len(envmap_hdr_path_list) - 1)
            intensity = random.uniform(cfg["intensity_range"][0], cfg["intensity_range"][1])
            rotation = [random.uniform(cfg["rotation_range"][0], cfg["rotation_range"][1]) for _ in range(3)]
        else:
            envmap_id = 0
            intensity = 1000.0
            rotation = [0.0, 0.0, 0.0]
        dome_prim_path = f"{self.root_prim_path}/DomeLight"
        envmap_hdr_path = envmap_hdr_path_list[envmap_id]

        if not is_prim_path_valid(dome_prim_path):
            self.dome_light_prim = UsdLux.DomeLight.Define(self.stage, dome_prim_path)
            UsdGeom.Xformable(self.dome_light_prim).AddRotateXYZOp().Set((rotation[0], rotation[1], rotation[2]))
        else:
            self.dome_light_prim.GetOrderedXformOps()[0].Set((rotation[0], rotation[1], rotation[2]))
        self.dome_light_prim.GetIntensityAttr().Set(intensity)
        self.dome_light_prim.GetTextureFileAttr().Set(envmap_hdr_path)
```

--------------------------------

### Keypoint Annotation Directory Structure

Source: https://internrobotics.github.io/InternDataEngine-Docs/config/assets.html

Displays the organization of action-specific keypoint folders within the Kps directory.

```text
Kps/
├── close_h/         # Close horizontally (e.g., laptop, pot, electric cooker)
├── close_v/         # Close vertically (e.g., microwave)
├── open_h/          # Open horizontally
├── open_v/          # Open vertically
├── pull/            # Pull (e.g., drawer)
├── push/            # Push
└── ...
```
Tokens:
Raw