### Start Basic Training

Source: https://github.com/tensorauto/opentau/blob/main/docs/source/tutorials/training.md

Use this command to start training with default accelerate configuration.

```bash
opentau-train --config_path=examples/pi05_config.json
```

--------------------------------

### Install Documentation Dependencies

Source: https://github.com/tensorauto/opentau/blob/main/docs/README.md

Installs project dependencies including those required for documentation generation using uv.

```bash
uv sync --extra doc
```

--------------------------------

### ROS 2 Client Example

Source: https://context7.com/tensorauto/opentau/llms.txt

Command-line example for launching the ROS 2 client to interact with the inference server.

```APIDOC
## ROS 2 Client

### Description
Launches the ROS 2 client to connect to the inference server and process commands.

### Usage
```bash
python src/opentau/scripts/grpc/client.py \
    --server_address 192.168.1.100:50051 \
    --prompt "pick up the red block" \
    --control_frequency 30.0 \
    --num_cameras 2 \
    --timeout 30.0
```

### Parameters
- **--server_address** (str) - The address of the inference server.
- **--prompt** (str) - The prompt for the action.
- **--control_frequency** (float) - The control frequency in Hz.
- **--num_cameras** (int) - The number of cameras to use.
- **--timeout** (float) - The timeout for the request in seconds.
```

--------------------------------

### Install OpenTau with URDF Support from Source

Source: https://github.com/tensorauto/opentau/blob/main/docs/source/tutorials/visualization.md

Install OpenTau with URDF support by cloning the repository and using uv for dependency management. Activate the virtual environment after installation.

```bash
git clone https://github.com/TensorAuto/OpenTau.git
cd OpenTau
uv sync --extra urdf
source .venv/bin/activate
```

--------------------------------

### Install All Project Dependencies with uv

Source: https://github.com/tensorauto/opentau/blob/main/docs/source/contributing.md

Install the project with all its dependencies, including optional extras. Run this command again if 'pyproject.toml' or 'uv.lock' are updated.

```bash
uv sync --all-extras
```

--------------------------------

### Install OpenTau

Source: https://context7.com/tensorauto/opentau/llms.txt

Install OpenTau from PyPI or build from source. Optional extras include development tools, LIBERO benchmark, and TensorRT. Docker images are also available. Configure distributed training using 'accelerate config'.

```bash
pip install opentau
```

```bash
pip install opentau[dev,libero,trt]
```

```bash
git clone https://github.com/TensorAuto/OpenTau.git
cd OpenTau
uv sync                          # base
uv sync --extra dev --extra libero  # with extras
source .venv/bin/activate
```

```bash
docker build -t opentau .
docker run -it --gpus all opentau /bin/bash
```

```bash
accelerate config
```

--------------------------------

### Start Training and Save Checkpoints

Source: https://github.com/tensorauto/opentau/blob/main/docs/source/tutorials/training.md

Initiate training with specified output directory, total steps, and saving frequency.

```bash
opentau-train --config_path=examples/pi05_config.json --output_dir=outputs/train/pi05 --steps 40 --log_freq 5 --save_freq 20
```

--------------------------------

### Start Training with Specific Accelerate Config

Source: https://github.com/tensorauto/opentau/blob/main/docs/source/tutorials/training.md

Specify a custom accelerate configuration file for training.

```bash
opentau-train --accelerate-config=configs/examples/accelerate_ddp_config.yaml --config_path=configs/examples/pi05_config.json
```

--------------------------------

### Launch Initial Training Run

Source: https://github.com/tensorauto/opentau/blob/main/notebooks/pi05_training.ipynb

Starts an initial training run using the Accelerate library. Specify the configuration path, output directory, and the number of steps.

```bash
! accelerate launch ../src/opentau/scripts/train.py --config_path=../configs/examples/pi05_training_config.json --output_dir=../outputs/train/pi05 --steps 40
```

--------------------------------

### Install OpenTau with PyPI

Source: https://github.com/tensorauto/opentau/blob/main/docs/source/installation.md

Install the OpenTau package directly from PyPI using pip. This is the simplest method for most users.

```bash
pip install opentau
```

--------------------------------

### Server Configuration Example

Source: https://github.com/tensorauto/opentau/blob/main/docs/source/tutorials/deployment.md

Example configuration file for the gRPC server, specifying port, worker threads, and message size limits. This configuration can be provided via a JSON file or command-line arguments.

```javascript
{
    "policy": {
        "type": "pi05",
        "pretrained_path": "outputs/train/pi05/checkpoints/000040",
        ...
    },
    "server": {
        "port": 50051,
        "max_workers": 4,
        "max_send_message_length_mb": 100,
        "max_receive_message_length_mb": 100
    },
    "resolution": [224, 224],
    "num_cams": 2,
    "max_state_dim": 32,
    "max_action_dim": 32,
    ...
}
```

--------------------------------

### Start Training with DeepSpeed ZeRO-2

Source: https://context7.com/tensorauto/opentau/llms.txt

Initiates training using DeepSpeed ZeRO-2 optimization. Requires specifying accelerate and training configuration files.

```bash
opentau-train \
    --accelerate-config=configs/examples/accelerate_deepspeed_config.yaml \
    --config_path=configs/examples/pi05_training_config.json
```

--------------------------------

### Install OpenTau with Extra Dependencies

Source: https://github.com/tensorauto/opentau/blob/main/docs/source/installation.md

Install OpenTau along with optional extra dependencies like development tools, TensorRT, or Libero. Use brackets to specify the desired extras.

```bash
pip install opentau[dev,libero,trt]
```

--------------------------------

### Install Specific Dependencies with uv

Source: https://github.com/tensorauto/opentau/blob/main/docs/source/installation.md

Install specific extra dependencies for OpenTau using uv. Specify the desired extras using the --extra flag.

```bash
uv sync --extra dev --extra libero
```

--------------------------------

### Deploy Policy as a gRPC Inference Server

Source: https://context7.com/tensorauto/opentau/llms.txt

Start a gRPC inference server for real-robot control. This requires a GPU and a deployment configuration file.

```bash
# Start the gRPC inference server (requires GPU)
python src/opentau/scripts/grpc/server.py \
    --config_path=configs/deployment_config.json \
    --server.port=50051 \
    --server.max_workers=8
```

--------------------------------

### Install Development Dependencies with uv

Source: https://github.com/tensorauto/opentau/blob/main/docs/source/contributing.md

Use 'uv' to install project dependencies, including the 'dev' extras. This tool helps manage dependencies and virtual environments.

```bash
uv sync --extra dev
```

--------------------------------

### Install Dependencies with uv

Source: https://github.com/tensorauto/opentau/blob/main/docs/source/installation.md

Use uv to synchronize all required Python dependencies for OpenTau. This command installs all necessary packages to run the project.

```bash
uv sync
```

--------------------------------

### Training with DeepSpeed ZeRO-2

Source: https://context7.com/tensorauto/opentau/llms.txt

Example of how to initiate training using DeepSpeed ZeRO-2 optimization, specifying accelerate configuration and training configuration paths.

```APIDOC
## Training with DeepSpeed ZeRO-2

### Description
Initiate training using DeepSpeed ZeRO-2 optimization, specifying accelerate configuration and training configuration paths.

### Command
```bash
opentau-train \
    --accelerate-config=configs/examples/accelerate_deepspeed_config.yaml \
    --config_path=configs/examples/pi05_training_config.json
```
```

--------------------------------

### Start OpenTau gRPC Server

Source: https://github.com/tensorauto/opentau/blob/main/docs/source/tutorials/deployment.md

Launch the OpenTau gRPC server using a Python script and specify a configuration file path.

```bash
python src/opentau/scripts/grpc/server.py --config_path=deployment_config.json
```

--------------------------------

### Interpreting profile_step.py Output

Source: https://github.com/tensorauto/opentau/blob/main/docs/source/tutorials/benchmarking.md

This is an example output from the profile_step.py script, showing a per-step timing breakdown across different phases. It helps in identifying performance bottlenecks.

```text
=========== profile_step results (rank 0) ===========
warmup=20 measured=200 ranks=8
batch_size=12 num_workers=16 prefetch_factor=8
wall-clock over full loop: 265.89s

phase            stats                                                           share
------------------------------------------------------------------------------------------
dataload_wait    mean=   1.24ms  median=   1.18ms  p95=   1.60ms                 0.1%
forward          mean= 378.86ms  median= 378.65ms  p95= 381.75ms                32.8%
bwd              mean= 706.32ms  median= 704.74ms  p95= 710.24ms                61.1%
unscale_clip     mean=  19.55ms  median=  19.43ms  p95=  21.12ms                 1.7%
optim_step       mean=  39.22ms  median=  39.17ms  p95=  39.81ms                 3.4%
zero_grad_sched  mean=   1.79ms  median=   1.74ms  p95=   2.74ms                 0.2%
backward_step    mean= 766.89ms  median= 765.58ms  p95= 770.31ms                66.3%
sync_gather      mean=   9.64ms  median=   6.47ms  p95=   9.78ms                 0.8%
total            mean=1156.63ms  median=1151.73ms  p95=1158.28ms                <-- total

throughput: 0.86 steps/s, 83.0 global samples/s
=====================================================
```

--------------------------------

### Install Pre-commit Hooks

Source: https://github.com/tensorauto/opentau/blob/main/docs/source/contributing.md

Install 'pre-commit' hooks to automatically run code formatting and linting checks as Git commit hooks. This helps maintain code consistency.

```bash
pre-commit install
```

--------------------------------

### Install OpenTau with URDF Support

Source: https://github.com/tensorauto/opentau/blob/main/docs/source/tutorials/visualization.md

Install the 'opentau' package with optional URDF support using pip. This is required for visualizing datasets with URDF models.

```bash
pip install "opentau[urdf]"
```

--------------------------------

### Define Dataset Mixture Configuration

Source: https://github.com/tensorauto/opentau/blob/main/docs/source/tutorials/datasets.md

Configure a dataset mixture using the 'dataset_mixture' key in your configuration file. This example shows how to specify datasets by 'repo_id' and their sampling weights.

```javascript
{
    "dataset_mixture": {
        "datasets": [
            {
                "repo_id": "physical-intelligence/libero"
            },
            {
                "repo_id": "lerobot/droid_100"
            }
        ],
        "weights": [
            0.3,
            0.7
        ],
        "action_freq": 30.0,
    },
    ...
}
```

--------------------------------

### Run gRPC Server

Source: https://github.com/tensorauto/opentau/blob/main/docs/source/tutorials/deployment.md

Command to start the OpenTau gRPC inference server. The configuration path is required, and server settings can be overridden via command-line arguments.

```bash
python src/opentau/scripts/grpc/server.py --config_path=/path/to/config.json
```

```bash
python src/opentau/scripts/grpc/server.py \
    --config_path=/path/to/config.json \
    --server.port=50051 \
    --server.max_workers=8
```

--------------------------------

### TrainPipelineConfig Example

Source: https://context7.com/tensorauto/opentau/llms.txt

Central configuration dataclass for all training, evaluation, and deployment workflows. This JSON defines dataset mixtures, policy parameters, training steps, and optimizer settings.

```json
{
    "dataset_mixture": {
        "datasets": [
            {
                "repo_id": "physical-intelligence/libero",
                "episodes": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
            }
        ],
        "weights": [1.0],
        "action_freq": 10.0,
        "image_resample_strategy": "nearest",
        "vector_resample_strategy": "nearest"
    },
    "policy": {
        "type": "pi05",
        "pretrained_path": "TensorAuto/pi05_base",
        "n_obs_steps": 1,
        "normalization_mapping": {
            "VISUAL": "IDENTITY",
            "STATE": "MIN_MAX",
            "ACTION": "MEAN_STD"
        },
        "chunk_size": 10,
        "predict_response": true,
        "n_action_steps": 10,
        "max_state_dim": 32,
        "max_action_dim": 32,
        "train_expert_only": true,
        "freeze_vision_encoder": true,
        "discrete_action_max_length": 100
    },
    "resume": false,
    "seed": 1000,
    "resolution": [224, 224],
    "num_cams": 2,
    "max_state_dim": 32,
    "max_action_dim": 32,
    "action_chunk": 10,
    "loss_weighting": {"MSE": 1, "CE": 1},
    "num_workers": 4,
    "batch_size": 2,
    "gradient_accumulation_steps": 1,
    "steps": 40000,
    "log_freq": 200,
    "save_checkpoint": true,
    "save_freq": 20000,
    "val_freq": 1000,
    "use_policy_training_preset": false,
    "optimizer": {
        "type": "adamw",
        "lr": 2.5e-05,
        "weight_decay": 1e-10,
        "grad_clip_norm": 10.0,
        "betas": [0.9, 0.95],
        "eps": 1e-08
    },
    "scheduler": {
        "type": "cosine_decay_with_warmup",
        "num_warmup_steps": 1000,
        "num_decay_steps": 30000,
        "peak_lr": 2.5e-05,
        "decay_lr": 2.5e-06
    },
    "wandb": {
        "enable": true,
        "project": "pi05",
        "notes": "PI05 training run"
    }
}
```

--------------------------------

### Configure Accelerate for Distributed Training

Source: https://github.com/tensorauto/opentau/blob/main/docs/source/installation.md

Configure the accelerate library for your distributed training setup. This command initiates an interactive configuration process.

```bash
accelerate config
```

--------------------------------

### Train with Explicit Output Directory and Checkpoint Frequency

Source: https://context7.com/tensorauto/opentau/llms.txt

Starts training with a specified output directory and frequency for saving checkpoints. Checkpoints include model weights and training configuration.

```bash
# Train with explicit output dir and checkpoint frequency
opentau-train \
    --config_path=configs/examples/pi05_training_config.json \
    --output_dir=outputs/train/pi05 \
    --steps=40000 \
    --save_freq=20000
```

--------------------------------

### Add a Package with uv

Source: https://github.com/tensorauto/opentau/blob/main/docs/source/contributing.md

Add a new package to your project's dependencies using 'uv'. This is the equivalent of 'pip install some-package'.

```bash
uv add some-package
```

--------------------------------

### Profile Training Step Breakdown

Source: https://context7.com/tensorauto/opentau/llms.txt

Use this script to get a per-step wall-clock time breakdown (forward, backward, optimizer, sync). It requires an accelerate configuration and a training pipeline configuration.

```bash
# 1. profile_step.py — per-step wall-clock breakdown (forward/backward/optimizer/sync)
accelerate launch \
    --config_file configs/examples/accelerate_ddp_config.yaml \
    src/opentau/scripts/profile_step.py \
    --config_path=configs/libero/reproduce_pi05_libero.json \
    --batch_size=12
# Output: mean/median/p95 for each phase; throughput in steps/s and samples/s
# Example: forward=32.8%, bwd=61.1%, optim_step=3.4%, sync=0.8%
```

--------------------------------

### Configure RoboCasa Policy Server Host and Port

Source: https://github.com/tensorauto/opentau/blob/main/docs/source/tutorials/robocasa.md

Example of running the OpenTau policy server with specific host and port configurations. These flags are parsed before standard OpenTau flags.

```bash
python -m opentau.scripts.robocasa.server \
    --robocasa_host 0.0.0.0 \
    --robocasa_port 8765 \
    --robocasa_action_dim 16 \
    --config_path /path/to/train_config.json
```

--------------------------------

### Run RoboCasa Policy Server

Source: https://github.com/tensorauto/opentau/blob/main/docs/source/tutorials/robocasa.md

Launches the OpenTau WebSocket policy server for RoboCasa. Ensure OpenTau is installed and a training config is available. The server uses MessagePack for communication and requires OpenCV for image decoding.

```bash
python -m opentau.scripts.robocasa.server \
    --config_path /path/to/train_config.json
```

--------------------------------

### ROS to LeRobot Conversion Config Example

Source: https://github.com/tensorauto/opentau/blob/main/docs/source/tutorials/ros_conversion.md

This JSON structure defines the configuration for converting ROS bag files to the LeRobot dataset format. Specify input/output paths, desired frame rate, joint order, and details for each dataset feature including its ROS topic, attribute, enum values, data type, and shape.

```json
{
"input_path": <path to ros bag folder>,
"output_path": <path to lerobot dataset output folder>,
"fps": <frames per second for the output dataset. The input dataset can be at high frequency like 1000 fps, but the output dataset is at 30 fps>,
"joint_order": <list of joint names in the order of the state vector>,
"dataset_features": {
    <feature_name in lerobot dataset>: {
            "ros_topic": <ros topic name stored in rosbag>,
            "topic_attribute":<attribute name stored in ros topic>,
            "enum_values": <enum vales stored in ros topic. Used to parse the field and apply appropriate preprocessing>,
            "dtype":<dtype of the feature in lerobot dataset>,
            "shape":<shape of the feature in lerobot dataset>
        },
    }
}
```

--------------------------------

### Annotations JSON Schema Example

Source: https://github.com/tensorauto/opentau/blob/main/docs/source/tutorials/attach_metadata.md

This JSON structure defines episode annotations, including quality and segment details with start frames, subtasks, and success status. Ensure episode IDs match the dataset and segment starts are strictly increasing integers.

```json
[
  {
    "episode_id": 0,
    "quality": 3,
    "segments": [
      {"start": 0,  "subtask": "approach the cup", "success": false},
      {"start": 50, "subtask": "pick up the cup",  "success": true}
    ]
  },
  {
    "episode_id": 7,
    "quality": 5,
    "segments": [
      {"start": 0, "subtask": "place the block", "success": true}
    ]
  }
]
```

--------------------------------

### Build HTML Documentation with Makefile

Source: https://github.com/tensorauto/opentau/blob/main/docs/README.md

Builds the HTML documentation from the 'docs' directory on Linux/macOS. Requires activating a virtual environment first.

```bash
source ../.venv/bin/activate
make html
```

--------------------------------

### Clone and Set Up Local Repository

Source: https://github.com/tensorauto/opentau/blob/main/docs/source/contributing.md

Clone your forked repository and add the base repository as a remote. Ensure your public SSH key is uploaded to GitHub.

```bash
git clone git@github.com:<your Github handle>/OpenTau.git
cd OpenTau
git remote add upstream https://github.com/TensorAuto/OpenTau.git
```

--------------------------------

### Build HTML Documentation with sphinx-build

Source: https://github.com/tensorauto/opentau/blob/main/docs/README.md

Builds the HTML documentation universally using sphinx-build from the 'docs' directory. Requires activating a virtual environment first.

```bash
source ../.venv/bin/activate
sphinx-build -b html source _build/html
```

--------------------------------

### Get Hugging Face Token

Source: https://github.com/tensorauto/opentau/blob/main/notebooks/pi05_evaluation_only.ipynb

Retrieves the Hugging Face token from environment variables.

```python
HF_TOKEN = os.environ.get("HF_TOKEN")
```

--------------------------------

### Open Built Documentation

Source: https://github.com/tensorauto/opentau/blob/main/docs/README.md

Opens the generated HTML documentation index file in the default web browser on Linux systems.

```bash
xdg-open docs/_build/html/index.html
```

--------------------------------

### StreamActionChunks RPC Example

Source: https://github.com/tensorauto/opentau/blob/main/docs/source/tutorials/deployment.md

Implement a generator function to stream observations to the server and process the incoming action chunks.

```python
def observation_stream():
    while True:
        yield create_observation_request(...)

for response in stub.StreamActionChunks(observation_stream()):
    process_action_chunk(response.action_chunk)
```

--------------------------------

### Run Attach Metadata Script (Production with OpenAI)

Source: https://github.com/tensorauto/opentau/blob/main/docs/source/tutorials/attach_metadata.md

Run the attach_metadata script for production, enabling OpenAI calls for memory summarization. Set your OPENAI_API_KEY in a .env file or as an environment variable. Adjust the model and delay as needed.

```bash
# .env
OPENAI_API_KEY=sk-...

python -m opentau.scripts.attach_metadata \
    --root /path/to/source/dataset \
    --annotations /path/to/annotations.json \
    --copy-to   /path/to/output/dataset \
    --model gpt-4o-mini \
    --delay 0.1
```

--------------------------------

### Run Async RoboCasa Client

Source: https://github.com/tensorauto/opentau/blob/main/docs/source/tutorials/robocasa.md

Launches the asynchronous RoboCasa client. Replace ENV_NAME with a registered RoboCasa kitchen task. Use --num-rollouts for total episodes and --num-parallel for parallel environment threads.

```bash
python -m robocasa.scripts.client_async ENV_NAME \
    --host localhost \
    --port 8765
```

--------------------------------

### Run SFT Training Command

Source: https://github.com/tensorauto/opentau/blob/main/docs/source/tutorials/RECAP.md

This command initiates the SFT training process. Replace placeholders with the actual paths to your accelerate configuration and JSON config file.

```bash
opentau-train --accelerate-config=<path/to/accelerate_config.yaml> --config_path=<path/to/config.json>
```

--------------------------------

### Launch Training with Accelerate

Source: https://github.com/tensorauto/opentau/blob/main/docs/source/tutorials/training.md

This command is equivalent to the opentau-train convenience wrapper, invoking accelerate launch directly.

```bash
accelerate launch --config_file configs/examples/accelerate_ddp_config.yaml src/opentau/scripts/train.py --config_path=configs/examples/pi05_config.json
```

--------------------------------

### Clone OpenTau Repository

Source: https://github.com/tensorauto/opentau/blob/main/docs/source/installation.md

Download the OpenTau source code from GitHub using git clone. Navigate into the cloned directory to proceed with the installation.

```bash
git clone https://github.com/TensorAuto/OpenTau.git
cd OpenTau
```

--------------------------------

### Convert Single Video (Exo Mode)

Source: https://github.com/tensorauto/opentau/blob/main/docs/source/tutorials/human_demo.md

Use this command to convert a single third-person demonstration video. Ensure the output path does not already exist.

```bash
python -m opentau.scripts.human_video_to_lerobot \
    /path/to/demo.mp4 \
    ./datasets/my_exo_dataset \
    --prompt "Pick up the red block"
```

--------------------------------

### Visualize Episode with URDF Model

Source: https://github.com/tensorauto/opentau/blob/main/docs/source/tutorials/visualization.md

Visualize a dataset episode using a specified URDF model file. Ensure 'opentau' is installed with URDF support.

```bash
opentau-dataset-viz --repo-id lerobot/droid_100 --episode-index 0 --urdf </path/to/your/model.urdf>
```

--------------------------------

### Convert Video with Landmark Overlay

Source: https://github.com/tensorauto/opentau/blob/main/docs/source/tutorials/human_demo.md

Generate a landmark-overlay video for visual inspection during the conversion process. Provide a path for the output overlay video.

```bash
python -m opentau.scripts.human_video_to_lerobot \
    /path/to/demo.mp4 \
    ./datasets/my_dataset \
    --prompt "Pick up the cup" \
    --overlay /path/to/overlay.mp4
```

--------------------------------

### Dataset Card Metadata

Source: https://github.com/tensorauto/opentau/blob/main/src/opentau/datasets/card_template.md

This section defines the metadata for the dataset card, including links to specifications and guides. It uses Jinja templating for dynamic content.

```markdown
# For reference on dataset card metadata, see the spec: https://github.com/huggingface/hub-docs/blob/main/datasetcard.md?plain=1
# Doc / guide: https://huggingface.co/docs/hub/datasets-cards
{{ card_data }}
```

--------------------------------

### Load and Create LeRobot Datasets

Source: https://context7.com/tensorauto/opentau/llms.txt

Load existing datasets from HuggingFace Hub or local directories, or create new datasets for recording demonstrations. Specify features, shapes, and data types when creating new datasets.

```python
from opentau.datasets.lerobot_dataset import LeRobotDataset
from opentau.configs.train import TrainPipelineConfig
from torch.utils.data import DataLoader

# Load an existing dataset from HuggingFace Hub
dataset = LeRobotDataset(cfg, repo_id="physical-intelligence/libero")
print(f"Episodes: {dataset.num_episodes}, Frames: {len(dataset)}")

dataloader = DataLoader(dataset, batch_size=8, num_workers=4)
sample = dataset[0]
# sample["observation.images.camera0"]:
# sample["observation.state"]:
# sample["action"]:
# sample["task"]:

# Load a subset of episodes
dataset = LeRobotDataset(cfg, repo_id="physical-intelligence/libero", episodes=[0, 1, 2, 3])

# Load from a local root directory (no Hub download)
dataset = LeRobotDataset(
    cfg,
    repo_id="my-org/my-dataset",
    root="~/.cache/huggingface/lerobot/my-org/my-dataset"
)

# Create a new dataset for recording human demonstrations
new_dataset = LeRobotDataset.create(
    repo_id="my-org/new-robot-dataset",
    fps=30,
    features={
        "observation.state": {"shape": (7,), "dtype": "float32", "names": None},
        "action": {"shape": (7,), "dtype": "float32", "names": None},
        "observation.images.camera0": {"shape": (3, 224, 224), "dtype": "video"},
    },
    use_videos=True,
)
```

--------------------------------

### Visualize Robot Demonstration Episodes with Rerun

Source: https://context7.com/tensorauto/opentau/llms.txt

Interactively visualize robot demonstration episodes using Rerun. Supports visualization from HuggingFace Hub datasets or local directories. Camera logging modes can be adjusted for file size and format.

```bash
opentau-dataset-viz --repo-id lerobot/droid_100 --episode-index 0
```

```bash
opentau-dataset-viz \
    --root ~/.cache/huggingface/lerobot/lerobot/droid_100 \
    --episode-index 0
```

```bash
opentau-dataset-viz \
    --repo-id lerobot/droid_100 \
    --episode-index 0 \
    --camera-log-mode asset_video
```

```bash
opentau-dataset-viz \
    --repo-id lerobot/droid_100 \
    --episode-index 0 \
    --save 1 \
    --output-dir ./rrd_output \
    --camera-log-mode asset_video
```

--------------------------------

### Resume Training Run

Source: https://github.com/tensorauto/opentau/blob/main/notebooks/pi05_training.ipynb

Resumes a previous training run using the Accelerate library. Specify the path to the saved training configuration, the output directory, and set resume to true. The steps parameter indicates the total steps to reach.

```bash
! accelerate launch ../src/opentau/scripts/train.py --config_path=../outputs/train/pi05/checkpoints/000040/train_config.json --output_dir=../outputs/train/pi05 --resume=true --steps 100
```

--------------------------------

### Profile Training Step with Accelerate

Source: https://github.com/tensorauto/opentau/blob/main/README.md

Use this script to profile the wall-clock time spent in each phase of a training step (forward, backward, optimizer, sync). Ensure you have accelerate configured and provide the path to your training configuration.

```bash
accelerate launch \
    --config_file configs/examples/accelerate_ddp_config.yaml \
    src/opentau/scripts/profile_step.py \
    --config_path=<your_training_config.json>
```

--------------------------------

### Standalone Python gRPC Client for OpenTau

Source: https://context7.com/tensorauto/opentau/llms.txt

Connect to the inference server, check its health, and get action chunks from observations. Ensure the server is running and accessible at the specified address.

```python
# Standalone Python gRPC client (no ROS 2 required)
from opentau.scripts.grpc.client import PolicyClient, ClientConfig
import numpy as np

config = ClientConfig(
    server_address="192.168.1.100:50051",
    timeout_seconds=5.0,
    max_retries=3,
    image_encoding="jpeg",   # "jpeg" | "png" | "raw"
    jpeg_quality=85,
)
client = PolicyClient(config)
assert client.connect(), "Failed to connect to inference server"

# Check server health
health = client.health_check()
# {"healthy": True, "model_name": "pi05", "device": "cuda:0",
#  "gpu_memory_used_gb": 12.3, "gpu_memory_total_gb": 80.0}

# Get action chunk from observations
images = [np.random.randint(0, 255, (224, 224, 3), dtype=np.uint8)] * 2
state  = np.zeros(32, dtype=np.float32)  # robot proprioceptive state

action_chunk, inference_ms = client.get_action_chunk(
    images=images,
    state=state,
    prompt="pick up the red block",
)
# action_chunk: np.ndarray of shape (chunk_length, action_dim)
print(f"Action chunk shape: {action_chunk.shape}, latency: {inference_ms:.1f}ms")

client.disconnect()
```

--------------------------------

### Calculate and Visualize Value Function Predictions

Source: https://context7.com/tensorauto/opentau/llms.txt

Use these commands to calculate value function predictions on rollout episodes and then visualize them. Ensure the configuration and dataset paths are correctly specified.

```bash
python -m opentau.scripts.calculate_value \
    --config_path=configs/recap/stage2_value.json \
    --dataset_mixture=configs/examples/value_config.json \
    --batch_size=20 \
    --output_file=outputs/values.json
```

```bash
python -m opentau.scripts.value_visualizer_app \
    --dataset-config=configs/examples/value_config.json \
    --values=outputs/values.json
```

--------------------------------

### OpenTau gRPC Streaming API for Real-time Control

Source: https://context7.com/tensorauto/opentau/llms.txt

Utilize the continuous streaming RPC for low-latency robot control. This example demonstrates sending observation requests and processing action chunks in a loop.

```python
import grpc
from opentau.scripts.grpc import robot_inference_pb2, robot_inference_pb2_grpc
import time, numpy as np
from PIL import Image
import io

channel = grpc.insecure_channel("192.168.1.100:50051")
stub = robot_inference_pb2_grpc.RobotPolicyServiceStub(channel)

def observation_stream(num_requests=100):
    """Generator that yields ObservationRequest messages."""
    for i in range(num_requests):
        request = robot_inference_pb2.ObservationRequest()
        request.request_id = f"req_{i}"
        request.timestamp_ns = time.time_ns()
        request.prompt = "wipe the table"

        # Encode and attach camera images
        for _ in range(2):
            fake_img = np.random.randint(0, 255, (224, 224, 3), dtype=np.uint8)
            buf = io.BytesIO()
            Image.fromarray(fake_img).save(buf, format="JPEG", quality=85)
            cam = robot_inference_pb2.CameraImage(image_data=buf.getvalue(), encoding="jpeg")
            request.images.append(cam)

        # Attach robot state
        request.robot_state.state.extend([0.0] * 32)
        yield request

# Streaming inference loop
for response in stub.StreamActionChunks(observation_stream()):
    actions = [list(av.values) for av in response.action_chunk]
    print(f"req {response.request_id}: {len(actions)} actions, "
          f"latency={response.inference_time_ms:.1f}ms")

# Single-shot RPC (non-streaming)
response = stub.GetActionChunk(
    robot_inference_pb2.ObservationRequest(prompt="open the drawer"),
    timeout=5.0
)
# response.action_chunk: repeated ActionVector messages
# response.inference_time_ms: float
# response.request_id: str

channel.close()
```

--------------------------------

### Visualize a Dataset Episode

Source: https://github.com/tensorauto/opentau/blob/main/docs/source/tutorials/visualization.md

Use this command to visualize the first episode of a specified dataset repository. Ensure the dataset is accessible via Hugging Face Hub.

```bash
opentau-dataset-viz --repo-id lerobot/droid_100 --episode-index 0
```

--------------------------------

### Overlay URDF Robot Model for Dataset Visualization

Source: https://context7.com/tensorauto/opentau/llms.txt

Visualize dataset episodes with an overlaid URDF robot model. Requires the 'opentau[urdf]' installation. Specify the paths to the URDF file and its package directory.

```bash
pip install "opentau[urdf]"
```

```bash
opentau-dataset-viz \
    --repo-id lerobot/droid_100 \
    --episode-index 0 \
    --urdf /path/to/robot_model.urdf \
    --urdf-package-dir /path/to/urdf_package/
```

--------------------------------

### Convert Single Video (Ego Mode)

Source: https://github.com/tensorauto/opentau/blob/main/docs/source/tutorials/human_demo.md

Convert a single first-person demonstration video (hands-focused). The `--mode ego` flag is essential for this.

```bash
python -m opentau.scripts.human_video_to_lerobot \
    /path/to/ego_demo.mp4 \
    ./datasets/my_ego_dataset \
    --prompt "Open the drawer" \
    --mode ego
```

--------------------------------

### Custom Image Feature Extractor for ROS Conversion

Source: https://github.com/tensorauto/opentau/blob/main/docs/source/tutorials/ros_conversion.md

Implement a custom feature extractor by inheriting from `FeatureExtractor` and overriding the `__call__` method. This example shows how to extract image data from ROS messages, convert it to a NumPy array, and handle potential errors. The script only supports video dtype for image features.

```python
class ImageExtractor(FeatureExtractor):
def __call__(self, msg: Any, ros_topic: str, attribute: str) -> Any:
    try:
        import io

        from PIL import Image

        image = Image.open(io.BytesIO(msg.data))
        # Convert to numpy array
        image_np = np.array(image)
        # Handle RGBA if necessary, or just ensure RGB
        if image_np.shape[-1] == 4:
            image_np = image_np[..., :3]
        return image_np

    except (KeyError, AttributeError, TypeError, Exception) as e:
        logging.warning(f"Error extracting {attribute} from {ros_topic}: {e}")
        return None
```

--------------------------------

### Login to Weights and Biases

Source: https://github.com/tensorauto/opentau/blob/main/docs/source/installation.md

Log in to your Weights and Biases account for experiment tracking. This command prompts for your API key.

```bash
wandb login
```

--------------------------------

### Launch Inference Script

Source: https://github.com/tensorauto/opentau/blob/main/notebooks/pi05_evaluation_only.ipynb

Launches the OpenTAU inference script using the accelerate library with a specified configuration file.

```bash
! accelerate launch ../src/opentau/scripts/inference.py --config_path=../configs/examples/pi05_config.json
```

--------------------------------

### Launch profile_step.py Script

Source: https://github.com/tensorauto/opentau/blob/main/docs/source/tutorials/benchmarking.md

Use this command to launch the profiling script with specified configuration and batch size. It mirrors the launch incantation for opentau-train.

```bash
accelerate launch \
    --config_file configs/examples/accelerate_ddp_config.yaml \
    src/opentau/scripts/profile_step.py \
    --config_path=configs/libero/reproduce_pi05_libero.json \
    --batch_size=12
```

--------------------------------

### Dataset Configuration for Subtask Response Script

Source: https://github.com/tensorauto/opentau/blob/main/docs/source/tutorials/datasets.md

Create a configuration file to specify datasets for processing by the 'add_subtask_response' script. Each dataset entry requires a 'repo_id' and a local 'root' path.

```javascript
{
    "datasets": [
        {
            "repo_id": "TensorAuto/ice-lemonade",
            "root": "/path/to/local/dataset"
        }
    ]
}
```

--------------------------------

### Compute Advantage and Percentiles

Source: https://github.com/tensorauto/opentau/blob/main/docs/source/tutorials/RECAP.md

Use this command to calculate the advantage for each data point and determine the epsilon threshold for policy training. Ensure the config path points to your value function configuration.

```bash
python lerobot/scripts/get_advantage_and_percentiles.py --config_path=<path/to/value_function_config.json> --batch_size=20 --dataloader_batch_size=20 --dataset_mixture=examples/advantage_config.json
```

--------------------------------

### Load Policy Configuration from Pretrained Source

Source: https://context7.com/tensorauto/opentau/llms.txt

Loads a policy configuration from either the HuggingFace Hub or a local directory. Supports overriding configuration parameters via CLI arguments.

```python
from opentau.configs.policies import PreTrainedConfig

# Load from HuggingFace Hub
config = PreTrainedConfig.from_pretrained("TensorAuto/tPi0.5-libero")
print(config.type)        # "pi05"
print(config.chunk_size)  # 10

# Load from a local checkpoint directory
config = PreTrainedConfig.from_pretrained(
    "outputs/train/pi05/checkpoints/000040"
)

# Load with CLI overrides (e.g., override device at load time)
config = PreTrainedConfig.from_pretrained(
    "TensorAuto/tPi0.5-libero",
    cli_overrides=["--device=cuda:0", "--n_obs_steps=1"]
)
```

--------------------------------

### Checkpointing and Resuming Training

Source: https://context7.com/tensorauto/opentau/llms.txt

Instructions on how to save training checkpoints and resume training from a saved checkpoint.

```APIDOC
## Checkpointing and Resuming Training

### Description
Save and resume training from checkpoints stored as `model.safetensors` + `train_config.json`.

### Train with explicit output dir and checkpoint frequency
```bash
opentau-train \
    --config_path=configs/examples/pi05_training_config.json \
    --output_dir=outputs/train/pi05 \
    --steps=40000 \
    --save_freq=20000
```

### Consolidate DeepSpeed sharded checkpoints
```bash
./convert_checkpoint.sh outputs/train/pi05/checkpoints/000020/
```

### Resume training from a checkpoint
```bash
opentau-train \
    --config_path=outputs/train/pi05/checkpoints/000020/train_config.json \
    --resume=true \
    --steps=100000
```
```

--------------------------------

### Profile Per-Step Time

Source: https://github.com/tensorauto/opentau/blob/main/docs/source/tutorials/benchmarking.md

Analyze where time is spent during each training step. This typically takes about 4 minutes at 200 steps. Ensure to replace <your_bs> with your actual batch size.

```bash
accelerate launch \
    --config_file configs/examples/accelerate_ddp_config.yaml \
    src/opentau/scripts/profile_step.py \
    --config_path=<your_config.json> \
    --batch_size=<your_bs>
```

--------------------------------

### Train Offline RL - Stage 1 (SFT)

Source: https://context7.com/tensorauto/opentau/llms.txt

Initiates Stage 1 of the offline RL training pipeline, which involves supervised fine-tuning (SFT) of pi0 on the full dataset. Requires an accelerate configuration and a stage 1 training configuration.

```bash
opentau-train \
    --accelerate-config=configs/examples/accelerate_ddp_config.yaml \
    --config_path=configs/recap/stage1_sft.json
```

--------------------------------

### Resume Training from Checkpoint

Source: https://github.com/tensorauto/opentau/blob/main/docs/source/tutorials/training.md

Continue training from a previously saved checkpoint, specifying the checkpoint directory and new total steps.

```bash
opentau-train --config_path=outputs/train/pi05/checkpoints/000040/train_config.json --resume=true --steps=100
```

--------------------------------

### Programmatic Dataset Mixture Usage

Source: https://context7.com/tensorauto/opentau/llms.txt

Demonstrates programmatic creation and usage of a `WeightedDatasetMixture`. It shows how to initialize the mixture with datasets and weights, and obtain a dataloader for iterating through batches.

```python
# Programmatic usage
from opentau.datasets.dataset_mixture import WeightedDatasetMixture
from opentau.configs.train import TrainPipelineConfig
import draccus

cfg = draccus.parse(TrainPipelineConfig, "configs/examples/pi05_training_config.json")

mixture = WeightedDatasetMixture(cfg, datasets=[ds1, ds2], weights=[0.7, 0.3], action_freq=30.0)
dataloader = mixture.get_dataloader()

for batch in dataloader:
    # batch contains standardized keys: camera0, camera1, state, actions, prompt
    print(batch["camera0"].shape)   # (B, C, H, W)
    print(batch["actions"].shape)   # (B, chunk_size, action_dim)
```

--------------------------------

### PreTrainedConfig.from_pretrained

Source: https://context7.com/tensorauto/opentau/llms.txt

Load a policy configuration from a local directory or HuggingFace Hub repository.

```APIDOC
## PreTrainedConfig.from_pretrained — Load Policy Config

### Description
Load a policy configuration from a local directory or HuggingFace Hub repository.

### Load from HuggingFace Hub
```python
from opentau.configs.policies import PreTrainedConfig

config = PreTrainedConfig.from_pretrained("TensorAuto/tPi0.5-libero")
print(config.type)        # "pi05"
print(config.chunk_size)  # 10
```

### Load from a local checkpoint directory
```python
config = PreTrainedConfig.from_pretrained(
    "outputs/train/pi05/checkpoints/000040"
)
```

### Load with CLI overrides
```python
config = PreTrainedConfig.from_pretrained(
    "TensorAuto/tPi0.5-libero",
    cli_overrides=["--device=cuda:0", "--n_obs_steps=1"]
)
```

### Save a config to a directory
```python
from pathlib import Path
config._save_pretrained(Path("my_policy_config/"))
# Writes: my_policy_config/config.json
```
```

--------------------------------

### Run Pre-commit Hooks on All Files

Source: https://github.com/tensorauto/opentau/blob/main/docs/source/contributing.md

Apply 'pre-commit' hooks to all files in the repository to reformat and lint the entire codebase. Use this if you have committed changes with incorrect formatting.

```bash
pre-commit run --all-files
```

--------------------------------

### Load Trained Policy Weights and Configuration

Source: https://context7.com/tensorauto/opentau/llms.txt

Loads a complete trained policy, including weights and configuration, from a local path or HuggingFace Hub. Allows for partial weight loading and inference.

```python
from opentau.policies.pi05 import PI05Policy
from opentau.policies.pi0 import PI0Policy

# Load pi05 from HuggingFace Hub (auto-downloads safetensors)
policy = PI05Policy.from_pretrained("TensorAuto/tPi0.5-libero")
policy.eval()  # already set to eval by default

# Load from a local checkpoint
policy = PI05Policy.from_pretrained(
    "outputs/train/pi05/checkpoints/000040",
    config=None,        # config auto-loaded from directory
    strict=False,       # allow partial weight loading
)

# Load pi0 checkpoint (for RECAP/offline RL workflows)
policy = PI0Policy.from_pretrained("lerobot/pi0")
```