### Install Docker on Ubuntu

Source: https://github.com/facebookresearch/home-robot/blob/main/docs/challenge.md

Configures the Apt repository and installs the Docker engine and plugins.

```bash
echo \
  "deb [arch="$(dpkg --print-architecture)" signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
  "$(. /etc/os-release && echo "$VERSION_CODENAME")" stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update
sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
```

--------------------------------

### Install Home-Robot Sim Library

Source: https://github.com/facebookresearch/home-robot/blob/main/src/home_robot_sim/README.md

Installs the home_robot_sim library in editable mode.

```bash
# Install home robot sim interfaces
pip install -e src/home_robot_sim
```

--------------------------------

### Install Project Packages

Source: https://github.com/facebookresearch/home-robot/blob/main/src/home_robot_spot/README.md

Install the home-robot, home-robot-hw, and home-robot-spot packages in editable mode.

```bash
pip install -e src/home_robot/. 
pip install -e src/home_robot_hw/. 
pip install -e src/home_robot_spot/. 
```

--------------------------------

### Install system dependencies

Source: https://github.com/facebookresearch/home-robot/blob/main/README.md

Install essential build tools on an Ubuntu system before setting up the environment.

```bash
sudo apt update
sudo apt install build-essential zip unzip
```

--------------------------------

### Install Project Dependencies

Source: https://github.com/facebookresearch/home-robot/blob/main/projects/instanceimagenav/README.md

Navigate to the project directory, install its requirements, and create a directory for scene datasets.

```bash
# ----------------------- This Project -----------------------
cd projects/instanceimagenav
pip install -r requirements.txt
mkdir -p data/scene_datasets
```

--------------------------------

### Install Habitat Sim v0.2.3

Source: https://github.com/facebookresearch/home-robot/blob/main/projects/instanceimagenav/README.md

Clone the Habitat Sim v0.2.3 repository, install its requirements, and then install it in headless mode with CUDA support.

```bash
# -------------------- Habitat Sim v0.2.3 --------------------
git clone --branch v0.2.3 https://github.com/facebookresearch/habitat-sim.git
cd habitat-sim
pip install -r requirements.txt
python setup.py install --headless --with-cuda
cd ..
```

--------------------------------

### Install targets

Source: https://github.com/facebookresearch/home-robot/blob/main/assets/hab_stretch/CMakeLists.txt

Defines installation paths for libraries and runtime executables.

```cmake
install(TARGETS ${PROJECT_NAME} ${PROJECT_NAME}_node
  ARCHIVE DESTINATION ${CATKIN_PACKAGE_LIB_DESTINATION}
  LIBRARY DESTINATION ${CATKIN_PACKAGE_LIB_DESTINATION}
  RUNTIME DESTINATION ${CATKIN_PACKAGE_BIN_DESTINATION}
)
```

--------------------------------

### Install Home Robot and Detectron2

Source: https://github.com/facebookresearch/home-robot/blob/main/projects/slap_manipulation/readme.md

Install the local home-robot packages and the Detectron2 dependency.

```bash
# return to HOME_ROBOT_ROOT
cd ../..

python -m pip install -e src/home_robot
python -m pip install -e src/home_robot_hw

# install Detectron2
python -m pip install 'git+https://github.com/facebookresearch/detectron2.git'
```

--------------------------------

### Install Habitat and PyTorch3D

Source: https://github.com/facebookresearch/home-robot/blob/main/src/home_robot_sim/README.md

Installs habitat-lab, habitat-baselines, and pytorch3d dependencies.

```bash
python -m pip install -e src/third_party/habitat-lab/habitat-lab
python -m pip install -e src/third_party/habitat-lab/habitat-baselines
python -m pip install "git+https://github.com/facebookresearch/pytorch3d.git"
```

--------------------------------

### Install Spot Sim2Real

Source: https://github.com/facebookresearch/home-robot/blob/main/src/home_robot_spot/README.md

Navigate to the spot-sim2real directory, checkout the no_habitat branch, and install its packages.

```bash
cd src/third_party/spot-sim2real
git checkout no_habitat
```

--------------------------------

### Install Demo Server Packages (pip)

Source: https://github.com/facebookresearch/home-robot/blob/main/projects/scannet_offline_eval/demo/README.md

Use this command to install the necessary Python packages for the demo server using pip.

```bash
pip install openai dash dash-bootstrap-components dash-extensions quart atomicwrites loguru
```

--------------------------------

### Setup Spot Sim2Real

Source: https://github.com/facebookresearch/home-robot/blob/main/docs/spot.md

Navigates to the spot-sim2real directory, checks out the main branch, and generates executables.

```bash
cd src/third_party/spot-sim2real
git checkout main
```

```bash
cd bd_spot_wrapper/ 
python generate_executables.py
pip install -e .
```

```bash
cd ../spot_rl_experiments 
python generate_executables.py
pip install -e .
```

```bash
# Generate module
cd ../perception_and_utils_root/ 
pip install -e . && cd ../
```

--------------------------------

### Install NVIDIA Container Toolkit

Source: https://github.com/facebookresearch/home-robot/blob/main/docs/challenge.md

Configures the NVIDIA repository and installs the toolkit if NVIDIA drivers are not detected.

```bash
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
  && curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
    sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
    sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list \
  && \
    sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit
sudo apt install -y nvidia-docker2
sudo systemctl daemon-reload
sudo systemctl restart docker
```

--------------------------------

### Install Habitat Lab and Baselines

Source: https://github.com/facebookresearch/home-robot/blob/main/projects/habitat_objectnav/README.md

Navigate to the habitat-lab directory, checkout the specific branch, and install the packages using pip. This sets up the necessary libraries for Habitat ObjectNav.

```bash
cd src/third_party/habitat-lab
git checkout home-robot_objectnav_support
pip install -e habitat-lab
pip install -e habitat-baselines
cd ../../..
```

--------------------------------

### Install and Configure Pre-commit Hooks

Source: https://github.com/facebookresearch/home-robot/blob/main/README.md

Install pre-commit and configure it for the Home Robot project to enforce code style before committing.

```bash
python -m pip install pre-commit
cd $HOME_ROBOT_ROOT
pre-commit install
```

--------------------------------

### Install Sophuspy with Pybind11

Source: https://github.com/facebookresearch/home-robot/blob/main/docs/spot.md

Installs sophuspy by first building and installing pybind11 from source, then installing sophuspy using the pybind11_DIR environment variable.

```bash
git clone https://github.com/pybind/pybind11.git
cd pybind11
mkdir build && cd build
cmake .. -DCMAKE_INSTALL_PREFIX=../install/
make -j8
make install
```

```bash
pybind11_DIR=$PWD/../install/share/cmake/pybind11/ pip3 install --user sophuspy
```

--------------------------------

### Install SuperGLUE

Source: https://github.com/facebookresearch/home-robot/blob/main/docs/spot.md

Installs the SuperGLUE model requirements. Ensure you are in the correct directory.

```bash
pip install -r src/home_robot/home_robot/agent/imagenav_agent/SuperGluePretrainedNetwork/requirements.txt
```

--------------------------------

### Install Home Robot on Robot-side

Source: https://github.com/facebookresearch/home-robot/blob/main/src/home_robot/README.md

Installs the package in editable mode directly into the current environment.

```sh
cd $HOME_ROBOT_ROOT/src/home_robot
pip install -e .
```

--------------------------------

### Install Habitat Lab v0.2.3

Source: https://github.com/facebookresearch/home-robot/blob/main/projects/instanceimagenav/README.md

Clone the Habitat Lab v0.2.3 repository and install it in editable mode, along with habitat-baselines.

```bash
# -------------------- Habitat Lab v0.2.3 --------------------
git clone --branch v0.2.3 https://github.com/facebookresearch/habitat-lab.git
cd habitat-lab
pip install -e habitat-lab
pip install -e habitat-baselines
cd ..
```

--------------------------------

### Initialize Environment and Load Data

Source: https://github.com/facebookresearch/home-robot/blob/main/src/home_robot/home_robot/mapping/semantic/vision_language_2d_demo.ipynb

Setup necessary imports and load observation and pose data from numpy files.

```python
import glob
import torch
import numpy as np
import warnings
from pathlib import Path
warnings.simplefilter("ignore")
```

```python
obs_paths = glob.glob("demo_data/obs_*.npy")
pose_delta_paths = glob.glob("demo_data/pose_delta_*.npy")
device = torch.device("cuda:0")

obs = np.stack([np.load(obs_path) for obs_path in obs_paths])

pose_delta = np.stack([np.load(pose_delta_path) 
                       for pose_delta_path in pose_delta_paths])
```

--------------------------------

### Install Detectron2

Source: https://github.com/facebookresearch/home-robot/blob/main/docs/spot.md

Installs Detectron2 in editable mode. Ensure CUDA_HOME is set correctly.

```bash
cd $HOME_ROBOT_ROOT
pip install -e src/third_party/detectron2/.
```

--------------------------------

### Install Home Robot on Server-side

Source: https://github.com/facebookresearch/home-robot/blob/main/src/home_robot/README.md

Creates a dedicated mamba environment from the environment.yml file and installs the package in editable mode.

```sh
cd $HOME_ROBOT_ROOT/src/home_robot
mamba env create -n home_robot -f environment.yml
conda activate home_robot
pip install -e .
```

--------------------------------

### Launch RViz for Mapping

Source: https://github.com/facebookresearch/home-robot/blob/main/docs/network.md

Starts the RViz visualization tool with the mapping demo configuration file.

```bash
rviz -d $HOME_ROBOT_ROOT/src/home_robot_hw/launch/mapping_demo.rviz
```

--------------------------------

### Install Detic and Detectron2

Source: https://github.com/facebookresearch/home-robot/blob/main/projects/instanceimagenav/README.md

Install Detectron2 and Detic requirements. Download Detic model weights and place them in the models directory.

```bash
# -------------------- Detic + Detectron2 --------------------
# Install Detectron2
pip install -e src/thrid_party/detectron2
# Install Detic requirements
pip install -r src/home_robot/home_robot/perception/detection/detic/Detic/requirements.txt
# download Detic model weights
mkdir -p src/home_robot/home_robot/perception/detection/detic/Detic/models
wget https://dl.fbaipublicfiles.com/detic/Detic_LCOCOI21k_CLIP_SwinB_896b32_4x_ft4x_max-size.pth \
  -O src/home_robot/home_robot/perception/detection/detic/Detic/models/Detic_LCOCOI21k_CLIP_SwinB_896b32_4x_ft4x_max-size.pth
```

--------------------------------

### Install miscellaneous files

Source: https://github.com/facebookresearch/home-robot/blob/main/assets/hab_stretch/CMakeLists.txt

Installs additional project files like launch or bag files to the share directory.

```cmake
install(FILES
  # myfile1
  # myfile2
  DESTINATION ${CATKIN_PACKAGE_SHARE_DESTINATION}
)
```

--------------------------------

### Install Project Files

Source: https://github.com/facebookresearch/home-robot/blob/main/src/home_robot_hw/CMakeLists.txt

Installs project files like launch and bag files to the package's share directory. Ensure files are listed and the destination is correctly set.

```cmake
# install(FILES
#   # myfile1
#   # myfile2
#   DESTINATION ${CATKIN_PACKAGE_SHARE_DESTINATION}
# )

```

--------------------------------

### Generate Spot Sim2Real Executables and Install

Source: https://github.com/facebookresearch/home-robot/blob/main/src/home_robot_spot/README.md

Generate executables for spot sim2real and install the packages in editable mode for both bd_spot_wrapper and spot_rl_experiments.

```bash
cd bd_spot_wrapper/
python generate_executables.py
pip install -e .
```

```bash
cd ../spot_rl_experiments
python generate_executables.py
pip install -e .
```

--------------------------------

### Install header files

Source: https://github.com/facebookresearch/home-robot/blob/main/assets/hab_stretch/CMakeLists.txt

Copies C++ header files from the include directory to the package installation path.

```cmake
install(DIRECTORY include/${PROJECT_NAME}/
  DESTINATION ${CATKIN_PACKAGE_INCLUDE_DESTINATION}
  FILES_MATCHING PATTERN "*.h"
  PATTERN ".svn" EXCLUDE
)
```

--------------------------------

### Example Usage of show_instance_image

Source: https://github.com/facebookresearch/home-robot/blob/main/projects/scannet_offline_eval/instance_map_figure.ipynb

Demonstrates how to call the show_instance_image function with sample data. This snippet is illustrative and requires the 'scene_obs' dictionary to be populated.

```python
idx = -1
# plt.imshow(
#     show_instance_image(
#         scene_obs['images'][idx],
#         scene_obs['instance_2ds'][idx],

```

--------------------------------

### Install Habitat-sim

Source: https://github.com/facebookresearch/home-robot/blob/main/docs/spot.md

Installs Habitat-sim and its dependencies using pip and mamba. Ensure the Conda environment is activated.

```bash
cd $HOME_ROBOT_ROOT/src/third_party/habitat-lab
pip install -e habitat-lab
mamba install habitat-sim headless -c conda-forge -c aihabitat --yes
```

--------------------------------

### Initialize ScanNet Dataset and Agent

Source: https://github.com/facebookresearch/home-robot/blob/main/projects/habitat_ovmm/test_svm_from_obs.ipynb

Setup for ScanNet dataset and Hydra-based configuration loading.

```python
# @hydra.main(config_path="/private/home/xiaohanzhang/home-robot/projects/scannet_offline_eval/configs/model", 
#             config_name="instancemap3d_top_down_detic")
scannet = ScanNetDataset(
    root_dir = '/private/home/ssax/home-robot/src/home_robot/home_robot/datasets/scannet/data',
    frame_skip = 180,
    split = 'val',
    n_classes=NUM_CLASSES_LONG,
    referit3d_config = ReferIt3dDataConfig(),
    scanrefer_config = ScanReferDataConfig(),    
)
from hydra import compose, initialize
from omegaconf import OmegaConf
# context initialization
with initialize(version_base=None, config_path="../scannet_offline_eval/configs/model", job_name="test_app"):
    cfg = compose(config_name="instancemap3d_top_down_detic")
    print(OmegaConf.to_yaml(cfg))
instance_agent = instantiate(cfg)
```

--------------------------------

### Setup Habitat Lab Data Directory

Source: https://github.com/facebookresearch/home-robot/blob/main/projects/habitat_ovmm/README.md

Creates a symbolic link for the Habitat Lab data directory.

```bash
cd /path/to/home-robot/src/third_party/habitat-lab/

# create soft link to data/ directory
ln -s /path/to/home-robot/data data
```

--------------------------------

### Install Docker on Ubuntu

Source: https://github.com/facebookresearch/home-robot/blob/main/docs/challenge.md

Commands to add Docker's official GPG key and install Docker on an Ubuntu system. This is a prerequisite for building and running Docker images.

```bash
sudo apt-get update
sudo apt-get install ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
sudo chmod a+r /etc/apt/keyrings/docker.gpg
```

--------------------------------

### Install Detic

Source: https://github.com/facebookresearch/home-robot/blob/main/docs/spot.md

Installs the Detic object detection model. This includes cloning the repository, downloading weights, and running the demo script.

```bash
cd src/home_robot/home_robot/perception/detection/detic/Detic
pip install -r requirements.txt
mkdir models
wget https://dl.fbaipublicfiles.com/detic/Detic_LCOCOI21k_CLIP_SwinB_896b32_4x_ft4x_max-size.pth -O models/Detic_LCOCOI21k_CLIP_SwinB_896b32_4x_ft4x_max-size.pth
```

```bash
wget https://eecs.engin.umich.edu/~fouhey/fun/desk/desk.jpg
python demo.py --config-file configs/Detic_LCOCOI21k_CLIP_SwinB_896b32_4x_ft4x_max-size.yaml --input desk.jpg --output out.jpg --vocabulary lvis --opts MODEL.WEIGHTS models/Detic_LCOCOI21k_CLIP_SwinB_896b32_4x_ft4x_max-size.pth
```

--------------------------------

### Launch Hello Stretch Hardware Drivers

Source: https://github.com/facebookresearch/home-robot/blob/main/docs/hardware_development.md

Use this command to start the necessary hardware drivers for the Hello Stretch robot. Ensure you are on the robot system.

```shell
roslaunch home_robot_hw startup_stretch_hector_slam.launch
```

--------------------------------

### Run Real-World OVMM Evaluation

Source: https://github.com/facebookresearch/home-robot/blob/main/README.md

Execute the OVMM example script for real-world evaluation after setting up grasping servers.

```bash
cd $HOME_ROBOT_ROOT
python projects/real_world_ovmm/eval_episode.py
```

--------------------------------

### Install Demo Server Packages (conda)

Source: https://github.com/facebookresearch/home-robot/blob/main/projects/scannet_offline_eval/demo/README.md

If you encounter dependency issues with pip, try installing heavier libraries from conda first, then use pip for the remaining packages.

```bash
mamba install -c conda-forge dash dash-bootstrap-components dash-extensions quart
pip install openai loguru atomicwrites
```

--------------------------------

### Initialize ScanNet Dataset Environment

Source: https://github.com/facebookresearch/home-robot/blob/main/projects/scannet_offline_eval/instance_map_figure.ipynb

Setup imports and environment configurations for working with ScanNet data.

```python
%load_ext autoreload
%autoreload 2
# Copyright (c) Meta Platforms, Inc. and affiliates.
#
# This source code is licensed under the MIT license found in the
# LICENSE file in the root directory of this source tree.
import argparse
import dataclasses
import random
import sys
import timeit
from typing import Tuple

import matplotlib.pyplot as plt
import numpy as np
import torch
from tqdm import tqdm

# from home_robot.mapping.voxel import SparseVoxelMap
from home_robot.utils.point_cloud_torch import unproject_masked_depth_to_xyz_coordinates
```

--------------------------------

### Run the Docker Container

Source: https://github.com/facebookresearch/home-robot/blob/main/docs/challenge.md

Starts the built Docker container with NVIDIA GPU support and host networking.

```bash
sudo docker run --runtime=nvidia --gpus all  --network="host" -e OPENBLAS_NUM_THREADS=1 -e NUMEXPR_NUM_THREADS=1 -e MKL_NUM_THREADS=1 --privileged  ovmm_baseline_submission
```

--------------------------------

### Install Common Python Packages

Source: https://github.com/facebookresearch/home-robot/blob/main/docs/spot.md

Installs a list of essential Python packages for the project. Ensure your environment is activated.

```bash
pip install bosdyn-api  bosdyn-client transforms3d einops gym==0.23.1 vtk scikit-image open3d natsort scikit-fmm pandas==2.1.1 atomicwrites loguru
```

--------------------------------

### Launch Robot Drivers and Visualization

Source: https://github.com/facebookresearch/home-robot/blob/main/docs/catkin.md

Commands to start drivers on the robot and visualization on the desktop.

```bash
roslaunch home_robot_hw startup_stretch_hector_slam.launch
```

```bash
source $HOME/catkin_ws/devel/setup.bash
roslaunch home_robot_hw visualization.launch
```

--------------------------------

### Example Local Evaluation Output

Source: https://github.com/facebookresearch/home-robot/blob/main/docs/challenge.md

This is an example of the output you might see when successfully evaluating the random agent locally, showing initialization and averaged metrics.

```text
2023-07-18 21:02:10,607 Initializing dataset OVMMDataset-v0
2023-07-18 21:02:10,713 initializing sim OVMMSim-v0
21:02:13,991 Initializing task OVMMNavToObjTask-v0
100%|██████████| 10/10 [00:36<00:00,  3.64s/it]
==================================================
Averaged metrics
==================================================
episode_count: 10.0
does_want_terminate: 1.0
num_steps: 84.8
find_object_phase_success: 0.0
pick_object_phase_success: 0.0
find_recep_phase_success: 0.0
overall_success: 0.0
partial_success: 0.0
==================================================
Metrics:
{'episode_count': 10.0, 'does_want_terminate': 1.0, 'num_steps': 84.8, 'find_object_phase_success': 0.0, 'pick_object_phase_success': 0.0, 'find_recep_phase_success': 0.0, 'overall_success': 0.0, 'partial_success': 0.0}
```

--------------------------------

### Run Install Script for Dependencies

Source: https://github.com/facebookresearch/home-robot/blob/main/README.md

Activate the Conda environment and run the install script to download submodules, model checkpoints, and build Detic.

```bash
conda activate home-robot
cd $HOME_ROBOT_ROOT
./install_deps.sh
```

--------------------------------

### Run OVMM Evaluation on Real Stretch Robot

Source: https://context7.com/facebookresearch/home-robot/llms.txt

Provides a complete example script for running OVMM evaluation on the real Stretch robot, including ROS initialization, environment and agent setup, and the main evaluation loop.

```python
#!/usr/bin/env python
"""Real-world OVMM evaluation example."""
from datetime import datetime
import rospy

from home_robot.agent.ovmm_agent.ovmm_agent import OpenVocabManipAgent
from home_robot.utils.config import load_config
from home_robot_hw.env.stretch_pick_and_place_env import StretchPickandPlaceEnv

def run_ovmm_evaluation():
    # Initialize ROS node
    rospy.init_node("eval_episode_stretch_ovmm")

    # Load configuration
    config = load_config(
        config_path="projects/real_world_ovmm/configs/agent/eval.yaml",
        visualize=True,
    )

    # Create environment
    env = StretchPickandPlaceEnv(
        config=config,
        cat_map_file="projects/real_world_ovmm/configs/example_cat_map.json",
        visualize_planner=True,
        ros_grasping=True,
    )

    # Create OVMM agent
    agent = OpenVocabManipAgent(config=config, device_id=0)

    # Reset agent and environment
    agent.reset()

    # Set up visualization directory
    if hasattr(agent, "planner"):
        now = datetime.now()
        agent.planner.set_vis_dir("real_world", now.strftime("%Y_%m_%d_%H_%M_%S"))

    # Reset environment with task
    env.reset(
        goal_find="chair",  # Start receptacle
        goal_obj="cup",     # Target object
        goal_place="table", # Goal receptacle
    )

    # Main evaluation loop
    max_steps = 200
    for step in range(max_steps):
        if rospy.is_shutdown():
            break

        print(f"Step {step + 1}")

        # Get observation
        obs = env.get_observation()

        # Agent computes action
        action, info, obs = agent.act(obs)

        # Execute action
        done = env.apply_action(action, info=info, prev_obs=obs)

        if done:
            print("Task completed successfully!")
            break

    # Get final metrics
    metrics = env.get_episode_metrics()
    print(f"Episode metrics: {metrics}")

    return metrics

if __name__ == "__main__":
    run_ovmm_evaluation()
```

--------------------------------

### Run the RPC Server

Source: https://github.com/facebookresearch/home-robot/blob/main/docs/challenge.md

Executes the evaluation script to start the RPC server.

```python
python src/home_robot_hw/home_robot_hw/utils/eval.py
```

--------------------------------

### Install Habitat-Lab Dependencies

Source: https://github.com/facebookresearch/home-robot/blob/main/projects/real_world_ovmm/README.md

Install habitat-lab and its associated baseline dependencies. This involves updating git submodules and installing packages in editable mode.

```sh
git submodule update --init --recursive src/third_party/habitat-lab
pip install -e habitat-lab
pip install -e src/third_party/habitat-lab/habitat-lab
pip install -e src/third_party/habitat-lab/habitat-baselines
```

--------------------------------

### Start Interactive CLI for Robot Interaction

Source: https://github.com/facebookresearch/home-robot/blob/main/docs/hardware_development.md

Launch an interactive command-line interface to interact with the robot from your workstation. This requires the home_robot_hw package.

```python
python -m home_robot_hw.remote.interactive_cli
```

--------------------------------

### Install MiDaS

Source: https://github.com/facebookresearch/home-robot/blob/main/docs/spot.md

Installs the MiDaS depth estimation model. This involves checking out the master branch, pulling updates, downloading weights, and installing the package.

```bash
cd $HOME_ROBOT_ROOT/src/third_party/MiDaS
git checkout master
git pull origin
```

```bash
cd weights
wget https://github.com/isl-org/MiDaS/releases/download/v3_1/dpt_beit_large_512.pt
```

```bash
cd ../
pip install -e .
pip install imutils
```

--------------------------------

### Download OVMM Dataset

Source: https://github.com/facebookresearch/home-robot/blob/main/projects/habitat_ovmm/README.md

Installs Git LFS and downloads the OVMM dataset. Ensure you have accepted the license on Hugging Face.

```bash
$HOME_ROBOT_ROOT/projects/habitat_ovmm/install.sh
```

--------------------------------

### Run Demo Server (Canned Trajectory)

Source: https://github.com/facebookresearch/home-robot/blob/main/projects/scannet_offline_eval/demo/README.md

Execute this command to start the demo server with a predefined trajectory. Ensure the trajectory is in the correct location and the server listens for updates.

```bash
python server.py
```

--------------------------------

### Install Pinocchio with Conda

Source: https://github.com/facebookresearch/home-robot/blob/main/docs/install_workstation.md

Install the Pinocchio library, an optional dependency for inverse kinematics, using conda. This ensures compatibility with existing boost installations.

```bash
conda install pinocchio>=2.6.17 -c conda-forge
```

--------------------------------

### Implement Custom Environment

Source: https://context7.com/facebookresearch/home-robot/llms.txt

Example environment implementation that defines the interface for simulation or real-world environments. It handles resetting, applying actions, and providing observations.

```python
from abc import ABC, abstractmethod
from typing import Any, Dict, Optional
from home_robot.core.abstract_env import Env
from home_robot.core.interfaces import Action, Observations

class CustomEnvironment(Env):
    """Example environment implementation."""

    def __init__(self, config):
        self.config = config
        self._episode_over = False
        self._metrics = {}

    def reset(self):
        """Reset environment to initial state."""
        self._episode_over = False
        self._metrics = {"steps": 0, "success": False}

    def apply_action(
        self,
        action: Action,
        info: Optional[Dict[str, Any]] = None,
        prev_obs: Optional[Observations] = None,
    ):
        """Execute action in the environment."""
        self._metrics["steps"] += 1
        # Execute the action and update internal state

    def get_observation(self) -> Observations:
        """Return current sensor observations."""
        import numpy as np
        return Observations(
            rgb=self._get_rgb(),
            depth=self._get_depth(),
            gps=self._get_position(),
            compass=self._get_heading(),
        )

    @property
    def episode_over(self) -> bool:
        """Check if episode has ended."""
        return self._episode_over

    def get_episode_metrics(self) -> Dict:
        """Return episode performance metrics."""
        return self._metrics
```

--------------------------------

### Initialize Submodules

Source: https://github.com/facebookresearch/home-robot/blob/main/src/home_robot_sim/README.md

Fetches the required habitat-lab submodule.

```bash
git submodule update --init --recursive src/third_party/habitat-lab
```

--------------------------------

### Initialize and use Discrete Action Planner

Source: https://github.com/facebookresearch/home-robot/blob/main/src/home_robot/home_robot/navigation_planner/README.md

Instantiate the DiscretePlanner with configuration parameters and execute a plan based on map and sensor data.

```python
from home_robot.agent.navigation_planner.discrete_planner import DiscretePlanner

# See discrete_planner.py for argument info
discrete_planner = DiscretePlanner(
    turn_angle=30.0,
    collision_threshold=0.20,
    obs_dilation_selem_radius=3,
    goal_dilation_selem_radius=10,
    map_size_cm=4800,
    map_resolution=5,
    visualize=True,
    print_images=True,
    dump_location="datadump",
    exp_name="exp"
)

# See discrete_planner.py for argument info
# 
# outputs discrete action in
# class DiscreteActions(Enum):
#    stop = 0
#    move_forward = 1
#    turn_left = 2
#    turn_right = 3
#
discrete_action, _ = planner.plan(
    obstacle_map,
    goal_map,
    sensor_pose,
    found_goal
)
```

--------------------------------

### Initialize Spot Robot Client

Source: https://github.com/facebookresearch/home-robot/blob/main/src/home_robot_spot/examples/main_old.txt

Instantiates the Spot client with configuration parameters, including dock ID and depth sensor settings. This client is used to control the robot's actions.

```python
planner = Shortcut(RRTConnect(navigation_space, navigation_space.is_valid))

spot = SpotClient(
    config=spot_config, dock_id=dock, use_midas=parameters["use_midas"], use_zero_depth=parameters['use_zero_depth']
)
try:
    # Turn on the robot using the client above
    spot.start()

    print("Sleep 1s")
    time.sleep(1)
    print("Start exploring!")
    x0, y0, theta0 = spot.current_position
    spot.reset_arm()
    spot.navigate_to([x0, y0, theta0], blocking=True)
```

--------------------------------

### Install executable scripts

Source: https://github.com/facebookresearch/home-robot/blob/main/assets/hab_stretch/CMakeLists.txt

Marks Python or other scripts for installation into the package binary destination.

```cmake
install(PROGRAMS
  scripts/my_python_script
  DESTINATION ${CATKIN_PACKAGE_BIN_DESTINATION}
)
```

--------------------------------

### Install PyTorch

Source: https://github.com/facebookresearch/home-robot/blob/main/docs/install_pytorch.md

Standard conda installation command for PyTorch with specific CUDA support.

```bash
conda install pytorch torchvision torchaudio pytorch-cuda=11.7 -c pytorch -c nvidia
```

--------------------------------

### Install PyTorch3d

Source: https://github.com/facebookresearch/home-robot/blob/main/docs/install_pytorch.md

Commands for installing PyTorch3d via conda or building directly from the source repository.

```bash
conda install pytorch3d -c pytorch3d
```

```bash
pip install "git+https://github.com/facebookresearch/pytorch3d.git"
```

--------------------------------

### Install Move Base Dependency

Source: https://github.com/facebookresearch/home-robot/blob/main/docs/catkin.md

Install the required move_base package using apt or mamba.

```bash
sudo apt install ros-noetic-move-base
# or
mamba install ros-noetic-move-base -c robostack
```

--------------------------------

### Initialize Catkin Workspace

Source: https://github.com/facebookresearch/home-robot/blob/main/docs/catkin.md

Commands to create a local workspace and link the home robot package.

```bash
# Create a catkin workspace
cd $HOME
mkdir catkin_ws
cd catkin_ws
catkin_init_workspace

# Link your home robot package to it
ln -s $HOME_ROBOT_ROOT/src/home_robot_hw $HOME/catkin_ws/src/home_robot_hw

# Clone the stretch_ros code
cd src
git clone git@github.com:hello-robot/stretch_ros.git
```

--------------------------------

### Initialize and Run Simulation Loop

Source: https://github.com/facebookresearch/home-robot/blob/main/src/home_robot_sim/notebooks/velocity_control_sim.ipynb

Sets up the environment and controller, then iterates through the simulation time horizon to execute control commands and record visualization frames.

```python
# initialize objects
xyt_goal = np.array(GOAL)

viz = Viz()
env = Env(SIM_HZ)
agent = Controller()
agent.set_goal(xyt_goal)

# simulate
t_control = 0
t_viz = 0
dt_control = 1 / CONTROL_HZ
dt_viz = 1 / VIZ_HZ
for t in np.linspace(0, TIME_HORIZON, int(TIME_HORIZON * SIM_HZ)):
    xyt = env.get_pose()
    
    vel_command = None
    if t >= t_control:
        vel_command = agent.forward(xyt)
        t_control += dt_control
    if t >= t_viz:
        viz.record_frame(xyt.copy(), xyt_goal.copy())
        t_viz += dt_viz
    
    env.step(vel_command)
```

--------------------------------

### Install OVMM Dependencies

Source: https://github.com/facebookresearch/home-robot/blob/main/projects/real_world_ovmm/README.md

Install additional dependencies required for OVMM. Ensure you are in the correct project directory.

```sh
pip install -r projects/real_world_ovmm/requirements.txt
```

--------------------------------

### Install Home Robot Packages

Source: https://github.com/facebookresearch/home-robot/blob/main/projects/instanceimagenav/README.md

Update the conda environment for home_robot, then install home_robot and home_robot_sim in editable mode.

```bash
# -------------------- Install Home-Robot --------------------
cd src/home_robot
conda env update --file environment.yml --prune
pip install -e .
cd ../src/home_robot_sim
pip install -e .
cd ../..
```

--------------------------------

### Install EvalAI CLI

Source: https://github.com/facebookresearch/home-robot/blob/main/docs/challenge.md

Install the EvalAI command-line interface. Ensure you use a version greater than or equal to 1.3.5.

```bash
pip install "evalai>=1.3.5"
```

--------------------------------

### Configure and Initialize Navigation Environment

Source: https://github.com/facebookresearch/home-robot/blob/main/src/home_robot_spot/examples/main_old.txt

Sets up the voxel map, robot kinematics model, and navigation space using configurations from YAML files. This is crucial for robot navigation and obstacle avoidance.

```python
# TODO add this to config
sspot_config = get_config("src/home_robot_spot/configs/default_config.yaml")[0]
parameters = get_config("src/home_robot_spot/configs/parameters.yaml")[0]
# Create voxel map
voxel_map = SparseVoxelMap(
    resolution=parameters["voxel_size"],
    local_radius=parameters["local_radius"],
    obs_min_height=parameters["obs_min_height"],
    obs_max_height=parameters["obs_max_height"],
    obs_min_density=parameters["obs_min_density"],
    smooth_kernel_size=parameters["smooth_kernel_size"],
)

# Create kinematic model (very basic for now - just a footprint)
robot_model = SimpleSpotKinematics()

# Create navigation space example
navigation_space = SparseVoxelMapNavigationSpace(
    voxel_map=voxel_map,
    robot=robot_model,
    step_size=parameters["step_size"],
    rotation_step_size=parameters["rotation_step_size"],
    dilate_frontier_size=parameters["dilate_frontier_size"],
    dilate_obstacle_size=parameters["dilate_obstacle_size"],
)
print(" - Created navigation space and environment")
print(f"   {navigation_space=}")
```

--------------------------------

### Initialize Environment and Imports

Source: https://github.com/facebookresearch/home-robot/blob/main/src/home_robot/home_robot/datasets/scannet/visualize_scannet.ipynb

Standard imports and autoreload configuration for the home-robot environment.

```python
%load_ext autoreload
%autoreload 2
# Copyright (c) Meta Platforms, Inc. and affiliates.
#
# This source code is licensed under the MIT license found in the
# LICENSE file in the root directory of this source tree.
import argparse
import dataclasses
import sys
import timeit
from typing import Tuple

import matplotlib.pyplot as plt
import numpy as np
import torch
from tqdm import tqdm
import pandas as pd
```

--------------------------------

### Install SLAP Prerequisites

Source: https://github.com/facebookresearch/home-robot/blob/main/projects/slap_manipulation/readme.md

Install PyTorch, PyG, and associated geometric deep learning libraries within the conda environment.

```bash
conda activate slap_base

python -m pip install torch==1.12.1+cu116 torchvision==0.13.1+cu116 torchaudio==0.12.1 --extra-index-url https://download.pytorch.org/whl/cu116
python -m pip install --no-index pyg_lib torch_scatter torch_sparse torch_cluster torch_spline_conv -f https://data.pyg.org/whl/torch-1.12.1+cu116.html
python -m pip install torch_geometric
```

--------------------------------

### Run Instance ImageNav Demo in Simulation

Source: https://github.com/facebookresearch/home-robot/blob/main/projects/instanceimagenav/README.md

Execute the simulation demo for the InstanceImageNav model. Results are saved to projects/instanceimagenav/datadump.

```bash
# in directory: /path/to/home-robot/projects/instanceimagenav
python eval_episode_habitat.py
```

--------------------------------

### Install Home Robot Packages

Source: https://github.com/facebookresearch/home-robot/blob/main/docs/spot.md

Installs the Home Robot packages in editable mode. Ensure the Conda environment is activated.

```bash
pip install -e src/home_robot/. 
pip install -e src/home_robot_hw/. 
pip install -e src/home_robot_spot/.
```

--------------------------------

### Launch Grasping Server on Server

Source: https://github.com/facebookresearch/home-robot/blob/main/projects/real_world_ovmm/README.md

Launch the simple grasping server on the server. This is a prerequisite for running the stand-alone pick and place script.

```python
python -m home_robot_hw.nodes.simple_grasp_server
```

--------------------------------

### Run Websocket Publisher Server

Source: https://github.com/facebookresearch/home-robot/blob/main/projects/scannet_offline_eval/demo/README.md

Start this Quart server to enable websocket connections for real-time data publishing. It creates a DirectoryWatcher server that publishes data on port 5000.

```bash
python -m components.publisher_server
```

--------------------------------

### Install Home Robot Packages

Source: https://github.com/facebookresearch/home-robot/blob/main/docs/install_workstation.md

Activate the 'home-robot' conda environment and install the core `home_robot` and `home_robot_hw` packages using pip in editable mode.

```bash
conda activate home-robot
```

```bash
python -m pip install -e src/home_robot
```

```bash
python -m pip install -e src/home_robot_hw
```

--------------------------------

### Install Home Robot ROS Package

Source: https://github.com/facebookresearch/home-robot/blob/main/docs/install_robot.md

Installs the Home Robot stack on the robot, including system dependencies, Python requirements, and ROS integration.

```sh
# Make sure ROS can find python properly
sudo apt install python-is-python3 pybind11-dev

# Clone the repository
git clone https://github.com/facebookresearch/home-robot
HOME_ROBOT_ROOT=$(realpath home-robot)

# Install requirements
cd $HOME_ROBOT_ROOT/src/home_robot
pip install -r requirements.txt

# Install the core home_robot package
pip install -e .

# Install SLAM dependency on the robot
sudo apt install ros-noetic-hector-slam 

# Create a catkin workspace  - this will contain your code
# It may already exist. If so, skip this step
cd $HOME
mkdir -p catkin_ws/src
cd catkin_ws
catkin_init_workspace

# Stretch ROS library
cd $HOME/catkin_ws/src
git clone https://github.com/hello-robot/stretch_ros.git --branch noetic

# Set up the python package for ROS
ln -s $HOME_ROBOT_ROOT/src/home_robot_hw $HOME/catkin_ws/src/home_robot_hw

# Rebuild ROS to make sure paths are correct
cd $HOME/catkin_ws
catkin_make
```

--------------------------------

### Source ROS Workspace Setup Zsh

Source: https://github.com/facebookresearch/home-robot/blob/main/docs/install_robot.md

Adds the ROS workspace setup script to your zsh configuration for easy access to ROS launch files.

```sh
source $HOME/catkin_ws/devel/setup.zsh
```

--------------------------------

### Manual Setup for OVMM Perception

Source: https://context7.com/facebookresearch/home-robot/llms.txt

Manually set up the OVMM perception module by reading category maps, building vocabularies, and initializing the OvmmPerception wrapper. Supports Detic or Grounded-SAM modules.

```python
# Method 2: Manual setup with vocabulary management
from home_robot.perception.constants import RearrangeDETICCategories

# Read category mappings from file
obj_id_to_name, rec_id_to_name = read_category_map_file(
    "projects/habitat_ovmm/configs/category_map.json"
)

# Build vocabulary for semantic detection
vocabulary = build_vocab_from_category_map(obj_id_to_name, rec_id_to_name)

# Create perception module
perception = OvmmPerception(
    config=config,
    gpu_device_id=0,
    verbose=True,
    confidence_threshold=0.5,
    module="detic",  # or "grounded_sam"
)
```

--------------------------------

### Create and Activate Conda Environment

Source: https://github.com/facebookresearch/home-robot/blob/main/docs/spot.md

Creates a new Conda environment named 'home-robot' using the provided environment file. If mamba is not installed, install it first.

```bash
mamba env create -n home-robot -f src/home_robot_spot/env.yaml
```

```bash
conda install -c conda-forge mamba --yes
```

```bash
conda activate home-robot
```

--------------------------------

### Run Contact Graspnet Server

Source: https://github.com/facebookresearch/home-robot/blob/main/README.md

Navigate to the Contact Graspnet directory, activate its environment, and run the graspnet ROS server.

```bash
cd $HOME_ROBOT_ROOT/src/third_party/contact_graspnet
conda activate contact_graspnet_env
python contact_graspnet/graspnet_ros_server.py  --local_regions --filter_grasps
```

--------------------------------

### Run Detic Demo Script

Source: https://github.com/facebookresearch/home-robot/blob/main/docs/install_workstation.md

Execute the Detic demo script to verify the installation. This command downloads a sample image and runs Detic with specified configurations and a custom vocabulary.

```bash
wget https://web.eecs.umich.edu/~fouhey/fun/desk/desk.jpg
python demo.py --config-file configs/Detic_LCOCOI21k_CLIP_SwinB_896b32_4x_ft4x_max-size.yaml --input desk.jpg --output out2.jpg --vocabulary custom --custom_vocabulary headphone,webcam,paper,coffe --confidence-threshold 0.3 --opts MODEL.WEIGHTS models/Detic_LCOCOI21k_CLIP_SwinB_896b32_4x_ft4x_max-size.pth
```

--------------------------------

### Configure ROS Package Build

Source: https://github.com/facebookresearch/home-robot/blob/main/src/home_robot_hw/CMakeLists.txt

Initializes the project, finds required catkin components, and sets up message/service generation.

```cmake
cmake_minimum_required(VERSION 2.8.3)
project(home_robot_hw)

find_package(catkin REQUIRED COMPONENTS
  actionlib
  actionlib_msgs
  geometry_msgs
  nav_msgs
  control_msgs
  trajectory_msgs
  rospy
  std_msgs
  std_srvs
  sensor_msgs
  tf
  tf2
  stretch_core
)

catkin_python_setup()

add_service_files(
   FILES
   GraspRequest.srv
)

generate_messages(
   DEPENDENCIES
   geometry_msgs
   std_msgs
   sensor_msgs
)
```