### Setup Conda Environment

Source: https://github.com/facebookresearch/ego4d/blob/main/ego4d/features/README.md

Initialize the environment and install dependencies from requirements.txt.

```sh
conda create --name ego4d_public
conda activate ego4d_public
pip install -r requirements.txt
```

--------------------------------

### Initialize Training Components

Source: https://github.com/facebookresearch/ego4d/blob/main/notebooks/CLEP_Features_Tutorial_CVPR_Presentation.ipynb

Setup for device, model, data loader, and optimizer.

```python
device = "cuda"
model = ClipModel().to(device)
```

```python
dloader = DataLoader(dset, batch_size=128, num_workers=10, pin_memory=False)  # use workers > 1 for efficiency
```

```python
optim = torch.optim.AdamW(
    model.parameters(),
    lr=0.0001,
    betas=(0.98, 0.9),
    eps=1e-6,
)
```

--------------------------------

### Launch review server

Source: https://github.com/facebookresearch/ego4d/blob/main/viz/narrations/recipes/README.md

Starts the Mephisto review interface with the prepared input files.

```console
$ ./1_gather_ids.sh 5 | ./3_prepare_input.sh | ./4_review.sh

$ ./1_gather_ids.sh 5 | ./2_dl_videos.sh | ./3_prepare_input.sh | ./4_review.sh
```

--------------------------------

### Initialize Environment and Imports

Source: https://github.com/facebookresearch/ego4d/blob/main/ego4d/internal/notebooks/EgoExo_Dev_Release_Example.ipynb

Setup necessary libraries and configure matplotlib for data visualization.

```python
%matplotlib inline
import cv2
import matplotlib.pyplot as plt
from matplotlib import rcParams

rcParams['figure.figsize'] = 36,128

import os
import random
import json
import pandas as pd
from tqdm.auto import tqdm
from collections import defaultdict
```

--------------------------------

### Install Ego4d Library

Source: https://github.com/facebookresearch/ego4d/blob/main/README.md

Run this command from the root of the Ego4d repository to install the library.

```bash
pip install .
```

--------------------------------

### Setup Human Pose Environment

Source: https://github.com/facebookresearch/ego4d/blob/main/ego4d/internal/human_pose/README.md

Creates and activates a conda environment for the human pose estimation pipeline and installs necessary Python packages. Python 3.9 is required.

```bash
cd ego4d/internal/human_pose/
conda create -n human_pose39 python=3.9 -y
conda activate human_pose39
pip install -r requirements.txt
pip install --upgrade numpy
```

--------------------------------

### Conda Environment Setup

Source: https://github.com/facebookresearch/ego4d/blob/main/ego4d/internal/colmap/README.md

Commands to create and activate a Python 3.9 environment and install required dependencies.

```bash
conda create -n colmap python=3.9
conda activate colmap
pip install -r ego4d/internal/colmap/requirements.txt
```

--------------------------------

### Repository Setup Commands

Source: https://github.com/facebookresearch/ego4d/blob/main/notebooks/moments_cvpr/MomentsWorkshop.ipynb

Shell commands for cloning the repository, downloading data, and setting up the conda environment.

```bash
git clone https://github.com/facebookresearch/Ego4d.git
```

```bash
ego4d --output_directory="~/ego4d_data" --datasets annotations video_540ss --metadata
```

```bash
ego4d -y --output_directory ~/ego4d_data/ --datasets video_540ss --video_uid_file ~/path/to/moments_mini_val_uids.csv
```

```bash
conda env create -n moments_cvpr -–file conda-env.yaml
```

```bash
pip install git+https://github.com/facebookresearch/pytorchvideo.git
```

```bash
jupyter notebook MomentsWorkshop.ipynb
```

--------------------------------

### Logging Utility Configuration

Source: https://github.com/facebookresearch/ego4d/blob/main/notebooks/moments_cvpr/MomentsWorkshop.ipynb

Setup for the project logger.

```python
# Utils

log: logging.Logger = logging.getLogger("Ego4dMoments")
# To see more verbose logging, uncomment below (i.e. print all INFO -> console)
# log.setLevel(logging.INFO)
# sh = logging.StreamHandler(sys.stdout)
# sh.setFormatter(logging.Formatter("[%(asctime)s] %(levelname)s %(message)s \t[%(filename)s.%(funcName)s:%(lineno)d]", datefmt="%y%m%d %H:%M:%S"))
```

--------------------------------

### Execute clustering examples

Source: https://github.com/facebookresearch/ego4d/blob/main/notebooks/egoexo/EgoExo_Atomic_Descriptions_Tutorial.ipynb

Example calls to the cluster_task function for various tasks.

```python
cluster_task("Soccer", 20)
# cluster_task("Soccer", 1.5, "aggl")
```

```python
# cluster_task("Cooking", 20)
# cluster_task("Cooking", 1.5, "aggl")
```

```python
# cluster_task("Bike Repair", 30)
```

```python
# cluster_task("Health", 20)
```

```python
# cluster_task("Basketball", 1.25, "aggl")
```

--------------------------------

### Install Ego4D via PyPI

Source: https://github.com/facebookresearch/ego4d/blob/main/README.md

Install the ego4d package using pip. Requires Python 3.10 or higher.

```bash
pip install ego4d --upgrade
```

--------------------------------

### Create Sample Data File

Source: https://github.com/facebookresearch/ego4d/blob/main/viz/narrations/review/README.md

Example of a sample data file in CSV format. This file can be used to test the Mephisto review command.

```text
This is good text, row1
This is bad text, row2
```

--------------------------------

### Install MMCV with Optional Requirements

Source: https://github.com/facebookresearch/ego4d/blob/main/ego4d/internal/human_pose/README.md

Installs the MMCV library with optional dependencies. Ensure you are in the correct directory before running.

```bash
mkdir tp
pushd tp
git clone git@github.com:rawalkhirodkar/mmlab.git
popd

pushd tp/mmlab/mmcv
pip install -r requirements/optional.txt
MMCV_WITH_OPS=1 pip install -e . -v
popd

pushd tp/mmlab/mmpose
pip install .
popd

pushd tp/mmlab/mmdetection
pip install .
popd

pip install "torch>=2.0.0"
```

--------------------------------

### Run the Ego4D visualization script

Source: https://github.com/facebookresearch/ego4d/blob/main/viz/narrations/README.md

Executes the visualization interface from the repository root after installing Mephisto and the Ego4D CLI.

```bash
./run_viz.sh
```

--------------------------------

### Create React App with Mephisto Review Template

Source: https://github.com/facebookresearch/ego4d/blob/main/viz/narrations/review/README.md

Command to generate a new React application using the mephisto-review template. Ensure Mephisto is installed before running.

```bash
npx create-react-app my-review --template mephisto-review
```

--------------------------------

### Install Ego4D CLI

Source: https://github.com/facebookresearch/ego4d/blob/main/ego4d/cli/README.md

Install the Ego4D CLI tool using pip. This command is used for downloading Ego4D datasets.

```bash
pip install ego4d
```

--------------------------------

### Setup and Utility Functions for Ego4D Visualization

Source: https://github.com/facebookresearch/ego4d/blob/main/notebooks/annotation_visualization.ipynb

Imports necessary libraries and defines utility functions for data manipulation and visualization. Includes functions for video description, string list deserialization, and flattening pandas Series.

```python
# Setup
import sys
if os.path.abspath(".") not in sys.path: # Allow us to use util files in the same dir
    sys.path.insert(0, os.path.abspath("."))

import av
import collections
import csv
import cv2
import functools
import json
import logging
import math
import matplotlib.collections as mc
import matplotlib.image as mpimg
import matplotlib.patches as mpatches
import matplotlib.pyplot as plt
import matplotlib.colors as mcolors

import numpy as np
import pandas as pd
import random
import uuid
import warnings

from celluloid import Camera
from IPython.display import HTML
from iopath.common.file_io import PathManager
from itertools import groupby
from pprint import pprint
from nb_video_utils import _get_frames

%matplotlib inline
plt.rcParams["animation.html"] = "jshtml"

pathmgr = PathManager()

warnings.filterwarnings('ignore')

def vid_df_des(df):
    return f"#{len(df)} {df.duration_sec.sum()/60/60:.1f}h"
def vid_des(videos):
    return f"#{len(videos)} {sum((x.duration_sec for x in videos))/60/60:.1f}h"
def deserialize_str_list(list_):
    list_ = list_[1:-2]
    items = list_.split("', '")
    return list(map(lambda z: z.strip("'"), items))
def to_1D(series):
    return pd.Series([x for _list in series for x in _list])
```

--------------------------------

### Mephisto Review Server Running Message

Source: https://github.com/facebookresearch/ego4d/blob/main/viz/narrations/review/README.md

Example output indicating the Mephisto review server is running and accessible via a local URL. Press CTRL+C to quit.

```text
Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)
```

--------------------------------

### Setup Imports and Utilities for Ego4D Notebook

Source: https://github.com/facebookresearch/ego4d/blob/main/notebooks/annotation_verification.ipynb

Imports necessary libraries for video processing, data handling, and UI elements. Includes utility functions for string deserialization and data aggregation.

```python
# Setup
import sys
if os.path.abspath(".") not in sys.path: # Allow us to use util files in the same dir
    sys.path.insert(0, os.path.abspath("."))

import av
import base64
import boto3
import cv2
import hashlib
import json
import math
import matplotlib.image as mpimg
import matplotlib.pyplot as plt
import progressbar
import pandas as pd
import random
import uuid
import warnings

from botocore.exceptions import NoCredentialsError
from enum import Enum, auto
from IPython.display import HTML, display
from ipywidgets.widgets import HBox, Layout
from ipywidgets.widgets import VBox
import ipywidgets as widgets
from iopath.common.file_io import PathManager
from nb_video_utils import _get_frames
from typing import Callable

%matplotlib inline
plt.rcParams["animation.html"] = "jshtml"

pathmgr = PathManager()

warnings.filterwarnings('ignore')

def vid_df_des(df):
    return f"#{len(df)} {df.duration_sec.sum()/60/60:.1f}h"
def vid_des(videos):
    return f"#{len(videos)} {sum((x.duration_sec for x in videos))/60/60:.1f}h"
def deserialize_str_list(list_):
    list_ = list_[1:-2]
    items = list_.split("', '")
    return list(map(lambda z: z.strip("'"'), items))
def to_1D(series):
    return pd.Series([x for _list in series for x in _list])

config = [
    {
        'section': 'Annotations Assessment (select one):',
        'options': [{
            'annotations_correct': {'btn_label': 'Correct', 'btn_color':'#47a84b'},
            'annotations_incorrect': {'btn_label':'Incorrect', 'btn_color':'#fb8e26'},
        }]
    }
]
option_base_color = "#b7c2ce"
label_frames_per_video = 10
fho_s3_path = "s3://ego4d-consortium-sharing/annotations/fho_220208.json"
```

--------------------------------

### Initialize PyTorch Lightning Trainer

Source: https://github.com/facebookresearch/ego4d/blob/main/notebooks/moments_cvpr/MomentsWorkshop.ipynb

Configure the PyTorch Lightning Trainer with parameters like number of nodes, GPUs, callbacks, and epoch/step limits. Ensure DDP strategy is uncommented for multi-GPU/node setups, excluding notebooks.

```python
trainer = pl.Trainer(
    num_nodes=inputs.num_nodes,
    gpus=inputs.num_gpus_per_node,
    # If using multiple GPU/node, you'll want to use DDP (but not in a notebook)
    # strategy=pl.strategies.DDPStrategy(find_unused_parameters=False),
    # logger=pl_logger,
    callbacks=[checkpoint],
    max_epochs=inputs.max_epochs,
    max_steps=inputs.max_steps,
    resume_from_checkpoint=inputs.resume_from_checkpoint,
    accumulate_grad_batches=inputs.accumulate_grad_batches,
)
```

--------------------------------

### Create Utility UI

Source: https://github.com/facebookresearch/ego4d/blob/main/notebooks/annotation_verification.ipynb

Defines control buttons for the labeling workflow, including start, submit, and download actions.

```python
def get_utility_ui(ui_elements, outputs, labeling_state):
    begin_btn = widgets.Button(description="Start Labeling")
    submit_btn = widgets.Button(description="Label Frame")
    download_btn = DownloadButton(filename='ego4d_annotation_labels.csv', contents=lambda: labeling_state['decisions_df'].to_csv(), description='Download CSV')
    skip_frame_btn = widgets.Button(description="Skip Frame")
    skip_video_btn = widgets.Button(description="Skip Video")
    
    utility_btns = [begin_btn, submit_btn, download_btn, skip_frame_btn, skip_video_btn]
    
    def on_begin_btn_clicked(b):
        if ui_elements['userid_input'].value == '':
            with outputs['logs']:
                print("Please enter your userid first")
        else:
            b.layout.display = 'none'
            ui_elements['userid_input'].disabled = True
            load_next_frame(ui_elements, outputs, labeling_state)
    
    def on_submit_btn_clicked(b):
        tagged_btns = [btn for btn in ui_elements['buttons'] if hasattr(btn, 'tag')]
        tagged_textboxes = [textbox for textbox in ui_elements['textboxes'] if hasattr(textbox, 'tag')]
        tagged_textareas = [textarea for textarea in ui_elements['textareas'] if hasattr(textarea, 'tag')]
        info_to_log = {
            **{btn.tag: btn.selected for btn in tagged_btns},
            **{textbox.tag: textbox.value for textbox in tagged_textboxes},
            **{textarea.tag: textarea.value for textarea in tagged_textareas},
            **labeling_state['render_identification_info']
        }
        labeling_state['decisions_df'] = labeling_state['decisions_df'].append(info_to_log, ignore_index=True)
        labeling_state['video_frames_labeled'] += 1
```

--------------------------------

### Configure COLMAP Settings

Source: https://github.com/facebookresearch/ego4d/blob/main/notebooks/COLMAP.ipynb

Instantiate a ColmapConfig object to specify input sources, synchronization options, frame sampling rates, camera models, and output directories. This example shows configuration for using metadata path and synchronized exo-centric views.

```python
config = ColmapConfig(
    # if using a metadata path, please
    # set video_source and take_id to None
    # as this will be derived for you
    # COMMENT/UNCOMMENT BEGIN
    in_metadata_path="s3://ego4d-consortium-sharing/internal/egoexo_pilot/unc/T1/metadata.json",
    in_videos=None,
    video_source=None,
    take_id=None,
    
    # whether we are taking synchronized exo-centric frames
    sync_exo_views=True,
    # where is the walkaround in the VRS file
    aria_walkthrough_start_sec=None,
    aria_walkthrough_end_sec=None,
    # COMMENT/UNCOMMENT END


    # FOR TESTING ONLY: uncomment the above to use specific paths.
    # NOTE: you cannot enable sync_exo_views if you are using specific paths
    # COMMENT/UNCOMMENT BEGIN
#     in_metadata_path=None,
#     video_source="penn",
#     take_id="0303_Violin_2",
#     in_videos={
#         "aria01": "s3://ego4d-penn/data/0303_Violin_2/ego/aria/c2e4b041-4e68-4b75-8338-f8c625429e75.vrs",
#         "cam01": "s3://ego4d-penn/data/0303_Violin_2/exo/gp01/GX010190.MP4",
#         "cam02": "s3://ego4d-penn/data/0303_Violin_2/exo/gp02/GX010175.MP4",
#         "cam03": "s3://ego4d-penn/data/0303_Violin_2/exo/gp03/GX010012.MP4",
#         "cam04": "s3://ego4d-penn/data/0303_Violin_2/exo/gp04/GX010195.MP4",
#         "mobile": "s3://ego4d-penn/data/0303_Violin_2/exo/mobile/GX010020.MP4",
#     },
#     sync_exo_views=False,
#     aria_walkthrough_start_sec=30.0,
#     aria_walkthrough_end_sec=200.0,
    # COMMENT/UNCOMMENT END

    output_dir=OUTPUT_DIR,  # where to save data

    # there are three rot_mode's:
    # - 0 => perform no rotation
    # - 1 => rotate aria to exo
    # - 2 => rotate exo to aria
    rot_mode=1,
    
    # refer to https://colmap.github.io/cameras.html
    camera_model="OPENCV_FISHEYE",
    
    # the inverse of the number of frames per second to sample
    # the mobile walkaround and aria walkaround
    frame_rate=0.25,
    
    # specific mobile frames to use
    # if None then all frames are considered using `frame_rate`
    mobile_frames=None,
    
    # whether to include the aria walkaround
    include_aria=True,
    # if aria_use_sync_info is True then aria_last_walkthrough_sec will be assigned 
    # based off timesync data
    aria_use_sync_info=False,
    # specific frames for the walkaround 
    # if not provided will use all frames in aria will be used (subsampled using `frame_rate`)
    aria_frames=None,
    
    # where to sample the exo videos from
    exo_from_frame=700,
    exo_to_frame=720,
    # specific frames for each exo video
    # if one is provided below, all must be provided and exo_from_frame/exo_to_frame must be None
    exo_frames=None,
    
    # the name of the configuration (a relative path/directory where all frames and config is saved to)
    # if none a name will be automatically constructed from the above configuration
    name=None,
    
    # misc
    aria_fps=30,      # aria is assumed to be (approx) 30fps by default
    exo_fps=None,     # if none, this will be determined from the video file
    mobile_fps=None,  # if none, this will be determined from the video file
    run_colmap=False, # whether we want to run colmap after
    colmap_bin=COLMAP_BIN, # where colmap is located, if None a default will be provided
    vrs_bin=VRS_BIN,       # where vrs is located, if None a default will be provided
    download_video_files=True, # whether we stream from S3 or download videos (changeme if timeout occurs)
    force_download=False, # if download_video_files is True, force_download will force a re-download of video files
)
```

--------------------------------

### Initialize Rerun and Load Device Calibration

Source: https://github.com/facebookresearch/ego4d/blob/main/notebooks/egoexo/EgoExo_Aria_Data_Tutorial.ipynb

Initializes the Rerun recording and loads the device calibration from a VRS file. This setup is necessary for visualizing Aria Glasses data.

```python
from projectaria_tools.core import data_provider
from projectaria_tools.utils.rerun_helpers import AriaGlassesOutline, ToTransform3D

###
# We are using here the projectaria_tools API for:
# - retrieving the DeviceCalibration and the POSE of each sensor
# - we are then plotting those POSE onto the Aria glasses outline
###

##
# Retrieve device calibration and plot sensors locations, orientations
vrs_file_path = os.path.join(ego_exo_project_path, 'aria01.vrs')
print(vrs_file_path)
assert os.path.exists(vrs_file_path), "We are not finding the required vrs file"

vrs_data_provider = data_provider.create_vrs_data_provider(vrs_file_path)
if not vrs_data_provider:
    print("Couldn't create data vrs_data_provider from vrs file")
    exit(1)

device_calibration = vrs_data_provider.get_device_calibration()

# Init rerun api
rr.init("Aria Glasses")
rec = rr.memory_recording()
```

--------------------------------

### Initialize Rerun API and Create Memory Recording

Source: https://github.com/facebookresearch/ego4d/blob/main/notebooks/egoexo/EgoExo_Aria_Data_Tutorial.ipynb

Initializes the Rerun API for visualization and creates a memory recording object. This is a common setup step for logging data.

```python
rr.init("Aria - Image undistortion")
rec = rr.memory_recording()
```

--------------------------------

### Fit Model with DataModule

Source: https://github.com/facebookresearch/ego4d/blob/main/notebooks/moments_cvpr/MomentsWorkshop.ipynb

Initiates the training process for the model using the provided data module. This command starts the training loop defined in the PyTorch Lightning model.

```python
trainer.fit(model, datamodule=data)
```

--------------------------------

### Run Human Pose Estimation Pipeline - Preprocess Mode

Source: https://github.com/facebookresearch/ego4d/blob/main/ego4d/internal/human_pose/README.md

Executes the human pose estimation pipeline in preprocess mode. Ensure you have completed the setup steps first. The repository root directory must be specified.

```bash
python3 ego4d/internal/human_pose/main.py --config-name unc_T1 mode=preprocess repo_root_dir=$PWD
```

--------------------------------

### Initialize Labeling State and UI Layout

Source: https://github.com/facebookresearch/ego4d/blob/main/notebooks/annotation_verification.ipynb

Sets up the initial state dictionary, UI element containers, and output widgets, then renders the interface.

```python
labeling_state = {
    'video_uid': None,
    'video_frames_labeled': 0,
#     'annotation_type': random.sample(list(AnnotationType), 1)[0],
    'annotation_type': AnnotationType.VQ,
    'decisions_df': pd.DataFrame([]),
    'render_identification_info': None
}

ui_elements = {
    'userid_input': None,
    'buttons': [],
    'textareas': [],
    'textboxes': [],
    'dropdowns': [],
}

outputs = {
    'downloads': widgets.Output(layout=Layout(height="auto")),
    'frames': widgets.Output(layout=Layout(height="auto")),
    'logs': widgets.Output(layout=Layout(height="auto"))
}

ui = [
    outputs['downloads'], # Render video download logging info
    get_selections_ui(ui_elements, labeling_state), # Render main selections row
    outputs['logs'], # Render logs
    outputs['frames'], # Render video frames
    get_options_ui(ui_elements), # Render options
    get_notes_textarea_ui(ui_elements), # Render notes textarea
    get_utility_ui(ui_elements, outputs, labeling_state) # Render utility row
]

# Initially disable UI
for elem in [*ui_elements['buttons'], *ui_elements['textareas'], *ui_elements['textboxes'], *ui_elements['dropdowns']]:
    elem.disabled = not hasattr(elem, 'enabled_at_start')

display(*ui)
```

--------------------------------

### Display EgoExo help options

Source: https://github.com/facebookresearch/ego4d/blob/main/ego4d/egoexo/download/README.md

View available command-line arguments and options for the downloader.

```bash
egoexo --help
```

--------------------------------

### Initialize Ego4D Dataset Configuration

Source: https://github.com/facebookresearch/ego4d/blob/main/notebooks/moments_cvpr/MomentsWorkshop.ipynb

Sets up video transformation parameters and loads metadata and annotation files using path managers.

```python
        self.audio_transform_type = audio_transform_type
        assert (label_id_map is not None) ^ (
            label_id_map_path is not None
        ), f"Either label_id_map or label_id_map_path required ({label_id_map_path} / {label_id_map})"  # noqa

        self.video_means = (0.45, 0.45, 0.45)
        self.video_stds = (0.225, 0.225, 0.225)
        self.video_crop_size = 224
        self.video_min_short_side_scale = 256
        self.video_max_short_side_scale = 320

        try:
            with g_pathmgr.open(metadata_path, "r") as f:
                metadata = json.load(f)
        except Exception:
            raise FileNotFoundError(
                f"{metadata_path} must be a valid metadata json for Ego4D"
            )

        self.video_metadata_map: Dict[str, Any] = {
            x["video_uid"]: x for x in metadata["videos"]
        }

        if not g_pathmgr.isfile(annotation_path):
            raise FileNotFoundError(f"{annotation_path} not found.")

        try:
            with g_pathmgr.open(annotation_path, "r") as f:
                moments_annotations = json.load(f)
        except Exception:
            raise FileNotFoundError(f"{annotation_path} must be json for Ego4D dataset")

        self.label_name_id_map: Dict[str, int]
        if label_id_map:
            self.label_name_id_map = label_id_map
        else:
            self.label_name_id_map = get_label_id_map(label_id_map_path)
            assert self.label_name_id_map

        self.num_classes: int = len(self.label_name_id_map)
        log.info(f"Label Classes: {self.num_classes}")

        self.imu_data: Optional[Ego4dImuDataBase] = None
        if imu:
            assert imu_path, "imu_path not provided"
            self.imu_data = Ego4dImuData(imu_path)
```

--------------------------------

### Install Project Aria Tools and Dependencies

Source: https://github.com/facebookresearch/ego4d/blob/main/notebooks/egoexo/tutorials/gaze_tutorial.ipynb

Install the required Python package for Project Aria Tools and OpenCV to enable data processing.

```bash
# install projectaria_tools
!git clone https://github.com/facebookresearch/projectaria_tools.git -b 1.5.5
!python3 -m pip install --upgrade pip
!python3 -m pip install projectaria-tools'[all]' --quiet

# install opencv-python
!pip install opencv-python --quiet
```

--------------------------------

### Initialize the review interface

Source: https://github.com/facebookresearch/ego4d/blob/main/viz/narrations/README.md

Creates the review interface directory using the Mephisto review template via create-react-app.

```bash
$ npx create-react-app review --template mephisto-review
```

--------------------------------

### Set up Dataset Path and Take Name

Source: https://github.com/facebookresearch/ego4d/blob/main/notebooks/egoexo/tutorials/gaze_tutorial.ipynb

Configure the root directory for the Ego-Exo4D dataset and specify the take name to load. Ensure the path is valid and exists.

```python
ego_exo_root = '/datasets01/egoexo4d/v2/' # Replace with your cli's download directory for Ego-Exo4D
take_name = 'fair_cooking_05_2'

ego_exo_project_path = os.path.join(ego_exo_root, 'takes', take_name)
assert os.path.exists(ego_exo_project_path), "Please do update your path to a valid EgoExo sequence folder."

use_general_gaze = True
if not use_general_gaze:
    assert os.path.exists(os.path.join(ego_exo_project_path, "eye_gaze", "personalized_eye_gaze.csv")),
    "personalized eye gaze not exists for this take"


# find aria number, for example, aria03.vrs will have aria_number = 3
pattern = re.compile(r'aria0(\d+)\.vrs')

for root, _, files in os.walk(ego_exo_project_path):
    for file in files:
        match = pattern.match(file)
        if match:
            aria_number = int(match.group(1))
```

--------------------------------

### Load CUDA and CUDNN Modules on FAIR Cluster

Source: https://github.com/facebookresearch/ego4d/blob/main/ego4d/internal/human_pose/README.md

Loads specific CUDA and CUDNN modules required for installation on the FAIR cluster. This should be done before installing dependencies.

```bash
module load cuda/11.2 cudnn/v8.1.1.33-cuda.11.0
```

--------------------------------

### Initialize Environment and Paths

Source: https://github.com/facebookresearch/ego4d/blob/main/notebooks/CLEP_Features_Tutorial_CVPR_Presentation.ipynb

Sets up necessary imports, environment variables, and file paths for processing Ego4D datasets.

```python
import random
random.seed(1234)

import os
os.environ["TOKENIZERS_PARALLELISM"] = "false"

import time
import json
import os
import math
from typing import List

import torch
import h5py
import numpy as np
from torch.nn import functional as F
from tqdm.auto import tqdm
from sentence_transformers import SentenceTransformer
import torch.nn as nn
from torch.utils.data import DataLoader

NARRATION_JSON_PATH = "/datasets01/ego4d_track2/v1/annotations/narration.json"
NARR_OUT_DIR = "/tmp/narrs/"
NARR_META_PATH = os.path.join(NARR_OUT_DIR, "meta.pt")
FEATURE_DIR = "/checkpoint/miguelmartin/ego4d_track2_features/full_scale/omnivore_video_swinL"
FEATURES_PER_SECOND = 30 / 16
FEATURE_DIM = 1536

VIDEO_UIDS = [x.split(".pt")[0] for x in os.listdir(FEATURE_DIR) if "yaml" not in x]
random.shuffle(VIDEO_UIDS)

EXAMPLE_VIDEO_UID = VIDEO_UIDS[0]
VIDEO_UIDS_EXAMPLE_SET = set(VIDEO_UIDS[0:100])

os.makedirs(NARR_OUT_DIR, exist_ok=True)
```

--------------------------------

### Prepare input files

Source: https://github.com/facebookresearch/ego4d/blob/main/viz/narrations/recipes/README.md

Generates annotation input files for the review server.

```console
$ echo 000786a7-3f9d-4fe6-bfb3-045b368f7d44 ... | ./3_prepare_input.sh
{ "info": ... }
{ "info": ... }
{ "info": ... }

$ ./3_prepare_input.sh ALL
```

--------------------------------

### Validate S3 Metadata

Source: https://github.com/facebookresearch/ego4d/blob/main/ego4d/internal/validation/README.md

Examples of running the validation tool against specific S3 buckets.

```bash
python ego4d/internal/validation/cli.py -i s3://ego4d-penn/egoexo/metadata_v2
python ego4d/internal/validation/cli.py -i s3://ego4d-utokyo/egoexo/metadata_v1
```

--------------------------------

### Download dataset to directory

Source: https://github.com/facebookresearch/ego4d/blob/main/ego4d/egoexo/download/README.md

Initiate a download of the recommended dataset set to a specified output directory.

```bash
egoexo -o <out-dir>
```

--------------------------------

### Set Up Ego4D CLI Output Directory and Paths

Source: https://github.com/facebookresearch/ego4d/blob/main/notebooks/transform_annotations.ipynb

Configure the output directory for the Ego4D CLI and define paths for metadata and annotations. Asserts verify the existence of these essential files.

```python
# Set your options here

import os

CLI_OUTPUT_DIR = "/Users/<userid>/ego4d" # Replace with the full path to the --output_directory you pass to the cli
VERSION = "v1"

METADATA_PATH = os.path.join(CLI_OUTPUT_DIR, "ego4d.json")
ANNOTATIONS_PATH = os.path.join(CLI_OUTPUT_DIR, VERSION, "annotations")

assert os.path.exists(METADATA_PATH), f"Metadata doesn't exist at {METADATA_PATH}. Is the CLI_OUTPUT_DIR right? Do you satisfy the pre-requisites?"
assert os.path.exists(os.path.join(ANNOTATIONS_PATH, "manifest.csv")),
    "Annotation metadata doesn't exist. Did you download it with the CLI?"

```

--------------------------------

### Get Stream Labels from StreamId

Source: https://github.com/facebookresearch/ego4d/blob/main/notebooks/egoexo/tutorials/gaze_tutorial.ipynb

Map specific sensor StreamIds to their corresponding labels.

```python
rgb_stream_id = StreamId("214-1")
slam_left_stream_id = StreamId("1201-1")
slam_right_stream_id = StreamId("1201-2")
rgb_stream_label = vrs_data_provider.get_label_from_stream_id(rgb_stream_id)
slam_left_stream_label = vrs_data_provider.get_label_from_stream_id(slam_left_stream_id)
slam_right_stream_label = vrs_data_provider.get_label_from_stream_id(slam_right_stream_id)
```

--------------------------------

### COLMAP Validation Output Format

Source: https://github.com/facebookresearch/ego4d/blob/main/ego4d/internal/colmap/README.md

Example of the text output generated by the COLMAP validation step.

```text
Cameras: 6
Images: 278
Registered images: 278
Points: 45824
Observations: 750872
Mean track length: 16.385999
Mean observations per image: 2700.978417
Mean reprojection error: 0.744607px
```

--------------------------------

### Get Number of Visual Objects

Source: https://github.com/facebookresearch/ego4d/blob/main/ego4d/internal/notebooks/EgoExo_Dev_Release_Example.ipynb

Returns the count of visual objects present in the egoexo data.

```python
len(egoexo["visual_objects"])
```

--------------------------------

### Initialize Video Readers for Ego4D Takes

Source: https://github.com/facebookresearch/ego4d/blob/main/ego4d/internal/notebooks/EgoExo_Dev_Release_Example.ipynb

Sets up VideoReader objects for each video stream within a take. Ensure RELEASE_DIR and VideoReader are correctly defined and imported.

```python
videos = {}
for k, temp in take["frame_aligned_videos"].items():
    for stream_id, v in temp.items():
        path = v["relative_path"]
        local_path = os.path.join(RELEASE_DIR, "takes", take["root_dir"], f"{v['relative_path']}")
        print(path, local_path)
        videos[(k, stream_id)] = VideoReader(
            local_path,
            resize=None,
            mean=None,
            frame_window_size=1,
            stride=1,
            gpu_idx=gpu_idx,
        )

for k, v in videos.items():
    print(f"{k}: {len(v)}")
n_frames = len(videos[k])
```

--------------------------------

### Get Local Video Path

Source: https://github.com/facebookresearch/ego4d/blob/main/notebooks/annotation_verification.ipynb

Constructs the local file system path for a given video UID.

```python
def get_local_video_path(video_uid):
    return os.path.join(CLI_OUTPUT_DIR, VERSION, 'full_scale', video_uid)
```

--------------------------------

### Initialize MPS Data Provider for Reprojection

Source: https://github.com/facebookresearch/ego4d/blob/main/notebooks/egoexo/EgoExo_Aria_Data_Tutorial.ipynb

Sets up the MpsDataProvider and Rerun environment for exocentric camera reprojection.

```python
import math
import torchvision # to read video
from projectaria_tools.core.calibration import CameraCalibration, KANNALA_BRANDT_K3 # Aria/GoPro Camera Calibration
from projectaria_tools.core import mps

## Configure the MpsDataProvider (interface used to retrieve Trajectory data)
mps_data_paths_provider = mps.MpsDataPathsProvider(ego_exo_project_path)
mps_data_paths = mps_data_paths_provider.get_data_paths()
mps_data_provider = mps.MpsDataProvider(mps_data_paths)

# Init rerun api
rr.init("Ego_Exo - image reprojection")
rec = rr.memory_recording()
```

--------------------------------

### labeled_video_dataset Helper Function

Source: https://github.com/facebookresearch/ego4d/blob/main/notebooks/moments_cvpr/MomentsWorkshop.ipynb

A helper function to create a LabeledVideoDataset object, simplifying the setup for Ucf101 and Kinetics datasets.

```APIDOC
## labeled_video_dataset

### Description
A helper function to create ``LabeledVideoDataset`` object for Ucf101 and Kinetics datasets.

### Method
```python
def labeled_video_dataset(
    data_path: str,
    clip_sampler: ClipSampler,
    video_sampler: Type[torch.utils.data.Sampler] = torch.utils.data.RandomSampler,
    transform: Optional[Callable[[Dict[str, Any]], Dict[str, Any]]] = None,
    video_path_prefix: str = "",
    decode_audio: bool = True,
    decoder: str = "pyav",
) -> LabeledVideoDataset:
```

### Parameters
#### Arguments
- **data_path** (str) - Required - Path to the data. Can be a file path or a directory.
- **clip_sampler** (ClipSampler) - Required - Defines how clips should be sampled from each video.
- **video_sampler** (Type[torch.utils.data.Sampler]) - Optional - Sampler for the internal video container. Defaults to ``torch.utils.data.RandomSampler``.
- **transform** (Callable) - Optional - A callable evaluated on the clip output before the clip is returned for preprocessing and augmentations.
- **video_path_prefix** (str) - Optional - Path to the root directory with the videos. Defaults to "".
- **decode_audio** (bool) - Optional - If True, also decode audio from video. Defaults to True.
- **decoder** (str) - Optional - Defines what type of decoder used to decode a video. Defaults to "pyav".

### Returns
- **LabeledVideoDataset** - An instance of the LabeledVideoDataset.
```

--------------------------------

### Define Annotation and Camera Pose Directories

Source: https://github.com/facebookresearch/ego4d/blob/main/notebooks/egoexo/Ego-Exo4D_EgoPose_Tutorial.ipynb

Sets up the directory paths for ego pose annotations (body or hand) and camera pose information. Includes a check to ensure the annotation type is valid.

```python
assert annotation_type in ["body", "hand"]
egopose_ann_dir = os.path.join(
    annotation_dir, f"ego_pose/train/{annotation_type}/annotation"
)
# egopose_pseudo_dir = os.path.join(annotation_dir, f"ego_pose/train/{annotation_type}/automatic")
egopose_camera_dir = os.path.join(annotation_dir, f"ego_pose/train/camera_pose/")
```

--------------------------------

### Convert Frame Index to Time

Source: https://github.com/facebookresearch/ego4d/blob/main/notebooks/Feature_Visualization_with_TSNE.ipynb

Calculates the start and end time in seconds for a given frame index range.

```python
def frame_idx_to_time(start, end, uid):
    t_start_frames = start*FEATURE_STRIDE
    t_end_frames = end*FEATURE_STRIDE+FEATURE_WINDOW_SIZE
    t_s = t_start_frames / FPS
    t_e = t_end_frames / FPS
    
    meta = meta_for_features[uid]["video_metadata"]
    vid_dur = meta["video_duration_sec"] + meta["video_start_sec"] 
    if t_e > vid_dur:
        return t_s, vid_dur
    return t_s, t_e
```

--------------------------------

### Get Distortion Coefficients and Intrinsics

Source: https://github.com/facebookresearch/ego4d/blob/main/notebooks/egoexo/Ego-Exo4D_EgoPose_Tutorial.ipynb

Extracts distortion coefficients and intrinsic camera parameters from a raw camera calibration dictionary.

```python
def get_distortion_and_intrinsics(_raw_camera):
    intrinsics = np.array(
        [
            [_raw_camera["intrinsics_0"], 0, _raw_camera["intrinsics_2"]],
            [0, _raw_camera["intrinsics_1"], _raw_camera["intrinsics_3"]],
            [0, 0, 1],
        ]
    )
    distortion_coeffs = np.array(
        [
            _raw_camera["intrinsics_4"],
            _raw_camera["intrinsics_5"],
            _raw_camera["intrinsics_6"],
            _raw_camera["intrinsics_7"],
        ]
    )
    return distortion_coeffs, intrinsics
```

--------------------------------

### Sample and Print Video Details

Source: https://github.com/facebookresearch/ego4d/blob/main/notebooks/annotation_visualization.ipynb

Samples a single video from the Ego4D dataset and prints its details. Ensure 'videos_df' is loaded and 'vq_ann_video_uids' is available.

```python
# Sample one video
vq_video_uid = random.sample(vq_ann_video_uids, 1)[0]
vq_video = videos_df[videos_df.video_uid == vq_video_uid].iloc[0]
print(f"Sampled Video: {vq_video}")
```

--------------------------------

### Get Visualization Image

Source: https://github.com/facebookresearch/ego4d/blob/main/notebooks/egoexo/Ego-Exo4D_EgoPose_Tutorial.ipynb

Generates a visualization overlay on a PIL image with keypoints and skeletons. Handles both body and hand annotations.

```python
def get_viz(pil_img, keypoints_map, ann, skeleton, pose_kpt_color, annot_type='body', is_aria=False):
    pts = get_coords(ann)
    ratio = 1
    if is_aria:
        ratio = 0.5           
    
    all_pts = list()
    for index, keypoints in enumerate(keypoints_map):
        kpname = keypoints["label"].lower()
        
        if kpname in pts:
            x, y = pts[kpname][0], pts[kpname][1]
            all_pts.append((x, y))                
        else:
            all_pts.append(())

    if annot_type=='body':
        draw_skeleton(pil_img, all_pts, skeleton)
    else:
        draw_skeleton_hands(pil_img, all_pts, skeleton, ratio)

    for index, keypoints in enumerate(keypoints_map):
        kpname = keypoints["label"].lower()
        if kpname in pts:
            x, y, pt_type = pts[kpname][0], pts[kpname][1], pts[kpname][2]
            color = tuple(pose_kpt_color[index])
            if pt_type == 1:
                if annot_type=='body':
                    draw_circle(pil_img, x, y, color)
                else:
                    draw_circle_hands(pil_img, x, y, color, ratio)
            else:
                if annot_type=='body':
                    draw_cross(pil_img, x, y, color)
                else:
                    draw_cross_hands(pil_img, x, y, color, ratio)
                
        else:
            pass
    return pil_img
```

--------------------------------

### Get Specific Frame from Video

Source: https://github.com/facebookresearch/ego4d/blob/main/notebooks/egoexo/Ego-Exo4D_EgoPose_Tutorial.ipynb

Extracts a specific frame from a video file using PyAV. Converts the frame to a PIL image.

```python
def get_frame(video_local_path, frame_idx):
    container = av.open(video_local_path)
    print(container.streams.video[0].frames)
    frame_count = 0
    for frame in tqdm(container.decode(video=0)):
        if frame_count == frame_idx:
            input_img = np.array(frame.to_image())
            pil_img = Image.fromarray(input_img)
            print(frame_count)
            return pil_img
        frame_count+=1
```

--------------------------------

### Create Options UI

Source: https://github.com/facebookresearch/ego4d/blob/main/notebooks/annotation_verification.ipynb

Constructs a grid of toggleable buttons based on a configuration object.

```python
def get_options_ui(ui_elements):
    btn_layout = Layout(width="100%", height="100px")
    btn_style = dict(font_weight='bold')
    
    def on_option_clicked(b):
        b.selected = not b.selected
        if b.selected:
            b.style.button_color = b.selected_color
        else:
            b.style.button_color = option_base_color
    
    ui_sections = []
    for section in config:
        ui_sections += [widgets.Label(value=section['section'])]
        buttons = []
        for btn_config in section['options']:
            for (decision_string, opt) in btn_config.items():
                newBtn = widgets.Button(description=opt['btn_label'], layout=btn_layout, style=btn_style, button_style='success')
                newBtn.style.button_color = option_base_color
                newBtn.reset_color = option_base_color
                newBtn.tag = decision_string
                newBtn.selected = False
                newBtn.selected_color = opt['btn_color']
                newBtn.on_click(on_option_clicked)
                buttons += [newBtn]
        btn_row = HBox()
        btn_row.children = buttons
        ui_sections += [btn_row]
        ui_elements['buttons'] += buttons
    
    grid = VBox(layout=Layout(width="95%"))
    grid.children = ui_sections
    return grid
```

--------------------------------

### Get Specific Frames for Multiple Streams

Source: https://github.com/facebookresearch/ego4d/blob/main/ego4d/internal/notebooks/EgoExo_Dev_Release_Example.ipynb

Calls the get_frames function to retrieve a single frame (specified by frame_idx) for multiple video streams.

```python
frames = get_frames({
    ("aria01", "rgb"): [frame_idx],
    ("cam01", "0"): [frame_idx],
    ("cam02", "0"): [frame_idx],
    ("cam03", "0"): [frame_idx],
    ("cam04", "0"): [frame_idx],
})
```

--------------------------------

### Get Video UIDs from File

Source: https://github.com/facebookresearch/ego4d/blob/main/notebooks/moments_cvpr/MomentsWorkshop.ipynb

Reads a file containing video UIDs, each on a new line, and returns them as a set of strings. Filters out empty lines.

```python
def get_video_uids(path):
    with g_pathmgr.open(path, "r") as f:
        return set([x for x in f.read().split('\n') if x])
```

--------------------------------

### Plot Single Track Segments

Source: https://github.com/facebookresearch/ego4d/blob/main/notebooks/annotation_visualization.ipynb

Renders a plot visualizing all segments from a single track. Segments are defined by start and end keys. Uses `matplotlib.collections.LineCollection` for plotting.

```python
def plot_segments(segments, start_key, end_key):
    ordered_segs = sorted(segments, key=lambda x: x[start_key])
    lines = [[(x[start_key], i), (x[end_key], i)] for i, x in enumerate(ordered_segs)]

    lc = mc.LineCollection(lines, linewidths=2)
    fig, ax = plt.subplots(figsize=(30, 10))
    ax.add_collection(lc)
    ax.autoscale()
    ax.set_xlabel('Frame', fontsize=15)
    ax.set_ylabel('Segment', fontsize=15)
    start, end = ax.get_xlim()
    stepsize = (end-start)/30
    ax.xaxis.set_ticks(np.arange(start, end, stepsize))
    plt.show()
```

--------------------------------

### Initialize PyAvReader for Video Processing

Source: https://context7.com/facebookresearch/ego4d/llms.txt

Initialize PyAvReader with preprocessing transforms for video analysis. Specify parameters like resize, crop, normalization, frame window size, stride, and output format. Use gpu_idx=-1 for CPU processing.

```python
from ego4d.research.readers import PyAvReader
import torch

# Initialize reader with preprocessing transforms
reader = PyAvReader(
    path="/path/to/video.mp4",
    resize=256,                    # Short side scale
    crop=224,                      # Center crop size
    mean=torch.tensor([0.45, 0.45, 0.45]),   # Normalization mean
    std=torch.tensor([0.225, 0.225, 0.225]), # Normalization std
    frame_window_size=32,          # Number of frames per clip
    stride=16,                     # Stride between clips
    gpu_idx=-1,                    # CPU processing
    axis_order="cthw",             # Output tensor format: channels, time, height, width
    uint8_scale=False,             # Output as float [0,1]
)

# Get number of clips available
num_clips = len(reader)
print(f"Video has {num_clips} clips")

# Read a specific clip by index
clip_data = reader[0]
video_tensor = clip_data["video"]        # Shape: (C, T, H, W)
frame_start = clip_data["frame_start_idx"]
frame_end = clip_data["frame_end_idx"]
print(f"Clip shape: {video_tensor.shape}")
print(f"Frames {frame_start} to {frame_end}")

# Iterate through all clips
for idx in range(len(reader)):
    clip = reader[idx]
    # Process clip["video"] with your model
    pass

# Get video metadata
from ego4d.research.readers import get_video_meta
meta = get_video_meta("/path/to/video.mp4")
print(f"Total frames: {len(meta['all_pts'])}")
print(f"Codec: {meta['codec']}")
print(f"Resolution: {meta['width']}x{meta['height']}")
```

--------------------------------

### Calculate Start and End Indices

Source: https://github.com/facebookresearch/ego4d/blob/main/notebooks/CLEP_Features_Tutorial_CVPR_Presentation.ipynb

Utility function to determine the feature range [t1, t2] based on frames per second and total feature count.

```python
def get_start_end_idx(t1: float, t2: float, feature_per_sec: float, nf: int):
    assert t2 >= 0
    x1 = min(
        max(0, math.floor(t1 * feature_per_sec)),
        nf - 1,
    )
    x2 = min(
       math.floor(t2 * feature_per_sec),
       nf - 1,
    )
    assert x2 >= x1
    return x1, x2 + 1
```

--------------------------------

### Configure Downloads with JSON

Source: https://context7.com/facebookresearch/ego4d/llms.txt

Use a JSON configuration file to define download parameters and override settings via the CLI.

```json
{
  "output_directory": "~/ego4d_data",
  "version": "v2_1",
  "datasets": ["full_scale", "annotations", "clips"],
  "benchmarks": ["FHO", "EM"],
  "video_uids": [
    "000a3525-6c98-4650-aaab-be7d2c7b9402",
    "001e3e4e-2c6e-4a0a-b07e-001e3e4e2c6e"
  ],
  "universities": ["cmu", "bristol"],
  "metadata": true,
  "aws_profile_name": "default",
  "assume_yes": false
}
```

```bash
# Use configuration file
ego4d --config_path ./download_config.json

# Override specific settings from command line
ego4d --config_path ./download_config.json --datasets annotations -y
```

--------------------------------

### Import Libraries for Data Processing

Source: https://github.com/facebookresearch/ego4d/blob/main/notebooks/egoexo/EgoExo_Atomic_Descriptions_Tutorial.ipynb

Imports essential Python libraries for data manipulation, file operations, and statistical analysis. Ensure these libraries are installed before running the code.

```python
import json
import time
import os
import traceback
import random
from collections import defaultdict

import numpy as np
import pandas as pd
from tqdm import tqdm
```

--------------------------------

### Initialize SlowFast Model Wrapper

Source: https://context7.com/facebookresearch/ego4d/llms.txt

Set up inference and model configurations for the SlowFast R101 model.

```python
import torch
from ego4d.features.models.slowfast import (
    load_model,
    get_transform,
    ModelConfig,
    PackPathway,
)
from ego4d.features.config import InferenceConfig

# Configure inference
inference_config = InferenceConfig(
    device="cuda",
    batch_size=4,
    fps=30,
    frame_window=32,
    stride=16,
)

# Configure model
model_config = ModelConfig(
    hub_path="slowfast_r101",     # PyTorchVideo hub model
    slowfast_alpha=4,             # Slow pathway temporal stride
    side_size=256,                # Short side resize
    crop_size=256,                # Center crop size
    mean=(0.45, 0.45, 0.45),
    std=(0.225, 0.225, 0.225),
)
```