### Download and Prepare Example Image for Octo Model (Python) Source: https://github.com/octo-models/octo/blob/main/examples/01_inference_pretrained.ipynb Downloads an example image from a specified URL, converts it to a NumPy array, resizes it, and displays it using Matplotlib. This prepares an input for model inference. ```python from PIL import Image import requests import matplotlib.pyplot as plt import numpy as np # download one example BridgeV2 image IMAGE_URL = "https://rail.eecs.berkeley.edu/datasets/bridge_release/raw/bridge_data_v2/datacol2_toykitchen7/drawer_pnp/01/2023-04-19_09-18-15/raw/traj_group0/traj0/images0/im_12.jpg" img = np.array(Image.open(requests.get(IMAGE_URL, stream=True).raw).resize((256, 256))) plt.imshow(img) ``` -------------------------------- ### Install Octo Model Dependencies (Python) Source: https://github.com/octo-models/octo/blob/main/examples/01_inference_pretrained.ipynb Installs necessary libraries and dependencies for running Octo models, including JAX with CUDA support and specific NumPy versions to resolve potential compatibility issues. ```python # run this block if you're using Colab # Download repo !git clone https://github.com/octo-models/octo.git %cd octo # Install repo !pip3 install -e . !pip3 install -r requirements.txt !pip3 install --upgrade "jax[cuda11_pip]==0.4.20" -f https://storage.googleapis.com/jax-releases/jax_cuda_releases.html !pip install numpy==1.21.1 # to fix colab AttributeError: module 'numpy' has no attribute '_no_nep50_warning', if the error still shows reload ``` -------------------------------- ### Install Visualization Dependencies for Octo Model (Python) Source: https://github.com/octo-models/octo/blob/main/examples/01_inference_pretrained.ipynb Installs 'mediapy' for video visualization and 'opencv-python' for image processing, which are commonly used when working with trajectory data and visual outputs from models. ```python # Install mediapy for visualization !pip install mediapy !pip install opencv-python ``` -------------------------------- ### Install Octo Environment and Dependencies (Bash) Source: https://github.com/octo-models/octo/blob/main/README.md These bash commands guide the installation of the Octo environment, including creating a Conda environment, activating it, and installing the package and its requirements. It also provides commands for installing GPU or TPU-accelerated JAX. No specific inputs or outputs are defined, but successful execution leads to a functional Octo installation. ```bash conda create -n octo python=3.10 conda activate octo pip install -e . pip install -r requirements.txt ``` ```bash # For GPU: pip install --upgrade "jax[cuda11_pip]==0.4.20" -f https://storage.googleapis.com/jax-releases/jax_cuda_releases.html ``` ```bash # For TPU pip install --upgrade "jax[tpu]==0.4.20" -f https://storage.googleapis.com/jax-releases/libtpu_releases.html ``` -------------------------------- ### Visualize Goal Images and Language Instructions Source: https://github.com/octo-models/octo/blob/main/examples/05_dataloading.ipynb Visualizes the goal images (primary and wrist) and prints the corresponding language instruction from the 'task' dictionary within a batch. This demonstrates how goal-relabeled data and textual instructions are stored and accessed. ```python # let's take a look at the "task" dict: it should now have both goal # images and language instructions! goal_primary = batch["task"]["image_primary"] goal_wrist = batch["task"]["image_wrist"] language_instruction = batch["task"]["language_instruction"] display(Image.fromarray(goal_primary[0])) display(Image.fromarray(goal_wrist[0])) print(language_instruction[0]) ``` -------------------------------- ### Prepare Task Dictionary for Octo Model Inference (Python) Source: https://github.com/octo-models/octo/blob/main/examples/01_inference_pretrained.ipynb Creates a 'task' dictionary for the Octo model, demonstrating how to prepare inputs for both goal-conditioned inference (using an image) and language-conditioned inference (using text). ```python WINDOW_SIZE = 2 # create `task` dict task = model.create_tasks(goals={"image_primary": goal_image[None]}) # for goal-conditioned task = model.create_tasks(texts=[language_instruction]) # for language conditioned ``` -------------------------------- ### Finetune Octo Model Configuration Example Source: https://github.com/octo-models/octo/blob/main/README.md Example configuration snippet for finetuning an Octo model. It demonstrates how to specify finetuning modes (e.g., 'full', 'head_only') and task types (e.g., 'image_conditioned', 'language_conditioned', 'multimodal'). ```bash --config=finetune_config.py:full,image_conditioned ``` -------------------------------- ### Octo Model Inference with Observation and Task (Python) Source: https://github.com/octo-models/octo/blob/main/examples/01_inference_pretrained.ipynb Prepares an observation dictionary including an image and a timestep mask, defines a task using natural language, and then uses the Octo model to sample actions. The output action is printed. ```python # create obs & task dict, run inference import jax # add batch + time horizon 1 img = img[np.newaxis,np.newaxis,...] observation = {"image_primary": img, "timestep_pad_mask": np.array([[True]])} task = model.create_tasks(texts=["pick up the fork"]) action = model.sample_actions( observation, task, unnormalization_statistics=model.dataset_statistics["bridge_dataset"]["action"], rng=jax.random.PRNGKey(0) ) print(action) # [batch, action_chunk, action_dim] ``` -------------------------------- ### Load Pre-trained Octo Model (Python) Source: https://github.com/octo-models/octo/blob/main/examples/01_inference_pretrained.ipynb Loads a pre-trained Octo model checkpoint from Hugging Face. This is a key step before performing any inference tasks with the model. ```python from octo.model.octo_model import OctoModel model = OctoModel.load_pretrained("hf://rail-berkeley/octo-small-1.5") ``` -------------------------------- ### Visualize Predicted vs. Ground Truth Actions with Matplotlib Source: https://github.com/octo-models/octo/blob/main/examples/01_inference_pretrained.ipynb This Python code visualizes predicted and ground truth actions using matplotlib. It plots each dimension of the actions over time, alongside an image strip showing sampled frames from the episode. Ensure matplotlib is installed for this functionality. ```python import matplotlib.pyplot as plt ACTION_DIM_LABELS = ['x', 'y', 'z', 'yaw', 'pitch', 'roll', 'grasp'] # build image strip to show above actions img_strip = np.concatenate(np.array(images[::3]), axis=1) # set up plt figure figure_layout = [ ['image'] * len(ACTION_DIM_LABELS), ACTION_DIM_LABELS ] plt.rcParams.update({'font.size': 12}) fig, axs = plt.subplot_mosaic(figure_layout) fig.set_size_inches([45, 10]) # plot actions pred_actions = np.array(pred_actions).squeeze() true_actions = np.array(true_actions).squeeze() for action_dim, action_label in enumerate(ACTION_DIM_LABELS): # actions have batch, horizon, dim, in this example we just take the first action for simplicity axs[action_label].plot(pred_actions[:, 0, action_dim], label='predicted action') axs[action_label].plot(true_actions[:, action_dim], label='ground truth') axs[action_label].set_title(action_label) axs[action_label].set_xlabel('Time in one episode') axs['image'].imshow(img_strip) axs['image'].set_xlabel('Time in one episode (subsampled)') plt.legend() ``` -------------------------------- ### Visualize Primary and Wrist Image Windows Source: https://github.com/octo-models/octo/blob/main/examples/05_dataloading.ipynb Visualizes a concatenated window of primary camera images and a window of wrist camera images from a batch. It also prints the padding mask for the wrist images, indicating which frames are considered padding (e.g., black frames). ```python # let's visualize a window of primary images display(Image.fromarray(np.concatenate(images_primary[0], axis=1))) # now a window of wrist images -- many datasets don't have wrist images, # so this will often be black display(Image.fromarray(np.concatenate(images_wrist[0], axis=1))) # pad_mask_dict also tells you which keys should be treated as padding # (e.g., if the wrist camera is black, the corresponding pad_mask_dict entry is False) print(batch["observation"]["pad_mask_dict"]["image_wrist"][0]) ``` -------------------------------- ### Configure and Weight Multiple OXE Datasets for Mixing Source: https://github.com/octo-models/octo/blob/main/examples/05_dataloading.ipynb Generates configurations and sample weights for mixing multiple OXE datasets. This function allows for combining diverse datasets and optionally specifies which camera views to load, such as 'primary' and 'wrist'. It also shows how to inspect the configuration for an individual dataset within the mix. ```python from octo.data.oxe import make_oxe_dataset_kwargs_and_weights from octo.data.dataset import make_interleaved_dataset dataset_kwargs_list, sample_weights = make_oxe_dataset_kwargs_and_weights( # you can pass your own list of dataset names and sample weights here, but we've # also provided a few named mixes for convenience. The Octo model was trained # using the "oxe_magic_soup" mix. "rtx", # can be local or on cloud storage (anything supported by TFDS) "gs://gresearch/robotics", # let's get a wrist camera! load_camera_views=("primary", "wrist"), ) # see `octo.data.dataset.make_dataset_from_rlds` for the meaning of these kwargs dataset_kwargs_list[0] ``` -------------------------------- ### Process and Inspect Dataset Batch Source: https://github.com/octo-models/octo/blob/main/examples/05_dataloading.ipynb Retrieves a batch of data from the configured dataset iterator and prints the top-level keys and the keys within the 'observation' and 'task' dictionaries. This helps in understanding the structure of the processed data, including images and actions. ```python # phew, that was a lot of configuration! Let's see what we got. batch = next(iterator) print("Top-level keys: ", batch.keys()) # should now have "image_primary" and "image_wrist"! print("Observation keys: ", batch["observation"].keys()) # should also have "image_primary" and "image_wrist", corresponding to future goal images print("Task keys: ", batch["task"].keys()) ``` -------------------------------- ### Prepare OXE Dataset for Training with Batching Source: https://github.com/octo-models/octo/blob/main/examples/05_dataloading.ipynb Transforms a loaded OXE dataset into a training-ready format by flattening trajectories into individual frames, shuffling them, and then batching them. This is crucial for feeding data into a machine learning model. It also demonstrates the shape of the batched image data. ```python # you should set these much higher in practice (as large as your memory can hold!) SHUFFLE_BUFFER_SIZE = 1000 BATCH_SIZE = 64 # turning a dataset of trajectories into a training-ready batched dataset train_dataset = ( dataset.flatten() # flattens trajectories into individual frames .shuffle(SHUFFLE_BUFFER_SIZE) # shuffles the frames .batch(BATCH_SIZE) # batches the frames ) batch = next(train_dataset.iterator()) images = batch["observation"]["image_primary"] # should be: (batch_size, window_size, height, width, channels) print(images.shape) Image.fromarray(np.concatenate(images.squeeze()[:5], axis=1)) ``` -------------------------------- ### Configure Environment for Octo Model (Python) Source: https://github.com/octo-models/octo/blob/main/examples/01_inference_pretrained.ipynb Sets the 'TOKENIZERS_PARALLELISM' environment variable to 'false' to disable parallel tokenization, which can sometimes resolve issues in specific environments like Colab. ```python import os os.environ['TOKENIZERS_PARALLELISM'] = 'false' ``` -------------------------------- ### Load Bridge Dataset Trajectory for Octo Model (Python) Source: https://github.com/octo-models/octo/blob/main/examples/01_inference_pretrained.ipynb Loads a trajectory from the Bridge dataset using TensorFlow Datasets. It then extracts images from the steps, resizes them, and identifies the goal image and language instruction for inference. ```python # create RLDS dataset builder builder = tfds.builder_from_directory(builder_dir='gs://gresearch/robotics/bridge/0.1.0/') ds = builder.as_dataset(split='train[:1]') # sample episode + resize to 256x256 (default third-person cam resolution) episode = next(iter(ds)) steps = list(episode['steps']) images = [cv2.resize(np.array(step['observation']['image']), (256, 256)) for step in steps] # extract goal image & language instruction goal_image = images[-1] language_instruction = steps[0]['observation']['natural_language_instruction'].numpy().decode() # visualize episode print(f'Instruction: {language_instruction}') mediapy.show_video(images, fps=10) ``` -------------------------------- ### Import Libraries for Trajectory Inference (Python) Source: https://github.com/octo-models/octo/blob/main/examples/01_inference_pretrained.ipynb Imports essential libraries for processing trajectory data and running inference, including OpenCV for image manipulation, JAX for numerical operations, TensorFlow Datasets for data loading, tqdm for progress bars, mediapy for visualization, and NumPy. ```python import cv2 import jax import tensorflow_datasets as tfds import tqdm import mediapy import numpy as np ``` -------------------------------- ### Process Image Observations from OXE Trajectories Source: https://github.com/octo-models/octo/blob/main/examples/05_dataloading.ipynb Extracts and processes image data from the primary camera observation within an OXE trajectory. It shows how to access the image array, check its shape, and visualize a sequence of images by concatenating them. ```python from PIL import Image import numpy as np traj = next(iterator) images = traj["observation"]["image_primary"] # should be: (traj_len, window_size, height, width, channels) # (window_size defaults to 1) print(images.shape) Image.fromarray(np.concatenate(images.squeeze()[-5:], axis=1)) ``` -------------------------------- ### Pretrain Octo Model with Bash Source: https://github.com/octo-models/octo/blob/main/README.md Command to reproduce Octo pretraining on robot trajectories. Requires specifying configuration, dataset directory, and dataset mix. The dataset is large (~1.2TB) and requires installation of the rlds_dataset_mod package and running a preparation script. ```bash python scripts/train.py --config scripts/configs/octo_pretrain_config.py: --name=octo --config.dataset_kwargs.oxe_kwargs.data_dir=... --config.dataset_kwargs.oxe_kwargs.data_mix=oxe_magic_soup ... ``` -------------------------------- ### Load Single OXE Dataset with Octo Source: https://github.com/octo-models/octo/blob/main/examples/05_dataloading.ipynb Loads a single dataset from the Open X-Embodiment (OXE) collection using the `octo.data` pipeline. It requires specifying the dataset name and the base directory for the data, which can be local or cloud-based. The function `make_single_dataset` returns a dataset object that can be iterated over to yield trajectories. ```python # minimum working example to load a single OXE dataset from octo.data.oxe import make_oxe_dataset_kwargs from octo.data.dataset import make_single_dataset dataset_kwargs = make_oxe_dataset_kwargs( # see octo/data/oxe/oxe_dataset_configs.py for available datasets # (this is a very small one for faster loading) "austin_buds_dataset_converted_externally_to_rlds", # can be local or on cloud storage (anything supported by TFDS) # "/path/to/base/oxe/directory", "gs://gresearch/robotics", ) dataset = make_single_dataset(dataset_kwargs, train=True) # load the train split iterator = dataset.iterator() ``` -------------------------------- ### Create Interleaved Dataset with Octo Model Source: https://github.com/octo-models/octo/blob/main/examples/05_dataloading.ipynb Configures and creates an interleaved dataset using `make_interleaved_dataset`. This function combines multiple datasets, applies specified transformations (frame and trajectory), and handles shuffling and batching. It requires lists of dataset keyword arguments, sample weights, and various configuration options for transformations and performance tuning. ```python dataset = make_interleaved_dataset( dataset_kwargs_list, sample_weights, train=True, # unlike our manual shuffling above, `make_interleaved_dataset` will shuffle # the JPEG-encoded images, so you should be able to fit a much larger buffer size shuffle_buffer_size=SHUFFLE_BUFFER_SIZE, batch_size=BATCH_SIZE, # see `octo.data.dataset.apply_trajectory_transforms` for full documentation # of these configuration options traj_transform_kwargs=dict( goal_relabeling_strategy="uniform", # let's get some goal images window_size=2, # let's get some history action_horizon=4, # let's get some future actions for action chunking subsample_length=100, # subsampling long trajectories improves shuffling a lot ), # see `octo.data.dataset.apply_frame_transforms` for full documentation # of these configuration options frame_transform_kwargs=dict( # let's apply some basic image augmentations -- see `dlimp.transforms.augment_image` # for full documentation of these configuration options image_augment_kwargs=dict( primary=dict( augment_order=["random_resized_crop", "random_brightness"], random_resized_crop=dict(scale=[0.8, 1.0], ratio=[0.9, 1.1]), random_brightness=[0.1], ) ), # providing a `resize_size` is highly recommended for a mixed dataset, otherwise # datasets with different resolutions will cause errors resize_size=dict( primary=(256, 256), wrist=(128, 128), ), # If parallelism options are not provided, they will default to tf.Data.AUTOTUNE. # However, we would highly recommend setting them manually if you run into issues # with memory or dataloading speed. Frame transforms are usually the speed # bottleneck (due to image decoding, augmentation, and resizing), so you can set # this to a very high value if you have a lot of CPU cores. Keep in mind that more # parallel calls also use more memory, though. num_parallel_calls=64, ), # Same spiel as above about performance, although trajectory transforms and data reading # are usually not the speed bottleneck. One reason to manually set these is if you want # to reduce memory usage (since autotune may spawn way more threads than necessary). traj_transform_threads=16, traj_read_threads=16, ) # Another performance knob to tune is the number of batches to prefetch -- again, # the default of tf.data.AUTOTUNE can sometimes use more memory than necessary. iterator = dataset.iterator(prefetch=1) ``` -------------------------------- ### Inspect Loaded Trajectories from OXE Dataset Source: https://github.com/octo-models/octo/blob/main/examples/05_dataloading.ipynb Inspects the structure and keys of a loaded trajectory from an OXE dataset. This helps understand the organization of observations and task-related information within each trajectory. It demonstrates accessing nested dictionaries for observations and tasks. ```python # make_single_dataset yields entire trajectories traj = next(iterator) print("Top-level keys: ", traj.keys()) print("Observation keys: ", traj["observation"].keys()) print("Task keys: ", traj["task"].keys()) ``` -------------------------------- ### Inspect Batch Data Shapes Source: https://github.com/octo-models/octo/blob/main/examples/05_dataloading.ipynb Extracts image data (primary and wrist) and action data from a batch. It then prints the shapes of these tensors to verify they match the expected dimensions for images (batch_size, window_size, height, width, channels) and actions (batch_size, window_size, action_horizon, action_dim). ```python from PIL import Image import numpy as np images_primary = batch["observation"]["image_primary"] images_wrist = batch["observation"]["image_wrist"] # should be: (batch_size, window_size (now 2), height, width, channels) print(images_primary.shape) print(images_wrist.shape) actions = batch["action"] # should be: (batch_size, window_size, action_horizon, action_dim) print(actions.shape) ``` -------------------------------- ### Collect Predicted and True Actions with Octo-Models Source: https://github.com/octo-models/octo/blob/main/examples/01_inference_pretrained.ipynb This Python code snippet iterates through a sequence of images, uses a model to predict actions based on observations, and collects both predicted and true actions. It handles action unnormalization and concatenates ground truth action components. ```python pred_actions, true_actions = [], [] for step in tqdm.trange(len(images) - (WINDOW_SIZE - 1)): input_images = np.stack(images[step:step+WINDOW_SIZE])[None] observation = { 'image_primary': input_images, 'timestep_pad_mask': np.full((1, input_images.shape[1]), True, dtype=bool) } # this returns *normalized* actions --> we need to unnormalize using the dataset statistics actions = model.sample_actions( observation, task, unnormalization_statistics=model.dataset_statistics["bridge_dataset"]["action"], rng=jax.random.PRNGKey(0) ) actions = actions[0] # remove batch dim pred_actions.append(actions) final_window_step = step + WINDOW_SIZE - 1 true_actions.append(np.concatenate( ( steps[final_window_step]['action']['world_vector'], steps[final_window_step]['action']['rotation_delta'], np.array(steps[final_window_step]['action']['open_gripper']).astype(np.float32)[None] ), axis=-1 )) ``` -------------------------------- ### Finetune Octo Model on Debug Dataset (Bash) Source: https://github.com/octo-models/octo/blob/main/README.md This bash command initiates the finetuning process for an Octo model using the debug dataset. It requires the 'scripts/finetune.py' script and specifies the pretrained model path. The '--debug' flag indicates a test run. The command's output indicates the progress and outcome of the finetuning process. ```bash python scripts/finetune.py --config.pretrained_path=hf://rail-berkeley/octo-small-1.5 --debug ``` -------------------------------- ### Finetune Octo Model with Bash Source: https://github.com/octo-models/octo/blob/main/README.md Command to run advanced finetuning of an Octo model. Allows changing hyperparameters via a config file and logs finetuning metrics. It requires specifying the path to a pretrained model. ```bash python scripts/finetune.py --config.pretrained_path=hf://rail-berkeley/octo-small-1.5 ``` -------------------------------- ### Load and Use Octo Model with Python Source: https://github.com/octo-models/octo/blob/main/README.md Python code snippet for loading a pretrained Octo model and performing inference. It demonstrates how to load a model from a Hugging Face path, create tasks from text, and sample actions based on observations. ```python from octo.model import OctoModel model = OctoModel.load_pretrained("hf://rail-berkeley/octo-small-1.5") task = model.create_tasks(texts=["pick up the spoon"]) action = model.sample_actions(observation, task, rng=jax.random.PRNGKey(0)) ``` -------------------------------- ### Load and Inspect a Pretrained Octo Model (Python) Source: https://github.com/octo-models/octo/blob/main/README.md This Python snippet demonstrates how to load a pretrained Octo model from Hugging Face and print its specification. It requires the 'octo-models' library. The input is a model identifier string, and the output is the model's specification. ```python from octo.model.octo_model import OctoModel model = OctoModel.load_pretrained("hf://rail-berkeley/octo-base-1.5") print(model.get_pretty_spec()) ``` === COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.