### Install MediaRef with Extras

Source: https://github.com/open-world-agents/mediaref/blob/main/README.md

Shows how to install the MediaRef package with optional dependencies for video decoding and HuggingFace datasets integration. Also includes installation for the uv package manager.

```bash
pip install mediaref                  # core: image loading + cloud-storage URIs (fsspec)
pip install 'mediaref[video]'         # + PyAV for video frame decoding
pip install 'mediaref[hf]'            # + HuggingFace datasets feature registration
pip install 'mediaref[video,hf]'      # all extras
```

```bash
For uv: uv add 'mediaref[video,hf]~=1.0.0'. MediaRef follows [semantic versioning](https://semver.org/); patch releases are bug-only, minor releases are backward-compatible. The wire schema (`uri`, `pts_ns`) is frozen for the life of Spec 1.x.
```

--------------------------------

### Install MediaRef and Extras

Source: https://context7.com/open-world-agents/mediaref/llms.txt

Install the core mediaref library, or with optional extras for video decoding, HuggingFace integration, or both. TorchCodec can be installed separately for GPU-accelerated decoding.

```bash
pip install mediaref                   # core: image loading + cloud-storage URIs (fsspec)
pip install 'mediaref[video]'          # + PyAV for video frame decoding
pip install 'mediaref[hf]'             # + HuggingFace datasets feature registration
pip install 'mediaref[video,hf]'       # all extras

# Optional TorchCodec backend (GPU-accelerated decoding)
pip install torchcodec
# If libavcodec shared library conflicts occur after installing torchcodec:
pip install patch-torchcodec && patch-torchcodec
```

--------------------------------

### Install Development Dependencies

Source: https://github.com/open-world-agents/mediaref/blob/main/CONTRIBUTING.md

Install MediaRef in development mode with all extras using uv or pip. Ensure video extras are included if working with video.

```bash
# With uv (recommended):
uv sync --all-extras --all-groups

# Or with pip:
pip install -e ".[video]"
pip install ipython pytest pytest-cov ruff
```

--------------------------------

### Verify TorchCodec installation

Source: https://github.com/open-world-agents/mediaref/blob/main/scripts/patch_torchcodec/README.md

After patching, TorchCodec should work without further configuration. This example shows a simple import statement.

```python
from torchcodec.decoders import VideoDecoder  # ✓ just works
```

--------------------------------

### Install Project Dependencies

Source: https://github.com/open-world-agents/mediaref/blob/main/examples/ros/README.md

Installs the necessary Python packages for the project using uv pip.

```bash
uv pip install -r requirements.txt
```

--------------------------------

### Install TorchCodec and patch-torchcodec

Source: https://github.com/open-world-agents/mediaref/blob/main/scripts/patch_torchcodec/README.md

Install torchcodec and patch-torchcodec packages. The patch-torchcodec package includes av and patchelf.

```bash
pip install torchcodec          # install torchcodec (with PyTorch)
pip install patch-torchcodec    # install patcher (av & patchelf included)
patch-torchcodec                # patches RPATH — done!
```

--------------------------------

### Direct Use of PyAV Video Decoder

Source: https://github.com/open-world-agents/mediaref/blob/main/docs/API.md

Use the PyAVVideoDecoder class directly for finer control over frame retrieval. This example demonstrates opening a video file and getting a specific frame at a given presentation timestamp in nanoseconds.

```python
from mediaref.decoders import PyAVVideoDecoder

with PyAVVideoDecoder("episode.mp4") as dec:
    frame = dec.get_frame_at_pts_ns(1_500_000_000)
```

--------------------------------

### Frame Selection Rule Example

Source: https://github.com/open-world-agents/mediaref/blob/main/docs/SPEC.md

Illustrates the frame selection logic based on presentation timestamps (pts_ns) for video playback.

```pseudocode
pts[i] ≤ t < pts[i+1]   ⇒   returns frame i
pts[N-1] ≤ t < end_stream_ns   ⇒   returns frame N-1 (last frame)
t < pts[0]   or   t ≥ end_stream_ns   ⇒   error
```

--------------------------------

### MediaRef Serialization Examples

Source: https://github.com/open-world-agents/mediaref/blob/main/docs/API.md

Illustrates serializing a MediaRef object to dictionary and JSON formats, and deserializing from these formats using Pydantic v2 methods.

```python
ref = MediaRef(uri="video.mp4", pts_ns=1_500_000_000)

ref.model_dump()        # {'uri': 'video.mp4', 'pts_ns': 1500000000}
ref.model_dump_json()   # '{"uri":"video.mp4","pts_ns":1500000000}'

MediaRef.model_validate({"uri": "video.mp4", "pts_ns": 0})
MediaRef.model_validate_json('{"uri":"video.mp4","pts_ns":0}')
```

--------------------------------

### MediaRef Message Format Example

Source: https://github.com/open-world-agents/mediaref/blob/main/examples/ros/README.md

Illustrates the JSON structure for a MediaRef message, typically stored as a std_msgs/String in a ROS bag.

```json
# MediaRef message format (std_msgs/String)
{"uri": "bag_mediaref.media/camera.mp4", "pts_ns": 1729561964520000000}
```

--------------------------------

### Reconstructing MediaRefs from LeRobot Episode

Source: https://github.com/open-world-agents/mediaref/blob/main/docs/API.md

Builds MediaRefs for an entire episode from a v3.0 LeRobotDataset shared mp4 shard. Requires video path, the starting timestamp, and frame timestamps.

```python
from mediaref.compat.lerobot import (
    from_videoframe, to_videoframe, lerobot_episode_to_refs,
)

# Build refs for an entire episode in a v3.0 LeRobotDataset shared mp4 shard
refs = lerobot_episode_to_refs(
    video_path="videos/observation.images.front_left/chunk-000/file-000.mp4",
    from_timestamp=12.34,                # meta.episodes[ep_idx][f"videos/{vid_key}/from_timestamp"]
    frame_timestamps=[0.0, 1/30, 2/30],  # episode-local timestamps
)
```

--------------------------------

### MediaRef: Cloud Storage URIs with fsspec

Source: https://context7.com/open-world-agents/mediaref/llms.txt

Access media files from cloud storage using MediaRef with fsspec. Requires installing the appropriate fsspec backend (e.g., s3fs, gcsfs). Credentials can be managed via environment variables or configuration files.

```python
from mediaref import MediaRef, batch_decode
import os

# --- S3 (credentials from env vars or ~/.aws/credentials) ---
os.environ["AWS_ACCESS_KEY_ID"] = "..."
os.environ["AWS_SECRET_ACCESS_KEY"] = "..."
ref_s3 = MediaRef(uri="s3://my-bucket/scene.mp4", pts_ns=1_500_000_000)
frame = ref_s3.to_ndarray()                # range read via s3fs — no full download
```

```python
# --- Google Cloud Storage ---
# pip install gcsfs
os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = "/path/to/key.json"
ref_gcs = MediaRef(uri="gs://my-bucket/video.mp4", pts_ns=0)
frame_gcs = ref_gcs.to_ndarray()
```

```python
# --- HuggingFace Hub (public datasets need no token) ---
# pip install huggingface_hub
ref_hf = MediaRef(uri="hf://datasets/open-world-agents/D2E-Original/video.mp4",
                  pts_ns=5_000_000_000)
frame_hf = ref_hf.to_ndarray()
```

```python
# --- Azure Blob Storage ---
# pip install adlfs
ref_az = MediaRef(uri="az://container/video.mp4", pts_ns=0)
```

```python
# --- Batch decode across cloud files (PyAV only; TorchCodec has no fsspec support) ---
refs = [
    MediaRef(uri="hf://datasets/me/clips/cam.mp4", pts_ns=int(i * 1e9))
    for i in range(10)
]
frames = batch_decode(refs)
```

--------------------------------

### Patch TorchCodec for CUDA Acceleration

Source: https://github.com/open-world-agents/mediaref/blob/main/README.md

Provides instructions to install and run the `patch-torchcodec` script, which is necessary to resolve potential conflicts between TorchCodec's FFmpeg dependencies and PyAV's bundled copies when using CUDA-accelerated decoding.

```bash
pip install patch-torchcodec && patch-torchcodec
```

--------------------------------

### Cloud Storage URI Decoding with MediaRef

Source: https://github.com/open-world-agents/mediaref/blob/main/docs/API.md

Decode media from cloud storage URIs using MediaRef. URIs like s3:// or hf:// are handled via fsspec, allowing direct reading without full downloads. Ensure the corresponding fsspec backend (e.g., s3fs) is installed.

```python
from mediaref import MediaRef, batch_decode

ref = MediaRef(uri="s3://my-bucket/episode.mp4", pts_ns=1_500_000_000)
frame = ref.to_ndarray()    # range read via fsspec — no full download

refs = [MediaRef(uri="hf://datasets/me/clips/cam.mp4", pts_ns=int(i*1e9)) for i in range(10)]
frames = batch_decode(refs)
```

--------------------------------

### Get Frames Played at Specific Times with PyAVVideoDecoder

Source: https://context7.com/open-world-agents/mediaref/llms.txt

Demonstrates how to retrieve frames based on their presentation timestamps (PTS) using PyAVVideoDecoder. Handles queries within the stream duration and notes that querying exactly at the end stream time raises a ValueError.

```python
from mediaref.video_decoder import PyAVVideoDecoder

with PyAVVideoDecoder("25fps_video.mp4") as dec:
    # 25 fps → frames at 0.00, 0.04, 0.08, ...
    b = dec.get_frames_played_at([0.00, 0.02, 0.04, 0.05])
    print(b.pts_seconds)
    # [0.0, 0.0, 0.04, 0.04]
    # query 0.02 returns frame 0 (displayed from 0.0 until 0.04)
    # query 0.05 returns frame 1 (displayed from 0.04 until 0.08)

    # Boundary: query exactly at end raises ValueError
    import contextlib
    with contextlib.suppress(ValueError):
        dec.get_frames_played_at([float(dec.metadata.end_stream_seconds)])
```

--------------------------------

### Create and Load Media References

Source: https://github.com/open-world-agents/mediaref/blob/main/README.md

Demonstrates creating MediaRef objects from various sources including local files, HTTP(S) URLs, cloud storage (S3), and specific video frames. It also shows how to load media data into NumPy arrays or PIL Images.

```python
from mediaref import MediaRef, DataURI, batch_decode
import numpy as np

# 1. Create references — local file, HTTP(S), cloud, or video frame.
ref = MediaRef(uri="image.png")
ref = MediaRef(uri="https://example.com/image.jpg")
ref = MediaRef(uri="s3://bucket/image.jpg")             # any fsspec scheme
ref = MediaRef(uri="video.mp4", pts_ns=1_000_000_000)   # frame at 1.0s

# 2. Load.
rgb = ref.to_ndarray()      # (H, W, 3) RGB
pil = ref.to_pil_image()
```

```python
# 3. Embed bytes inside a MediaRef (self-contained reference).
ref = MediaRef(uri=DataURI.from_image(rgb, format="png"))
```

```python
# 4. Batch-decode many frames from one video — opens the container once.
refs = [MediaRef(uri="video.mp4", pts_ns=int(i*1e9)) for i in range(10)]
frames = batch_decode(refs)
```

```python
# 5. Serialize for storage in any string-based format.
json_str = ref.model_dump_json()   # '{"uri":"...","pts_ns":...}'
```

--------------------------------

### Instantiate MediaRef with Different URIs

Source: https://github.com/open-world-agents/mediaref/blob/main/docs/API.md

Demonstrates creating MediaRef instances with various URI types including local paths, remote URLs, S3 URIs, and data URIs.

```python
from mediaref import MediaRef, DataURI

ref = MediaRef(uri="image.png")
ref = MediaRef(uri="https://example.com/img.jpg")
ref = MediaRef(uri="s3://bucket/clip.mp4", pts_ns=1_500_000_000)
ref = MediaRef(uri="data:image/png;base64,iVBORw0…")

# A DataURI instance is accepted directly — its `uri` string is extracted.
ref = MediaRef(uri=DataURI.from_image(rgb, format="png"))
```

--------------------------------

### Construct and Use MediaRef Objects

Source: https://context7.com/open-world-agents/mediaref/llms.txt

Demonstrates construction of MediaRef objects for various media types (images, videos) and URIs (local, HTTP, S3, embedded). Shows how to access properties like is_video, is_cloud_uri, and load media into different formats (ndarray, PIL Image).

```python
from mediaref import MediaRef, DataURI
import numpy as np

# --- Construction ---
ref_img   = MediaRef(uri="image.png")                              # still image, local path
ref_http  = MediaRef(uri="https://example.com/photo.jpg")         # HTTP image
ref_s3    = MediaRef(uri="s3://my-bucket/clip.mp4",               # cloud video frame
                     pts_ns=1_500_000_000)                         # at 1.5 s
ref_abs   = MediaRef(uri="/data/recordings/video.mkv",
                     pts_ns=0)                                     # absolute path, first frame
ref_embed = MediaRef(uri=DataURI.from_image(np.zeros((64, 64, 3), dtype=np.uint8), format="png"))

# --- Properties ---
print(ref_s3.is_video)          # True
print(ref_img.is_video)         # False
print(ref_s3.is_cloud_uri)      # True  (fsspec-routed)
print(ref_http.is_remote)       # True  (http/https)
print(ref_embed.is_embedded)    # True  (data: URI)
print(ref_img.is_relative_path) # True

# --- Loading ---
rgb   = ref_img.to_ndarray()                    # (H, W, 3) uint8 RGB
bgr   = ref_img.to_ndarray(format="bgr")        # (H, W, 3) uint8 BGR  — for OpenCV
gray  = ref_img.to_ndarray(format="gray")       # (H, W) uint8
rgba  = ref_img.to_ndarray(format="rgba")       # (H, W, 4) uint8
pil   = ref_img.to_pil_image()                  # PIL.Image (rgb by default)
pil_g = ref_img.to_pil_image(format="gray")

# --- Relative path resolution ---
rel = MediaRef(uri="clips/episode1.mkv", pts_ns=2_000_000_000)
abs_ref = rel.resolve_relative_path("/data/recordings")
# abs_ref.uri == "/data/recordings/clips/episode1.mkv"

remote = MediaRef(uri="https://example.com/img.png")
remote.resolve_relative_path("/data", on_unresolvable="ignore")  # returned unchanged

# --- Serialization (wire format) ---
ref = MediaRef(uri="video.mp4", pts_ns=1_500_000_000)
print(ref.model_dump())       # {'uri': 'video.mp4', 'pts_ns': 1500000000}
print(ref.model_dump_json())  # '{"uri":"video.mp4","pts_ns":1500000000}'

# round-trip
MediaRef.model_validate({"uri": "video.mp4", "pts_ns": 0})
MediaRef.model_validate_json('{"uri":"video.mp4","pts_ns":0}')

# Still image: pts_ns is omitted from wire format
print(MediaRef(uri="image.png").model_dump_json())  # '{"uri":"image.png","pts_ns":null}'
```

--------------------------------

### Using MediaRefFeature with Hugging Face Datasets

Source: https://github.com/open-world-agents/mediaref/blob/main/docs/API.md

Demonstrates how to create a Hugging Face Dataset with MediaRef objects and save/load it. Ensure `mediaref.hf` is imported before loading to avoid errors.

```python
from datasets import Dataset, Features, load_from_disk
from mediaref import MediaRef
from mediaref.hf import MediaRefFeature

ds = Dataset.from_dict(
    {"frame": [MediaRef(uri="video.mp4", pts_ns=0),
               MediaRef(uri="video.mp4", pts_ns=33_333_333)]},
    features=Features({"frame": MediaRefFeature()}),
)
ds.save_to_disk("path/to/ds")
load_from_disk("path/to/ds")[0]["frame"].to_ndarray()  # round-trips as MediaRef
```

--------------------------------

### Clone MediaRef Repository

Source: https://github.com/open-world-agents/mediaref/blob/main/CONTRIBUTING.md

Clone the MediaRef repository and navigate into the project directory.

```bash
git clone https://github.com/open-world-agents/mediaref.git
cd mediaref
```

--------------------------------

### Create DataURI from File

Source: https://github.com/open-world-agents/mediaref/blob/main/docs/API.md

Wraps the raw bytes of a file on disk into a DataURI.

```python
from mediaref import MediaRef, DataURI
from PIL import Image
import cv2
import numpy as np

# File on disk
ref = MediaRef(uri=DataURI.from_file("photo.png"))
```

--------------------------------

### Encode MediaRef, dict, or URI string

Source: https://context7.com/open-world-agents/mediaref/llms.txt

The MediaRefFeature's encode_example method accepts MediaRef objects, dictionaries, or plain URI strings. It returns a dictionary representation suitable for serialization.

```python
from mediaref import MediaRef
from mediaref.hf import MediaRefFeature

feat = MediaRefFeature()
print(feat.encode_example(MediaRef(uri="v.mp4", pts_ns=0)))
print(feat.encode_example({"uri": "v.mp4", "pts_ns": 0}))
print(feat.encode_example("image.png"))
```

--------------------------------

### Run Tests with Coverage

Source: https://github.com/open-world-agents/mediaref/blob/main/CONTRIBUTING.md

Run all tests and generate an HTML report of code coverage. This helps identify areas of the code not tested.

```bash
pytest --cov=mediaref --cov-report=html
```

--------------------------------

### Direct decoder use

Source: https://github.com/open-world-agents/mediaref/blob/main/docs/API.md

Provides direct access to decoder classes for finer control over media decoding.

```APIDOC
## Direct decoder use

### Description
For finer control, use the decoder classes directly.

### Methods
#### `PyAVVideoDecoder`
- **Initialization**: `PyAVVideoDecoder(uri: str)`
- **Method**: `get_frame_at_pts_ns(pts_ns: int) -> np.ndarray`

#### `TorchCodecVideoDecoder`
- **Initialization**: `TorchCodecVideoDecoder(uri: str)`
- **Method**: `get_frame_at_pts_ns(pts_ns: int) -> np.ndarray`
```

--------------------------------

### Decode MediaRef References using Python

Source: https://github.com/open-world-agents/mediaref/blob/main/examples/ros/README.md

Demonstrates how to parse MediaRef JSON messages and decode them into image frames using the MediaRef library and pyav.

```python
# Decoding
from mediaref import MediaRef, batch_decode
refs = [MediaRef.model_validate_json(msg.data) for msg in messages]
frames = batch_decode(refs, decoder="pyav")  # numpy arrays [H, W, 3] uint8
```

--------------------------------

### Resolve Relative Path for MediaRef

Source: https://github.com/open-world-agents/mediaref/blob/main/docs/API.md

Shows how to resolve a MediaRef with a relative URI against a base path. Unresolvable URIs are handled based on the `on_unresolvable` parameter.

```python
ref = MediaRef(uri="relative/video.mkv", pts_ns=123456)
ref.resolve_relative_path("/data/recordings")
# MediaRef(uri='/data/recordings/relative/video.mkv', pts_ns=123456)

remote = MediaRef(uri="https://example.com/image.jpg")
remote.resolve_relative_path("/data", on_unresolvable="ignore")  # returned unchanged
```

--------------------------------

### Create DataURI from file or existing URI

Source: https://context7.com/open-world-agents/mediaref/llms.txt

Creates a DataURI from a local file or by parsing an existing data URI string. Allows overriding the format if the file extension is missing or incorrect.

```python
# --- From file on disk ---
duri_file = DataURI.from_file("image.png")
# with explicit format override
duri_file2 = DataURI.from_file("image_noextension", format="png")

# --- Parse existing data URI string ---
uri_str = "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAA..."
duri_parsed = DataURI.from_uri(uri_str)
```

--------------------------------

### Convert ROS bag to MediaRef bag (CLI)

Source: https://context7.com/open-world-agents/mediaref/llms.txt

Converts ROS1/ROS2 bags with CompressedImage topics to MediaRef-based bags, re-encoding images to H.264 mp4 and storing messages as MediaRef JSON strings. Requires 'rosbags', 'tqdm', 'pyav', and 'opencv-python'.

```bash
# Install dependencies (no ROS install required)
pip install rosbags tqdm pyav opencv-python

# Convert a ROS1 bag (auto-detects format)
python examples/ros/bag_to_mediaref.py recording.bag

# Convert a ROS2 bag directory
python examples/ros/bag_to_mediaref.py ros2_bag_dir/ -o output_bag/ --fps 30

# Decode frames from the resulting bag
python examples/ros/mediaref_decode.py recording_mediaref.bag
```

--------------------------------

### Enable HuggingFace feature patch

Source: https://context7.com/open-world-agents/mediaref/llms.txt

Use the mediaref CLI to patch the datasets package for auto-registration of MediaRefFeature. Re-run after 'pip upgrade datasets'.

```bash
mediaref enable-hf-feature
```

--------------------------------

### Create DataURI from OpenCV BGR Image

Source: https://github.com/open-world-agents/mediaref/blob/main/docs/API.md

Creates a DataURI from an OpenCV image (BGR format), explicitly setting `input_format` to 'bgr'.

```python
from mediaref import MediaRef, DataURI
from PIL import Image
import cv2
import numpy as np

# OpenCV BGR — input_format is REQUIRED
bgr = cv2.imread("photo.jpg")
ref = MediaRef(uri=DataURI.from_image(bgr, format="png", input_format="bgr"))
```

--------------------------------

### Run All Tests

Source: https://github.com/open-world-agents/mediaref/blob/main/CONTRIBUTING.md

Execute all tests in the project using pytest. This is a basic command to ensure all tests pass.

```bash
pytest
```

--------------------------------

### HuggingFace Datasets Integration with MediaRefFeature

Source: https://context7.com/open-world-agents/mediaref/llms.txt

Integrate MediaRef as a first-class feature type in HuggingFace datasets. This allows datasets with MediaRef columns to be saved, loaded, and pushed to the Hub while preserving lazy decoding capabilities.

```python
# pip install 'mediaref[hf]'
from datasets import Dataset, Features, load_from_disk, load_dataset
from mediaref import MediaRef
from mediaref.hf import MediaRefFeature  # REQUIRED before any load call

# --- Build a dataset with MediaRef columns ---
data = {
    "step": [0, 1, 2, 3],
    "frame": [
        MediaRef(uri="video.mp4", pts_ns=0),
        MediaRef(uri="video.mp4", pts_ns=33_333_333),
        MediaRef(uri="video.mp4", pts_ns=66_666_666),
        MediaRef(uri="video.mp4", pts_ns=100_000_000),
    ],
}
ds = Dataset.from_dict(
    data,
    features=Features({"step": ..., "frame": MediaRefFeature()}),
)

# --- Access returns a MediaRef object ---
row = ds[0]
print(row["frame"])
rgb = row["frame"].to_ndarray()              # lazy decode (H, W, 3)

# --- Save and reload (round-trip) ---
ds.save_to_disk("path/to/dataset")
ds2 = load_from_disk("path/to/dataset")     # requires mediaref.hf imported first
assert isinstance(ds2[0]["frame"], MediaRef)

# --- decode=False: get raw dicts instead of MediaRef objects ---
ds_raw = Dataset.from_dict(
    data,
    features=Features({"step": ..., "frame": MediaRefFeature(decode=False)}),
)
print(ds_raw[0]["frame"])

# --- push_to_hub / load_dataset round-trip ---
ds.push_to_hub("my-org/my-dataset")
from mediaref.hf import MediaRefFeature  # noqa: F811 — ensure registered
ds_loaded = load_dataset("my-org/my-dataset", split="train")
```

--------------------------------

### JSON Representation of MediaRef

Source: https://github.com/open-world-agents/mediaref/blob/main/docs/SPEC.md

Illustrates different ways to represent a MediaRef in JSON, including for video frames and still images.

```json
{"uri": "video.mp4", "pts_ns": 1500000000}
```

```json
{"uri": "image.png"}                          // pts_ns absent ⇒ still image
```

```json
{"uri": "image.png", "pts_ns": null}          // equivalent to the above
```

--------------------------------

### Convert lerobot VideoFrame to MediaRef

Source: https://context7.com/open-world-agents/mediaref/llms.txt

Converts a lerobot VideoFrame dictionary to a MediaRef object. Requires the 'mediaref' library but not 'lerobot'.

```python
from mediaref.compat.lerobot import from_videoframe
from mediaref import MediaRef

vf = {"path": "videos/observation.mp4", "timestamp": 0.5}
ref = from_videoframe(vf)
print(ref)
```

--------------------------------

### Convert ROS Bag to MediaRef Format

Source: https://github.com/open-world-agents/mediaref/blob/main/examples/ros/README.md

Converts a ROS bag containing CompressedImage messages to the MediaRef format, creating a new bag with video references.

```bash
# Convert bag to MediaRef format
uv run bag_to_mediaref.py input.bag
```

--------------------------------

### Patch TorchCodec using command line

Source: https://github.com/open-world-agents/mediaref/blob/main/scripts/patch_torchcodec/README.md

Various command-line options are available for patching TorchCodec. The default option patches RPATH, which is recommended.

```bash
patch-torchcodec               # Patch RPATH (default, recommended)
```

```bash
patch-torchcodec --env-only    # Symlinks only (requires LD_LIBRARY_PATH)
```

```bash
patch-torchcodec --status      # Check current setup status
```

```bash
patch-torchcodec --verify      # Verify TorchCodec works
```

```bash
patch-torchcodec --quiet       # Silent mode
```

--------------------------------

### Convert lerobot v3.0 episode to MediaRefs

Source: https://context7.com/open-world-agents/mediaref/llms.txt

Converts a v3.0 LeRobotDataset episode layout to a list of MediaRef objects. Handles shared mp4 shards and calculates absolute timestamps. Requires the 'mediaref' library but not 'lerobot'.

```python
from mediaref.compat.lerobot import lerobot_episode_to_refs
from mediaref import MediaRef, batch_decode

refs = lerobot_episode_to_refs(
    video_path="videos/observation.images.front_left/chunk-000/file-000.mp4",
    from_timestamp=12.34,
    frame_timestamps=[0.0, 1/30, 2/30, 3/30],
)
print(refs[0])
print(refs[1])

frames = batch_decode(refs)
```

--------------------------------

### PyAV: Sparse Timestamp Queries

Source: https://context7.com/open-world-agents/mediaref/llms.txt

Decode video frames at specific timestamps using PyAVVideoDecoder. Supports metadata extraction, frame retrieval by time, and range decoding with optional FPS resampling.

```python
with PyAVVideoDecoder("episode.mp4") as dec:
    # metadata
    meta = dec.metadata
    print(meta.width, meta.height)         # e.g. 1920 1080
    print(meta.num_frames)
    print(float(meta.duration_seconds))
    print(float(meta.begin_stream_seconds))
    print(float(meta.end_stream_seconds))

    # frames at specific seconds
    batch = dec.get_frames_played_at([0.0, 1.0, 2.5])
    print(batch.data.shape)                # (3, 3, H, W)
    print(batch.pts_seconds)              # actual PTS of each returned frame

    # convert NCHW → HWC for display
    frame_hwc = np.transpose(batch.data[0], (1, 2, 0))  # (H, W, 3) RGB

    # continuous range at native fps
    range_batch = dec.get_frames_played_in_range(0.0, 2.0)
    print(range_batch.data.shape)

    # range with fps resampling
    range_30fps = dec.get_frames_played_in_range(0.0, 2.0, fps=30.0)
```

```python
with PyAVVideoDecoder("video.mkv") as dec:
    single_batch = dec.get_frames_played_at([1_500_000_000 / 1e9])  # pts_ns → seconds
    frame = single_batch.data[0]           # (3, H, W)
```

--------------------------------

### Create DataURI from PIL Image

Source: https://github.com/open-world-agents/mediaref/blob/main/docs/API.md

Constructs a DataURI from a PIL Image object, specifying JPEG format and quality.

```python
from mediaref import MediaRef, DataURI
from PIL import Image
import cv2
import numpy as np

# PIL.Image
pil_img = Image.open("photo.png")
ref = MediaRef(uri=DataURI.from_image(pil_img, format="jpeg", quality=90))
```

--------------------------------

### MediaRef Methods

Source: https://github.com/open-world-agents/mediaref/blob/main/docs/API.md

Methods available on MediaRef objects for decoding media into different formats.

```APIDOC
## `to_ndarray`

### Description
Decodes the media reference into a NumPy array.

### Method
`to_ndarray(format="rgb") -> np.ndarray`

### Parameters
#### Query Parameters
- **format** (string) - Optional - Specifies the output format. Defaults to "rgb".

### Response
#### Success Response (200)
- **np.ndarray**: The decoded media as a NumPy array.
```

```APIDOC
## `to_pil_image`

### Description
Decodes the media reference into a PIL Image object.

### Method
`to_pil_image() -> PIL.Image`

### Response
#### Success Response (200)
- **PIL.Image**: The decoded media as a PIL Image.
```

--------------------------------

### Display MediaRef Messages from Bag

Source: https://github.com/open-world-agents/mediaref/blob/main/examples/ros/README.md

Processes a MediaRef-converted ROS bag and displays a specified number of decoded MediaRef messages.

```bash
# Display MediaRef messages
uv run mediaref_decode.py input_mediaref.bag -n 30
```

--------------------------------

### Run Specific Test

Source: https://github.com/open-world-agents/mediaref/blob/main/CONTRIBUTING.md

Execute a single, specific test case. Useful for debugging a particular test failure. The -x flag stops after the first failure, -v for verbose output, and -s to show print statements.

```bash
pytest tests/test_loading.py::TestToNdarrayImage::test_to_ndarray_from_file -xvs
```

--------------------------------

### Convert MediaRef to lerobot VideoFrame

Source: https://context7.com/open-world-agents/mediaref/llms.txt

Converts a MediaRef object to a lerobot VideoFrame dictionary. Raises ValueError if MediaRef has no timestamp. Requires the 'mediaref' library but not 'lerobot'.

```python
from mediaref.compat.lerobot import to_videoframe
from mediaref import MediaRef

ref2 = MediaRef(uri="videos/observation.mp4", pts_ns=500_000_000)
vf_out = to_videoframe(ref2)
print(vf_out)

try:
    to_videoframe(MediaRef(uri="image.png"))
except ValueError as e:
    print(e)
```

--------------------------------

### MediaRef Class Definition

Source: https://github.com/open-world-agents/mediaref/blob/main/docs/API.md

Defines the MediaRef Pydantic v2 model with URI and optional nanosecond presentation timestamp.

```python
class MediaRef(BaseModel):
    uri: str
    pts_ns: int | None = None
```

--------------------------------

### Converting LeRobot VideoFrame to MediaRef

Source: https://github.com/open-world-agents/mediaref/blob/main/docs/API.md

Converts a single LeRobot VideoFrame dictionary to a MediaRef object. This function handles the conversion of path and timestamp information.

```python
from mediaref.compat.lerobot import (
    from_videoframe, to_videoframe, lerobot_episode_to_refs,
)

# Convert a single VideoFrame dict
ref = from_videoframe({"path": "videos/clip.mp4", "timestamp": 0.5})
# MediaRef(uri='videos/clip.mp4', pts_ns=500000000)
```

--------------------------------

### Create DataURI from NumPy array

Source: https://context7.com/open-world-agents/mediaref/llms.txt

Embeds image data directly into a MediaRef. Supports various input formats like RGB, BGR, and RGBA, and allows specifying output format and JPEG quality.

```python
from mediaref import MediaRef, DataURI
from PIL import Image
import cv2, numpy as np

# --- From numpy RGB array ---
rgb = np.random.randint(0, 255, (100, 100, 3), dtype=np.uint8)
duri = DataURI.from_image(rgb, format="png")
ref = MediaRef(uri=duri)
print(ref.is_embedded)          # True
print(duri.mimetype)            # "image/png"
print(duri.is_image)            # True
print(len(duri))                # URI length in bytes

# --- From OpenCV BGR array (input_format REQUIRED) ---
bgr = cv2.imread("photo.jpg")
duri_bgr = DataURI.from_image(bgr, format="png", input_format="bgr")
ref_bgr = MediaRef(uri=duri_bgr)

# --- JPEG with quality setting ---
duri_jpeg = DataURI.from_image(rgb, format="jpeg", quality=85)
ref_jpeg = MediaRef(uri=duri_jpeg)

# --- RGBA with alpha channel (PNG only preserves alpha) ---
rgba = np.random.randint(0, 255, (64, 64, 4), dtype=np.uint8)
duri_rgba = DataURI.from_image(rgba, format="png", input_format="rgba")

# --- From PIL.Image ---
pil_img = Image.open("photo.png")
duri_pil = DataURI.from_image(pil_img, format="jpeg", quality=90)
```

--------------------------------

### Batch Decode Media References

Source: https://github.com/open-world-agents/mediaref/blob/main/docs/API.md

Decode multiple MediaRef video frames efficiently by grouping references that share a URI. This method is significantly faster than per-reference decoding when references cluster on the same video file. Supports both CPU (PyAV) and GPU-accelerated (TorchCodec) decoding.

```python
from mediaref import MediaRef, batch_decode

refs = [MediaRef(uri="episode.mp4", pts_ns=int(i * 1e9)) for i in range(10)]
frames = batch_decode(refs)                          # default: PyAV (CPU)
frames = batch_decode(refs, decoder="torchcodec")    # GPU-accelerated
```

--------------------------------

### Create DataURI from NumPy RGB Image

Source: https://github.com/open-world-agents/mediaref/blob/main/docs/API.md

Generates a DataURI from a NumPy array representing an RGB image, specifying PNG format.

```python
from mediaref import MediaRef, DataURI
from PIL import Image
import cv2
import numpy as np

# numpy RGB
rgb = np.random.randint(0, 255, (100, 100, 3), dtype=np.uint8)
ref = MediaRef(uri=DataURI.from_image(rgb, format="png"))
```

--------------------------------

### Arrow/Parquet Schema for MediaRef

Source: https://github.com/open-world-agents/mediaref/blob/main/docs/SPEC.md

Defines the structure for a MediaRef in Arrow or Parquet format, where pts_ns is nullable.

```arrow
struct<uri: string, pts_ns: int64>   -- pts_ns nullable
```

--------------------------------

### BibTeX Citation for MediaRef

Source: https://github.com/open-world-agents/mediaref/blob/main/README.md

Use this BibTeX entry to cite the MediaRef software in academic publications. It includes author, title, version, year, DOI, and URL.

```bibtex
@software{mediaref,
  author  = {Choi, Suhwan},
  title   = {MediaRef: a portable frame-level media reference primitive},
  version = {1.0.0},
  year    = {2026},
  doi     = {10.5281/zenodo.19892316},
  url     = {https://github.com/open-world-agents/MediaRef}
}
```

--------------------------------

### Programmatically read MediaRef strings from ROS bag

Source: https://context7.com/open-world-agents/mediaref/llms.txt

Reads MediaRef JSON strings from a converted ROS bag, deserializes them into MediaRef objects, and then batch decodes the frames. Requires 'rosbags' and 'mediaref'.

```python
from rosbags.rosbag1 import Reader
from rosbags.typesys import Stores, get_typestore
from mediaref import MediaRef, batch_decode

typestore = get_typestore(Stores.ROS1_NOETIC)
refs = []

with Reader("recording_mediaref.bag") as reader:
    for conn, timestamp, rawdata in reader.messages():
        if conn.msgtype == "std_msgs/msg/String":
            msg = typestore.deserialize_ros1(rawdata, conn.msgtype)
            ref = MediaRef.model_validate_json(msg.data)
            refs.append(ref)

frames = batch_decode(refs)
print(f"Decoded {len(frames)} frames, shape: {frames[0].shape}")
```

--------------------------------

### MediaRefFeature Configuration

Source: https://github.com/open-world-agents/mediaref/blob/main/docs/API.md

Shows the definition of the MediaRefFeature dataclass. The `decode` parameter controls whether accessing a column returns a `MediaRef` instance or a raw dictionary.

```python
@dataclass
class MediaRefFeature:
    decode: bool = True
```

--------------------------------

### CompressedImage vs. MediaRef Message Format

Source: https://github.com/open-world-agents/mediaref/blob/main/examples/ros/README.md

Compares the typical size and structure of a ROS CompressedImage message with a MediaRef message.

```json
# CompressedImage: ~50KB per message
{"format": "jpeg", "data": b"\xff\xd8\xff\xe0..."}
```

```json
# MediaRef: ~60 bytes per message
{"data": '{"uri": "bag.media/camera.mp4", "pts_ns": 123456789}'}
```

--------------------------------

### Check HuggingFace feature patch status

Source: https://context7.com/open-world-agents/mediaref/llms.txt

Verify if the MediaRef feature patch is enabled for the HuggingFace datasets library.

```bash
mediaref status
```

--------------------------------

### Inspect ROS Bag Contents

Source: https://github.com/open-world-agents/mediaref/blob/main/examples/ros/README.md

Analyzes a ROS bag to display information about its topics and sample messages.

```bash
# Inspect bag contents
uv run bag_info.py input.bag -n 5
```

--------------------------------

### MediaRef Properties

Source: https://github.com/open-world-agents/mediaref/blob/main/docs/API.md

Properties available on MediaRef objects.

```APIDOC
## MediaRef Properties

### Description
Properties providing metadata about the media reference.

### Properties
- **uri** (string): Full data URI string.
- **mimetype** (string): The MIME type of the media (e.g., "image/png").
- **is_image** (boolean): True if the MIME type is an image type.
- **len(data_uri)** (integer): The length of the data URI in bytes.
```

--------------------------------

### batch_decode

Source: https://github.com/open-world-agents/mediaref/blob/main/docs/API.md

Efficiently decodes multiple MediaRef video frames by grouping references with the same URI.

```APIDOC
## `batch_decode`

### Description
Decodes many `MediaRef` video frames efficiently by grouping refs that share a URI, opening each container once, and seeking through the requested timestamps in order. Significantly faster than per-ref decoding when refs cluster on the same video file.

### Method
`batch_decode(refs, decoder="pyav") -> list[np.ndarray]`

### Parameters
#### Path Parameters
- **refs** (list[MediaRef]) - Required - A list of MediaRef objects to decode.
#### Query Parameters
- **decoder** (string) - Optional - The decoder backend to use. Options are "pyav" (default, CPU) or "torchcodec" (GPU-accelerated). Defaults to "pyav".

### Response
#### Success Response (200)
- **list[np.ndarray]**: A list of decoded video frames as NumPy arrays.
```

--------------------------------

### Decode and Save MediaRef Frames as Images

Source: https://github.com/open-world-agents/mediaref/blob/main/examples/ros/README.md

Decodes MediaRef messages from a ROS bag and saves the extracted frames as image files in a specified output directory.

```bash
# Decode frames and save as images
uv run mediaref_decode.py input_mediaref.bag --decode -n 100 -o frames/
```

--------------------------------

### MediaRef Class

Source: https://github.com/open-world-agents/mediaref/blob/main/docs/API.md

The MediaRef class represents a reference to a media file, including its URI and an optional presentation timestamp. It provides methods for loading media, resolving relative paths, and validating URIs.

```APIDOC
## MediaRef Class

### Description
A two-field Pydantic v2 model representing a media reference with a URI and an optional nanosecond presentation timestamp.

### Properties
- `uri` (str): The Uniform Resource Identifier for the media.
- `pts_ns` (int | None): The presentation timestamp in nanoseconds.

### Methods
- `to_ndarray(format="rgb") -> np.ndarray`:
  Loads the media as a numpy array. Supported formats include "rgb", "bgr", "rgba", "bgra", "gray".
- `to_pil_image(format="rgb") -> PIL.Image`:
  Loads the media as a PIL Image. Supported formats include "rgb", "rgba", "gray".
- `resolve_relative_path(base_path, on_unresolvable="warn") -> MediaRef`:
  Resolves a relative path against a base path, returning a new MediaRef.
- `validate_uri() -> bool`:
  Checks if the URI exists (local files only).

### Serialization
- `model_dump()`: Returns a dictionary representation of the MediaRef.
- `model_dump_json()`: Returns a JSON string representation of the MediaRef.
- `MediaRef.model_validate(data)`: Validates and creates a MediaRef from a dictionary.
- `MediaRef.model_validate_json(json_data)`: Validates and creates a MediaRef from a JSON string.
```

--------------------------------

### Direct use of PyAVVideoDecoder and TorchCodecVideoDecoder

Source: https://context7.com/open-world-agents/mediaref/llms.txt

Provides lower-level access to video decoding functionalities. These classes offer methods like `get_frames_played_at` and `get_frames_played_in_range`, returning `FrameBatch` objects with NCHW formatted data.

```python
from mediaref.video_decoder import PyAVVideoDecoder, TorchCodecVideoDecoder
import numpy as np
```

--------------------------------

### Efficiently decode multiple video frames with batch_decode

Source: https://context7.com/open-world-agents/mediaref/llms.txt

Optimizes video decoding by grouping MediaRefs by URI, opening each container once, and processing timestamps monotonically. Supports PyAV and TorchCodec backends, including cloud URIs with PyAV.

```python
from mediaref import MediaRef, batch_decode, cleanup_cache
import numpy as np

# --- Basic usage: 10 frames from one video ---
refs = [MediaRef(uri="episode.mp4", pts_ns=int(i * 1e9)) for i in range(10)]
frames = batch_decode(refs)                        # list of 10 (H, W, 3) uint8 arrays
print(np.stack(frames).shape)                     # (10, H, W, 3)

# --- GPU-accelerated with TorchCodec ---
frames_gpu = batch_decode(refs, decoder="torchcodec")

# --- Mixed videos: refs may span multiple files ---
refs_multi = (
    [MediaRef(uri="cam0.mp4", pts_ns=int(i * 33_333_333)) for i in range(5)] +
    [MediaRef(uri="cam1.mp4", pts_ns=int(i * 33_333_333)) for i in range(5)]
)
# Both videos are opened once each, regardless of interleaving
frames_multi = batch_decode(refs_multi)

# --- Cloud URIs (PyAV backend only; TorchCodec does not support fsspec) ---
cloud_refs = [
    MediaRef(uri="s3://my-bucket/episode.mp4", pts_ns=int(i * 1e9))
    for i in range(5)
]
cloud_frames = batch_decode(cloud_refs)            # range reads via fsspec

hf_refs = [
    MediaRef(uri="hf://datasets/me/clips/cam.mp4", pts_ns=int(i * 1e9))
    for i in range(3)
]
hf_frames = batch_decode(hf_refs)

# --- Cleanup decoder cache between long sessions ---
refs_batch1 = [MediaRef(uri="video.mp4", pts_ns=int(i * 1e9)) for i in range(100)]
batch_decode(refs_batch1)
cleanup_cache()   # frees PyAV container from memory; safe to call with no-op if [video] not installed

# --- Empty input ---
assert batch_decode([]) == []
```

--------------------------------

### Direct decoder use — PyAVVideoDecoder / TorchCodecVideoDecoder

Source: https://context7.com/open-world-agents/mediaref/llms.txt

For advanced control over video decoding, `PyAVVideoDecoder` and `TorchCodecVideoDecoder` can be used directly. These classes inherit from `BaseVideoDecoder` and provide methods like `get_frames_played_at` and `get_frames_played_in_range` for frame retrieval.

```APIDOC
## Direct decoder use — `PyAVVideoDecoder` / `TorchCodecVideoDecoder`

For lower-level control, use the decoder classes directly. Both inherit from `BaseVideoDecoder` and expose `get_frames_played_at(seconds)` (list of timestamps → `FrameBatch`) and `get_frames_played_in_range(start, stop, fps=None)`. `FrameBatch` carries `.data` in NCHW format `(N, 3, H, W)`, `.pts_seconds`, and `.duration_seconds`.

```python
from mediaref.video_decoder import PyAVVideoDecoder, TorchCodecVideoDecoder
import numpy as np

```
```

--------------------------------

### TorchCodec: GPU-accelerated Decoding

Source: https://context7.com/open-world-agents/mediaref/llms.txt

Decode video frames using TorchCodec for GPU acceleration. Returns frames as torch.Tensor, which can be converted to NumPy arrays.

```python
with TorchCodecVideoDecoder("episode.mp4") as dec:
    batch = dec.get_frames_played_at([0.0, 0.5, 1.0])
    frames_np = batch.data.numpy()         # TorchCodec returns torch.Tensor; convert if needed
```

--------------------------------

### MediaRef JSON Message Structure

Source: https://github.com/open-world-agents/mediaref/blob/main/examples/ros/README.md

Shows the standard JSON format for MediaRef messages, including the video URI and presentation timestamp.

```json
{
"uri": "relative/path/to/video.mp4", 
"pts_ns": 1729561964520000000
}
```

--------------------------------

### Decode DataURI to image arrays or PIL Image

Source: https://context7.com/open-world-agents/mediaref/llms.txt

Converts a DataURI back into various image formats, including NumPy arrays (RGB, BGR, RGBA) and PIL Images. Also provides access to the raw decoded bytes.

```python
# --- Decode back to arrays / PIL ---
rgb_out  = duri.to_ndarray()              # (H, W, 3) RGB  (default)
bgr_out  = duri.to_ndarray(format="bgr") # (H, W, 3) BGR
rgba_out = duri.to_ndarray(format="rgba")
pil_out  = duri.to_pil_image()           # PIL.Image RGB

# Access raw decoded bytes
raw_bytes = duri.decoded_data   # bytes (base64-decoded)
```

--------------------------------

### cleanup_cache

Source: https://github.com/open-world-agents/mediaref/blob/main/docs/API.md

Clears the PyAV container cache.

```APIDOC
## `cleanup_cache`

### Description
Clears the PyAV container cache. Call between long-running decode sessions if you want to release decoder memory before automatic eviction.

### Method
`cleanup_cache()`
```