TorchCodec

https://meta-pytorch.org/torchcodec/
Admin
TorchCodec is a Python library for decoding video and audio data into PyTorch tensors on CPU and...
Tokens:74,713
Snippets:666
Trust Score:6.4
Update:2 months ago
Show doc for...
Tokens:
Raw
### Benchmark output example

Source: https://meta-pytorch.org/torchcodec/stable/generated_examples/decoding/approximate_mode.html

Example output showing the performance difference between seek modes.

```text
Sampling clips with seek_mode='exact':
med = 208.62ms +- 22.31
Sampling clips with seek_mode='approximate':
med = 142.84ms +- 27.40
```

--------------------------------

### Encoding Output Example

Source: https://meta-pytorch.org/torchcodec/stable/generated_examples/encoding/video_encoding.html

Example output showing the file size difference between fast and slow encoding presets.

```text
Size of fast encoded file: 7253009 bytes
Size of slow encoded file: 2110051 bytes
```

--------------------------------

### Metadata and accuracy output example

Source: https://meta-pytorch.org/torchcodec/stable/generated_examples/decoding/approximate_mode.html

Example output showing metadata differences and frame verification results.

```text
Metadata of short video with seek_mode='exact':
VideoStreamMetadata:
  duration_seconds_from_header: 13.8
  begin_stream_seconds_from_header: 0
  bit_rate: 505790
  codec: h264
  stream_index: 0
  duration_seconds: 13.8
  begin_stream_seconds: 0
  begin_stream_seconds_from_content: 0
  end_stream_seconds_from_content: 13.8
  width: 640
  height: 360
  num_frames_from_header: 345
  num_frames_from_content: 345
  average_fps_from_header: 25
  pixel_aspect_ratio: 1
  rotation: None
  color_primaries: smpte170m
  color_space: smpte170m
  color_transfer_characteristic: smpte170m
  pixel_format: yuv420p
  end_stream_seconds: 13.8
  num_frames: 345
  average_fps: 25

Metadata of short video with seek_mode='approximate':
VideoStreamMetadata:
  duration_seconds_from_header: 13.8
  begin_stream_seconds_from_header: 0
  bit_rate: 505790
  codec: h264
  stream_index: 0
  duration_seconds: 13.8
  begin_stream_seconds: 0
  begin_stream_seconds_from_content: None
  end_stream_seconds_from_content: None
  width: 640
  height: 360
  num_frames_from_header: 345
  num_frames_from_content: None
  average_fps_from_header: 25
  pixel_aspect_ratio: 1
  rotation: None
  color_primaries: smpte170m
  color_space: smpte170m
  color_transfer_characteristic: smpte170m
  pixel_format: yuv420p
  end_stream_seconds: 13.8
  num_frames: 345
  average_fps: 25

Frame seeking is the same for this video!
```

--------------------------------

### Setup and Video Preparation

Source: https://meta-pytorch.org/torchcodec/stable/_downloads/4bf13eb02a73a239a7e7af99b5c000a9/approximate_mode.ipynb

Downloads a sample video and creates a long version by repeating it 100 times using ffmpeg.

```python
import torch
import requests
import tempfile
from pathlib import Path
import shutil
import subprocess
from time import perf_counter_ns


# Video source: https://www.pexels.com/video/dog-eating-854132/
# License: CC0. Author: Coverr.
url = "https://videos.pexels.com/video-files/854132/854132-sd_640_360_25fps.mp4"
response = requests.get(url, headers={"User-Agent": ""})
if response.status_code != 200:
    raise RuntimeError(f"Failed to download video. {response.status_code = }.")

temp_dir = tempfile.mkdtemp()
short_video_path = Path(temp_dir) / "short_video.mp4"
with open(short_video_path, 'wb') as f:
    for chunk in response.iter_content():
        f.write(chunk)

long_video_path = Path(temp_dir) / "long_video.mp4"
ffmpeg_command = [
    "ffmpeg",
    "-stream_loop", "99",  # repeat video 100 times
    "-i", f"{short_video_path}",
    "-c", "copy",
    f"{long_video_path}"
]
subprocess.run(ffmpeg_command, check=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)

from torchcodec.decoders import VideoDecoder
print(f"Short video duration: {VideoDecoder(short_video_path).metadata.duration_seconds} seconds")
print(f"Long video duration: {VideoDecoder(long_video_path).metadata.duration_seconds / 60} minutes")
```

--------------------------------

### Setup Boilerplate and Helper Functions

Source: https://meta-pytorch.org/torchcodec/stable/_downloads/f189e474be55fc74900ac94fda37f6f0/transforms.ipynb

Utility functions for downloading videos and plotting frames using torchvision and matplotlib.

```python
import torch
import requests
import tempfile
from pathlib import Path
import shutil
from time import perf_counter_ns


def store_video_to(url: str, local_video_path: Path):
    response = requests.get(url, headers={"User-Agent": ""})
    if response.status_code != 200:
        raise RuntimeError(f"Failed to download video. {response.status_code = }.")

    with open(local_video_path, 'wb') as f:
        for chunk in response.iter_content():
            f.write(chunk)


def plot(frames: torch.Tensor, title : str | None = None):
    try:
        from torchvision.utils import make_grid
        from torchvision.transforms.v2.functional import to_pil_image
        import matplotlib.pyplot as plt
    except ImportError:
        print("Cannot plot, please run `pip install torchvision matplotlib`")
        return

    plt.rcParams["savefig.bbox"] = "tight"
    dpi = 300
    fig, ax = plt.subplots(figsize=(800 / dpi, 600 / dpi), dpi=dpi)
    ax.imshow(to_pil_image(make_grid(frames)))
    ax.set(xticklabels=[], yticklabels=[], xticks=[], yticks=[])
    if title is not None:
        ax.set_title(title, fontsize=6)
    plt.tight_layout()
```

--------------------------------

### Benchmark Output Examples

Source: https://meta-pytorch.org/torchcodec/stable/generated_examples/decoding/transforms.html

Sample output showing performance differences between decoder and torchvision transforms.

```text
decoder transforms:    times_med = 1935.59ms +- 40.12
torchvision transform: times_med = 4075.54ms +- 110.53
```

```text
decoder transforms:    times_med = 10765.18ms +- 10.20
torchvision transform: times_med = 12557.40ms +- 6.29
```

--------------------------------

### Setup Benchmarking and Video Processing Utilities

Source: https://meta-pytorch.org/torchcodec/stable/_downloads/1cff79ade5aa64baf192aceab5e06959/parallel_decoding.ipynb

Defines utility functions for benchmarking, splitting indices, and generating a long video file for testing purposes.

```python
import torch
import requests
import tempfile
from pathlib import Path
import subprocess
from time import perf_counter_ns
from datetime import timedelta

from joblib import Parallel, delayed, cpu_count
from torchcodec.decoders import VideoDecoder


def bench(f, *args, num_exp=3, warmup=1, **kwargs):
    """Benchmark a function by running it multiple times and measuring execution time."""
    for _ in range(warmup):
        f(*args, **kwargs)

    times = []
    for _ in range(num_exp):
        start = perf_counter_ns()
        result = f(*args, **kwargs)
        end = perf_counter_ns()
        times.append(end - start)

    return torch.tensor(times).float(), result


def report_stats(times, unit="s"):
    """Report median and standard deviation of benchmark times."""
    mul = {
        "ns": 1,
        "µs": 1e-3,
        "ms": 1e-6,
        "s": 1e-9,
    }[unit]
    times = times * mul
    std = times.std().item()
    med = times.median().item()
    print(f"median = {med:.2f}{unit} ± {std:.2f}")
    return med


def split_indices(indices: list[int], num_chunks: int) -> list[list[int]]:
    """Split a list of indices into approximately equal chunks."""
    chunk_size = len(indices) // num_chunks
    chunks = []

    for i in range(num_chunks - 1):
        chunks.append(indices[i * chunk_size:(i + 1) * chunk_size])

    # Last chunk may be slightly larger
    chunks.append(indices[(num_chunks - 1) * chunk_size:])
    return chunks


def generate_long_video(temp_dir: str):
    # Video source: https://www.pexels.com/video/dog-eating-854132/
    # License: CC0. Author: Coverr.
    url = "https://videos.pexels.com/video-files/854132/854132-sd_640_360_25fps.mp4"
    response = requests.get(url, headers={"User-Agent": ""})
    if response.status_code != 200:
        raise RuntimeError(f"Failed to download video. {response.status_code = }.")

    short_video_path = Path(temp_dir) / "short_video.mp4"
    with open(short_video_path, 'wb') as f:
        for chunk in response.iter_content():
            f.write(chunk)

    # Create a longer video by repeating the short one 50 times
    long_video_path = Path(temp_dir) / "long_video.mp4"
    ffmpeg_command = [
        "ffmpeg", "-y",
        "-stream_loop", "49",  # repeat video 50 times
        "-i", str(short_video_path),
        "-c", "copy",
        str(long_video_path)
    ]
    subprocess.run(ffmpeg_command, check=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)

    return short_video_path, long_video_path


temp_dir = tempfile.mkdtemp()
short_video_path, long_video_path = generate_long_video(temp_dir)

decoder = VideoDecoder(long_video_path, seek_mode="approximate")
metadata = decoder.metadata

short_duration = timedelta(seconds=VideoDecoder(short_video_path).metadata.duration_seconds)
long_duration = timedelta(seconds=metadata.duration_seconds)
print(f"Original video duration: {int(short_duration.total_seconds() // 60)}m{int(short_duration.total_seconds() % 60):02d}s")
print(f"Long video duration: {int(long_duration.total_seconds() // 60)}m{int(long_duration.total_seconds() % 60):02d}s")
print(f"Video resolution: {metadata.width}x{metadata.height}")
print(f"Average FPS: {metadata.average_fps:.1f}")
print(f"Total frames: {metadata.num_frames}")
```

--------------------------------

### FFprobe commands and frame mapping sample

Source: https://meta-pytorch.org/torchcodec/stable/_sources/generated_examples/decoding/custom_frame_mappings.rst.txt

Example commands to extract frame metadata using ffprobe and a sample of the resulting frame mapping structure.

```none
Running ffprobe:
ffprobe -i /tmp/tmp0af6ckf5/long_video.mp4 -select_streams 0 -show_frames -show_entries frame=pts,duration,key_frame -of json

Running ffprobe:
ffprobe -i /tmp/tmp0af6ckf5/short_video.mp4 -select_streams 0 -show_frames -show_entries frame=pts,duration,key_frame -of json

Sample of fields in custom frame mappings:
frame['key_frame'] = 1, frame['pts'] = 0, frame['duration'] = 1
frame['key_frame'] = 0, frame['pts'] = 1, frame['duration'] = 1
frame['key_frame'] = 0, frame['pts'] = 2, frame['duration'] = 1
```

--------------------------------

### Benchmark Output

Source: https://meta-pytorch.org/torchcodec/stable/generated_examples/decoding/custom_frame_mappings.html

Example output showing performance differences between custom mappings and exact seek mode.

```text
Running benchmarks on short_video.mp4
Decoding frames with custom_frame_mappings:
med = 18.40ms +- 0.05
Decoding frames with seek_mode='exact':
med = 18.61ms +- 0.05

Running benchmarks on long_video.mp4
Decoding frames with custom_frame_mappings:
med = 40.63ms +- 0.09
Decoding frames with seek_mode='exact':
med = 54.06ms +- 0.58
```

--------------------------------

### Benchmark output results

Source: https://meta-pytorch.org/torchcodec/stable/generated_examples/decoding/custom_frame_mappings.html

Example output showing performance differences between custom frame mappings and exact seek mode for short and long videos.

```text
Running benchmarks on short_video.mp4
Creating a VideoDecoder object with custom_frame_mappings:
med = 6.20ms +- 0.03
Creating a VideoDecoder object with seek_mode='exact':
med = 6.44ms +- 0.03

Running benchmarks on long_video.mp4
Creating a VideoDecoder object with custom_frame_mappings:
med = 28.34ms +- 0.11
Creating a VideoDecoder object with seek_mode='exact':
med = 42.87ms +- 0.68
```

--------------------------------

### Example Output of Comparison

Source: https://meta-pytorch.org/torchcodec/stable/generated_examples/decoding/basic_cuda_example.html

Shows the expected output when comparing CPU and CUDA decoded frames.

```text
frames_equal=False
mean_abs_diff=tensor(0.6004, device='cuda:0')
max_abs_diff=tensor(2., device='cuda:0')
```

--------------------------------

### Example output for CUDA verification

Source: https://meta-pytorch.org/torchcodec/stable/generated_examples/decoding/basic_cuda_example.html

Sample output showing PyTorch version and detected Nvidia GPU properties.

```text
torch.__version__='2.13.0+cu126'
torch.cuda.is_available()=True
torch.cuda.get_device_properties(0)=_CudaDeviceProperties(name='NVIDIA A10G', major=8, minor=6, total_memory=22587MB, multi_processor_count=80, uuid=70924de9-4c20-9e7a-328a-9dda007113f1, pci_bus_id=0, pci_device_id=30, pci_domain_id=0, L2_cache_size=6MB)
```

--------------------------------

### Download and Setup Video Data

Source: https://meta-pytorch.org/torchcodec/stable/_downloads/c8852e7e664672bdb64c6ba771a9bd30/basic_example.ipynb

Downloads a sample video and defines a plotting utility for visualization. Requires torchvision and matplotlib for the plotting function.

```python
import torch
import requests


# Video source: https://www.pexels.com/video/dog-eating-854132/
# License: CC0. Author: Coverr.
url = "https://videos.pexels.com/video-files/854132/854132-sd_640_360_25fps.mp4"
response = requests.get(url, headers={"User-Agent": ""})
if response.status_code != 200:
    raise RuntimeError(f"Failed to download video. {response.status_code = }.")

raw_video_bytes = response.content


def plot(frames: torch.Tensor, title: str | None = None):
    try:
        from torchvision.utils import make_grid
        from torchvision.transforms.v2.functional import to_pil_image
        import matplotlib.pyplot as plt
    except ImportError:
        print("Cannot plot, please run `pip install torchvision matplotlib`")
        return

    plt.rcParams["savefig.bbox"] = 'tight'
    fig, ax = plt.subplots()
    ax.imshow(to_pil_image(make_grid(frames)))
    ax.set(xticklabels=[], yticklabels=[], xticks=[], yticks=[])
    if title is not None:
        ax.set_title(title)
    plt.tight_layout()
```

--------------------------------

### Verify PyTorch CUDA Configuration

Source: https://meta-pytorch.org/torchcodec/stable/_downloads/0d9d7e92cb34559510d86a36f73e1e53/basic_cuda_example.ipynb

Check the installed PyTorch version, CUDA availability, and device properties to ensure the environment is correctly set up.

```python
import torch

print(f"{torch.__version__=}")
print(f"{torch.cuda.is_available()=}")
print(f"{torch.cuda.get_device_properties(0)=}")
```

--------------------------------

### Accessing clip start timestamps

Source: https://meta-pytorch.org/torchcodec/stable/_sources/generated_examples/decoding/sampling.rst.txt

Retrieves the start timestamps for all clips in the batch.

```Python
clip_starts = clips.pts_seconds[:, 0]
clip_starts
```

--------------------------------

### Configure multiple video and audio streams

Source: https://meta-pytorch.org/torchcodec/stable/_downloads/ccd7596cf7d161af13c32c87cfc7f878/multi_stream_encoding.ipynb

Shows how to add multiple video streams targeting different devices (CPU/CUDA) and multiple audio streams with distinct settings.

```python
encoder = Encoder()

# Two video streams: one on CPU, one on CUDA
cpu_video = encoder.add_video(
    height=1080, width=1920, frame_rate=30,
    device="cpu",
)
cuda_video = encoder.add_video(
    height=720, width=1280, frame_rate=30,
    device="cuda",
)

# Two audio streams with different settings
audio_en = encoder.add_audio(sample_rate=44100, num_channels=2)
audio_fr = encoder.add_audio(sample_rate=44100, num_channels=2)

with encoder.open_file("multi_stream_output.mkv"):
    cpu_video.add_frames(cpu_frames)
    cuda_video.add_frames(cuda_frames)
    audio_en.add_samples(english_samples)
    audio_fr.add_samples(french_samples)
```

--------------------------------

### Prepare video files for testing

Source: https://meta-pytorch.org/torchcodec/stable/_downloads/515bb6477ed8fb530cccbe1b67ef6f0c/custom_frame_mappings.ipynb

Downloads a sample video and uses ffmpeg to create a longer version for testing purposes.

```python
import tempfile
from pathlib import Path
import subprocess
import requests

# Video source: https://www.pexels.com/video/dog-eating-854132/
# License: CC0. Author: Coverr.
url = "https://videos.pexels.com/video-files/854132/854132-sd_640_360_25fps.mp4"
response = requests.get(url, headers={"User-Agent": ""})
if response.status_code != 200:
    raise RuntimeError(f"Failed to download video. {response.status_code = }.")

temp_dir = tempfile.mkdtemp()
short_video_path = Path(temp_dir) / "short_video.mp4"
with open(short_video_path, 'wb') as f:
    for chunk in response.iter_content():
        f.write(chunk)

long_video_path = Path(temp_dir) / "long_video.mp4"
ffmpeg_command = [
    "ffmpeg",
    "-stream_loop", "50",  # repeat video 50 times to get a ~12 min video
    "-i", f"{short_video_path}",
    "-c", "copy",
    f"{long_video_path}"
]
subprocess.run(ffmpeg_command, check=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)

from torchcodec.decoders import VideoDecoder
print(f"Short video duration: {VideoDecoder(short_video_path).metadata.duration_seconds} seconds")
print(f"Long video duration: {VideoDecoder(long_video_path).metadata.duration_seconds / 60} minutes")
```

--------------------------------

### Get Multiple Frames

Source: https://meta-pytorch.org/torchcodec/stable/_modules/torchcodec/decoders/_video_decoder.html

Retrieves a batch of frames at specified indices.

```python
def get_frames_at(self, indices: torch.Tensor | list[int]) -> FrameBatch:
    """Return frames at the given indices.

    Args:
        indices (torch.Tensor or list of int): The indices of the frames to retrieve.

    Returns:
        FrameBatch: The frames at the given indices.
    """

    data, pts_seconds, duration_seconds = core.get_frames_at_indices(
        self._decoder, frame_indices=indices
    )

    return FrameBatch(
        data=data,
        pts_seconds=pts_seconds,
        duration_seconds=duration_seconds,
    )
```

--------------------------------

### Get Single Frame

Source: https://meta-pytorch.org/torchcodec/stable/_modules/torchcodec/decoders/_video_decoder.html

Retrieves a single frame at a specific index.

```python
def get_frame_at(self, index: int) -> Frame:
    """Return a single frame at the given index.

    .. note::

        If you need to decode multiple frames, we recommend using the batch
        methods instead, since they are faster:
        :meth:`~torchcodec.decoders.VideoDecoder.get_frames_at`,
        :meth:`~torchcodec.decoders.VideoDecoder.get_frames_in_range`,
        :meth:`~torchcodec.decoders.VideoDecoder.get_frames_played_at`,
        :meth:`~torchcodec.decoders.VideoDecoder.get_frames_played_in_range`.

    Args:
        index (int): The index of the frame to retrieve.

    Returns:
        Frame: The frame at the given index.
    """
    data, pts_seconds, duration_seconds = core.get_frame_at_index(
        self._decoder, frame_index=index
    )
    return Frame(
        data=data,
        pts_seconds=pts_seconds.item(),
        duration_seconds=duration_seconds.item(),
    )
```

--------------------------------

### Initialize VideoDecoder with pre-downloaded bytes

Source: https://meta-pytorch.org/torchcodec/stable/_downloads/f0caa5fce0f20da68647d009fb8dd8d0/file_like.ipynb

Downloads a sample video to inspect its metadata and establish a baseline for decoding performance.

```python
from torchcodec.decoders import VideoDecoder

nasa_url = "https://download.pytorch.org/torchaudio/tutorial-assets/stream-api/NASAs_Most_Scientifically_Complex_Space_Observatory_Requires_Precision-MP4.mp4"

pre_downloaded_raw_video_bytes = get_url_content(nasa_url)
decoder = VideoDecoder(pre_downloaded_raw_video_bytes)

print(f"Video size in MB: {len(pre_downloaded_raw_video_bytes) // 1024 // 1024}")
print(decoder.metadata)
```

--------------------------------

### Get VideoDecoder Length

Source: https://meta-pytorch.org/torchcodec/stable/_modules/torchcodec/decoders/_video_decoder.html

Returns the total number of frames in the video stream.

```python
    def __len__(self) -> int:
        return self._num_frames
```

--------------------------------

### Filtering clips by timestamp

Source: https://meta-pytorch.org/torchcodec/stable/_sources/generated_examples/decoding/sampling.rst.txt

Filters the batch to include only clips that start after five seconds.

```Python
clips_starting_after_five_seconds = clips[clip_starts > 5]
clips_starting_after_five_seconds
```

--------------------------------

### Example output of FrameBatch

Source: https://meta-pytorch.org/torchcodec/stable/generated_examples/decoding/sampling.html

Represents the structure and data format returned by a sampling operation.

```text
FrameBatch:
  data (shape): torch.Size([3, 4, 3, 360, 640])
  pts_seconds: tensor([[2.0000, 2.4800, 3.0000, 3.4800],
        [3.0000, 3.4800, 4.0000, 4.4800],
        [4.0000, 4.4800, 5.0000, 5.4800]], dtype=torch.float64)
  duration_seconds: tensor([[0.0400, 0.0400, 0.0400, 0.0400],
        [0.0400, 0.0400, 0.0400, 0.0400],
        [0.0400, 0.0400, 0.0400, 0.0400]], dtype=torch.float64)
```

--------------------------------

### FrameBatch Output Representation

Source: https://meta-pytorch.org/torchcodec/stable/generated_examples/decoding/sampling.html

Example output structure of a FrameBatch object returned by the sampler.

```text
FrameBatch:
  data (shape): torch.Size([5, 4, 3, 360, 640])
  pts_seconds: tensor([[11.3600, 11.4800, 11.6000, 11.7200],
        [10.2000, 10.3200, 10.4400, 10.5600],
        [ 9.8000,  9.9200, 10.0400, 10.1600],
        [ 9.6000,  9.7200,  9.8400,  9.9600],
        [ 8.4400,  8.5600,  8.6800,  8.8000]], dtype=torch.float64)
  duration_seconds: tensor([[0.0400, 0.0400, 0.0400, 0.0400],
        [0.0400, 0.0400, 0.0400, 0.0400],
        [0.0400, 0.0400, 0.0400, 0.0400],
        [0.0400, 0.0400, 0.0400, 0.0400],
        [0.0400, 0.0400, 0.0400, 0.0400]], dtype=torch.float64)
```

--------------------------------

### Regular Index Sampling

Source: https://meta-pytorch.org/torchcodec/stable/_modules/torchcodec/samplers/_index_based.html

Samples clips from a video using regularly spaced starting indices.

```python
def clips_at_regular_indices(
    decoder: VideoDecoder,
    *,
    num_clips: int = 1,
    num_frames_per_clip: int = 1,
    num_indices_between_frames: int = 1,
    sampling_range_start: int = 0,
    sampling_range_end: int | None = None,  # interval is [start, end).
    policy: Literal["repeat_last", "wrap", "error"] = "repeat_last",
) -> FrameBatch:
    # See docstring below
    torch._C._log_api_usage_once("torchcodec.samplers.clips_at_regular_indices")
    return _generic_index_based_sampler(
        kind="regular",
        decoder=decoder,
        num_clips=num_clips,
        num_frames_per_clip=num_frames_per_clip,
        num_indices_between_frames=num_indices_between_frames,
        sampling_range_start=sampling_range_start,
        sampling_range_end=sampling_range_end,
        policy=policy,
    )
```

--------------------------------

### Sample with wrap Policy

Source: https://meta-pytorch.org/torchcodec/stable/generated_examples/decoding/sampling.html

Demonstrates sampling near the end of a video using the wrap policy to handle out-of-bounds timestamps.

```python
torch.manual_seed(0)
clips = clips_at_random_timestamps(
    decoder,
    num_clips=1,
    num_frames_per_clip=5,
    seconds_between_frames=0.4,
    sampling_range_start=end_of_video - 1,
    sampling_range_end=end_of_video,
    policy="wrap",
)
clips.pts_seconds
```

```python
tensor([[13.7600, 13.7600, 13.7600, 13.7600, 13.7600]], dtype=torch.float64)
```

--------------------------------

### Random Index Sampling

Source: https://meta-pytorch.org/torchcodec/stable/_modules/torchcodec/samplers/_index_based.html

Samples clips from a video using randomly selected starting indices.

```python
def clips_at_random_indices(
    decoder: VideoDecoder,
    *,
    num_clips: int = 1,
    num_frames_per_clip: int = 1,
    num_indices_between_frames: int = 1,
    sampling_range_start: int = 0,
    sampling_range_end: int | None = None,  # interval is [start, end).
    policy: Literal["repeat_last", "wrap", "error"] = "repeat_last",
) -> FrameBatch:
    # See docstring below
    torch._C._log_api_usage_once("torchcodec.samplers.clips_at_random_indices")
    return _generic_index_based_sampler(
        kind="random",
        decoder=decoder,
        num_clips=num_clips,
        num_frames_per_clip=num_frames_per_clip,
        num_indices_between_frames=num_indices_between_frames,
        sampling_range_start=sampling_range_start,
        sampling_range_end=sampling_range_end,
        policy=policy,
    )
```

--------------------------------

### Prepare video and audio data for encoding

Source: https://meta-pytorch.org/torchcodec/stable/_downloads/ccd7596cf7d161af13c32c87cfc7f878/multi_stream_encoding.ipynb

Downloads a sample video and generates a synthetic sine-wave audio track to be used as input for the encoder.

```python
import subprocess
import tempfile
from pathlib import Path

import requests
import torch
from torchcodec.decoders import VideoDecoder
from torchcodec.encoders import Encoder


# Video source: https://www.pexels.com/video/adorable-cats-on-the-lawn-4977395/
# Author: Altaf Shah.
url = "https://videos.pexels.com/video-files/4977395/4977395-hd_1920_1080_24fps.mp4"

response = requests.get(url, headers={"User-Agent": ""})
if response.status_code != 200:
    raise RuntimeError(f"Failed to download video. {response.status_code = }.")

decoder = VideoDecoder(response.content)
frames = decoder.get_frames_in_range(0, 60).data
frame_rate = decoder.metadata.average_fps

# Generate a 440 Hz sine wave that lasts as long as the video
audio_sample_rate = 16000
duration_seconds = len(frames) / frame_rate
t = torch.linspace(
    0, duration_seconds, int(audio_sample_rate * duration_seconds),
    dtype=torch.float32,
)
audio_samples = torch.sin(2 * torch.pi * 440 * t).unsqueeze(0)  # shape: (1, num_samples)
```

--------------------------------

### Configure Sampling Range

Source: https://meta-pytorch.org/torchcodec/stable/_downloads/98c8aa10d85bcd2ddc4f4456ad29dcbe/sampling.ipynb

Uses sampling_range_start and sampling_range_end to restrict the start times of sampled clips.

```python
from torchcodec.samplers import clips_at_regular_timestamps

clips = clips_at_regular_timestamps(
    decoder,
    seconds_between_clip_starts=1,
    num_frames_per_clip=4,
    seconds_between_frames=0.5,
    sampling_range_start=2,
    sampling_range_end=5
)
clips
```

--------------------------------

### WavDecoder.__init__

Source: https://meta-pytorch.org/torchcodec/stable/_modules/torchcodec/decoders/_wav_decoder.html

Initializes a new WavDecoder instance to decode audio from a specified source.

```APIDOC
## WavDecoder(source)

### Description
Initializes the decoder with a source, which can be a file path, raw bytes, a torch.Tensor, or a file-like object.

### Parameters
- **source** (str | Path | io.RawIOBase | io.BufferedReader | bytes | torch.Tensor) - Required - The source of the audio data.
```

--------------------------------

### WavDecoder Implementation

Source: https://meta-pytorch.org/torchcodec/stable/_modules/torchcodec/decoders/_wav_decoder.html

The WavDecoder class handles initialization from various sources and provides methods to retrieve audio samples within specific time ranges.

```python
# Copyright (c) Meta Platforms, Inc. and affiliates.
# All rights reserved.
#
# This source code is licensed under the BSD-style license found in the
# LICENSE file in the root directory of this source tree.


import io
import json
from pathlib import Path

import torch
from torchcodec import _core, AudioSamples
from torchcodec._core._decoder_utils import create_wav_decoder
from torchcodec._core._metadata import AudioStreamMetadata



[docs]
class WavDecoder:
    """A fast decoder for WAV audio files.

    This is a lightweight, high-performance alternative to
    :class:`~torchcodec.decoders.AudioDecoder` that is specialized for WAV
    files. See :ref:`sphx_glr_generated_examples_decoding_performance_tips.py`
    for more details.

    Unlike :class:`~torchcodec.decoders.AudioDecoder`, this decoder does not
    support resampling (``sample_rate`` parameter) or channel remixing
    (``num_channels`` parameter). If you need those features, use
    :class:`~torchcodec.decoders.AudioDecoder`.

    Returned samples are float samples normalized in [-1, 1].

    Args:
        source (str, ``Pathlib.path``, bytes, ``torch.Tensor`` or file-like
            object): The source of the audio:

            - If ``str``: a path to a local WAV file.
            - If ``Pathlib.path``: a path to a local WAV file.
            - If ``bytes`` object or ``torch.Tensor``: the raw WAV data.
            - If file-like object: we read audio data from the object on
              demand. The object must expose the methods ``read(self, size:
              int) -> bytes`` and ``seek(self, offset: int, whence: int) ->
              int``.

    Attributes:
        metadata (AudioStreamMetadata): Metadata of the audio stream.
        stream_index (int): The stream index. Always 0 for WAV files.
    """

    def __init__(
        self,
        source: str | Path | io.RawIOBase | io.BufferedReader | bytes | torch.Tensor,
    ):
        torch._C._log_api_usage_once("torchcodec.decoders.WavDecoder")

        self._decoder = create_wav_decoder(source)
        self._source = source
        self.stream_index = 0

        metadata_json = _core.get_wav_metadata_from_decoder(self._decoder)
        metadata_dict = json.loads(metadata_json)

        self.metadata = AudioStreamMetadata(
            sample_rate=metadata_dict["sampleRate"],
            num_channels=metadata_dict["numChannels"],
            sample_format=metadata_dict["sampleFormat"],
            duration_seconds=metadata_dict["durationSeconds"],
            stream_index=metadata_dict["streamIndex"],
            codec=metadata_dict["codec"],
            bit_rate=metadata_dict["bitRate"],
            duration_seconds_from_header=metadata_dict["durationSecondsFromHeader"],
            begin_stream_seconds=metadata_dict["beginStreamSeconds"],
            begin_stream_seconds_from_header=None,  # WAV format lacks stream start time metadata
        )



[docs]
    def get_all_samples(self) -> AudioSamples:
        """Returns all the audio samples from the source.

        To decode samples in a specific range, use
        :meth:`~torchcodec.decoders.WavDecoder.get_samples_played_in_range`.

        Returns:
            AudioSamples: The samples within the file.
        """
        return self.get_samples_played_in_range()



[docs]
    def get_samples_played_in_range(
        self, start_seconds: float = 0.0, stop_seconds: float | None = None
    ) -> AudioSamples:
        """Returns audio samples in the given range.

        Samples are in the half open range [start_seconds, stop_seconds).

        To decode all the samples from beginning to end, you can call this
        method while leaving ``start_seconds`` and ``stop_seconds`` to their
        default values, or use
        :meth:`~torchcodec.decoders.WavDecoder.get_all_samples` as a more
        convenient alias.

        Args:
            start_seconds (float): Time, in seconds, of the start of the
                range. Default: 0.
            stop_seconds (float or None): Time, in seconds, of the end of the
                range. As a half open range, the end is excluded. Default: None,
                which decodes samples until the end.

        Returns:
            AudioSamples: The samples within the specified range.
        """
        if stop_seconds is not None and not start_seconds <= stop_seconds:
            raise ValueError(
                f"Invalid start seconds: {start_seconds}. It must be less than or equal to stop seconds ({stop_seconds})."
            )
```

--------------------------------

### Get Item Operator

Source: https://meta-pytorch.org/torchcodec/stable/_modules/torchcodec/decoders/_video_decoder.html

Retrieves a frame or batch of frames using integer indexing or slicing.

```python
def __getitem__(self, key: numbers.Integral | slice) -> Tensor:
    """Return frame or frames as tensors, at the given index or range.

    .. note::

        If you need to decode multiple frames, we recommend using the batch
        methods instead, since they are faster:
        :meth:`~torchcodec.decoders.VideoDecoder.get_frames_at`,
        :meth:`~torchcodec.decoders.VideoDecoder.get_frames_in_range`,
        :meth:`~torchcodec.decoders.VideoDecoder.get_frames_played_at`, and
        :meth:`~torchcodec.decoders.VideoDecoder.get_frames_played_in_range`.

    Args:
        key(int or slice): The index or range of frame(s) to retrieve.

    Returns:
        torch.Tensor: The frame or frames at the given index or range.
    """
    if isinstance(key, numbers.Integral):
        return self._getitem_int(int(key))
    elif isinstance(key, slice):
        return self._getitem_slice(key)

    raise TypeError(
        f"Unsupported key type: {type(key)}. Supported types are int and slice."
    )
```

--------------------------------

### Manual Codec Selection

Source: https://meta-pytorch.org/torchcodec/stable/_downloads/e8868f804bffbeae56ed6b335353e809/video_encoding.ipynb

Demonstrates how to specify different codecs like libx264 or hevc to control output quality and file size, followed by verification using ffprobe.

```python
# H.264 encoding
h264_output = tempfile.NamedTemporaryFile(suffix=".mp4", delete=False).name
encoder = Encoder()
vs = encoder.add_video(
    height=frames.shape[2], width=frames.shape[3],
    frame_rate=frame_rate, codec="libx264",
)
with encoder.open_file(h264_output):
    vs.add_frames(frames)

# H.265 encoding
hevc_output = tempfile.NamedTemporaryFile(suffix=".mp4", delete=False).name
encoder = Encoder()
vs = encoder.add_video(
    height=frames.shape[2], width=frames.shape[3],
    frame_rate=frame_rate, codec="hevc",
)
with encoder.open_file(hevc_output):
    vs.add_frames(frames)

# Now let's use ffprobe to verify the codec used in the output files
import subprocess

for output, name in [(h264_output, "h264_output"), (hevc_output, "hevc_output")]:
    result = subprocess.run(
        [
            "ffprobe",
            "-v",
            "error",
            "-select_streams",
            "v:0",
            "-show_entries",
            "stream=codec_name",
            "-of",
            "default=noprint_wrappers=1:nokey=1",
            output,
        ],
        capture_output=True,
        text=True,
    )
    print(f"Codec used in {name}: {result.stdout.strip()}")
```

--------------------------------

### Build Clip Timestamps

Source: https://meta-pytorch.org/torchcodec/stable/_modules/torchcodec/samplers/_time_based.html

Calculates a list of timestamps for all clips based on start times and frame intervals.

```python
def _build_all_clips_timestamps(
    *,
    clip_start_seconds: torch.Tensor,  # 1D float tensor
    num_frames_per_clip: int,
    seconds_between_frames: float,
    end_stream_seconds: float,
    policy_fun: _POLICY_FUNCTION_TYPE,
) -> list[float]:

    all_clips_timestamps: list[float] = []
    for start_seconds in clip_start_seconds:
        clip_timestamps = [
            timestamp
            for i in range(num_frames_per_clip)
            if (timestamp := start_seconds + i * seconds_between_frames)
            < end_stream_seconds
        ]

        if len(clip_timestamps) < num_frames_per_clip:
            clip_timestamps = policy_fun(clip_timestamps, num_frames_per_clip)
        all_clips_timestamps += clip_timestamps

    return all_clips_timestamps
```

--------------------------------

### VideoDecoder benchmark output

Source: https://meta-pytorch.org/torchcodec/stable/generated_examples/decoding/approximate_mode.html

Example output showing the performance difference between exact and approximate seek modes.

```text
Creating a VideoDecoder object with seek_mode='exact' on a short video:
med = 6.46ms +- 0.03
Creating a VideoDecoder object with seek_mode='approximate' on a short video:
med = 5.81ms +- 0.03

Creating a VideoDecoder object with seek_mode='exact' on a long video:
med = 76.70ms +- 2.04
Creating a VideoDecoder object with seek_mode='approximate' on a long video:
med = 8.44ms +- 0.04
```

--------------------------------

### Initialize File-like Object Wrapper

Source: https://meta-pytorch.org/torchcodec/stable/_sources/generated_examples/decoding/file_like.rst.txt

Imports necessary modules for creating custom file-like object wrappers.

```Python
from pathlib import Path
import tempfile
```

--------------------------------

### Decode samples within a range

Source: https://meta-pytorch.org/torchcodec/stable/generated_examples/decoding/audio_decoding.html

Decodes a specific segment of the audio stream using start and stop times.

```python
samples = decoder.get_samples_played_in_range(start_seconds=10, stop_seconds=70)

print(samples)
play_5s(samples)
```

```text
AudioSamples:
  data (shape): torch.Size([2, 2646000])
  pts_seconds: 10.0
  duration_seconds: 60.0
  sample_rate: 44100
```

--------------------------------

### Validate sampling range

Source: https://meta-pytorch.org/torchcodec/stable/_modules/torchcodec/samplers/_index_based.html

Calculates and validates the start and end frame indices for sampling, supporting negative indexing.

```python
def _validate_sampling_range_index_based(
    *,
    num_indices_between_frames,
    num_frames_per_clip,
    sampling_range_start,
    sampling_range_end,
    num_frames_in_video,
):
    if sampling_range_start < 0:
        sampling_range_start = num_frames_in_video + sampling_range_start

    if sampling_range_start >= num_frames_in_video:
        raise ValueError(
            f"sampling_range_start ({sampling_range_start}) must be smaller than "
            f"the number of frames ({num_frames_in_video})."
        )

    clip_span = _get_clip_span(
        num_indices_between_frames=num_indices_between_frames,
        num_frames_per_clip=num_frames_per_clip,
    )

    if sampling_range_end is None:
        sampling_range_end = max(num_frames_in_video - clip_span + 1, 1)
        if sampling_range_start >= sampling_range_end:
            raise ValueError(
                f"We determined that sampling_range_end should be {sampling_range_end}, "
                "but it is smaller than or equal to sampling_range_start "
                f"({sampling_range_start})."
            )
    else:
        if sampling_range_end < 0:
            # Support negative values so that -1 means last frame.
            sampling_range_end = num_frames_in_video + sampling_range_end
        sampling_range_end = min(sampling_range_end, num_frames_in_video)
        if sampling_range_start >= sampling_range_end:
            raise ValueError(
                f"sampling_range_start ({sampling_range_start}) must be smaller than "
                f"sampling_range_end ({sampling_range_end})."
            )

    return sampling_range_start, sampling_range_end
```

--------------------------------

### Initialize and add video to encoder

Source: https://meta-pytorch.org/torchcodec/stable/_modules/torchcodec/encoders/_video_encoder.html

Configures an encoder instance and adds a video stream using provided codec and quality parameters.

```python
encoder = Encoder()
video = encoder.add_video(
    **self._get_add_video_kwargs(
        codec=codec,
        pixel_format=pixel_format,
        crf=crf,
        preset=preset,
        extra_options=extra_options,
    )
)
with encoder.open_file(dest):
    video.add_frames(self._frames)
```

--------------------------------

### Sample video clips with different policies

Source: https://meta-pytorch.org/torchcodec/stable/_downloads/98c8aa10d85bcd2ddc4f4456ad29dcbe/sampling.ipynb

Demonstrates how to use the 'repeat_last' and 'wrap' policies in clips_at_random_timestamps to handle sampling near the end of a video.

```python
torch.manual_seed(0)
clips = clips_at_random_timestamps(
    decoder,
    num_clips=1,
    num_frames_per_clip=5,
    seconds_between_frames=0.4,
    sampling_range_start=end_of_video - 1,
    sampling_range_end=end_of_video,
    policy="repeat_last",
)
clips.pts_seconds
```

```python
torch.manual_seed(0)
clips = clips_at_random_timestamps(
    decoder,
    num_clips=1,
    num_frames_per_clip=5,
    seconds_between_frames=0.4,
    sampling_range_start=end_of_video - 1,
    sampling_range_end=end_of_video,
    policy="wrap",
)
clips.pts_seconds
```

--------------------------------

### Initialize VideoDecoder

Source: https://meta-pytorch.org/torchcodec/stable/_downloads/c8852e7e664672bdb64c6ba771a9bd30/basic_example.ipynb

Creates a decoder instance from raw video bytes. Local file paths are also supported as input.

```python
from torchcodec.decoders import VideoDecoder

# You can also pass a path to a local file!
decoder = VideoDecoder(raw_video_bytes)
```

--------------------------------

### Sample clips at random timestamps

Source: https://meta-pytorch.org/torchcodec/stable/_modules/torchcodec/samplers/_time_based.html

Samples a specified number of clips starting at random timestamps within a defined range.

```python
def clips_at_random_timestamps(
    decoder,
    *,
    num_clips: int = 1,
    num_frames_per_clip: int = 1,
    seconds_between_frames: float | None = None,
    # None means "begining", which may not always be 0
    sampling_range_start: float | None = None,
    sampling_range_end: float | None = None,  # interval is [start, end).
    policy: Literal["repeat_last", "wrap", "error"] = "repeat_last",
) -> FrameBatch:
    # See docstring below
    torch._C._log_api_usage_once("torchcodec.samplers.clips_at_random_timestamps")
    return _generic_time_based_sampler(
        kind="random",
        decoder=decoder,
        num_clips=num_clips,
        seconds_between_clip_starts=None,
        num_frames_per_clip=num_frames_per_clip,
        seconds_between_frames=seconds_between_frames,
        sampling_range_start=sampling_range_start,
        sampling_range_end=sampling_range_end,
        policy=policy,
    )
```

--------------------------------

### Filter Clips by Timestamp

Source: https://meta-pytorch.org/torchcodec/stable/_downloads/98c8aa10d85bcd2ddc4f4456ad29dcbe/sampling.ipynb

Demonstrates filtering a FrameBatch object using PyTorch indexing based on clip start times.

```python
clip_starts = clips.pts_seconds[:, 0]
clip_starts
```

```python
clips_starting_after_five_seconds = clips[clip_starts > 5]
clips_starting_after_five_seconds
```

--------------------------------

### Encode video and audio streams

Source: https://meta-pytorch.org/torchcodec/stable/_downloads/ccd7596cf7d161af13c32c87cfc7f878/multi_stream_encoding.ipynb

Initializes the encoder, adds video and audio streams, and writes the prepared data to an output file.

```python
output_path = tempfile.NamedTemporaryFile(suffix=".mp4", delete=False).name
encoder = Encoder()
video_stream = encoder.add_video(
    height=frames.shape[2], width=frames.shape[3], frame_rate=frame_rate,
)
audio_stream = encoder.add_audio(sample_rate=audio_sample_rate, num_channels=1)

with encoder.open_file(output_path):
    video_stream.add_frames(frames)
    audio_stream.add_samples(audio_samples)

print(f"Encoded video + audio to {output_path}")
print(f"Output size: {Path(output_path).stat().st_size} bytes")
```

--------------------------------

### get_frames_in_range(start: int, stop: int, step: int = 1) -> FrameBatch

Source: https://meta-pytorch.org/torchcodec/stable/generated/torchcodec.decoders.VideoDecoder.html

Retrieves multiple frames within a specified index range.

```APIDOC
## get_frames_in_range(start, stop, step=1)

### Description
Return multiple frames at the given index range [start, stop).

### Parameters
- **start** (int) - Required - Index of the first frame to retrieve.
- **stop** (int) - Required - End of indexing range (exclusive).
- **step** (int) - Optional - Step size between frames.

### Returns
- **FrameBatch** - The frames within the specified range.
```

--------------------------------

### Download Sample Video File

Source: https://meta-pytorch.org/torchcodec/stable/_downloads/0d9d7e92cb34559510d86a36f73e1e53/basic_cuda_example.ipynb

Use urllib to retrieve a sample H.264 video file for testing the decoding pipeline.

```python
import urllib.request

video_file = "video.mp4"
urllib.request.urlretrieve(
    "https://download.pytorch.org/torchaudio/tutorial-assets/stream-api/NASAs_Most_Scientifically_Complex_Space_Observatory_Requires_Precision-MP4_small.mp4",
    video_file,
)
```

--------------------------------

### Validate Time-Based Sampling Range

Source: https://meta-pytorch.org/torchcodec/stable/_modules/torchcodec/samplers/_time_based.html

Calculates and validates the start and end timestamps for sampling, ensuring clips remain within the stream duration.

```python
def _validate_sampling_range_time_based(
    *,
    num_frames_per_clip,
    seconds_between_frames,
    sampling_range_start,
    sampling_range_end,
    begin_stream_seconds,
    end_stream_seconds,
):

    if sampling_range_start is None:
        sampling_range_start = begin_stream_seconds
    else:
        if sampling_range_start < begin_stream_seconds:
            raise ValueError(
                f"sampling_range_start ({sampling_range_start}) must be at least {begin_stream_seconds}"
            )
        if sampling_range_start >= end_stream_seconds:
            raise ValueError(
                f"sampling_range_start ({sampling_range_start}) must be smaller than {end_stream_seconds}"
            )

    if sampling_range_end is None:
        # We allow a clip to start anywhere within
        # [sampling_range_start, sampling_range_end)
        # When sampling_range_end is None, we want to automatically set it to
        # the largest possible value such that the sampled frames in any clip
        # are within the bounds of the video duration (in other words, we don't
        # want to have to resort to the `policy`).
        # I.e. we want to guarantee that for all frames in any clip we have
        # pts < end_stream_seconds.
        #
        # The frames of a clip will be sampled at the following pts:
        # clip_timestamps = [
        #  clip_start + 0 * seconds_between_frames,
        #  clip_start + 1 * seconds_between_frames,
        #  clip_start + 2 * seconds_between_frames,
        #  ...
        #  clip_start + (num_frames_per_clip - 1) * seconds_between_frames,
        # ]
        # To guarantee that any such value is < end_stream_seconds, we only need
        # to guarantee that
        # clip_start < end_stream_seconds - (num_frames_per_clip - 1) * seconds_between_frames
        #
        # So that's the value of sampling_range_end we want to use.
        sampling_range_end = (
            end_stream_seconds - (num_frames_per_clip - 1) * seconds_between_frames
        )
    elif sampling_range_end <= begin_stream_seconds:
        raise ValueError(
            f"sampling_range_end ({sampling_range_end}) must be at least {begin_stream_seconds}"
        )

    if sampling_range_start >= sampling_range_end:
        raise ValueError(
            f"sampling_range_start ({sampling_range_start}) must be smaller than sampling_range_end ({sampling_range_end})"
        )

    sampling_range_end = min(sampling_range_end, end_stream_seconds)

    return sampling_range_start, sampling_range_end
```
Tokens:
Raw