### Install Dependencies for Synchformer

Source: https://github.com/v-iashin/synchformer/blob/main/example.ipynb

Use these commands to clone the repository and install necessary Python packages. Ensure pip is updated first. Note potential issues with specific av package versions.

```python
# !git clone https://github.com/v-iashin/Synchformer.git
# !pip install pip==23  # run this first
# # NOTE: `av>=9.1.1` causing worse accuracy (see issue #11),
# # but the installation of <= 10.0.0 versions to Google Colab is difficult (help is needed).
# !pip install omegaconf==2.0.6 av==10.0 einops timm==0.6.7
```

--------------------------------

### Synchformer Model Initialization and Inference

Source: https://context7.com/v-iashin/synchformer/llms.txt

Demonstrates how to load the Synchformer model from a configuration file, prepare input data for visual and audio streams, and perform inference to get predicted offset logits.

```APIDOC
## Synchformer — Main synchronization model

### Description
The `Synchformer` class wraps audio and visual feature extractors, linear projection heads, and a `GlobalTransformer` to predict the temporal offset class between audio and video. The forward pass returns a `(loss, logits)` tuple; `logits` shape is `(B, num_offset_classes)`, typically 21 classes spanning -2 to +2 seconds.

### Method Signature
```python
model(vis: torch.Tensor, aud: torch.Tensor) -> Tuple[Optional[torch.Tensor], torch.Tensor]
```

### Parameters
- **vis** (torch.Tensor) - Visual input tensor with shape `(B, S, Tv, C, H, W)`.
- **aud** (torch.Tensor) - Audio input tensor with shape `(B, S, 1, F, Ta)`.

### Returns
- **loss** (Optional[torch.Tensor]) - The computed loss if targets are provided, otherwise None.
- **logits** (torch.Tensor) - The predicted logits for each offset class, with shape `(B, num_offset_classes)`.

### Request Example
```python
import torch
from omegaconf import OmegaConf
from utils.utils import instantiate_from_config

# Load config and build the model
cfg = OmegaConf.load('./configs/sync.yaml')
model = instantiate_from_config(cfg.model)
model = model.to('cuda')
model.eval()

# Input shapes:
#   vis: (B, S, Tv, C, H, W) = (1, 14, 16, 3, 224, 224)  — 14 segments × 16 frames
#   aud: (B, S, 1, F, Ta)    = (1, 14, 1, 128, 66)        — 14 segments × mel spectrogram
B, S = 1, 14
vis = torch.randn(B, S, 16, 3, 224, 224, device='cuda')
aud = torch.randn(B, S, 1, 128, 66, device='cuda')

with torch.no_grad():
    with torch.autocast('cuda', enabled=True):
        loss, logits = model(vis, aud)  # loss=None when targets=None

probs = torch.softmax(logits, dim=-1)
predicted_class = logits.argmax(dim=-1).item()   # integer offset class index
# Expected output shape: logits = torch.Size([1, 21])
print(f'Predicted offset class: {predicted_class}')
```
```

--------------------------------

### Instantiate and Run Synchformer Model

Source: https://context7.com/v-iashin/synchformer/llms.txt

Loads a Synchformer model from configuration and performs a forward pass with random visual and audio inputs. Use this to get initial predictions without calculating loss.

```python
import torch
from omegaconf import OmegaConf
from utils.utils import instantiate_from_config

# Load config and build the model
cfg = OmegaConf.load('./configs/sync.yaml')
model = instantiate_from_config(cfg.model)
model = model.to('cuda')
model.eval()

# Input shapes:
#   vis: (B, S, Tv, C, H, W) = (1, 14, 16, 3, 224, 224)  — 14 segments × 16 frames
#   aud: (B, S, 1, F, Ta)    = (1, 14, 1, 128, 66)        — 14 segments × mel spectrogram
B, S = 1, 14
vis = torch.randn(B, S, 16, 3, 224, 224, device='cuda')
aud = torch.randn(B, S, 1, 128, 66, device='cuda')

with torch.no_grad():
    with torch.autocast('cuda', enabled=True):
        loss, logits = model(vis, aud)  # loss=None when targets=None

probs = torch.softmax(logits, dim=-1)
predicted_class = logits.argmax(dim=-1).item()   # integer offset class index
# Expected output shape: logits = torch.Size([1, 21])
print(f'Predicted offset class: {predicted_class}')
```

--------------------------------

### Install Conda Environment for Synchformer

Source: https://github.com/v-iashin/synchformer/blob/main/README.md

Creates and activates a conda environment using the provided yml file. It's recommended to check the 'av' package version after installation, as specific versions might be required for reproducibility.

```bash
conda env create -f conda_env.yml
# check the installation, if `av` is > 9.0.0, you may need to downgrade to reproduce the results
conda activate synchformer
conda list av
# # Name                    Version                   Build  Channel
# av                        9.0.0                    pypi_0    pypi
```

--------------------------------

### Initialize Device for PyTorch

Source: https://github.com/v-iashin/synchformer/blob/main/example.ipynb

Sets the PyTorch device to CUDA if available, otherwise defaults to CPU. This is a standard setup for deep learning tasks.

```python
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
```

--------------------------------

### Install Synchformer Dependencies

Source: https://github.com/v-iashin/synchformer/blob/main/example.ipynb

Installs necessary Python packages for the Synchformer project. This includes PyTorch, torchvision, torchaudio, and other specialized libraries. Ensure your conda environment is activated.

```bash
pip install -r requirements.txt
```

--------------------------------

### Process Video with FFmpeg

Source: https://github.com/v-iashin/synchformer/blob/main/example.ipynb

Constructs and executes FFmpeg commands to process video files, including changing frame rate, resizing, and converting audio. Requires ffmpeg to be installed and in the system's PATH.

```python
cmd = f'{which_ffmpeg()}'
    cmd += ' -hide_banner -loglevel panic'
    cmd += f' -y -i {path}'
    # 1) change fps, 2) resize: min(H,W)=MIN_SIDE (vertical vids are supported), 3) change audio framerate
    cmd += f" -vf fps={vfps},scale=iw*{in_size}/'min(iw,ih)':ih*{in_size}/'min(iw,ih)',crop='trunc(iw/2)'*2:'trunc(ih/2)'*2"
    cmd += f" -ar {afps}"
    cmd += f' {new_path}'
    subprocess.call(cmd.split())
    cmd = f'{which_ffmpeg()}'
    cmd += ' -hide_banner -loglevel panic'
    cmd += f' -y -i {new_path}'
    cmd += f' -acodec pcm_s16le -ac 1'
    cmd += f' {new_path.replace(".mp4", ".wav")}'
    subprocess.call(cmd.split())
    return new_path
```

--------------------------------

### Run Synchformer Prediction with Negative Offset

Source: https://github.com/v-iashin/synchformer/blob/main/README.md

Applies a negative temporal offset to the audio track of a video and runs the Synchformer prediction. Requires specifying the experiment name, video path, offset in seconds, and the start index for the visual track. Note potential 'av' package version issues.

```bash
python example.py \
  --exp_name "24-01-04T16-39-21" \
  --vid_path "./data/vggsound/h264_video_25fps_256side_16000hz_aac/ZYc410CE4Rg_0_10000.mp4" \
  --offset_sec -2.0 \
  --v_start_i_sec 4.0

# Prediction Results:
# p=0.8291 (12.7734), "-2.00" (0)
# ...
```

--------------------------------

### Run Training Stages with main.py

Source: https://context7.com/v-iashin/synchformer/llms.txt

Entry point for training scripts, dispatching to different actions based on configuration. Supports overriding parameters via CLI and multi-GPU training.

```bash
# Stage 1: Contrastive pre-training of segment-level feature extractors (AVCLIP)
python main.py \
    config=./configs/segment_avclip.yaml \
    logging.logdir=/path/to/logs/ \
    data.vids_path=/path/to/lrs3/h264_uncropped_25fps_256side_16000hz_aac/ \
    data.dataset.target=dataset.lrs.LRS3 \
    training.base_batch_size=2

# Stage 2: Train synchronization module (feature extractors frozen)
S1_CKPT_ID="23-12-22T16-04-18"
python main.py \
    config=./configs/sync.yaml \
    logging.logdir=/path/to/logs/ \
    data.vids_path=/path/to/lrs3/h264_uncropped_25fps_256side_16000hz_aac/ \
    data.dataset.target=dataset.lrs.LRS3 \
    model.params.vfeat_extractor.params.ckpt_path="/path/to/logs/${S1_CKPT_ID}/checkpoints/epoch_best.pt" \
    model.params.afeat_extractor.params.ckpt_path="/path/to/logs/${S1_CKPT_ID}/checkpoints/epoch_best.pt" \
    training.base_batch_size=16

# Multi-GPU training (4 GPUs on one node)
torchrun --standalone --nnodes=1 --nproc-per-node=4 main.py \
    config=./configs/sync.yaml \
    logging.logdir=/path/to/logs/ \
    data.vids_path=/path/to/vggsound/ \
    data.dataset.target=dataset.vggsound.VGGSound \
    training.base_batch_size=16

# Stage 3: Fine-tune for synchronizability (binary classification)
python main.py \
    config=./configs/ft_synchability.yaml \
    training.finetune=True \
    ckpt_path="/path/to/logs/${S2_CKPT_ID}/${S2_CKPT_ID}.pt" \
    data.dataset.target=dataset.audioset.AudioSet \
    data.vids_path=/path/to/audioset/ \
    logging.logdir=/path/to/logs/
```

--------------------------------

### End-to-end Inference with Synchformer (CLI)

Source: https://context7.com/v-iashin/synchformer/llms.txt

Demonstrates how to run end-to-end inference on a video file using the Synchformer model via the command line. This includes downloading a pre-trained model and specifying input video path and inference parameters.

```bash
# Download pre-trained model automatically, then run inference:
python example.py \
  --exp_name "24-01-04T16-39-21" \
  --vid_path "./data/vggsound/h264_video_25fps_256side_16000hz_aac/3qesirWAGt4_20000_30000.mp4" \
  --offset_sec 1.6 \
  --v_start_i_sec 0.0 \
  --device cuda:0

# Expected output:
# Ground Truth offset (sec): 1.60 (18)
# Prediction Results:
# p=0.8076 (11.5469), "1.60" (18)
# p=0.0821 (9.1234), "1.40" (17)
# ...
```

--------------------------------

### Train SynchFormer Model

Source: https://github.com/v-iashin/synchformer/blob/main/README.md

Use this command to initiate the training of the SynchFormer model. Ensure you have the correct paths for configuration, logging, and video data. Consider adding `logging.use_wandb=True` for Weights & Biases logging. For DDP issues, try downgrading `av` to `8.1.0`.

```bash
python ./main.py \
     config=./configs/segment_avclip.yaml \
     logging.logdir=/path/to/logging_dir/ \
     data.vids_path=/path/to/lrs3/h264_uncropped_25fps_256side_16000hz_aac/ \
     data.dataset.target=dataset.lrs.LRS3 \
     training.base_batch_size=2
```

--------------------------------

### Re-encode Video Utility

Source: https://github.com/v-iashin/synchformer/blob/main/example.ipynb

A Python function to re-encode videos using FFmpeg. It allows specifying video and audio frame rates, and input size. Ensure FFmpeg is installed and accessible in your environment.

```python
import subprocess
from pathlib import Path

import torch
import torchaudio
import torchvision
from omegaconf import OmegaConf

from dataset.dataset_utils import get_video_and_audio
from dataset.transforms import make_class_grid, quantize_offset
from utils.utils import check_if_file_exists_else_download, which_ffmpeg
from scripts.train_utils import get_model, get_transforms, prepare_inputs


def reencode_video(path, vfps=25, afps=16000, in_size=256):
    assert which_ffmpeg() != '', 'Is ffmpeg installed? Check if the conda environment is activated.'
    new_path = Path.cwd() / 'vis' / f'{Path(path).stem}_{vfps}fps_{in_size}side_{afps}hz.mp4'
    new_path.parent.mkdir(exist_ok=True)
    new_path = str(new_path)
    cmd = f'{which_ffmpeg()}'
    # no info/error printing
    
```

--------------------------------

### Download Pre-trained Synchronization Models

Source: https://github.com/v-iashin/synchformer/blob/main/README.md

Links to download pre-trained synchronization models. Each entry includes configuration and checkpoint files.

```text
23-12-23T18-33-57 | LRS3 ('Full Scene') | LRS3 ('Full Scene') | LRS3 ('Full Scene') | 86.6 / 99.6 | [config](https://a3s.fi/swift/v1/AUTH_a235c0f452d648828f745589cde1219a/sync/sync_models/23-12-23T18-33-57/cfg-23-12-23T18-33-57.yaml) / [ckpt](https://a3s.fi/swift/v1/AUTH_a235c0f452d648828f745589cde1219a/sync/sync_models/23-12-23T18-33-57/23-12-23T18-33-57.pt) (md5: `4415276...`)
```

```text
24-01-02T10-00-53 | VGGSound            |            VGGSound |     VGGSound-Sparse | 43.8 / 60.2 | [config](https://a3s.fi/swift/v1/AUTH_a235c0f452d648828f745589cde1219a/sync/sync_models/24-01-02T10-00-53/cfg-24-01-02T10-00-53.yaml) / [ckpt](https://a3s.fi/swift/v1/AUTH_a235c0f452d648828f745589cde1219a/sync/sync_models/24-01-02T10-00-53/24-01-02T10-00-53.pt) (md5: `19592ed...`)
```

```text
24-01-04T16-39-21 | AudioSet            |            AudioSet |     VGGSound-Sparse | 47.2 / 67.4 | [config](https://a3s.fi/swift/v1/AUTH_a235c0f452d648828f745589cde1219a/sync/sync_models/24-01-04T16-39-21/cfg-24-01-04T16-39-21.yaml) / [ckpt](https://a3s.fi/swift/v1/AUTH_a235c0f452d648828f745589cde1219a/sync/sync_models/24-01-04T16-39-21/24-01-04T16-39-21.pt) (md5: `54037d2...`)
```

--------------------------------

### Load Video and Audio with get_video_and_audio

Source: https://context7.com/v-iashin/synchformer/llms.txt

Loads RGB frames and audio from an MP4 file. Use `start_sec` and `end_sec` to load specific time windows.

```python
from dataset.dataset_utils import get_video_and_audio

# Load full video
rgb, audio, meta = get_video_and_audio('./data/vggsound/h264_video_25fps_256side_16000hz_aac/sample.mp4',
                                       get_meta=True)
print(f"RGB: {rgb.shape}, dtype={rgb.dtype}")       # e.g. (250, 3, 256, 256), torch.uint8
print(f"Audio: {audio.shape}, dtype={audio.dtype}") # e.g. (400000,), torch.float32
print(f"Video FPS: {meta['video']['fps'][0]}")       # 25.0
print(f"Audio sample rate: {meta['audio']['framerate'][0]}")  # 16000.0

# Load a time window (3–8 seconds)
rgb_clip, audio_clip, meta = get_video_and_audio('./path/to/video.mp4',
                                                  get_meta=True, start_sec=3.0, end_sec=8.0)
print(f"Clip RGB: {rgb_clip.shape}")    # (~125, 3, H, W)
print(f"Clip audio: {audio_clip.shape}")  # (~80000,)
```

--------------------------------

### Instantiate and Configure Synchformer Model

Source: https://context7.com/v-iashin/synchformer/llms.txt

Builds a Synchformer model, optionally freezes feature extractors, moves it to the target device, and wraps it in DDP if initialized. Loads checkpoint weights.

```python
import torch
from omegaconf import OmegaConf
from scripts.train_utils import get_model

cfg = OmegaConf.load('./configs/sync.yaml')
device = torch.device('cuda:0')
torch.cuda.set_device(device)

# model is the (possibly DDP-wrapped) model, model_without_ddp is always the unwrapped module
model, model_without_ddp = get_model(cfg, device)

print(f'Trainable params: {sum(p.numel() for p in model.parameters() if p.requires_grad):,}')
# Feature extractors are frozen by default in sync.yaml (is_trainable: False)

# Load a checkpoint
ckpt = torch.load('./logs/sync_models/24-01-04T16-39-21/24-01-04T16-39-21.pt',
                  map_location='cpu', weights_only=False)
model_without_ddp.load_state_dict(ckpt['model'])
model.eval()
```

--------------------------------

### Resume SynchFormer Training

Source: https://github.com/v-iashin/synchformer/blob/main/README.md

To resume a previous training session, specify the checkpoint ID and set `training.resume="latest"`. Ensure the configuration path points to the correct experiment folder.

```bash
CKPT_ID="xx-xx-xxTxx-xx-xx"  # replace this with the exp folder name

python main.py \
    config="/path/to/logging_dir/$CKPT_ID/cfg-$CKPT_ID.yaml" \
    training.resume="latest"
```

--------------------------------

### Download Synchronizability Prediction Model

Source: https://github.com/v-iashin/synchformer/blob/main/README.md

Link to download the synchronizability prediction model checkpoint and its configuration file.

```text
`24-01-22T20-34-52` | AudioSet | VGGSound-Sparse | 73.5 | 0.83 | [config](https://a3s.fi/swift/v1/AUTH_a235c0f452d648828f745589cde1219a/sync/sync_models/24-01-22T20-34-52/cfg-24-01-22T20-34-52.yaml) / [ckpt](https://a3s.fi/swift/v1/AUTH_a235c0f452d648828f745589cde1219a/sync/sync_models/24-01-22T20-34-52/24-01-22T20-34-52.pt) (md5: `b1cb346...`)
```

--------------------------------

### Initialize GlobalTransformerWithSyncabilityHead

Source: https://context7.com/v-iashin/synchformer/llms.txt

Instantiates the `GlobalTransformerWithSyncabilityHead` for binary synchronizability prediction. The model is set to evaluation mode and then used for a forward pass with dummy visual and audio tokens to demonstrate output shape.

```python
import torch
from model.sync_model import GlobalTransformerWithSyncabilityHead

sync_head_model = GlobalTransformerWithSyncabilityHead(
    tok_pdrop=0.0,
    embd_pdrop=0.1,
    resid_pdrop=0.1,
    attn_pdrop=0.1,
    n_layer=3,
    n_head=8,
    n_embd=768,
)
sync_head_model.eval()

B, Sv, Sa, D = 2, 112, 84, 768
vis_tokens = torch.randn(B, Sv, D)
aud_tokens = torch.randn(B, Sa, D)

with torch.no_grad():
    logits_sync = sync_head_model(vis_tokens, aud_tokens)
    # shape: (B, 2) — binary syncability classification
    probs = torch.softmax(logits_sync, dim=-1)
    is_syncable = probs[:, 1] > 0.5
    print(f'Syncability logits: {logits_sync}')
    print(f'Is syncable: {is_syncable.tolist()}')  # e.g. [True, False]
```

--------------------------------

### Download Pretrained Checkpoints

Source: https://context7.com/v-iashin/synchformer/llms.txt

Utility function to check for the existence of checkpoint or config files and automatically download them if missing. Supports progress bar display. Files are saved under ./logs/sync_models/<exp_name>/.

```python
from utils.utils import check_if_file_exists_else_download

# Download a pre-trained synchronization model (AudioSet-trained, ICASSP 2024)
exp_name = "24-01-04T16-39-21"
check_if_file_exists_else_download(f'./logs/sync_models/{exp_name}/cfg-{exp_name}.yaml')
check_if_file_exists_else_download(f'./logs/sync_models/{exp_name}/{exp_name}.pt')
# Files are saved under ./logs/sync_models/<exp_name>/

# Available checkpoint IDs:
# "24-01-04T16-39-21" — Synchformer trained on AudioSet (best for general use)
# "23-12-23T18-33-57" — Synchformer trained on LRS3 (86.6% Acc@1)
# "24-01-02T10-00-53" — Synchformer trained on VGGSound
# "24-01-22T20-34-52" — Synchronizability model (fine-tuned from AudioSet)
```

--------------------------------

### Resume Training Synchformer Model

Source: https://github.com/v-iashin/synchformer/blob/main/README.md

Use this command to resume training from a checkpoint. Ensure the CKPT_ID and config path are correctly set.

```bash
CKPT_ID="xx-xx-xxTxx-xx-xx"  # replace this with an exp folder name

python main.py \
    config="/path/to/logging_dir/$CKPT_ID/cfg-$CKPT_ID.yaml" \
    training.resume="True" training.finetune="False"
```

--------------------------------

### Prepare Data Command

Source: https://github.com/v-iashin/synchformer/blob/main/README.md

This command is used to prepare data following the SparseSync procedure. Refer to the SparseSync repository for detailed instructions on data preparation for LRS3 and VGGSound datasets.

```bash
bash ./scripts/prepare_data.sh
```

--------------------------------

### main.py

Source: https://context7.com/v-iashin/synchformer/llms.txt

The main training script for Synchformer, which dispatches to different training stages (feature extraction pre-training, synchronization model training, fine-tuning) based on configuration.

```APIDOC
## main.py

### Description
The main training script dispatches to either AVCLIP feature extractor pre-training or synchronization model training based on `cfg.action`. Configuration is loaded from a YAML file and can be overridden via CLI key-value pairs using OmegaConf's merge mechanism.

### Usage Examples

**Stage 1: Contrastive pre-training of segment-level feature extractors (AVCLIP)**
```bash
python main.py \
    config=./configs/segment_avclip.yaml \
    logging.logdir=/path/to/logs/ \
    data.vids_path=/path/to/lrs3/h264_uncropped_25fps_256side_16000hz_aac/ \
    data.dataset.target=dataset.lrs.LRS3 \
    training.base_batch_size=2
```

**Stage 2: Train synchronization module (feature extractors frozen)**
```bash
S1_CKPT_ID="23-12-22T16-04-18"
python main.py \
    config=./configs/sync.yaml \
    logging.logdir=/path/to/logs/ \
    data.vids_path=/path/to/lrs3/h264_uncropped_25fps_256side_16000hz_aac/ \
    data.dataset.target=dataset.lrs.LRS3 \
    model.params.vfeat_extractor.params.ckpt_path="/path/to/logs/${S1_CKPT_ID}/checkpoints/epoch_best.pt" \
    model.params.afeat_extractor.params.ckpt_path="/path/to/logs/${S1_CKPT_ID}/checkpoints/epoch_best.pt" \
    training.base_batch_size=16
```

**Multi-GPU training (4 GPUs on one node)**
```bash
torchrun --standalone --nnodes=1 --nproc-per-node=4 main.py \
    config=./configs/sync.yaml \
    logging.logdir=/path/to/logs/ \
    data.vids_path=/path/to/vggsound/ \
    data.dataset.target=dataset.vggsound.VGGSound \
    training.base_batch_size=16
```

**Stage 3: Fine-tune for synchronizability (binary classification)**
```bash
python main.py \
    config=./configs/ft_synchability.yaml \
    training.finetune=True \
    ckpt_path="/path/to/logs/${S2_CKPT_ID}/${S2_CKPT_ID}.pt" \
    data.dataset.target=dataset.audioset.AudioSet \
    data.vids_path=/path/to/audioset/ \
    logging.logdir=/path/to/logs/
```
```

--------------------------------

### Initialize GenerateMultipleSegments Transform

Source: https://context7.com/v-iashin/synchformer/llms.txt

Initializes the GenerateMultipleSegments transform for splitting video into fixed-length segments. Use `is_start_random=True` for training.

```python
import torch
from dataset.transforms import GenerateMultipleSegments

transform = GenerateMultipleSegments(
    segment_size_vframes=16,   # 16 frames per segment @ 25fps = 0.64s
    n_segments=14,             # 14 segments total
    is_start_random=False,     # center crop (use True for training)
    step_size_seg=0.5,         # 50% overlap between segments
)

item = {
    'video': torch.randint(0, 256, (125, 3, 224, 224), dtype=torch.uint8),  # 5s clip
    'audio': torch.zeros(80000),                                              # 5s @ 16kHz
    'meta': {'video': {'fps': [25]}, 'audio': {'framerate': [16000]}},
    'path': '/path/to/video.mp4',
}

item = transform(item)
print(f"Segmented video shape: {item['video'].shape}")  # (14, 16, 3, 224, 224)
print(f"Segmented audio shape: {item['audio'].shape}")  # (14, 5120)
```

--------------------------------

### Initialize TemporalCropAndOffset Transform

Source: https://context7.com/v-iashin/synchformer/llms.txt

Shows the import statement for `TemporalCropAndOffset`, a `torch.nn.Module` used for temporal cropping and injecting synthetic audio offsets. This transform is part of the data pipeline for creating training data.

```python
import torch
from dataset.transforms import TemporalCropAndOffset
```

--------------------------------

### Evaluate Synchronizability with Torchrun

Source: https://github.com/v-iashin/synchformer/blob/main/README.md

Run this command to evaluate the synchronizability of the model. Ensure S3_CKPT_ID is set to your checkpoint ID and adjust paths accordingly. The a/vfeat_extractor paths are set to null as they are included in the synchronization model checkpoint.

```bash
S3_CKPT_ID="xx-xx-xxTxx-xx-xx"

torchrun --standalone --nnodes=1 --nproc-per-node=1 --master_addr=localhost --master_port=1234 \
scripts/test_syncability.py \
    config_sync="/path/to/logging_dir/${S3_CKPT_ID}/cfg-${S3_CKPT_ID}.yaml" \
    ckpt_path_sync="/path/to/logging_dir/${S3_CKPT_ID}/${S3_CKPT_ID}_best.pt" \
    logging.logdir="/path/to/logging_dir" \
    training.finetune=False \
    training.run_test_only=True \
    data.dataset.target=dataset.vggsound.VGGSoundSparsePickedCleanTest \
    data.vids_path="path/to/vggsound/h264_video_25fps_256side_16000hz_aac/" \
    data.n_segments=14 \
    data.dataset.params.iter_times=25 \
    data.dataset.params.load_fixed_offsets_on="[]" \
    logging.log_code_state=False \
    model.params.afeat_extractor.params.ckpt_path=null \
    model.params.vfeat_extractor.params.ckpt_path=null \
    training.base_batch_size=8 \
    logging.log_frequency=4 \
    logging.use_wandb=False
```

--------------------------------

### Build Audio Augmentation Pipeline

Source: https://context7.com/v-iashin/synchformer/llms.txt

Creates a pipeline of stochastic audio augmentation transforms applied during training. Each transform has a probability of application and specific parameters.

```python
import torch
from dataset.transforms import (
    AudioRandomVolume, AudioRandomPitchShift, AudioRandomLowpassFilter,
    AudioRandomGaussNoise, AudioRandomReverb,
)
import torchvision

# Build an augmentation pipeline applied with probability 0.2 each
aug_pipeline = torchvision.transforms.Compose([
    AudioRandomReverb(p=0.2),                              # wet reverb
    AudioRandomVolume(p=0.2, gain=2.0, gain_type='amplitude'),  # random loudness
    AudioRandomPitchShift(p=0.2, shift=1000),              # ±1000 cents pitch shift
    AudioRandomLowpassFilter(p=0.2, cutoff_freq=100),      # low-pass filter
    AudioRandomGaussNoise(p=0.2, amplitude=0.01),          # additive Gaussian noise
])

item = {
    'audio': torch.randn(80000),  # 5s @ 16kHz
    'meta': {'audio': {'framerate': [16000]}},
}
item = aug_pipeline(item)
print(f"Augmented audio: {item['audio'].shape}")  # (80000,)
```

--------------------------------

### Create Offset Classification Grid and Quantize Offset

Source: https://context7.com/v-iashin/synchformer/llms.txt

Demonstrates the use of `make_class_grid` to create a uniform grid of offset values and `quantize_offset` to snap an arbitrary offset to the nearest grid element. This is crucial for defining discrete offset classes.

```python
from dataset.transforms import make_class_grid, quantize_offset

# Standard 21-class grid from -2.0s to +2.0s (step = 0.2s)
grid = make_class_grid(-2.0, 2.0, 21)
print(grid)
# tensor([-2.0000, -1.8000, -1.6000, ..., 1.8000,  2.0000])

# Snap an arbitrary offset to the nearest grid element
grid_val, class_idx = quantize_offset(grid, 1.55)
print(f'Nearest grid value: {grid_val:.2f}s, class index: {class_idx}')
# Nearest grid value: 1.60s, class index: 18
```

--------------------------------

### End-to-end Inference with Synchformer (Programmatic)

Source: https://context7.com/v-iashin/synchformer/llms.txt

Shows how to perform end-to-end inference programmatically. This involves loading a pre-trained model, preparing input data with necessary transforms, and decoding the prediction.

```python
# Programmatic usage
import torch, torchvision
from omegaconf import OmegaConf
from scripts.train_utils import get_model, get_transforms, prepare_inputs
from dataset.dataset_utils import get_video_and_audio
from dataset.transforms import make_class_grid
from example import patch_config, decode_single_video_prediction

exp_name = "24-01-04T16-39-21"
cfg = OmegaConf.load(f'./logs/sync_models/{exp_name}/cfg-{exp_name}.yaml')
cfg = patch_config(cfg)
device = torch.device('cuda:0')

_, model = get_model(cfg, device)
ckpt = torch.load(f'./logs/sync_models/{exp_name}/{exp_name}.pt', map_location='cpu', weights_only=False)
model.load_state_dict(ckpt['model'])
model.eval()

rgb, audio, meta = get_video_and_audio('./path/to/video.mp4', get_meta=True)
item = dict(video=rgb, audio=audio, meta=meta, path='./path/to/video.mp4', split='test',
            targets={'v_start_i_sec': 0.0, 'offset_sec': 1.6})

grid = make_class_grid(-cfg.data.max_off_sec, cfg.data.max_off_sec,
                       cfg.model.params.transformer.params.off_head_cfg.params.out_features)
item = get_transforms(cfg, ['test'])['test'](item)
batch = torch.utils.data.default_collate([item])
aud, vid, targets = prepare_inputs(batch, device)

with torch.no_grad():
    with torch.autocast('cuda', enabled=cfg.training.use_half_precision):
        _, logits = model(vid, aud)

decode_single_video_prediction(logits, grid, item)
```

--------------------------------

### Run Synchformer Prediction with Positive Offset

Source: https://github.com/v-iashin/synchformer/blob/main/README.md

Applies a positive temporal offset to the audio track of a video and runs the Synchformer prediction. Requires specifying the experiment name, video path, and the offset in seconds. Note potential 'av' package version issues.

```bash
python example.py \
  --exp_name "24-01-04T16-39-21" \
  --vid_path "./data/vggsound/h264_video_25fps_256side_16000hz_aac/3qesirWAGt4_20000_30000.mp4" \
  --offset_sec 1.6

# Prediction Results:
# p=0.8076 (11.5469), "1.60" (18)
# ...
```

--------------------------------

### Initialize TemporalCropAndOffset Transform

Source: https://context7.com/v-iashin/synchformer/llms.txt

Initializes the TemporalCropAndOffset transform for video data augmentation. It applies temporal cropping and grid-based offsets.

```python
transform = TemporalCropAndOffset(
    crop_len_sec=5.0,
    max_off_sec=2.0,
    offset_type='grid',
    grid_size=21,
    do_offset=True,
    segment_size_vframes=16,
    n_segments=14,
    step_size_seg=0.5,
    vfps=25.0,
)

v_fps, a_fps = 25, 16000
duration = 10.0
item = {
    'video': torch.randint(0, 256, (int(duration * v_fps), 3, 224, 224), dtype=torch.uint8),
    'audio': torch.zeros(int(duration * a_fps)),
    'targets': {},
    'meta': {
        'video': {'fps': [v_fps], 'duration': [duration]},
        'audio': {'framerate': [a_fps], 'duration': [duration]},
    },
    'path': '/path/to/video.mp4',
    'split': 'train',
}
item = transform(item)
print(f"Video shape: {item['video'].shape}")    # (125, 3, 224, 224)  — 5s × 25fps
print(f"Audio shape: {item['audio'].shape}")    # (80000,)            — 5s × 16000hz
print(f"Offset sec: {item['targets']['offset_sec']}")
print(f"Offset label (class idx): {item['targets']['offset_target']}")
```

--------------------------------

### Synchformer Forward Pass with Loss Calculation

Source: https://context7.com/v-iashin/synchformer/llms.txt

Illustrates the `Synchformer.forward` method, which includes the optional calculation of loss when target classes are provided. This is useful for training and evaluation.

```APIDOC
## Synchformer.forward — Forward pass with optional loss

### Description
Runs the full pipeline: extract visual features → extract audio features → project both to a shared embedding dimension → flatten segment sequences → pass through `GlobalTransformer` → return `(loss, logits)`. This method can compute the loss if target labels are supplied.

### Method Signature
```python
model(vis: torch.Tensor, aud: torch.Tensor, targets: torch.Tensor) -> Tuple[torch.Tensor, torch.Tensor]
```

### Parameters
- **vis** (torch.Tensor) - Visual input tensor with shape `(B, S, Tv, C, H, W)`.
- **aud** (torch.Tensor) - Audio input tensor with shape `(B, S, 1, F, Ta)`.
- **targets** (torch.Tensor) - Ground truth class indices for the batch, shape `(B,)`.

### Returns
- **loss** (torch.Tensor) - The computed cross-entropy loss.
- **logits** (torch.Tensor) - The predicted logits for each offset class, with shape `(B, num_offset_classes)`.

### Request Example
```python
import torch
from omegaconf import OmegaConf
from utils.utils import instantiate_from_config
from dataset.transforms import make_class_grid, quantize_offset

cfg = OmegaConf.load('./configs/sync.yaml')
model = instantiate_from_config(cfg.model).to('cuda').eval()

vis = torch.randn(2, 14, 16, 3, 224, 224, device='cuda')  # batch of 2
aud = torch.randn(2, 14, 1, 128, 66, device='cuda')
# Target: class index for each item in the batch (0 to num_off_cls-1)
targets = torch.tensor([10, 5], device='cuda')  # class index 10 = 0.0s, 5 = -1.0s

with torch.no_grad():
    with torch.autocast('cuda'):
        loss, logits = model(vis, aud, targets=targets)

print(f'Loss: {loss.item():.4f}')         # cross-entropy loss
print(f'Logits shape: {logits.shape}')    # torch.Size([2, 21])
print(f'Predicted classes: {logits.argmax(dim=-1).tolist()}')
```
```

--------------------------------

### check_if_file_exists_else_download

Source: https://context7.com/v-iashin/synchformer/llms.txt

Utility function to check for the existence of checkpoint or config files and automatically download them from object storage if they are missing. It also supports displaying a progress bar during download.

```APIDOC
## check_if_file_exists_else_download

### Description
Checks whether a checkpoint or config file exists at the given path and automatically downloads it from the project's object storage if missing. Supports progress bar display.

### Parameters
- **path** (str) - The path to the file to check for or download.

### Returns
None. The function modifies the filesystem by downloading the file if it's missing.

### Usage Example
```python
from utils.utils import check_if_file_exists_else_download

# Download a pre-trained synchronization model (AudioSet-trained, ICASSP 2024)
exp_name = "24-01-04T16-39-21"
check_if_file_exists_else_download(f'./logs/sync_models/{exp_name}/cfg-{exp_name}.yaml')
check_if_file_exists_else_download(f'./logs/sync_models/{exp_name}/{exp_name}.pt')
# Files are saved under ./logs/sync_models/<exp_name>/

# Available checkpoint IDs:
# "24-01-04T16-39-21" — Synchformer trained on AudioSet (best for general use)
# "23-12-23T18-33-57" — Synchformer trained on LRS3 (86.6% Acc@1)
# "24-01-02T10-00-53" — Synchformer trained on VGGSound
# "24-01-22T20-34-52" — Synchronizability model (fine-tuned from AudioSet)
```
```

--------------------------------

### Build Train/Test Transform Pipelines

Source: https://context7.com/v-iashin/synchformer/llms.txt

Instantiates torchvision.transforms.Compose pipelines from YAML configuration. Each entry maps to a target class path and parameters.

```python
import torchvision
from omegaconf import OmegaConf
from scripts.train_utils import get_transforms

cfg = OmegaConf.load('./configs/sync.yaml')
cfg.data.vids_path = '/path/to/vggsound/h264_video_25fps_256side_16000hz_aac'

transforms = get_transforms(cfg, which_transforms=['train', 'test'])
```

--------------------------------

### Load Synchformer Model

Source: https://github.com/v-iashin/synchformer/blob/main/example.ipynb

Loads a Synchformer model from specified configuration and checkpoint paths. It includes logic to download missing files and prepares the model for evaluation.

```python
cfg_path = f'./logs/sync_models/{exp_name}/cfg-{exp_name}.yaml'
ckpt_path = f'./logs/sync_models/{exp_name}/{exp_name}.pt'

# if the model does not exist try to download it from the server
check_if_file_exists_else_download(cfg_path)
check_if_file_exists_else_download(ckpt_path)

# load config
cfg = OmegaConf.load(cfg_path)

# patch config
cfg = patch_config(cfg)

_, model = get_model(cfg, device)
ckpt = torch.load(ckpt_path, map_location=torch.device('cpu'))
model.load_state_dict(ckpt['model'])
model.eval()
print('Model loaded.')
```

--------------------------------

### Set Video Processing Parameters

Source: https://github.com/v-iashin/synchformer/blob/main/example.ipynb

Defines key parameters for video processing, including video and audio frame rates, and input size for scaling.

```python
vfps = 25
afps = 16000
in_size = 256
exp_name = '24-01-04T16:39:21'
```

--------------------------------

### Build Audio Mel Spectrogram Pipeline

Source: https://context7.com/v-iashin/synchformer/llms.txt

Constructs the audio preprocessing pipeline including Mel spectrogram extraction, log scaling, padding/truncation, and normalization. Ensure sample rate and other parameters match your audio data.

```python
import torch
import torchvision
from dataset.transforms import AudioMelSpectrogram, AudioLog, PadOrTruncate, AudioNormalizeAST

# Build the pipeline manually (same as in sync.yaml)
audio_pipeline = torchvision.transforms.Compose([
    AudioMelSpectrogram(sample_rate=16000, win_length=400, hop_length=160, n_fft=1024, n_mels=128),
    AudioLog(eps=1e-6),
    PadOrTruncate(max_spec_t=66),
    AudioNormalizeAST(mean=-4.2677393, std=4.5689974),
])

item = {
    'audio': torch.zeros(80000),  # 5s @ 16kHz raw waveform
    'meta': {'audio': {'framerate': [16000.0]}},
}
item = audio_pipeline(item)
print(f"Mel spectrogram shape: {item['audio'].shape}")  # (128, 66) — [n_mels, time_frames]
```

--------------------------------

### Train Audio-Visual Synchronization Module

Source: https://github.com/v-iashin/synchformer/blob/main/README.md

This command trains the synchronization module while keeping feature extractors frozen. It requires specifying the paths to the pre-trained feature extractors from a previous stage.

```bash
S1_CKPT_ID="xx-xx-xxTxx-xx-xx"  # replace this with an exp folder name
EPOCH="best"

python main.py \
    config=./configs/sync.yaml \
    logging.logdir=/path/to/logging_dir/ \
    data.vids_path=/path/to/lrs3/h264_uncropped_25fps_256side_16000hz_aac/ \
    data.dataset.target=dataset.lrs.LRS3 \
    model.params.vfeat_extractor.params.ckpt_path="/path/to/logging_dir/${S1_CKPT_ID}/checkpoints/epoch_${EPOCH}.pt" \
    model.params.afeat_extractor.params.ckpt_path="/path/to/logging_dir/${S1_CKPT_ID}/checkpoints/epoch_${EPOCH}.pt" \
    training.base_batch_size=16
```

--------------------------------

### Download Segment-level Feature Extractors

Source: https://github.com/v-iashin/synchformer/blob/main/README.md

Links to download pre-trained segment-level feature extractors. Each entry provides configuration and checkpoint files.

```text
`23-12-22T16-04-18` | LRS3 ('Full Scene') | [config](https://a3s.fi/swift/v1/AUTH_a235c0f452d648828f745589cde1219a/sync/sync_models/23-12-22T16-04-18/cfg-23-12-22T16-04-18.yaml) / [ckpt](https://a3s.fi/swift/v1/AUTH_a235c0f452d648828f745589cde1219a/sync/sync_models/23-12-22T16-04-18/checkpoints/epoch_best.pt) (md5: `20b6e55...`)
```

```text
`23-12-22T16-10-50` | VGGSound            | [config](https://a3s.fi/swift/v1/AUTH_a235c0f452d648828f745589cde1219a/sync/sync_models/23-12-22T16-10-50/cfg-23-12-22T16-10-50.yaml) / [ckpt](https://a3s.fi/swift/v1/AUTH_a235c0f452d648828f745589cde1219a/sync/sync_models/23-12-22T16-10-50/checkpoints/epoch_best.pt) (md5: `a9979df...`)
```

```text
`23-12-22T16-13-38` | Audioset            | [config](https://a3s.fi/swift/v1/AUTH_a235c0f452d648828f745589cde1219a/sync/sync_models/23-12-22T16-13-38/cfg-23-12-22T16-13-38.yaml) / [ckpt](https://a3s.fi/swift/v1/AUTH_a235c0f452d648828f745589cde1219a/sync/sync_models/23-12-22T16-13-38/checkpoints/epoch_best.pt) (md5: `4a566f2...`)
```