### Install DeepGaze

Source: https://github.com/matthias-k/deepgaze/blob/main/_autodocs/README.md

Standard installation command for the library.

```bash
pip install deepgaze_pytorch
```

--------------------------------

### Install DeepGaze with CLIP support

Source: https://github.com/matthias-k/deepgaze/blob/main/_autodocs/README.md

Installation command including dependencies for DeepGazeMSDB.

```bash
pip install deepgaze_pytorch git+https://github.com/openai/CLIP.git einops
```

--------------------------------

### Install DeepGaze and Dependencies

Source: https://github.com/matthias-k/deepgaze/blob/main/_autodocs/DOCUMENTATION.txt

Commands to install the core library and optional dependencies for specific models.

```bash
pip install deepgaze_pytorch
pip install git+https://github.com/openai/CLIP.git  # For DeepGazeMSDB
pip install einops  # For DINOv2
```

--------------------------------

### DeepGazeMSDB Usage Example

Source: https://github.com/matthias-k/deepgaze/blob/main/_autodocs/models.md

Full workflow for preparing inputs and running inference with both specific and averaged dataset parameters.

```python
import torch
import numpy as np
from scipy.misc import face
from scipy.ndimage import zoom
from scipy.special import logsumexp
from deepgaze_pytorch import DeepGazeMSDB, MSDBDataset

device = 'cuda'
model = DeepGazeMSDB(pretrained=True).to(device)

image = face()  # shape (768, 1024, 3)
image_tensor = torch.tensor([image.transpose(2, 0, 1)]).float().to(device)

# Centerbias preprocessing
centerbias_template = np.load('centerbias_mit1003.npy')
centerbias = zoom(
    centerbias_template,
    (image.shape[0] / centerbias_template.shape[0],
     image.shape[1] / centerbias_template.shape[1]),
    order=0, mode='nearest'
)
centerbias -= logsumexp(centerbias)
centerbias_tensor = torch.tensor([centerbias]).float().to(device)

# For known dataset (MIT1003) with specific display calibration (35 PPD)
with torch.no_grad():
    log_density = model(
        image_tensor,
        centerbias_tensor,
        pixel_per_dva=35.0,
        dataset=MSDBDataset.MIT1003
    )

# For unknown dataset using averaged parameters (e.g., custom setup at 21.7 PPD)
with torch.no_grad():
    log_density = model(
        image_tensor,
        centerbias_tensor,
        pixel_per_dva=21.7,
        dataset=None
    )
```

--------------------------------

### DeepGazeI Usage Example

Source: https://github.com/matthias-k/deepgaze/blob/main/_autodocs/models.md

Full workflow for preparing inputs and generating a saliency prediction using DeepGazeI.

```python
import torch
import numpy as np
from scipy.ndimage import zoom
from scipy.special import logsumexp
from deepgaze_pytorch import DeepGazeI

device = 'cuda'
model = DeepGazeI(pretrained=True).to(device)

# Load an image (e.g., scipy.misc.face())
image = np.random.randint(0, 256, (768, 1024, 3), dtype=np.uint8)

# Load centerbias (download or create)
centerbias_template = np.zeros((1024, 1024))  # or load from file
centerbias = zoom(
    centerbias_template,
    (image.shape[0] / centerbias_template.shape[0],
     image.shape[1] / centerbias_template.shape[1]),
    order=0,
    mode='nearest'
)
centerbias -= logsumexp(centerbias)

# Prepare tensors
image_tensor = torch.tensor([image.transpose(2, 0, 1)]).float().to(device)
centerbias_tensor = torch.tensor([centerbias]).float().to(device)

# Predict
with torch.no_grad():
    log_density = model(image_tensor, centerbias_tensor)
```

--------------------------------

### Initialize MixtureModel ensemble

Source: https://github.com/matthias-k/deepgaze/blob/main/_autodocs/modules.md

Example of wrapping multiple DeepGazeIIIMixture instances into a single MixtureModel ensemble.

```python
# DeepGazeIIE creates 4 DeepGazeIIIMixture instances
# and wraps them in MixtureModel for 4x ensemble
backbone_models = [
    build_deepgaze_mixture(BACKBONES[0], components=30),
    build_deepgaze_mixture(BACKBONES[1], components=30),
    build_deepgaze_mixture(BACKBONES[2], components=30),
    build_deepgaze_mixture(BACKBONES[3], components=30),
]
model = MixtureModel(backbone_models)
```

--------------------------------

### Install CLIP Requirements

Source: https://github.com/matthias-k/deepgaze/blob/main/_autodocs/features.md

Installs the required OpenAI CLIP package.

```bash
pip install git+https://github.com/openai/CLIP.git
```

--------------------------------

### DeepGazeIIE Usage Example

Source: https://github.com/matthias-k/deepgaze/blob/main/_autodocs/models.md

Complete workflow for loading the model, preparing image and centerbias tensors, and performing inference.

```python
import torch
from scipy.misc import face
from deepgaze_pytorch import DeepGazeIIE

device = 'cuda'
model = DeepGazeIIE(pretrained=True).to(device)

image = face()  # shape (768, 1024, 3)
image_tensor = torch.tensor([image.transpose(2, 0, 1)]).float().to(device)

# Centerbias preprocessing (same as DeepGazeI)
import numpy as np
from scipy.ndimage import zoom
from scipy.special import logsumexp

centerbias_template = np.load('centerbias_mit1003.npy')  # or np.zeros((1024, 1024))
centerbias = zoom(
    centerbias_template,
    (image.shape[0] / centerbias_template.shape[0],
     image.shape[1] / centerbias_template.shape[1]),
    order=0, mode='nearest'
)
centerbias -= logsumexp(centerbias)
centerbias_tensor = torch.tensor([centerbias]).float().to(device)

# Predict
with torch.no_grad():
    log_density = model(image_tensor, centerbias_tensor)
```

--------------------------------

### DeepGazeIII Usage Example

Source: https://github.com/matthias-k/deepgaze/blob/main/_autodocs/models.md

Complete workflow for preparing inputs and predicting the next fixation location.

```python
import torch
import numpy as np
from scipy.misc import face
from scipy.ndimage import zoom
from scipy.special import logsumexp
from deepgaze_pytorch import DeepGazeIII

device = 'cuda'
model = DeepGazeIII(pretrained=True).to(device)

image = face()  # shape (768, 1024, 3)
image_tensor = torch.tensor([image.transpose(2, 0, 1)]).float().to(device)

# Fixation history (must have at least 4 fixations)
fixation_history_x = np.array([512, 300, 500, 200, 200, 700], dtype=np.float32)
fixation_history_y = np.array([384, 300, 100, 300, 100, 500], dtype=np.float32)

x_hist_tensor = torch.tensor([fixation_history_x[model.included_fixations]]).float().to(device)
y_hist_tensor = torch.tensor([fixation_history_y[model.included_fixations]]).float().to(device)

# Centerbias
centerbias_template = np.zeros((1024, 1024))
centerbias = zoom(
    centerbias_template,
    (image.shape[0] / centerbias_template.shape[0],
     image.shape[1] / centerbias_template.shape[1]),
    order=0, mode='nearest'
)
centerbias -= logsumexp(centerbias)
centerbias_tensor = torch.tensor([centerbias]).float().to(device)

# Predict next fixation
with torch.no_grad():
    log_density = model(image_tensor, centerbias_tensor, x_hist_tensor, y_hist_tensor)
```

--------------------------------

### Perform saliency prediction with DeepGazeIIE

Source: https://github.com/matthias-k/deepgaze/blob/main/_autodocs/quickstart.md

Example showing model initialization, centerbias creation, and inference on an image.

```python
import torch
import numpy as np
from scipy.ndimage import zoom
from scipy.special import logsumexp
from deepgaze_pytorch import DeepGazeIIE

# Setup
device = 'cuda' if torch.cuda.is_available() else 'cpu'
model = DeepGazeIIE(pretrained=True).to(device)
model.eval()

# Load an image (e.g., from PIL)
from PIL import Image
image = Image.open('path/to/image.jpg').convert('RGB')
image_np = np.array(image)  # shape (H, W, 3), values [0, 255]

# Create simple centerbias (Gaussian at center)
H, W = image_np.shape[:2]
y_coords = np.linspace(-1, 1, H)
x_coords = np.linspace(-1, 1, W)
X, Y = np.meshgrid(x_coords, y_coords)
centerbias = -0.5 * (X**2 + Y**2) / 0.1  # Gaussian sigma=0.316
centerbias -= logsumexp(centerbias)  # Normalize

# Prepare tensors
image_tensor = torch.tensor(image_np.transpose(2, 0, 1)).float().to(device).unsqueeze(0)
centerbias_tensor = torch.tensor(centerbias).float().to(device).unsqueeze(0)

# Predict
with torch.no_grad():
    log_density = model(image_tensor, centerbias_tensor)  # shape (1, H, W)

# Convert to probability
saliency_map = torch.exp(log_density[0])  # (H, W)
saliency_map_np = saliency_map.cpu().numpy()

# Visualize
import matplotlib.pyplot as plt
plt.imshow(image_np)
plt.imshow(saliency_map_np, alpha=0.5, cmap='hot')
plt.colorbar()
plt.savefig('saliency.png')
```

--------------------------------

### Install Einops Requirement

Source: https://github.com/matthias-k/deepgaze/blob/main/_autodocs/features.md

Installs the einops package required for tensor reshaping.

```bash
pip install einops
```

--------------------------------

### Run DeepGaze MSDB Inference

Source: https://github.com/matthias-k/deepgaze/blob/main/README.md

Demonstrates loading the DeepGaze MSDB model and performing inference with either dataset-specific or generalized parameters. Requires a centerbias template and the pixel_per_dva value for the display setup.

```python
import numpy as np
from scipy.misc import face
from scipy.ndimage import zoom
from scipy.special import logsumexp
import torch

import deepgaze_pytorch
from deepgaze_pytorch import MSDBDataset

DEVICE = 'cuda'

model = deepgaze_pytorch.DeepGazeMSDB(pretrained=True).to(DEVICE)

image = face()

# load precomputed centerbias log density (from MIT1003) over a 1024x1024 image
# you can download the centerbias from https://github.com/matthias-k/DeepGaze/releases/download/v1.0.0/centerbias_mit1003.npy
# alternatively, you can use a uniform centerbias via `centerbias_template = np.zeros((1024, 1024))`.
centerbias_template = np.load('centerbias_mit1003.npy')
# rescale to match image size
centerbias = zoom(centerbias_template, (image.shape[0]/centerbias_template.shape[0], image.shape[1]/centerbias_template.shape[1]), order=0, mode='nearest')
# renormalize log density
centerbias -= logsumexp(centerbias)

image_tensor = torch.tensor([image.transpose(2, 0, 1)]).to(DEVICE)
centerbias_tensor = torch.tensor([centerbias]).to(DEVICE)

# For a known dataset (e.g., MIT1003), use dataset-specific parameters:
log_density_prediction = model(image_tensor, centerbias_tensor, pixel_per_dva=35.0, dataset=MSDBDataset.MIT1003)

# For a new/unknown dataset, use averaged parameters for generalization:
log_density_prediction = model(image_tensor, centerbias_tensor, pixel_per_dva=35.0, dataset=None)
```

--------------------------------

### Calculate log-likelihood example

Source: https://github.com/matthias-k/deepgaze/blob/main/_autodocs/metrics.md

Demonstrates how to compute the log-likelihood using random tensors for predictions and fixation masks.

```python
import torch
from deepgaze_pytorch.metrics import log_likelihood

# Predictions: (8, 768, 1024)
log_density = torch.randn(8, 768, 1024)

# Fixation masks: (8, 768, 1024)
fixation_mask = torch.zeros(8, 768, 1024)
fixation_mask[0, 100:105, 200:205] = 1  # 25 fixations in first image

# Weights: (8,)
weights = torch.ones(8) / 8

ll = log_likelihood(log_density, fixation_mask, weights)
print(f"Log-likelihood: {ll:.3f} bits")
```

--------------------------------

### Initialize CLIP ResNet Feature Extractor

Source: https://github.com/matthias-k/deepgaze/blob/main/_autodocs/api-index.md

Requires the CLIP repository to be installed via pip. Loads 1792-channel features.

```python
# Requires: pip install git+https://github.com/openai/CLIP.git
from deepgaze_pytorch.features.clip_resnet import CLIPResNet50x64
backbone = CLIPResNet50x64()  # Loads 1792-channel features
```

--------------------------------

### Initialize DINOv2 Feature Extractor

Source: https://github.com/matthias-k/deepgaze/blob/main/_autodocs/api-index.md

Requires the einops library to be installed. Configures the extractor with specific transformer blocks.

```python
# Requires: pip install einops
from deepgaze_pytorch.features.dino import DINOv2_ViTB14, DINOTransformersFeatureExtractor
backbone = DINOv2_ViTB14()
extractor = DINOTransformersFeatureExtractor(backbone, ['1.blocks.10', '1.blocks.6'])
```

--------------------------------

### Use ImageDatasetSampler with DataLoader

Source: https://github.com/matthias-k/deepgaze/blob/main/_autodocs/data.md

Example of using the sampler to group images by resolution in a DataLoader.

```python
dataset = ImageDataset(...)
sampler = ImageDatasetSampler(dataset, batch_size=8, shuffle=True)
loader = torch.utils.data.DataLoader(
    dataset,
    batch_size=8,
    sampler=sampler,
)
```

--------------------------------

### Execute full training loop

Source: https://github.com/matthias-k/deepgaze/blob/main/_autodocs/data.md

Demonstrates setting up the dataset, optimizer, and running the training and evaluation loop.

```python
import torch
from deepgaze_pytorch import DeepGazeIIE
from deepgaze_pytorch.training import train_epoch, eval_epoch
from deepgaze_pytorch.data import ImageDataset, FixationMaskTransform

device = 'cuda'
model = DeepGazeIIE(pretrained=True).to(device)

# Setup data
dataset = ImageDataset(...)
transform = FixationMaskTransform(sparse=False)
dataset.transform = transform
loader = torch.utils.data.DataLoader(dataset, batch_size=8)

# Setup optimizer (only train readout heads)
optimizer = torch.optim.Adam(
    [p for p in model.parameters() if p.requires_grad],
    lr=1e-4
)

# Training loop
for epoch in range(100):
    loss = train_epoch(model, loader, optimizer, device)
    print(f"Epoch {epoch}: loss={loss:.4f}")
    
    if epoch % 10 == 0:
        metrics = eval_epoch(model, loader, baseline_information_gain=0, device=device)
        print(f"Metrics: {metrics}")
```

--------------------------------

### Configure Paths and Device

Source: https://github.com/matthias-k/deepgaze/blob/main/train_deepgaze3.ipynb

Set up directory paths and target computation device.

```python
dataset_directory = Path('pysaliency_datasets')
train_directory = Path('train_deepgaze3')
```

```python
device = 'cuda'
```

--------------------------------

### Apply FixationMaskTransform in Compose

Source: https://github.com/matthias-k/deepgaze/blob/main/_autodocs/data.md

Example of integrating the transform into a torchvision Compose pipeline.

```python
transform = FixationMaskTransform(sparse=False)
batch_transform = transforms.Compose([
    transform,
])

dataset = ImageDataset(..., transform=batch_transform)
```

--------------------------------

### Initialize and Run DeepGaze III

Source: https://github.com/matthias-k/deepgaze/blob/main/README.md

Demonstrates loading the pretrained DeepGaze III model, preparing input tensors including centerbias and fixation history, and generating a log density prediction.

```python
import matplotlib.pyplot as plt
import numpy as np
from scipy.misc import face
from scipy.ndimage import zoom
from scipy.special import logsumexp
import torch

import deepgaze_pytorch

DEVICE = 'cuda'

# you can use DeepGazeI or DeepGazeIIE
model = deepgaze_pytorch.DeepGazeIII(pretrained=True).to(DEVICE)

image = face()

# location of previous scanpath fixations in x and y (pixel coordinates), starting with the initial fixation on the image.
fixation_history_x = np.array([1024//2, 300, 500, 200, 200, 700])
fixation_history_y = np.array([768//2, 300, 100, 300, 100, 500])

# load precomputed centerbias log density (from MIT1003) over a 1024x1024 image
# you can download the centerbias from https://github.com/matthias-k/DeepGaze/releases/download/v1.0.0/centerbias_mit1003.npy
# alternatively, you can use a uniform centerbias via `centerbias_template = np.zeros((1024, 1024))`.
centerbias_template = np.load('centerbias_mit1003.npy')
# rescale to match image size
centerbias = zoom(centerbias_template, (image.shape[0]/centerbias_template.shape[0], image.shape[1]/centerbias_template.shape[1]), order=0, mode='nearest')
# renormalize log density
centerbias -= logsumexp(centerbias)

image_tensor = torch.tensor([image.transpose(2, 0, 1)]).to(DEVICE)
centerbias_tensor = torch.tensor([centerbias]).to(DEVICE)
x_hist_tensor = torch.tensor([fixation_history_x[model.included_fixations]]).to(DEVICE)
y_hist_tensor = torch.tensor([fixation_history_y[model.included_fixations]]).to(DEVICE)

log_density_prediction = model(image_tensor, centerbias_tensor, x_hist_tensor, y_hist_tensor)

f, axs = plt.subplots(nrows=1, ncols=2, figsize=(8, 3))
axs[0].imshow(image)
axs[0].plot(fixation_history_x, fixation_history_y, 'o-', color='red')
axs[0].scatter(fixation_history_x[-1], fixation_history_y[-1], 100, color='yellow', zorder=100)
axs[0].set_axis_off()
axs[1].matshow(log_density_prediction.detach().cpu().numpy()[0, 0])  # first image in batch, first (and only) channel
axs[1].plot(fixation_history_x, fixation_history_y, 'o-', color='red')
axs[1].scatter(fixation_history_x[-1], fixation_history_y[-1], 100, color='yellow', zorder=100)
axs[1].set_axis_off()
```

--------------------------------

### Initialize and Run DeepGazeIIE

Source: https://github.com/matthias-k/deepgaze/blob/main/_autodocs/DOCUMENTATION.txt

Instantiate the DeepGazeIIE model and perform inference on input images.

```python
from deepgaze_pytorch import DeepGazeIIE
import torch

model = DeepGazeIIE(pretrained=True).cuda()
log_density = model(image, centerbias)  # (B,H,W)
```

--------------------------------

### Initialize DeepGaze III Model

Source: https://github.com/matthias-k/deepgaze/blob/main/train_deepgaze3.ipynb

Configures the DeepGaze III model with specific feature layers and networks, then moves it to the target device and sets up the optimizer and learning rate scheduler.

```python
model = DeepGazeIII(
    features=FeatureExtractor(RGBDenseNet201(), [
            '1.features.denseblock4.denselayer32.norm1',
            '1.features.denseblock4.denselayer32.conv1',
            '1.features.denseblock4.denselayer31.conv2',
        ]),
    saliency_network=build_saliency_network(2048),
    scanpath_network=None,
    fixation_selection_network=build_fixation_selection_network(scanpath_features=0),
    downsample=1.5,
    readout_factor=4,
    saliency_map_factor=4,
    included_fixations=[],
)

model = model.to(device)
optimizer = torch.optim.Adam(model.parameters(), lr=0.001)
lr_scheduler = torch.optim.lr_scheduler.MultiStepLR(optimizer, milestones=[15, 30, 45, 60, 75, 90, 105, 120])
```

--------------------------------

### Initialize and iterate ImageDataset

Source: https://github.com/matthias-k/deepgaze/blob/main/_autodocs/data.md

Demonstrates loading a dataset, configuring the ImageDataset, and iterating via a PyTorch DataLoader.

```python
import pysaliency
from deepgaze_pytorch.data import ImageDataset, ImageDatasetSampler, FixationMaskTransform

# Load pysaliency dataset (e.g., MIT1003)
dataset = pysaliency.get_dataset('MIT1003')
centerbias = pysaliency.models.FixationAverageCenterbias(dataset)

# Create image dataset
image_dataset = ImageDataset(
    stimuli=dataset.stimuli,
    fixations=dataset.fixations,
    centerbias_model=centerbias,
    cached=True,
)

# Create dataloader
loader = torch.utils.data.DataLoader(
    image_dataset,
    batch_size=8,
    sampler=ImageDatasetSampler(image_dataset, batch_size=8, shuffle=True),
)

# Iterate
for batch in loader:
    image = batch['image']  # (B, 3, H, W)
    fixations = batch['x'], batch['y']  # (B, N)
    centerbias = batch['centerbias']  # (B, H, W)
```

--------------------------------

### Interpret included_fixations Parameter

Source: https://github.com/matthias-k/deepgaze/blob/main/_autodocs/data.md

Examples showing how the included_fixations parameter determines the length of historical fixation data.

```python
# included_fixations = -2
# Includes the last 2 previous fixations before the target fixation
# x_hist = [x_{t-2}, x_{t-1}]
# y_hist = [y_{t-2}, y_{t-1}]

# included_fixations = -4
# Includes the last 4 previous fixations (used by DeepGaze III)
# x_hist = [x_{t-4}, x_{t-3}, x_{t-2}, x_{t-1}]
# y_hist = [y_{t-4}, y_{t-3}, y_{t-2}, y_{t-1}]
```

--------------------------------

### Initialize DeepGazeMSDB

Source: https://github.com/matthias-k/deepgaze/blob/main/_autodocs/models.md

Instantiate the model with pretrained weights.

```python
DeepGazeMSDB(pretrained=True)
```

--------------------------------

### Initialize DeepGazeIIE

Source: https://github.com/matthias-k/deepgaze/blob/main/_autodocs/models.md

Instantiate the model with pretrained weights.

```python
DeepGazeIIE(pretrained=True)
```

--------------------------------

### Initialize and run DeepGazeMSDB

Source: https://github.com/matthias-k/deepgaze/blob/main/_autodocs/api-index.md

Instantiates the multi-scale dataset-bias model, requiring pixel-per-dva and optional dataset identifiers.

```python
model = DeepGazeMSDB(pretrained=True)
output = model(
    image: (B,3,H,W),
    centerbias: (B,H,W),
    pixel_per_dva: float,
    dataset: int | None = None
) -> (B,H,W)
```

--------------------------------

### Initialize DeepGazeMSDB Model

Source: https://github.com/matthias-k/deepgaze/blob/main/_autodocs/types.md

Instantiates the DeepGazeMSDB model utilizing CLIP and DINOv2 backbones.

```python
DeepGazeMSDB(pretrained=True)
# Features:
#   - Backbones: CLIP (1792 ch) + DINOv2 (768 ch)
#   - Concatenated channels: 2560
#   - Scale resolutions (PPD): 5 scales
#   - Scale resolutions (size): 5 scales
#   - Per-dataset parameters: sigma, center_bias_weight, priority_scaling
#   - Supported datasets: 5 (MIT1003, CAT2000, COCO, DAEMONS, FIGRIM)
```

--------------------------------

### Calculate AUC score

Source: https://github.com/matthias-k/deepgaze/blob/main/_autodocs/metrics.md

Example usage for calculating the AUC score given model predictions and fixation masks.

```python
from deepgaze_pytorch.metrics import auc

auc_score = auc(log_density, fixation_mask, weights)
print(f"AUC: {auc_score:.3f}")
```

--------------------------------

### Initialize DeepGazeIII Model

Source: https://github.com/matthias-k/deepgaze/blob/main/_autodocs/models.md

Instantiate the model with pretrained weights.

```python
DeepGazeIII(pretrained=True)
```

--------------------------------

### Calculate NSS score

Source: https://github.com/matthias-k/deepgaze/blob/main/_autodocs/metrics.md

Example usage for calculating the NSS score given model predictions and fixation masks.

```python
from deepgaze_pytorch.metrics import nss

nss_score = nss(log_density, fixation_mask, weights)
print(f"NSS: {nss_score:.3f}")
```

--------------------------------

### Initialize DeepGazeIIE Model

Source: https://github.com/matthias-k/deepgaze/blob/main/_autodocs/types.md

Instantiates the DeepGazeIIE model using an ensemble of four backbones.

```python
DeepGazeIIE(pretrained=True)
# Features:
#   - Backbones: 4 (ShapeNet, EfficientNet, DenseNet, ResNext)
#   - Components per backbone: 30 (3 instances × 10 folds)
#   - Total components: 120
#   - Ensemble type: Log-space mixture
```

--------------------------------

### Initialize DeepGazeI Model

Source: https://github.com/matthias-k/deepgaze/blob/main/_autodocs/models.md

Instantiate the DeepGazeI model with optional pretrained weights.

```python
DeepGazeI(pretrained=True)
```

--------------------------------

### Initialize and run DeepGazeIIE

Source: https://github.com/matthias-k/deepgaze/blob/main/_autodocs/api-index.md

Instantiates the ensemble saliency model with four backbones and defines the expected input tensor shapes.

```python
model = DeepGazeIIE(pretrained=True)
output = model(image: (B,3,H,W), centerbias: (B,H,W)) -> (B,H,W)
```

--------------------------------

### Run DeepGaze IIE Inference

Source: https://github.com/matthias-k/deepgaze/blob/main/README.md

Shows how to initialize the pre-trained DeepGaze IIE model and process an image with a centerbias template.

```python
import numpy as np
from scipy.misc import face
from scipy.ndimage import zoom
from scipy.special import logsumexp
import torch

import deepgaze_pytorch

DEVICE = 'cuda'

# you can use DeepGazeI or DeepGazeIIE
model = deepgaze_pytorch.DeepGazeIIE(pretrained=True).to(DEVICE)

image = face()

# load precomputed centerbias log density (from MIT1003) over a 1024x1024 image
# you can download the centerbias from https://github.com/matthias-k/DeepGaze/releases/download/v1.0.0/centerbias_mit1003.npy
# alternatively, you can use a uniform centerbias via `centerbias_template = np.zeros((1024, 1024))`.
centerbias_template = np.load('centerbias_mit1003.npy')
# rescale to match image size
centerbias = zoom(centerbias_template, (image.shape[0]/centerbias_template.shape[0], image.shape[1]/centerbias_template.shape[1]), order=0, mode='nearest')
# renormalize log density
centerbias -= logsumexp(centerbias)

image_tensor = torch.tensor([image.transpose(2, 0, 1)]).to(DEVICE)
centerbias_tensor = torch.tensor([centerbias]).to(DEVICE)

log_density_prediction = model(image_tensor, centerbias_tensor)
```

--------------------------------

### Initialize and run DeepGazeI

Source: https://github.com/matthias-k/deepgaze/blob/main/_autodocs/api-index.md

Instantiates the AlexNet-based saliency model and defines the expected input tensor shapes.

```python
model = DeepGazeI(pretrained=True)
output = model(image: (B,3,H,W), centerbias: (B,H,W)) -> (B,H,W)
```

--------------------------------

### Initialize DeepGazeIII Model

Source: https://github.com/matthias-k/deepgaze/blob/main/_autodocs/types.md

Instantiates the DeepGazeIII model with a DenseNet201 backbone and scanpath network support.

```python
DeepGazeIII(pretrained=True)
# Features:
#   - Backbone: DenseNet201
#   - Backbone channels: 2048
#   - Mixture components: 10
#   - Included fixations: [-1, -2, -3, -4] (last 4)
#   - Has scanpath network: Yes
#   - Readout factor: 4
#   - Saliency map factor: 4
```

--------------------------------

### Initialize DeepGazeI Model

Source: https://github.com/matthias-k/deepgaze/blob/main/_autodocs/types.md

Instantiates the DeepGazeI model using the AlexNet backbone.

```python
DeepGazeI(pretrained=True)
# Features:
#   - Backbone: AlexNet (features['1.features.10'])
#   - Channels: 256
#   - Readout factor: 4
#   - Saliency map factor: 4
#   - Initial sigma: 8.0
#   - Learnable: sigma, center_bias_weight, readout_network
```

--------------------------------

### Initialize and run DeepGazeIII

Source: https://github.com/matthias-k/deepgaze/blob/main/_autodocs/api-index.md

Instantiates the scanpath model, requiring fixation history coordinates in addition to image and centerbias inputs.

```python
model = DeepGazeIII(pretrained=True)
output = model(
    image: (B,3,H,W),
    centerbias: (B,H,W),
    x_hist: (B,N),
    y_hist: (B,N)
) -> (B,H,W)
```

--------------------------------

### Execute Model Training

Source: https://github.com/matthias-k/deepgaze/blob/main/train_deepgaze3.ipynb

Runs the training process using the initialized model, data loaders, optimizer, and scheduler.

```python
_train(train_directory / 'pretraining',
    model,
    train_loader, train_baseline_log_likelihood,
    validation_loader, val_baseline_log_likelihood,
    optimizer, lr_scheduler,
    minimum_learning_rate=1e-7,
    device=device,
)
```

--------------------------------

### Import DeepGaze Models

Source: https://github.com/matthias-k/deepgaze/blob/main/_autodocs/README.md

Import the available DeepGaze model architectures and the MSDB dataset class.

```python
from deepgaze_pytorch import DeepGazeI, DeepGazeIIE, DeepGazeIII, DeepGazeMSDB, MSDBDataset
```

--------------------------------

### Load SALICON Data and Baseline

Source: https://github.com/matthias-k/deepgaze/blob/main/train_deepgaze3.ipynb

Load SALICON training/validation data and compute the center bias baseline model.

```python
SALICON_train_stimuli, SALICON_train_fixations = pysaliency.get_SALICON_train(location=dataset_directory)
SALICON_val_stimuli, SALICON_val_fixations = pysaliency.get_SALICON_val(location=dataset_directory)

# parameters taken from an early fit for MIT1003. Since SALICON has many more fixations, the bandwidth won't be too small
SALICON_centerbias = BaselineModel(stimuli=SALICON_train_stimuli, fixations=SALICON_train_fixations, bandwidth=0.0217, eps=2e-13, caching=False)

# takes quite some time, feel free to set to zero
train_baseline_log_likelihood = SALICON_centerbias.information_gain(SALICON_train_stimuli, SALICON_train_fixations, verbose=True, average='image')
val_baseline_log_likelihood = SALICON_centerbias.information_gain(SALICON_val_stimuli, SALICON_val_fixations, verbose=True, average='image')
```

--------------------------------

### Initialize ImageDataset

Source: https://github.com/matthias-k/deepgaze/blob/main/_autodocs/api-index.md

Creates a PyTorch Dataset for image and fixation data with optional caching.

```python
dataset = ImageDataset(stimuli, fixations, centerbias_model, cached=True)
batch = dataset[0]  # {'image', 'x', 'y', 'centerbias', 'weight'}
```

--------------------------------

### Forward Method Signature

Source: https://github.com/matthias-k/deepgaze/blob/main/_autodocs/models.md

The forward method requires image, centerbias, and pixel_per_dva calibration parameters.

```python
output = model(
    image: torch.Tensor,
    centerbias: torch.Tensor,
    pixel_per_dva: float | List[float],
    dataset: int | None = None
) -> torch.Tensor
```

--------------------------------

### Initialize DeepGazeII architecture

Source: https://github.com/matthias-k/deepgaze/blob/main/_autodocs/api-index.md

Instantiates the base DeepGazeII saliency architecture with a feature extractor and readout network.

```python
model = DeepGazeII(features, readout_network, ...)
output = model(x: (B,3,H,W), centerbias: (B,H,W)) -> (B,H,W)
```

--------------------------------

### Initialize DeepGazeIIIMixture ensemble

Source: https://github.com/matthias-k/deepgaze/blob/main/_autodocs/api-index.md

Creates an ensemble model using multiple saliency, scanpath, and fixation networks.

```python
model = DeepGazeIIIMixture(features, sal_nets, scan_nets, fix_nets, finalizers)
output = model(x, centerbias, x_hist=None, y_hist=None) -> (B,1,H,W)
```

--------------------------------

### DeepGazeIIE Constructor

Source: https://github.com/matthias-k/deepgaze/blob/main/_autodocs/models.md

Initializes the DeepGazeIIE model instance, optionally loading pretrained weights.

```APIDOC
## DeepGazeIIE()

### Description
Initializes the DeepGazeIIE model. The model uses an ensemble of multiple backbones to predict saliency.

### Parameters
- **pretrained** (bool) - Optional - If True, loads pretrained weights from GitHub release v1.0.0. Defaults to True.
```

--------------------------------

### Import Sampling Utility

Source: https://github.com/matthias-k/deepgaze/blob/main/Examples.ipynb

Imports the utility function for sampling from log density predictions.

```python
from pysaliency.models import sample_from_logdensity
```

--------------------------------

### Execute DeepGaze III training

Source: https://github.com/matthias-k/deepgaze/blob/main/train_deepgaze3.ipynb

Initiates the training process for the DeepGaze III model with specified directories, loaders, and optimization parameters.

```python
_train(train_directory / 'MIT1003_scanpath' / f'crossval-10-{crossval_fold}',
    model,
    train_loader, train_baseline_log_likelihood,
    validation_loader, val_baseline_log_likelihood,
    optimizer, lr_scheduler,
    minimum_learning_rate=1e-7,
    device=device,
    startwith=train_directory / 'MIT1003_scanpath_partially_frozen_saliency_network' / f'crossval-10-{crossval_fold}' / 'final.pth'
)
```

--------------------------------

### Finalizer.__init__

Source: https://github.com/matthias-k/deepgaze/blob/main/_autodocs/modules.md

Initializes the Finalizer module with parameters for Gaussian smoothing and center bias weighting.

```APIDOC
## Finalizer.__init__

### Description
Initializes the Finalizer module to post-process readout predictions into normalized saliency maps.

### Parameters
- **sigma** (float) - Required - Standard deviation of Gaussian filter for smoothing
- **kernel_size** (int) - Optional - Size of Gaussian kernel; if None, computed from sigma and truncate
- **learn_sigma** (bool) - Optional - If True, sigma becomes a learnable parameter
- **center_bias_weight** (float) - Optional - Initial weight multiplying the center bias
- **learn_center_bias_weight** (bool) - Optional - If True, center_bias_weight becomes learnable
- **saliency_map_factor** (int) - Optional - Downsampling factor to apply to centerbias before addition
```

--------------------------------

### Model Inference

Source: https://github.com/matthias-k/deepgaze/blob/main/_autodocs/README.md

How to initialize a DeepGaze model and perform a forward pass to obtain log-density predictions.

```APIDOC
## Model Inference

### Description
Initialize a DeepGaze model (e.g., DeepGazeIIE) and compute the log-probability density of fixations for a given image and centerbias.

### Usage
```python
from deepgaze_pytorch import DeepGazeIIE

model = DeepGazeIIE(pretrained=True).to('cuda')
model.eval()

# image: (B, 3, H, W), values [0, 255]
# centerbias: (B, H, W), log-probability density
log_density = model(image, centerbias)  # shape (B, H, W)
```
```

--------------------------------

### Prepare Datasets

Source: https://github.com/matthias-k/deepgaze/blob/main/train_deepgaze3.ipynb

Utility functions to create PyTorch DataLoaders for spatial and scanpath datasets, optionally caching to LMDB.

```python
def prepare_spatial_dataset(stimuli, fixations, centerbias, batch_size, path=None):
    if path is not None:
        path.mkdir(parents=True, exist_ok=True)
        lmdb_path = str(path)
    else:
        lmdb_path = None

    dataset = ImageDataset(
        stimuli=stimuli,
        fixations=fixations,
        centerbias_model=centerbias,
        transform=FixationMaskTransform(sparse=False),
        average='image',
        lmdb_path=lmdb_path,
    )

    loader = torch.utils.data.DataLoader(
        dataset,
        batch_sampler=ImageDatasetSampler(dataset, batch_size=batch_size),
        pin_memory=False,
        num_workers=0,
    )

    return loader
```

```python
def prepare_scanpath_dataset(stimuli, fixations, centerbias, batch_size, path=None):
    if path is not None:
        path.mkdir(parents=True, exist_ok=True)
        lmdb_path = str(path)
    else:
        lmdb_path = None

    dataset = FixationDataset(
        stimuli=stimuli,
        fixations=fixations,
        centerbias_model=centerbias,
        included_fixations=[-1, -2, -3, -4],
        allow_missing_fixations=True,
        transform=FixationMaskTransform(sparse=False),
        average='image',
        lmdb_path=lmdb_path,
    )

    loader = torch.utils.data.DataLoader(
        dataset,
        batch_sampler=ImageDatasetSampler(dataset, batch_size=batch_size),
        pin_memory=False,
        num_workers=0,
    )

    return loader
```

--------------------------------

### Perform Basic Prediction

Source: https://github.com/matthias-k/deepgaze/blob/main/_autodocs/README.md

Initialize a pretrained DeepGazeIIE model and perform inference on an image with centerbias.

```python
import torch
from deepgaze_pytorch import DeepGazeIIE

model = DeepGazeIIE(pretrained=True).to('cuda')
model.eval()

# RGB image: (B, 3, H, W), values [0, 255]
# Centerbias: (B, H, W), log-probability density
log_density = model(image, centerbias)  # shape (B, H, W)
```

--------------------------------

### Prepare Spatial Datasets

Source: https://github.com/matthias-k/deepgaze/blob/main/train_deepgaze3.ipynb

Creates data loaders for training and validation sets, utilizing LMDB caching for efficient data access.

```python
train_loader = prepare_spatial_dataset(SALICON_train_stimuli, SALICON_train_fixations, SALICON_centerbias, batch_size=4, path=train_directory / 'lmdb_cache' / 'SALICON_train')
validation_loader = prepare_spatial_dataset(SALICON_val_stimuli, SALICON_val_fixations, SALICON_centerbias, batch_size=4, path=train_directory / 'lmdb_cache' / 'SALICON_val')
```

--------------------------------

### Handle Missing Centerbias Files

Source: https://github.com/matthias-k/deepgaze/blob/main/_autodocs/quickstart.md

Fallback to zero-initialized arrays when centerbias files are missing.

```python
# If centerbias.npy doesn't exist, use zeros or uniform
import numpy as np
from scipy.special import logsumexp

H, W = 768, 1024
centerbias = np.zeros((H, W))
centerbias -= logsumexp(centerbias)  # Normalize to log-probability
```

--------------------------------

### Initialize DeepGazeII Model

Source: https://github.com/matthias-k/deepgaze/blob/main/_autodocs/modules.md

Defines the constructor for the DeepGazeII baseline saliency model.

```python
class DeepGazeII(torch.nn.Module):
    def __init__(
        self,
        features: FeatureExtractor,
        readout_network: nn.Module,
        downsample: int = 2,
        readout_factor: int = 16,
        saliency_map_factor: int = 2,
        initial_sigma: float = 8.0,
    )
```

--------------------------------

### Load MIT1003 Dataset

Source: https://github.com/matthias-k/deepgaze/blob/main/train_deepgaze3.ipynb

Initializes the MIT1003 dataset using the pysaliency library.

```python
mit_stimuli_orig, mit_scanpaths_orig = pysaliency.external_datasets.mit.get_mit1003_with_initial_fixation(location=dataset_directory, replace_initial_invalid_fixations=True)
```

--------------------------------

### Initialize DeepGazeIIIMixture Model

Source: https://github.com/matthias-k/deepgaze/blob/main/_autodocs/modules.md

Defines the constructor for the DeepGazeIIIMixture ensemble model.

```python
class DeepGazeIIIMixture(torch.nn.Module):
    def __init__(
        self,
        features: FeatureExtractor,
        saliency_networks: List[nn.Module],
        scanpath_networks: List[nn.Module],
        fixation_selection_networks: List[nn.Module],
        finalizers: List[Finalizer],
        downsample: int = 2,
        readout_factor: int = 2,
        saliency_map_factor: int = 2,
        included_fixations: List[int] = -2,
        initial_sigma: float = 8.0,
    )
```

--------------------------------

### DeepGazeMSDB Constructor

Source: https://github.com/matthias-k/deepgaze/blob/main/_autodocs/models.md

Initializes the DeepGazeMSDB model instance.

```APIDOC
## DeepGazeMSDB(pretrained=True)

### Description
Initializes the DeepGazeMSDB model. If pretrained is True, it loads weights from the GitHub release v1.2.0.

### Parameters
- **pretrained** (bool) - Optional - If True, loads pretrained weights. Defaults to True.
```

--------------------------------

### DeepGazeMSDB

Source: https://github.com/matthias-k/deepgaze/blob/main/_autodocs/api-index.md

Initializes the DeepGazeMSDB multi-scale dataset-bias model with CLIP and DINOv2.

```APIDOC
## DeepGazeMSDB

### Description
Multi-scale dataset-bias model with CLIP + DINOv2.

### Signature
`DeepGazeMSDB(pretrained: bool = True)`

### Usage
```python
model = DeepGazeMSDB(pretrained=True)
output = model(image: (B,3,H,W), centerbias: (B,H,W), pixel_per_dva: float, dataset: int | None = None) -> (B,H,W)
```
```

--------------------------------

### Optimize Execution Device

Source: https://github.com/matthias-k/deepgaze/blob/main/_autodocs/quickstart.md

Configure the model to use GPU acceleration if available.

```python
# Use GPU if available
device = 'cuda' if torch.cuda.is_available() else 'cpu'
model = model.to(device)

# Or use CPU but with fewer model components
# (though this requires architectural changes)
```

--------------------------------

### DeepGazeIII Constructor

Source: https://github.com/matthias-k/deepgaze/blob/main/_autodocs/models.md

Initializes the DeepGazeIII model instance.

```APIDOC
## DeepGazeIII(pretrained=True)

### Description
Initializes the DeepGazeIII model. If pretrained is True, it loads weights from the GitHub release v1.1.0.

### Parameters
- **pretrained** (bool) - Optional - If True, loads pretrained weights.
```

--------------------------------

### DeepGazeIII

Source: https://github.com/matthias-k/deepgaze/blob/main/_autodocs/api-index.md

Initializes the DeepGazeIII scanpath model with fixation history conditioning.

```APIDOC
## DeepGazeIII

### Description
Scanpath model with fixation history conditioning.

### Signature
`DeepGazeIII(pretrained: bool = True)`

### Usage
```python
model = DeepGazeIII(pretrained=True)
output = model(image: (B,3,H,W), centerbias: (B,H,W), x_hist: (B,N), y_hist: (B,N)) -> (B,H,W)
```
```

--------------------------------

### DeepGazeIIE

Source: https://github.com/matthias-k/deepgaze/blob/main/_autodocs/api-index.md

Initializes the DeepGazeIIE ensemble saliency model with 4 backbones.

```APIDOC
## DeepGazeIIE

### Description
Ensemble saliency model with 4 backbones.

### Signature
`DeepGazeIIE(pretrained: bool = True)`

### Usage
```python
model = DeepGazeIIE(pretrained=True)
output = model(image: (B,3,H,W), centerbias: (B,H,W)) -> (B,H,W)
```
```

--------------------------------

### Initialize FixationDataset

Source: https://github.com/matthias-k/deepgaze/blob/main/_autodocs/api-index.md

Creates a PyTorch Dataset for scanpath data, allowing configuration of fixation history length.

```python
dataset = FixationDataset(stimuli, fixations, included_fixations=-4)
batch = dataset[0]  # {'image', 'x', 'y', 'x_hist', 'y_hist', 'centerbias', 'weight'}
```

--------------------------------

### DeepGazeI

Source: https://github.com/matthias-k/deepgaze/blob/main/_autodocs/api-index.md

Initializes the DeepGazeI saliency model using an AlexNet backbone.

```APIDOC
## DeepGazeI

### Description
Saliency model using AlexNet backbone.

### Signature
`DeepGazeI(pretrained: bool = True)`

### Usage
```python
model = DeepGazeI(pretrained=True)
output = model(image: (B,3,H,W), centerbias: (B,H,W)) -> (B,H,W)
```
```

--------------------------------

### Initialize Conv2dMultiInput

Source: https://github.com/matthias-k/deepgaze/blob/main/_autodocs/modules.md

Defines a convolution layer that accepts a list of input channel counts and combines the results.

```python
class Conv2dMultiInput(nn.Module):
    def __init__(
        self,
        in_channels: List[int],
        out_channels: int,
        kernel_size: Tuple[int, int],
        bias: bool = True,
    )
```

--------------------------------

### Initialize and apply Conv2dMultiInput

Source: https://github.com/matthias-k/deepgaze/blob/main/_autodocs/api-index.md

Processes multiple input tensors through a convolution layer, supporting None values for skipped inputs.

```python
conv = Conv2dMultiInput([256, 0, 512], out_channels=128, kernel_size=(1,1))
output = conv([tensor1: (B,256,H,W), None, tensor3: (B,512,H,W)])
```

--------------------------------

### Import Dependencies

Source: https://github.com/matthias-k/deepgaze/blob/main/train_deepgaze3.ipynb

Essential imports for DeepGaze III, including PyTorch modules, pysaliency utilities, and custom deepgaze_pytorch layers.

```python
from collections import OrderedDict
import os
from pathlib import Path
import shutil

from imageio.v3 import imread, imwrite
from PIL import Image
import pysaliency
from pysaliency.baseline_utils import BaselineModel, CrossvalidatedBaselineModel
import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.utils import model_zoo

from tqdm import tqdm


from deepgaze_pytorch.layers import (
    Conv2dMultiInput,
    LayerNorm,
    LayerNormMultiInput,
    Bias,
    FlexibleScanpathHistoryEncoding
)

from deepgaze_pytorch.modules import DeepGazeIII, FeatureExtractor
from deepgaze_pytorch.features.densenet import RGBDenseNet201
from deepgaze_pytorch.data import ImageDataset, ImageDatasetSampler, FixationDataset, FixationMaskTransform
from deepgaze_pytorch.training import _train
```

--------------------------------

### Define RGBInception3 backbone

Source: https://github.com/matthias-k/deepgaze/blob/main/_autodocs/features.md

Sequential wrapper for the Inception-v3 architecture.

```python
class RGBInception3(nn.Sequential)
```

--------------------------------

### Initialize Center Bias Model

Source: https://github.com/matthias-k/deepgaze/blob/main/train_deepgaze3.ipynb

Configures the center bias model with parameters optimized for MIT1003.

```python
# parameters optimized on MIT1003 for maximum leave-one-image-out crossvalidation log-likelihood
MIT1003_centerbias = CrossvalidatedBaselineModel(
    mit_stimuli_twosize,
    mit_fixations_twosize,
    bandwidth=10**-1.6667673342543432,
    eps=10**-14.884189168516073,
    caching=False,
)
```

--------------------------------

### Export dataset to LMDB

Source: https://github.com/matthias-k/deepgaze/blob/main/_autodocs/data.md

Creates an LMDB database for efficient disk-based caching of images and centerbias. Requires pysaliency stimuli and a centerbias model.

```python
def _export_dataset_to_lmdb(
    stimuli: pysaliency.FileStimuli,
    centerbias_model: pysaliency.Model,
    lmdb_path: str,
    write_frequency: int = 100,
):
    """Create LMDB database with images and precomputed centerbias.
    
    Args:
        stimuli: Image dataset
        centerbias_model: Centerbias computation model
        lmdb_path: Path to create LMDB (directory or file)
        write_frequency: Batch writes for efficiency
    """
```

```python
# Precompute LMDB once
from deepgaze_pytorch.data import _export_dataset_to_lmdb
import pysaliency

dataset = pysaliency.get_dataset('MIT1003')
centerbias = pysaliency.models.FixationAverageCenterbias(dataset)

_export_dataset_to_lmdb(
    dataset.stimuli,
    centerbias,
    lmdb_path='/path/to/mit1003.lmdb'
)

# Then use in dataset creation
image_dataset = ImageDataset(
    stimuli=dataset.stimuli,
    fixations=dataset.fixations,
    centerbias_model=centerbias,
    lmdb_path='/path/to/mit1003.lmdb',  # Loads from cache
)
```

--------------------------------

### Define RGBEfficientNetB5 backbone

Source: https://github.com/matthias-k/deepgaze/blob/main/_autodocs/features.md

Sequential wrapper for the EfficientNet-B5 architecture.

```python
class RGBEfficientNetB5(nn.Sequential)
```

--------------------------------

### Serialize image and centerbias

Source: https://github.com/matthias-k/deepgaze/blob/main/_autodocs/data.md

Serializes image file and centerbias data into a pickle format for LMDB storage.

```python
def _encode_filestimulus_item(filename: str, centerbias: np.ndarray) -> bytes:
    """Encode image file and centerbias as pickle."""
```