### Setup test data

Source: https://github.com/dpird-dma/omnicloudmask/blob/main/tests/Visual tests.ipynb

Executes a setup script to prepare the necessary test data for visual testing. This is a prerequisite for running subsequent examples.

```python
%run setup_test_data.py
```

--------------------------------

### Install Dependencies

Source: https://github.com/dpird-dma/omnicloudmask/blob/main/benchmarking/README.md

Installs the required development dependencies for benchmarking tools. Run this before executing notebooks or scripts.

```bash
uv sync --group dev
```

--------------------------------

### Install Dependencies

Source: https://github.com/dpird-dma/omnicloudmask/blob/main/examples/sentinel2_planetary_computer.ipynb

Install the necessary libraries for querying Planetary Computer and using OmniCloudMask.

```bash
pip install pystac-client planetary-computer
```

--------------------------------

### Install uv and Sync Dependencies

Source: https://github.com/dpird-dma/omnicloudmask/blob/main/python_package_publishing.md

Installs the 'uv' tool using a curl script and then synchronizes project dependencies.

```bash
curl -LsSf https://astral.sh/uv/install.sh | sh
uv sync
```

--------------------------------

### Install OmniCloudMask from Source

Source: https://github.com/dpird-dma/omnicloudmask/blob/main/docs/installation.md

Install OmniCloudMask directly from its GitHub repository using pip.

```bash
pip install git+https://github.com/DPIRD-DMA/OmniCloudMask.git
```

--------------------------------

### Install h5py

Source: https://github.com/dpird-dma/omnicloudmask/blob/main/examples/planetscope_hyperspectral.ipynb

Install the h5py library, which is required for reading HDF5 files.

```bash
pip install h5py
```

--------------------------------

### Install OmniCloudMask using pip

Source: https://github.com/dpird-dma/omnicloudmask/blob/main/README.md

Install the OmniCloudMask library using pip. This is the primary method for installation.

```bash
pip install omnicloudmask
```

--------------------------------

### Install OmniCloudMask using uv

Source: https://github.com/dpird-dma/omnicloudmask/blob/main/docs/installation.md

Install OmniCloudMask using the uv package installer. This is an alternative to pip.

```bash
uv add omnicloudmask
```

--------------------------------

### Clone Repository and Install Dependencies

Source: https://github.com/dpird-dma/omnicloudmask/blob/main/docs/contributing.md

Clone the OmniCloudMask repository and install development dependencies using uv.

```bash
git clone https://github.com/DPIRD-DMA/OmniCloudMask.git
cd OmniCloudMask
uv sync --group dev
```

--------------------------------

### Install Package from Test PyPI

Source: https://github.com/dpird-dma/omnicloudmask/blob/main/python_package_publishing.md

Installs the 'omnicloudmask' package from Test PyPI into a new environment, also including packages from the main PyPI.

```bash
pip install -i https://test.pypi.org/simple/ omnicloudmask --extra-index-url https://pypi.org/simple
```

--------------------------------

### Install OmniCloudMask with Legacy Model Support (uv)

Source: https://github.com/dpird-dma/omnicloudmask/blob/main/docs/installation.md

Install the 'legacy' extra for OmniCloudMask using uv to support older model versions.

```bash
uv add omnicloudmask --extra legacy
```

--------------------------------

### Install Earthdata Access Library

Source: https://github.com/dpird-dma/omnicloudmask/blob/main/examples/hls.ipynb

Install the earthaccess library to interact with NASA Earthdata. This is a prerequisite for accessing HLS data.

```bash
pip install earthaccess
```

--------------------------------

### Install pytest and coverage

Source: https://github.com/dpird-dma/omnicloudmask/blob/main/tests/test_coverage_commands.md

Installs the necessary Python packages for running tests and collecting code coverage.

```bash
pip install pytest
pip install coverage
```

--------------------------------

### Create Hard Negative Directory

Source: https://github.com/dpird-dma/omnicloudmask/blob/main/training/Get datasets/5 Download OCM Hard negatives.ipynb

Creates a directory for storing hard negative examples if it doesn't already exist.

```python
hard_negative_dir = base_dataset_dir / "Hard negative"
hard_negative_dir.mkdir(parents=True, exist_ok=True)
```

--------------------------------

### Import Libraries and Setup

Source: https://github.com/dpird-dma/omnicloudmask/blob/main/validation dataset/Landsat/Get stats.ipynb

Imports necessary libraries including rasterio, pandas, pathlib, numpy, and sys. It also appends the parent directory to the system path to import the omnicloudmask module.

```python
import rasterio as rio
import pandas as pd
from pathlib import Path
import numpy as np
import sys

sys.path.append(Path.cwd().parents[1].as_posix())
import omnicloudmask
```

--------------------------------

### Install OmniCloudMask with Legacy Model Support (pip)

Source: https://github.com/dpird-dma/omnicloudmask/blob/main/docs/installation.md

Install the 'legacy' extra for OmniCloudMask using pip to support older model versions (1.0-3.0) built with fastai.

```bash
pip install omnicloudmask[legacy]
```

--------------------------------

### Define Dataset Directory

Source: https://github.com/dpird-dma/omnicloudmask/blob/main/validation dataset/PlanetScope/OCM inference.ipynb

Sets the path for the dataset directory. This is a common setup step before organizing or accessing data.

```python
dataset_dir = Path.cwd() / "dataset"
```

--------------------------------

### Import Libraries

Source: https://github.com/dpird-dma/omnicloudmask/blob/main/examples/hls.ipynb

Import necessary libraries for data handling, authentication, parallel processing, and visualization. Ensure all required packages are installed.

```python
from concurrent.futures import ThreadPoolExecutor

import earthaccess
import matplotlib.pyplot as plt
import numpy as np
import rasterio as rio
from rasterio.env import Env

from omnicloudmask import predict_from_array
```

--------------------------------

### Build OmniCloudMask Docker Image

Source: https://github.com/dpird-dma/omnicloudmask/blob/main/docs/installation.md

Clone the repository and build the Docker image locally for OmniCloudMask. Ensure Docker and NVIDIA Container Toolkit (for GPU) are installed.

```bash
git clone https://github.com/DPIRD-DMA/OmniCloudMask
cd OmniCloudMask
docker build -f docker/Dockerfile -t omnicloudmask:local .
```

--------------------------------

### Install OmniCloudMask with Legacy Model Support (conda)

Source: https://github.com/dpird-dma/omnicloudmask/blob/main/docs/installation.md

Install OmniCloudMask and fastai from conda-forge to support legacy model versions.

```bash
conda install conda-forge::omnicloudmask conda-forge::fastai
```

--------------------------------

### Import Libraries

Source: https://github.com/dpird-dma/omnicloudmask/blob/main/training/Get datasets/1 Download CloudSEN12.ipynb

Imports necessary libraries for data handling, image processing, and parallel operations. Ensure these are installed before running.

```python
import tacoreader
import rasterio as rio
from tqdm.auto import tqdm
from pathlib import Path
from multiprocessing.pool import ThreadPool
import time
from threading import Thread
import numpy as np
from typing import Optional
```

--------------------------------

### Import Core Libraries for OCM Model Training

Source: https://github.com/dpird-dma/omnicloudmask/blob/main/training/Train OCM models.ipynb

Imports essential libraries for deep learning, image processing, and model saving. Ensure these libraries are installed before running.

```python
import torch
import rasterio as rio
from fastai.vision.all import * # type: ignore
from pathlib import Path
from safetensors.torch import save_file
import timm
from rasterio.enums import Resampling
from rasterio.errors import NotGeoreferencedWarning
import warnings
import numpy as np
import random
import cv2
from collections import defaultdict
```

--------------------------------

### Run Experiment Across Resolutions

Source: https://github.com/dpird-dma/omnicloudmask/blob/main/benchmarking/spatial_context.ipynb

Iterates through different resolutions, pre-loading images and labels, and then running threaded predictions for each patch size. This setup optimizes for performance by loading data once and overlapping I/O with computation.

```python
results = {}
pixel_accuracy_10m: dict[
    int, np.ndarray
] = {}  # patch_size -> (eval_size, eval_size) mean correctness

for res_name, factor in RESOLUTION_FACTORS.items():
    print(f"\n{'=' * 60}")
    print(f"Resolution: {res_name} (downsample factor: {factor})")
    print(f"{'=' * 60}")

    # Pre-load all images (downsampled) and 10m labels into memory
    images: list[np.ndarray] = []
    labels_10m: list[np.ndarray] = []
    for img_path, label_path in tqdm(scenes, desc=f"Loading {res_name}"):
        with rasterio.open(img_path) as src:
            img = src.read()
        with rasterio.open(label_path) as src:
            label = src.read(1)
        labels_10m.append(label)
        if factor > 1:
            img = downsample_image(img, factor)
        images.append(img)

    effective_size = min(images[0].shape[1], images[0].shape[2])
    print(
        f"Loaded {len(images)} scenes, images at {effective_size}\u00d7{effective_size} px"
    )

    # Include the full downsampled image size so all resolutions
    # cover their maximum available ground distance
    patch_sizes = sorted(set(PATCH_SIZES + [effective_size]))

    # Evaluate each patch size with threaded predictions
    res_results = {}
    for patch_size in patch_sizes:
        if patch_size > effective_size:
            continue

        all_preds: list[np.ndarray] = []
        all_labels: list[np.ndarray] = []

        with ThreadPoolExecutor(max_workers=N_WORKERS) as pool:
            futures = {
                pool.submit(
                    predict_scene,
                    images[i],
                    labels_10m[i],
                    patch_size,
                    EVAL_SIZE,
                    factor,
                ): i
                for i in range(len(images))
            }
            for future in tqdm(
                as_completed(futures),
                total=len(futures),
                desc=f"{res_name} {patch_size}px",
                leave=False,
            ):
                preds_stack, labels_stack = future.result()
                all_preds.append(preds_stack.ravel())
                all_labels.append(labels_stack.ravel())
                if factor == 1:
                    # Accumulate correctness across all rotations for smoother heatmap
                    correct = (
                        (preds_stack == labels_stack).astype(np.float32).sum(axis=0)
                    )
                    if patch_size in pixel_accuracy_10m:
                        pixel_accuracy_10m[patch_size] += correct
                    else:
                        pixel_accuracy_10m[patch_size] = correct.copy()

        y_pred = np.concatenate(all_preds)
        y_true = np.concatenate(all_labels)
        metrics = compute_metrics(y_true, y_pred)
        res_results[patch_size] = metrics
```

--------------------------------

### Import Libraries for Maxar Cloud Masking

Source: https://github.com/dpird-dma/omnicloudmask/blob/main/examples/maxar.ipynb

Imports necessary libraries for data handling, visualization, and cloud masking. Ensure these are installed before running.

```python
from pathlib import Path

import matplotlib.pyplot as plt
import numpy as np
import rasterio as rio
import requests
from rasterio.enums import Resampling
from tqdm.auto import tqdm

from omnicloudmask import predict_from_array
```

--------------------------------

### Pre-generate Scenes for Benchmarking

Source: https://github.com/dpird-dma/omnicloudmask/blob/main/benchmarking/benchmarking.ipynb

Generates and stores synthetic scenes of various sizes before starting the main benchmarking loop. This ensures that the time taken for scene generation does not influence the performance measurements.

```python
# Pre-generate all scenes so array creation isn't included in timings
all_sizes = sorted(set(s for sizes in SCENE_SIZES.values() for s in sizes))
scenes: dict[int, np.ndarray] = {size: make_scene(size) for size in all_sizes}
warmup_scene = make_scene(64)
```

--------------------------------

### Get OCM version

Source: https://github.com/dpird-dma/omnicloudmask/blob/main/validation dataset/Landsat/OCM inference.ipynb

Retrieves and displays the installed version of the omnicloudmask library.

```python
ocm_version = omnicloudmask.__version__
ocm_version
```

--------------------------------

### Serve Documentation Locally

Source: https://github.com/dpird-dma/omnicloudmask/blob/main/benchmarking/README.md

Builds and serves the project documentation locally with live reloading. Access the docs at http://127.0.0.1:8000.

```bash
uv run sphinx-autobuild docs docs/_build/html
```

--------------------------------

### Create Output Directories

Source: https://github.com/dpird-dma/omnicloudmask/blob/main/training/Get datasets/4 Make CloudSEN12 super res.ipynb

Creates directories for storing raw super-resolution output and tiled super-resolution output. The `exist_ok=True` argument prevents errors if the directories already exist.

```python
super_res_raw_dir = base_dataset_dir / "CloudSEN12 high super res raw"
super_res_raw_dir.mkdir(exist_ok=True)

super_res_tile_dir = base_dataset_dir / "CloudSEN12 high super res tiles"
super_res_tile_dir.mkdir(exist_ok=True)
```

--------------------------------

### Create Download Directories

Source: https://github.com/dpird-dma/omnicloudmask/blob/main/validation dataset/PlanetScope/Download PlanetScope.ipynb

Creates the main download directory and subdirectories for scenes and UDM2 files if they do not already exist.

```python
download_dir = Path.cwd() / "dataset"
download_dir.mkdir(exist_ok=True)
```

```python
scenes_dir = download_dir / "Scenes"
scenes_dir.mkdir(exist_ok=True)
udm2_dir = download_dir / "UDM2"
udm2_dir.mkdir(exist_ok=True)
```

--------------------------------

### Install OmniCloudMask from conda-forge

Source: https://github.com/dpird-dma/omnicloudmask/blob/main/docs/installation.md

Install OmniCloudMask using the conda package manager from the conda-forge channel.

```bash
conda install conda-forge::omnicloudmask
```

--------------------------------

### Configuration and Scene Discovery

Source: https://github.com/dpird-dma/omnicloudmask/blob/main/benchmarking/spatial_context.ipynb

Sets up dataset paths, evaluation parameters, and identifies image-label pairs for benchmarking. Adjust DATASET_DIR to your local path.

```python
DATASET_DIR = Path("/media/nick/4TB Working 7/Datasets/OCM datasets v3/CloudSEN12 test")
EVAL_SIZE = 32  # Center pixels to evaluate
PATCH_SIZES = [32, 48, 64, 96, 128, 192, 256, 320, 384, 448, 509]
RESOLUTION_FACTORS = {"10m": 1, "20m": 2, "30m": 3, "40m": 4, "50m": 5}
CLASS_NAMES = {0: "Clear", 1: "Thick Cloud", 2: "Thin Cloud", 3: "Cloud Shadow"}
N_WORKERS = 4  # Threads for concurrent predict_from_array calls

# Discover scenes: pair each L2A image with its label
l2a_files = sorted(DATASET_DIR.glob("*_l2a.tif"))
scenes = []
for img_path in l2a_files:
    label_path = img_path.parent / img_path.name.replace("_image_l2a", "_label")
    if label_path.exists():
        scenes.append((img_path, label_path))

print(f"Found {len(scenes)} scenes")
print(f"Patch sizes: {PATCH_SIZES}")
print(f"Eval size: {EVAL_SIZE}\u00d7{EVAL_SIZE} pixels")
print(f"Resolutions: {list(RESOLUTION_FACTORS.keys())}")
print(f"Workers: {N_WORKERS}")
```

--------------------------------

### Set Up Directories for Batch Processing

Source: https://github.com/dpird-dma/omnicloudmask/blob/main/examples/planetscope.ipynb

Defines the input directory for PlanetScope scenes and the output directory for saving generated cloud masks.

```python
scenes_dir = Path("/example/path/planetscope_scenes")
output_dir = Path("/example/path/output")

scene_paths = list(scenes_dir.glob("*.tif"))
print(f"Found {len(scene_paths)} scenes")
```

--------------------------------

### Run Documentation Generator

Source: https://github.com/dpird-dma/omnicloudmask/blob/main/benchmarking/README.md

Executes the script to generate the performance documentation page and plot from benchmark results. This should be run after new results are added.

```bash
uv run benchmarking/generate_docs_table.py
```

--------------------------------

### Configure Logging

Source: https://github.com/dpird-dma/omnicloudmask/blob/main/training/Get datasets/3 Download CloudSEN12 Planitary computer.ipynb

Sets up basic logging to display warning messages and their descriptions. This is useful for monitoring the script's execution and identifying potential issues.

```python
import logging
logging.basicConfig(level=logging.WARNING, format="%(levelname)s: %(message)s")
```

--------------------------------

### Initialize STAC Client and Fetch Scene

Source: https://github.com/dpird-dma/omnicloudmask/blob/main/examples/sentinel2_planetary_computer.ipynb

Initialize the Planetary Computer STAC client and retrieve a specific Sentinel-2 L2A scene item using its ID.

```python
catalog = pystac_client.Client.open(
    "https://planetarycomputer.microsoft.com/api/stac/v1",
    modifier=planetary_computer.sign_inplace,
)

ITEM_ID = "S2C_MSIL2A_20260104T015411_R117_T51KYB_20260104T055910"
item = catalog.get_collection("sentinel-2-l2a").get_item(ITEM_ID)
print(f"Found: {item.id}  (cloud cover: {item.properties['eo:cloud_cover']}%)")
```

--------------------------------

### Check OmniCloudMask library version

Source: https://github.com/dpird-dma/omnicloudmask/blob/main/tests/Visual tests.ipynb

This snippet checks and displays the installed version of the OmniCloudMask library.

```python
import omnicloudmask

omnicloudmask.__version__
```

--------------------------------

### Import Libraries

Source: https://github.com/dpird-dma/omnicloudmask/blob/main/examples/sentinel2_planetary_computer.ipynb

Import all required libraries for data handling, STAC querying, and cloud masking.

```python
from concurrent.futures import ThreadPoolExecutor

import numpy as np
import planetary_computer
import pystac_client
import rasterio
from rasterio.enums import Resampling

from omnicloudmask import predict_from_array
```

--------------------------------

### Print OmniCloudMask Version

Source: https://github.com/dpird-dma/omnicloudmask/blob/main/validation dataset/Landsat/Get stats.ipynb

Prints the installed version of the OmniCloudMask library. This is useful for tracking dependencies and ensuring compatibility.

```python
ocm_version = omnicloudmask.__version__
print(f"OmniCloudMask version: {ocm_version}")
```

--------------------------------

### Configure DataBlock for OCM model training

Source: https://github.com/dpird-dma/omnicloudmask/blob/main/training/Train OCM models.ipynb

Sets up the DataBlock with image opening, mask processing, weight sampling, and specified transformations for training.

```python
dblock = DataBlock(
    blocks=[
        TransformBlock([open_image_func]),
        MaskBlock(codes=[0, 1, 2, 3]),
        TransformBlock([sample_weights]),
    ],
    n_inp=1,
    get_items=multi_dataset_getter,
    get_y=[label_func, lambda x: x],
    splitter=FuncSplitter(is_validation_item),
    batch_tfms=batch_tfms,
    item_tfms=[
        Resize(original_image_size, method="squish")
    ],  # required to resize the 2 masks to 509
)
```

--------------------------------

### Set Scene Path and Create Directory

Source: https://github.com/dpird-dma/omnicloudmask/blob/main/examples/planetscope_hyperspectral.ipynb

Defines the local path for saving the downloaded scene and ensures the parent directory exists.

```python
scene_path = Path("PlanetScope data", sample_scene_url.split("/")[-1])
scene_path.parent.mkdir(parents=True, exist_ok=True)
```

--------------------------------

### Import Necessary Libraries

Source: https://github.com/dpird-dma/omnicloudmask/blob/main/benchmarking/spatial_context.ipynb

Imports all required libraries for data manipulation, image processing, and model prediction. Ensure these are installed in your environment.

```python
import json
import warnings
from concurrent.futures import ThreadPoolExecutor, as_completed
from datetime import datetime, timezone
from pathlib import Path

import matplotlib.pyplot as plt
import numpy as np
import rasterio
import torch
from scipy.ndimage import zoom
from sklearn.metrics import balanced_accuracy_score, f1_score
from tqdm.auto import tqdm

import omnicloudmask
from omnicloudmask import predict_from_array
```

--------------------------------

### Create Dataset Directory

Source: https://github.com/dpird-dma/omnicloudmask/blob/main/validation dataset/Landsat/Download PixBox Landsat 8.ipynb

Ensures the 'dataset' directory exists for storing downloaded files. If it already exists, it remains unchanged.

```python
dataset_dir = Path("dataset")
dataset_dir.mkdir(exist_ok=True)
```

--------------------------------

### Validate Dataset Directory Existence

Source: https://github.com/dpird-dma/omnicloudmask/blob/main/training/Train OCM models.ipynb

Iterates through the dataset directories and asserts that each directory exists. This prevents training from starting with missing data.

```python
for dataset_dir in dataset_dirs:
    assert dataset_dir.exists(), \
        (f"Training data directory {dataset_dir} does not exist.")
```

--------------------------------

### Configure Logging

Source: https://github.com/dpird-dma/omnicloudmask/blob/main/training/Get datasets/3 Download CloudSEN12 Planitary computer.ipynb

Sets up basic logging to display warning messages and above. Use this to control the verbosity of log output during data processing.

```python
logging.basicConfig(level=logging.WARNING, format="%(levelname)s: %(message)s")
```

--------------------------------

### Helper function to get sysctl value

Source: https://github.com/dpird-dma/omnicloudmask/blob/main/benchmarking/benchmarking.ipynb

A utility function to retrieve values from the `sysctl` command on macOS. It handles potential errors and timeouts.

```python
import subprocess

def _sysctl(key: str) -> str:
    """Return the value of a sysctl key, or empty string on failure."""
    try:
        result = subprocess.run(
            ["sysctl", "-n", key],
            capture_output=True,
            text=True,
            timeout=5
        )
        return result.stdout.strip() if result.returncode == 0 else ""
    except Exception:
        return ""
```

--------------------------------

### Get total system RAM in GB

Source: https://github.com/dpird-dma/omnicloudmask/blob/main/benchmarking/benchmarking.ipynb

Calculates and returns the total system RAM in gigabytes using standard library functions for cross-platform compatibility.

```python
import os

def get_ram_gb() -> float:
    """Get total system RAM in GB using stdlib only."""
    try:
        pages = os.sysconf("SC_PHYS_PAGES")
        page_size = os.sysconf("SC_PAGE_SIZE")
        return round(pages * page_size / (1024**3), 1)
    except (ValueError, OSError):
        return 0.0
```

--------------------------------

### Set up directories for scenes and predictions

Source: https://github.com/dpird-dma/omnicloudmask/blob/main/validation dataset/Landsat/OCM inference.ipynb

Defines the directories for Landsat 8 scenes and OCM predictions, creating the prediction directory if it doesn't exist. The prediction directory name includes the OCM version.

```python
scenes_dir = dataset_dir / "Landsat8_L1"
preds_dir = dataset_dir / f"OCM preds v{ocm_version}"
preds_dir.mkdir(exist_ok=True)
```

--------------------------------

### Get CPU model name

Source: https://github.com/dpird-dma/omnicloudmask/blob/main/benchmarking/benchmarking.ipynb

Retrieves the CPU model name, providing specific details for macOS and a fallback for other systems using `/proc/cpuinfo` or `platform.processor()`.

```python
import platform

def get_cpu_name() -> str:
    """Get CPU model name cross-platform."""
    if platform.system() == "Darwin":
        name = _sysctl("machdep.cpu.brand_string")
        if name:
            return name
    try:
        with open("/proc/cpuinfo") as f:
            for line in f:
                if line.startswith("model name"):
                    return line.split(":")[1].strip()
    except FileNotFoundError:
        pass
    return platform.processor() or "Unknown"
```

--------------------------------

### Create Raw Dataset Directory

Source: https://github.com/dpird-dma/omnicloudmask/blob/main/training/Get datasets/2 Download kappaset.ipynb

Creates a directory to store the raw downloaded zip file and its extracted contents. The directory is created with parent directories if they do not exist.

```python
kappaset_raw_dir = base_dataset_dir / "Kappaset raw"
kappaset_raw_dir.mkdir(parents=True, exist_ok=True)
```

--------------------------------

### Predict with Specific Model Version

Source: https://github.com/dpird-dma/omnicloudmask/blob/main/docs/usage.md

Use older model versions by specifying the `model_version` parameter. Note that versions 1-3 require legacy extra installation.

```python
mask = predict_from_array(input_array, model_version=3.0)
```

--------------------------------

### Upload Package to Test PyPI

Source: https://github.com/dpird-dma/omnicloudmask/blob/main/python_package_publishing.md

Uploads the built package to the Test PyPI repository. Requires user token authentication.

```bash
uv publish --publish-url https://test.pypi.org/legacy/
```

--------------------------------

### Get DataFrame Shape

Source: https://github.com/dpird-dma/omnicloudmask/blob/main/validation dataset/Sentinel-2/Get stats.ipynb

Retrieves and displays the current shape (number of rows and columns) of the validation data DataFrame. This is useful for tracking data dimensions during processing.

```python
val_data.shape
```

--------------------------------

### Import necessary libraries

Source: https://github.com/dpird-dma/omnicloudmask/blob/main/validation dataset/Throughput/Downlaod scenes.ipynb

Imports the gdown library for downloading from Google Drive, pathlib for path manipulation, and zipfile for handling zip archives.

```python
import gdown
from pathlib import Path
import zipfile
```

--------------------------------

### Get MPS GPU name for Apple Silicon

Source: https://github.com/dpird-dma/omnicloudmask/blob/main/benchmarking/benchmarking.ipynb

Retrieves the specific chip name for Apple Silicon GPUs when using the MPS backend, ensuring accurate labeling.

```python
def get_mps_gpu_name() -> str:
    """Get Apple Silicon chip name for MPS GPU labelling."""
    name = _sysctl("machdep.cpu.brand_string")
    return f"{name} (MPS)" if name else "Apple MPS"
```

--------------------------------

### Experiment Progress Log (50m, 64px)

Source: https://github.com/dpird-dma/omnicloudmask/blob/main/benchmarking/spatial_context.ipynb

Shows the progress of an experiment run with a 50m model, 64px resolution, and 16px context size.

```text
50m 64px:   0%|          | 0/975 [00:00<?, ?it/s]
```

```text
    64px  ctx=  16px  boa=0.7175
```

--------------------------------

### Call Process Dataset Function

Source: https://github.com/dpird-dma/omnicloudmask/blob/main/training/Get datasets/1 Download CloudSEN12.ipynb

These examples show how to call the `process_dataset` function with different processing levels and label options. Ensure the `image_types` variable is defined before calling.

```python
process_dataset(processing_level="l1c", image_types=image_types)
```

```python
process_dataset(processing_level="l2a", image_types=image_types, label=False)
```

--------------------------------

### Define Sample Scene URL

Source: https://github.com/dpird-dma/omnicloudmask/blob/main/examples/planetscope_hyperspectral.ipynb

Specifies the URL for downloading a sample PlanetScope hyperspectral scene in HDF5 format.

```python
sample_scene_url = "https://storage.googleapis.com/open-cogs/planet-stac/release1-basic-radiance/20250302_030003_92_4001_basic_radiance.h5"
```

--------------------------------

### Get Training and Validation Images

Source: https://github.com/dpird-dma/omnicloudmask/blob/main/training/Train OCM models.ipynb

Calls the multi_dataset_getter function to retrieve all training and validation image paths. The print_counts argument is set to True to display detailed information about the dataset loading process.

```python
train_and_val_images = multi_dataset_getter(list(dataset_dirs), print_counts=True)
```

--------------------------------

### Specify Google Drive for Model Downloads

Source: https://github.com/dpird-dma/omnicloudmask/blob/main/docs/troubleshooting.md

Use 'google_drive' as the model download source if Hugging Face downloads fail.

```python
mask = predict_from_array(
    input_array,
    model_download_source="google_drive",
)
```

--------------------------------

### Experiment Progress Log (50m, 32px)

Source: https://github.com/dpird-dma/omnicloudmask/blob/main/benchmarking/spatial_context.ipynb

Shows the progress of an experiment run with a 50m model, 32px resolution, and 0px context size.

```text
50m 32px:   0%|          | 0/975 [00:00<?, ?it/s]
```

```text
    32px  ctx=   0px  boa=0.6214
```

--------------------------------

### Display Configuration Path

Source: https://github.com/dpird-dma/omnicloudmask/blob/main/training/Train OCM models.ipynb

Prints the path where the configuration file is saved. Useful for locating the configuration file after it has been written.

```python
config_path
```

--------------------------------

### Set up local directory for Sentinel-2 data

Source: https://github.com/dpird-dma/omnicloudmask/blob/main/examples/sentinel2_safe.ipynb

Defines and creates a local directory to store downloaded Sentinel-2 scenes. Ensures the directory exists.

```python
test_data_dir = Path.cwd() / "Sentinel-2 data"
test_data_dir.mkdir(exist_ok=True, parents=True)
```

--------------------------------

### Experiment Progress Log (50m, 48px)

Source: https://github.com/dpird-dma/omnicloudmask/blob/main/benchmarking/spatial_context.ipynb

Shows the progress of an experiment run with a 50m model, 48px resolution, and 8px context size.

```text
50m 48px:   0%|          | 0/975 [00:00<?, ?it/s]
```

```text
    48px  ctx=   8px  boa=0.6801
```

--------------------------------

### Experiment Progress Log (64px)

Source: https://github.com/dpird-dma/omnicloudmask/blob/main/benchmarking/spatial_context.ipynb

Shows the progress of an experiment run with 64px resolution and a context size of 16px.

```text
    64px  ctx=  16px  boa=0.7275
```

```text
40m 96px:   0%|          | 0/975 [00:00<?, ?it/s]
```

--------------------------------

### Download Kappaset Dataset with Progress Bar

Source: https://github.com/dpird-dma/omnicloudmask/blob/main/training/Get datasets/2 Download kappaset.ipynb

Downloads the dataset from the specified URL. It checks for existing files and shows a progress bar during download. Requires the 'requests' and 'tqdm' libraries.

```python
# Full path for the downloaded file
download_path = kappaset_raw_dir / "kappaset.zip"

# Get the file size first
response = requests.head(dataset_url)
total_size = int(response.headers.get("content-length", 0))

# Download with progress bar
response = requests.get(dataset_url, stream=True)
response.raise_for_status()
if not download_path.exists():
    with (
        download_path.open("wb") as file,
        tqdm(
            desc="Downloading",
            total=total_size,
            unit="B",
            unit_scale=True,
            unit_divisor=1024,
        ) as pbar,
    ):
        for chunk in response.iter_content(chunk_size=8192):
            file.write(chunk)
            pbar.update(len(chunk))

    print(f"Downloaded to: {download_path.absolute()}")
```

--------------------------------

### Load Planet API Key

Source: https://github.com/dpird-dma/omnicloudmask/blob/main/validation dataset/PlanetScope/Download PlanetScope.ipynb

Reads the Planet API key from a 'Planet API key.txt' file in the current working directory. Ensure this file exists and contains your API key.

```python
# You need to have a Planet API key in a file called "Planet API key.txt"
# in the current working directory
api_key_path = Path.cwd() / "Planet API key.txt"
api_key = api_key_path.read_text().strip()
```

--------------------------------

### Experiment Progress Log (50m, 102px)

Source: https://github.com/dpird-dma/omnicloudmask/blob/main/benchmarking/spatial_context.ipynb

Shows the progress of an experiment run with a 50m model, 102px resolution, and 35px context size.

```text
50m 102px:   0%|          | 0/975 [00:00<?, ?it/s]
```

```text
   102px  ctx=  35px  boa=0.7596
```

--------------------------------

### Create scene directory

Source: https://github.com/dpird-dma/omnicloudmask/blob/main/validation dataset/Throughput/Downlaod scenes.ipynb

Creates a directory named 'speed test scenes' in the current working directory. The directory is created if it doesn't exist.

```python
scene_dir = Path.cwd() / "speed test scenes"
scene_dir.mkdir(exist_ok=True, parents=True)
```

--------------------------------

### Experiment Progress Log (50m, 96px)

Source: https://github.com/dpird-dma/omnicloudmask/blob/main/benchmarking/spatial_context.ipynb

Shows the progress of an experiment run with a 50m model, 96px resolution, and 32px context size.

```text
50m 96px:   0%|          | 0/975 [00:00<?, ?it/s]
```

```text
    96px  ctx=  32px  boa=0.7513
```

--------------------------------

### Define Training Configuration

Source: https://github.com/dpird-dma/omnicloudmask/blob/main/training/Train OCM models.ipynb

Creates a dictionary containing various configuration parameters for model training. This includes model type, training parameters, and weights for different labels.

```python
config = {
    "model_version": model_version,
    "model_type": model_type,
    "use_bf16": use_bf16,
    "demo_mode": demo_mode,
    "original_image_size": original_image_size,
    "max_clip_image_clip_size": max_clip_image_clip_size,
    "min_clip_image_size": min_clip_image_size,
    "limited_band_read_list": limited_band_read_list,
    "native_band_scales": native_band_scales,
    "gradient_accumulation_batch_size": gradient_accumulation_batch_size,
    "batch_size": batch_size,
    "learning_rate": learning_rate,
    "high_label_weight": high_label_weight,
    "scribble_label_weight": scribble_label_weight,
    "tiles_2k_weight": tiles_2k_label_weight,
    "kappaset_weight": kappaset_label_weight,
    "super_res_weight": super_res_label_weight,
    "high_pc_label_weight": high_pc_label_weight,
    "hard_negative_label_weight": hard_negative_label_weight,
    "freeze_epochs": freeze_epochs,
    "unfrozen_epochs": unfrozen_epochs,
    "limit_training_images": limit_training_images,
}
```

--------------------------------

### Create Dataset Directories

Source: https://github.com/dpird-dma/omnicloudmask/blob/main/training/Get datasets/1 Download CloudSEN12.ipynb

Creates the necessary subdirectories for different CloudSEN12 dataset types ('high', 'scribble', '2k', 'validation', 'test') within the base directory. Directories are created if they do not already exist.

```python
high_dir = base_dataset_dir / "CloudSEN12 high"
scribble_dir = base_dataset_dir / "CloudSEN12 scribble"
two_k_dir = base_dataset_dir / "CloudSEN12 2k"
validation_dir = base_dataset_dir / "CloudSEN12 validation"
test_dir = base_dataset_dir / "CloudSEN12 test"

high_dir.mkdir(exist_ok=True, parents=True)
scribble_dir.mkdir(exist_ok=True, parents=True)
two_k_dir.mkdir(exist_ok=True, parents=True)
validation_dir.mkdir(exist_ok=True, parents=True)
test_dir.mkdir(exist_ok=True, parents=True)
```

--------------------------------

### Format Code with Ruff

Source: https://github.com/dpird-dma/omnicloudmask/blob/main/docs/contributing.md

Use ruff to format the codebase according to project style guidelines.

```bash
ruff format .
```

--------------------------------

### Initialize Learner

Source: https://github.com/dpird-dma/omnicloudmask/blob/main/training/Train OCM models.ipynb

Creates a `Learner` object with the specified data loaders, model, loss function, metrics, and callbacks. This is the core object for training.

```python
learner = Learner(
    dls=dl,
    model=model,
    loss_func=CrossEntropyLossFlatImageTypeWeighted(),
    metrics=[DiceMultiStrip],
    cbs=callbacks,
)
```

--------------------------------

### Experiment Progress Log (48px)

Source: https://github.com/dpird-dma/omnicloudmask/blob/main/benchmarking/spatial_context.ipynb

Shows the progress of an experiment run with 48px resolution and a context size of 8px.

```text
48px  ctx=   8px  boa=0.6965
```

```text
40m 64px:   0%|          | 0/975 [00:00<?, ?it/s]
```

--------------------------------

### Create Timm Model Architecture

Source: https://github.com/dpird-dma/omnicloudmask/blob/main/training/Train OCM models.ipynb

Initializes a timm model with specified architecture, pretrained weights, and number of input channels. This model will be used as the backbone for the UNet.

```python
timm_model = partial(
    timm.create_model,
    model_type,
    pretrained=True,
    in_chans=num_input_channels,
)
model = create_unet_model(
    img_size=(509, 509),
    arch=timm_model,
    n_out=4,
    pretrained=True,
    act_cls=torch.nn.Mish,
)
```

--------------------------------

### Create directory for CloudSEN12 high planetary computer data

Source: https://github.com/dpird-dma/omnicloudmask/blob/main/training/Get datasets/3 Download CloudSEN12 Planitary computer.ipynb

Creates the directory to store CloudSEN12 'high' data from Planetary Computer. The directory is created if it does not already exist.

```python
high_planetary_computer_dir = base_dataset_dir / "CloudSEN12 high planetary computer"
high_planetary_computer_dir.mkdir(exist_ok=True, parents=True)
```

--------------------------------

### Experiment Progress Log (96px)

Source: https://github.com/dpird-dma/omnicloudmask/blob/main/benchmarking/spatial_context.ipynb

Shows the progress of an experiment run with 96px resolution and a context size of 32px.

```text
    96px  ctx=  32px  boa=0.7640
```

```text
40m 127px:   0%|          | 0/975 [00:00<?, ?it/s]
```

--------------------------------

### Benchmarking Inference Times

Source: https://github.com/dpird-dma/omnicloudmask/blob/main/benchmarking/benchmarking.ipynb

This script benchmarks inference times across different devices and data types. It iterates through specified devices and data types, performs warm-up runs, and then times the inference for various scene sizes, dynamically adjusting batch size for optimal performance. Results include size, mean time, standard deviation, and batch size.

```python
results: dict[tuple[str, torch.dtype], list[tuple]] = {}

for device in devices:
    for dtype in DTYPES[device]:
        dtype_name = str(dtype).split(".")[-1]
        key = (device, dtype)

        # Warm up this device/dtype before any timing
        timed_predict(warmup_scene, batch_size=1, device=device, dtype=dtype)

        print(f"\n{device} / {dtype_name}")
        print(f"{ 'size':>10}  {'bs':>4}  {'reps':>4}  {'mean_s':>8}  {'std_s':>8}")
        print("-" * 46)

        config_results = []
        for size in SCENE_SIZES[device]:
            scene = scenes[size]
            repeats = REPEATS_SMALL if size <= 500 else REPEATS_LARGE

            # Only one patch when scene <= 1000 px, so bs=1
            if size <= 1000:
                bs = 1
            else:
                bs = find_optimal_batch_size(scene, device, dtype)

            # Warmup with chosen batch size, then time
            timed_predict(scene, batch_size=bs, device=device, dtype=dtype)

            times = []
            for _ in range(repeats):
                t = timed_predict(scene, batch_size=bs, device=device, dtype=dtype)
                times.append(t)
            mean_s = float(np.mean(times))
            std_s = float(np.std(times, ddof=1))
            config_results.append((size, mean_s, std_s, bs))
            print(f"{size:>10}  {bs:>4}  {repeats:>4}  {mean_s:>8.3f}  {std_s:>8.3f}")

        results[key] = config_results
```

--------------------------------

### Placeholder for Tiling Logic

Source: https://github.com/dpird-dma/omnicloudmask/blob/main/training/Get datasets/4 Make CloudSEN12 super res.ipynb

This is a placeholder comment indicating where logic to grab the top-left of each image and mask, and save it, would be implemented.

```python
# grab the top left of each image and mask and save it
```

--------------------------------

### Create a partial function for 20m resolution loading

Source: https://github.com/dpird-dma/omnicloudmask/blob/main/examples/sentinel2_safe.ipynb

Creates a specialized loader function `load_s2_20m` using `functools.partial` to pre-set the resolution to 20.0 meters for Sentinel-2 data loading. This can speed up inference.

```python
load_s2_20m = partial(load_s2, resolution=20.0)
```

--------------------------------

### Configure PlanetScope Data Loader

Source: https://github.com/dpird-dma/omnicloudmask/blob/main/validation dataset/PlanetScope/OCM inference.ipynb

Creates a partial function for loading PlanetScope multiband data. It specifies the band order and resampling resolution for consistent data input.

```python
ps_loader = partial(load_multiband, band_order=[6, 4, 8], resample_res=10)
```

--------------------------------

### Save Training Configuration to File

Source: https://github.com/dpird-dma/omnicloudmask/blob/main/training/Train OCM models.ipynb

Saves the training configuration dictionary to a JSON file. This allows for easy reproducibility and management of training settings.

```python
with open(config_path, "w") as f:
    json.dump(config, f, indent=4)
```

--------------------------------

### Generate Markdown Documentation Page

Source: https://github.com/dpird-dma/omnicloudmask/blob/main/benchmarking/spatial_context.ipynb

Writes markdown content to a file, including explanations of spatial context, recommendations for patch size, and metric definitions. Requires Path.

```python
DOCS_DIR = Path("..") / "docs"

# --- Build docs page ---
docs_page = DOCS_DIR / "spatial-context.md"
docs_page.write_text(
    """# Spatial Context

When processing a small area of interest, the size of the patch you feed to OmniCloudMask has a large effect on accuracy. Follow these recommendations to get the best results.

## When this matters

These guidelines apply when:

- You have deliberately chosen a small `patch_size` for `predict_from_array` or `predict_from_load_func`.
- Your input scene is small enough that OmniCloudMask automatically reduces the patch size to match the scene dimensions.
- You are deciding whether to process imagery at native resolution or downsample for speed.

If you are running large inputs such as Sentinel-2 or Landsat scenes with the default `patch_size=1000`, you already have plenty of spatial context and can skip this page.

## Recommendations

### Use at least 96×96 pixels

If you only need to classify a small area, expand the input to the model to **at least 96×96 pixels** centred on your area of interest. Most of the accuracy gain happens between 32×32 and 96×96. Below 96×96, the model has too little surrounding context and performance drops sharply.

### Go larger when you can

Accuracy continues to improve beyond 96×96 but at a slower rate. If your workflow allows larger patches without hurting runtime, use them. The default `patch_size=1000` tends to give reliable results. Larger patches also reduce the proportion of overlap pixels when tiling a large scene, which can speed up processing.

### Expect thin cloud and cloud shadow to suffer the most from small patches

Clear and thick cloud classes are relatively easy for the model to identify even with minimal context. Thin cloud and cloud shadow are much harder and see the largest accuracy improvements as patch size grows. If your use case depends on accurate thin cloud or shadow detection, err on the side of larger patches.

## How the results were produced

Evaluated on the CloudSEN12 High test set ({n_scenes} scenes). For each scene, the model sees a center patch of a given size, but only the center 32×32 pixels of the prediction are scored against the ground truth. This isolates the effect of spatial context on accuracy for a fixed-size area of interest.

The experiment is repeated at simulated {resolutions_str} resolution (bilinear downsampling of the native 10m Sentinel-2 imagery) with predictions upsampled back to 10m so all resolutions are compared against the same 10m ground truth.

### Metrics

**BOA (Balanced Overall Accuracy)** is the mean of per-class recall, averaged across all classes. Unlike overall accuracy, BOA gives equal weight to each class regardless of how many pixels it covers, so rare classes like thin cloud and cloud shadow are not drowned out by the dominant clear class.

$$BOA = 0.5 \\left( \\\\frac{{TP}}{{TP + FN}} + \\\\frac{{TN}}{{TN + FP}} \\\\right) $$

**Dice** (equivalent to F1 score) is reported per class.

$$Dice = \\\\frac{{2TP}}{{2TP + FP + FN}} $$

""")
```

--------------------------------

### Organize Scene Files

Source: https://github.com/dpird-dma/omnicloudmask/blob/main/validation dataset/PlanetScope/Download PlanetScope.ipynb

Moves all downloaded '.tif' files ending with '8b' (likely the 8-bit ortho-analytic scenes) into the 'Scenes' subdirectory.

```python
# move 8b.tif files into scenes dir
for scene_file in scenes_dir.glob("*8b.tif"):
    scene_file.rename(scenes_dir / scene_file.name)
```

--------------------------------

### Create DataLoaders for the training dataset

Source: https://github.com/dpird-dma/omnicloudmask/blob/main/training/Train OCM models.ipynb

Initializes DataLoaders for the training dataset using the configured DataBlock, specifying batch size, number of workers, and memory pinning.

```python
dl = dblock.dataloaders(
    size=original_image_size,
    source=dataset_dirs,
    bs=batch_size,
    num_workers=6,
    pin_memory=True,
)
```

--------------------------------

### Import necessary libraries

Source: https://github.com/dpird-dma/omnicloudmask/blob/main/benchmarking/benchmarking.ipynb

Imports essential libraries for the benchmarking process, including system utilities, numerical computation, deep learning frameworks, and the OmniCloudMask library.

```python
import os
import platform
import time
from datetime import datetime, timezone

import numpy as np
import torch

import omnicloudmask
from omnicloudmask import predict_from_array
```

--------------------------------

### Upload Package to PyPI

Source: https://github.com/dpird-dma/omnicloudmask/blob/main/python_package_publishing.md

Uploads the final, tested package to the main Python Package Index (PyPI). Ensure credentials are set up.

```bash
uv publish
```

--------------------------------

### Experiment Progress Log (127px)

Source: https://github.com/dpird-dma/omnicloudmask/blob/main/benchmarking/spatial_context.ipynb

Shows the progress of an experiment run with 127px resolution and a context size of 47px.

```text
   127px  ctx=  47px  boa=0.7715
```

--------------------------------

### Prime model for compilation

Source: https://github.com/dpird-dma/omnicloudmask/blob/main/validation dataset/Throughput/OCM inference.ipynb

Runs inference on a single scene to pre-compile the models. This is a necessary step before measuring performance with `compile_models=True`.

```python
# First prime the model with a single scene to ensure it is compiled
predict_from_load_func(
    scene_paths=scenes[:1],
    load_func=load_s2,
    batch_size=4,
    inference_dtype="bf16",
    compile_models=True,
)
```

--------------------------------

### Download Sentinel-2 scene if not present

Source: https://github.com/dpird-dma/omnicloudmask/blob/main/examples/sentinel2_safe.ipynb

Fetches a specified Sentinel-2 scene archive from the Copernicus Data Space to the local directory if it doesn't already exist. The function `fetch_single_sentinel_product` is used for this purpose.

```python
product_id = "S2A_MSIL1C_20230304T020441_N0509_R017_T50HNH_20230304T051523"
scene_dir = test_data_dir / (product_id + ".SAFE")
if not scene_dir.exists():
    fetch_single_sentinel_product(
        product_id,
        test_data_dir,
    )
scene_dir
```

--------------------------------

### Create partial function for opening images

Source: https://github.com/dpird-dma/omnicloudmask/blob/main/training/Train OCM models.ipynb

Creates a partial function 'open_image_func' from 'open_img' with predefined 'img_size' and 'use_bf16' arguments.

```python
open_image_func = partial(open_img, img_size=original_image_size, use_bf16=use_bf16)
```