Copernicus Foundation Model

https://github.com/zhu-xlab/copernicus-fm
Admin
A unified foundation model for Earth vision that processes any spectral or non-spectral sensor...
Tokens:37,288
Snippets:240
Trust Score:9.8
Update:5 months ago
Show doc for...
Tokens:
Raw
### Setup Copernicus-FM Environment and Dependencies

Source: https://github.com/zhu-xlab/copernicus-fm/blob/main/Copernicus-FM/README.md

Create a conda environment named 'copernicusfm' with Python 3.10 and install necessary packages. Alternatively, use the provided requirements.txt.

```bash
conda create -n copernicusfm python=3.10
conda activate copernicusfm
pip install torch torchvision timm einops rasterio pyproj # or pip install -r requirements.txt
```

--------------------------------

### Install Dependencies

Source: https://github.com/zhu-xlab/copernicus-fm/blob/main/Copernicus-Pretrain/data_loading/demo_dataloading.ipynb

Installs necessary libraries for data loading and processing, including kornia, rasterio, and gzip.

```bash
!pip install kornia rasterio gzip # torch torchvision
```

--------------------------------

### Install scikit-learn

Source: https://github.com/zhu-xlab/copernicus-fm/blob/main/Copernicus-Embed-025deg/demo_embed_map.ipynb

Installs or upgrades the scikit-learn library. It's recommended to use a virtual environment.

```python
!pip install -U scikit-learn
```

--------------------------------

### Sample Metadata Output

Source: https://github.com/zhu-xlab/copernicus-fm/blob/main/Copernicus-Pretrain/data_loading/demo_dataloading.ipynb

Example output showing the structure of the metadata dictionary for a single data sample, including file paths for various Copernicus sensor data.

```json
[{'s1_grd': ['0772176_-96.00_44.00/0865831_-95.92_43.96/20221121'], 's2_toa': ['0772176_-96.00_44.00/0865831_-95.92_43.96/20210303'], 's3_olci': ['0772176_-96.00_44.00/20210127'], 's5p_co': ['0772176_-96.00_44.00/20211201'], 's5p_no2': ['0772176_-96.00_44.00/20210901'], 's5p_o3': ['0772176_-96.00_44.00/20210201'], 's5p_so2': ['0772176_-96.00_44.00/20211001'], 'dem': ['0772176_-96.00_44.00']}]
```

--------------------------------

### Run Local Training Script

Source: https://github.com/zhu-xlab/copernicus-fm/blob/main/Copernicus-FM/README.md

Execute the local training script for Copernicus-FM. This example shows training on 2 GPUs. A debug option for CPU is also available.

```bash
# train locally on 2 GPUs for example
bash ./src/train_cfm_local_gpu.sh

# debug locally with cpu
# bash ./src/train_cfm_local.sh
```

--------------------------------

### Install einops Library

Source: https://github.com/zhu-xlab/copernicus-fm/blob/main/Copernicus-Bench/src/foundation_models/CROMA/README.md

Install the einops library, which is required for using CROMA models. Any relatively recent version should work.

```bash
pip install einops
```

--------------------------------

### Install WebDataset Dependency

Source: https://github.com/zhu-xlab/copernicus-fm/blob/main/Copernicus-Pretrain/data_loading/demo_dataloading.ipynb

Installs the webdataset library, which is required for loading the data shards.

```bash
# install dependencies
!pip install webdataset
```

--------------------------------

### Download and Extract Copernicus-Pretrain Data Subset

Source: https://github.com/zhu-xlab/copernicus-fm/blob/main/Copernicus-Pretrain/data_loading/demo_dataloading.ipynb

Use these shell commands to download the 100-grid subset of the Copernicus-Pretrain dataset and extract the GeoTiff files. Ensure you have `wget` and `unzip` installed.

```shell
# download and extract the 100-grid subset
!mkdir -p ../data/
!wget https://huggingface.co/datasets/wangyi111/Copernicus-Pretrain/resolve/main/example_100_grids/fnames_100_union.json.gz -P ../data/
!wget https://huggingface.co/datasets/wangyi111/Copernicus-Pretrain/resolve/main/example_100_grids/example_100_geotiff.zip -P ../data/
!unzip ../data/example_100_geotiff.zip -d ../data/example_100_geotiff/
!rm ../data/example_100_geotiff.zip
```

--------------------------------

### Install Dependencies with Conda

Source: https://github.com/zhu-xlab/copernicus-fm/blob/main/Copernicus-Bench/README.md

Installs necessary dependencies for the Copernicus-Bench project using Conda and pip. Ensure you activate the correct Conda environment before installation.

```bash
conda create -n copernicusbench python=3.10
conda activate copernicusbench
pip install -U openmim
pip install torch==2.1.2
mim install mmcv==2.1.0 mmsegmentation==1.2.2
pip install -e .
```

--------------------------------

### Initialize CopernicusPretrain DataLoader

Source: https://github.com/zhu-xlab/copernicus-fm/blob/main/Copernicus-Pretrain/data_loading/demo_dataloading.ipynb

Initializes the CopernicusPretrain dataset loader with specified shard paths, batch size, and shuffling options. This setup is for streaming data from local tar archives.

```python
from copernicuspretrain_dataset_webdataset import CopernicusPretrain

shards_path = '../data/example_100_webdataset/example-{000000..000009}.tar'
data_size = 100
batch_size = 1

copernicus_pretrain = CopernicusPretrain(shards_path, batch_size=batch_size, num_workers=2, shuffle=10, shardshuffle=True, resampled=True)
dataloader = copernicus_pretrain.get_dataloader()

# # Unbatch, shuffle between workers, then rebatch. This may explode memory usage?!
# dataloader = dataloader.unbatched().shuffle(100).batched(batch_size)
```

--------------------------------

### Visualize a Dataset Example

Source: https://github.com/zhu-xlab/copernicus-fm/blob/main/Copernicus-Bench/demo_load_dataset.ipynb

Visualizes a single data sample from the dataset, including multiple time steps of imagery and its corresponding segmentation mask. It also maps numerical labels to class names.

```python
# visualize one example
ID = 23
img = dataset_train_cls[ID]['image']
img = img.permute(1, 2, 0).numpy()
img = normalize(img)
#img = np.stack([img[:,:,0], img[:,:,1], (img[:,:,0]+img[:,:,1])/2], axis=-1)
label = dataset_train_cls[ID]['label']

ts_imgs = dataset_train_seg_se[ID]['image']
for i in range(len(ts_imgs)):
    ts_imgs[i] = ts_imgs[i].permute(1, 2, 0).numpy()
    ts_imgs[i] = normalize(ts_imgs[i])

#label2 = dataset_train_seg_se[ID]['label']

print(label)
#print(label2)
# label id to class
ml_label = []
for i in range(23):
    if label[i] == 1:
        ml_label.append(LC100_ID_TO_NAME[i])
label = ml_label

label_seg = dataset_train_seg[ID]['mask']

plt.figure(figsize=(15, 5))

plt.subplot(1, 5, 1)
plt.imshow(ts_imgs[0])
plt.title('Time 1 (static)')
plt.axis('off')
plt.subplot(1, 5, 2)
plt.imshow(ts_imgs[1])
plt.title('Time 2')
plt.axis('off')
plt.subplot(1, 5, 3)
plt.imshow(ts_imgs[2])
plt.title('Time 3')
plt.axis('off')
plt.subplot(1, 5, 4)
plt.imshow(ts_imgs[3])
plt.title('Time 4')
plt.axis('off')

plt.subplot(1, 5, 5)
plt.imshow(label_seg)
plt.title('Label')
plt.axis('off')

plt.subplots_adjust(bottom=-0.4)  # Reduce bottom margin
plt.suptitle(label, y=-0.0)

# save pdf
#plt.savefig('./assets/vis_lc100_s3_2.pdf', bbox_inches='tight')
```

--------------------------------

### Authenticate Google Earth Engine

Source: https://github.com/zhu-xlab/copernicus-fm/blob/main/Copernicus-Pretrain/data_collection/README.md

Authenticate your Google Earth Engine account after installing the API. Use `--auth_mode notebook` if browser authentication fails on a remote server.

```bash
earthengine authenticate --auth_mode notebook
```

--------------------------------

### Setup DataLoaders and Extract Coordinates

Source: https://github.com/zhu-xlab/copernicus-fm/blob/main/Copernicus-Bench/demo_load_dataset.ipynb

Initializes PyTorch DataLoaders for annual NO2 and O3 satellite data. It then iterates through the training, validation, and test dataloaders to extract and concatenate geographical coordinates.

```python
dataloader_train = torch.utils.data.DataLoader(dataset_train_no2_annual, batch_size=16, shuffle=False, num_workers=4)
dataloader_val = torch.utils.data.DataLoader(dataset_val_no2_annual, batch_size=16, shuffle=False, num_workers=4)
dataloader_test = torch.utils.data.DataLoader(dataset_test_no2_annual, batch_size=16, shuffle=False, num_workers=4)
# get the coordinates
coords_train = []
for i, data in enumerate(tqdm(dataloader_train)):
    meta = data['meta']
    coords_train.append(meta[:, :2].numpy())
coords_train = np.concatenate(coords_train, axis=0)

coords_val = []
for i, data in enumerate(tqdm(dataloader_val)):
    meta = data['meta']
    coords_val.append(meta[:, :2].numpy())
coords_val = np.concatenate(coords_val, axis=0)

coords_test = []
for i, data in enumerate(tqdm(dataloader_test)):
    meta = data['meta']
    coords_test.append(meta[:, :2].numpy())
coords_test = np.concatenate(coords_test, axis=0)
```

--------------------------------

### Load LC100Cls-S3 and LC100Seg-S3 Datasets

Source: https://github.com/zhu-xlab/copernicus-fm/blob/main/Copernicus-Bench/demo_load_dataset.ipynb

Initializes CoBenchLC100ClsS3 and CoBenchLC100SegS3 datasets for training, validation, and testing. Specify the root directory, desired bands, and data split.

```python
from src.datasets.cobench_lc100clss3_wrapper import CoBenchLC100ClsS3
from src.datasets.cobench_lc100segs3_wrapper import CoBenchLC100SegS3

root_dir_cls = './data/copernicusbench/lc100_s3olci/lc100_s3'
bands_cls = ['Oa08_radiance', 'Oa06_radiance', 'Oa04_radiance']
dataset_train_cls = CoBenchLC100ClsS3(
            root=root_dir_cls, split="train", bands=bands_cls, mode='static', transforms=None
        )
dataset_train_cls_se = CoBenchLC100SegS3(
            root=root_dir_cls, split="train", bands=bands_cls, mode='series', transforms=None
        )    

root_dir_seg = './data/copernicusbench/lc100_s3olci/lc100_s3'
bands_seg = ['Oa08_radiance', 'Oa06_radiance', 'Oa04_radiance']
dataset_train_seg = CoBenchLC100SegS3(
            root=root_dir_seg, split="train", bands=bands_seg, mode='static', transforms=None
        )
dataset_train_seg_se = CoBenchLC100SegS3(
            root=root_dir_seg, split="train", bands=bands_seg, mode='series', transforms=None
        )


LC100_ID_TO_NAME = {
    0: "unknown",
    1: "shrubs",
    2: "herbaceous vegetation",
    3: "cultivated and managed vegetation/agriculture",
    4: "urban / built-up",
    5: "bare / sparse vegetation",
    6: "snow and ice",
    7: "permanent water bodies",
    8: "herbaceous wetland",
    9: "moss and lichen",
    10: "closed forest, evergreen needle leaf",
    11: "closed forest, evergreen broad leaf",
    12: "closed forest, deciduous needle leaf",
    13: "closed forest, deciduous broad leaf",
    14: "closed forest, mixed",
    15: "closed forest, other",
    16: "open forest, evergreen needle leaf",
    17: "open forest, evergreen broad leaf",
    18: "open forest, deciduous needle leaf",
    19: "open forest, deciduous broad leaf",
    20: "open forest, mixed",
    21: "open forest, other",
    22: "oceans, seas",
}

dataset_val_cls = CoBenchLC100ClsS3(
            root=root_dir_cls, split="val", bands=bands_cls, mode='static', transforms=None
        )

dataset_test_cls = CoBenchLC100ClsS3(
            root=root_dir_cls, split="test", bands=bands_cls, mode='static', transforms=None
        )
```

--------------------------------

### Initialize CopernicusPretrain Dataset and DataLoader

Source: https://github.com/zhu-xlab/copernicus-fm/blob/main/Copernicus-Pretrain/data_loading/demo_dataloading.ipynb

Sets up the CopernicusPretrain dataset and a DataLoader for iterating through samples. Note that batch_size must be 1 due to varying numbers of images per grid. It also suppresses rasterio logging.

```python
from torch.utils.data import DataLoader
from copernicuspretrain_dataset_geotiff import CopernicusPretrain
import logging
logging.getLogger("rasterio").setLevel(logging.ERROR)

fnames_path = '../data/example_100_geotiff/fnames_100_union.json.gz'
root_dir = '../data/example_100_geotiff/'

CopernicusPretrain = CopernicusPretrain(
    fnames_path, root_dir, 
    transform_s1=None, transform_s2=None, transform_s3=None, transform_s5p=None, transform_dem=None
    )
dataloader = DataLoader(CopernicusPretrain, batch_size=1, shuffle=True, num_workers=2) # batch size can only be 1 because of varying number of images per grid

for i, (sample, meta_data) in enumerate(dataloader):
    #print(i)
    print('Grid ID:', meta_data['dem'][0])
    print(sample.keys())
    print(meta_data.keys())

    
    print('### S1 GRD ###')
    print('Number of s1 local patches:', len(meta_data['s1_grd']), '  ', 'Number of time stamps for first local patch:', len(meta_data['s1_grd'][0]))
    print('Example for one image:', sample['s1_grd'][0][0].shape, meta_data['s1_grd'][0][0])
    print('### S2 TOA ###')
    print('Number of s2 local patches:', len(meta_data['s2_toa']), '  ', 'Number of time stamps for first local patch:', len(meta_data['s2_toa'][0]))
    print('Example for one image:', sample['s2_toa'][0][0].shape, meta_data['s2_toa'][0][0])
    print('### S3 OLCI ###')
    print('Number of s3 time stamps:', len(meta_data['s3_olci']))
    print('Example for one image:', sample['s3_olci'][0].shape, meta_data['s3_olci'][0])
    print('### S5P ###')
    print('Number of s5p time stamps for CO/NO2/O3/SO2:', len(meta_data['s5p_co']), len(meta_data['s5p_no2']), len(meta_data['s5p_o3']), len(meta_data['s5p_so2']))
    print('Example for one CO image:', sample['s5p_co'][0].shape, meta_data['s5p_co'][0])
    print('Example for one NO2 image:', sample['s5p_no2'][0].shape, meta_data['s5p_no2'][0])
    print('Example for one O3 image:', sample['s5p_o3'][0].shape, meta_data['s5p_o3'][0])
    print('Example for one SO2 image:', sample['s5p_so2'][0].shape, meta_data['s5p_so2'][0])
    print('### DEM ###')
    print('One DEM image for the grid:', sample['dem'].shape, meta_data['dem'][0])

    break
```

--------------------------------

### Import Libraries for Data Collection

Source: https://github.com/zhu-xlab/copernicus-fm/blob/main/Copernicus-Pretrain/data_collection/utils/count_all_fnames_vis.ipynb

Imports necessary Python libraries for file handling, data manipulation, and visualization. Ensure these libraries are installed before running the script.

```python
import os
import json
import csv
from tqdm import tqdm
import gzip
import matplotlib.pyplot as plt
import numpy as np
```

--------------------------------

### Loading DFC2020-S1/S2 Datasets

Source: https://github.com/zhu-xlab/copernicus-fm/blob/main/Copernicus-Bench/demo_load_dataset.ipynb

Initializes CoBenchDFC2020S12 datasets for Sentinel-1 and Sentinel-2. Specify the root directory, split, desired bands, and modality.

```python
### DFC2020-S1/S2
from src.datasets.cobench_dfc2020s12_wrapper import CoBenchDFC2020S12

root_dir = './data/copernicusbench/dfc2020_s1s2/dfc2020_s1s2'

bands_s1 = ['VV', 'VH']
bands_s2 = ['B04', 'B03', 'B02']

dataset_train_s1 = CoBenchDFC2020S12(
            root=root_dir,
            split="train",
            bands=bands_s1,
            modality='s1',
            transforms=None
        )

dataset_train_s2 = CoBenchDFC2020S12(
            root=root_dir,
            split="train",
            bands=bands_s2,
            modality='s2',
            transforms=None
        )

dataset_val_s2 = CoBenchDFC2020S12(
            root=root_dir,
            split="val",
            bands=bands_s2,
            modality='s2',
            transforms=None
        )

dataset_test_s2 = CoBenchDFC2020S12(
            root=root_dir,
            split="test",
            bands=bands_s2,
            modality='s2',
            transforms=None
        )
```

--------------------------------

### Save Grid GeoDataFrame to Parquet

Source: https://github.com/zhu-xlab/copernicus-fm/blob/main/Copernicus-Pretrain/data_collection/utils/sample_era5_grids_land.ipynb

Saves the processed grid GeoDataFrame to a parquet file for efficient storage and retrieval. Includes commented-out examples for saving to CSV.

```python
# save the grid_gdf to parquet file
grid_gdf.to_parquet('grid_0_25_globe.parquet')
# read and open the parguet file
#grid_gdf_2 = gpd.read_parquet('grid_0_25_globe.parquet')

# save the grid_gdf to csv
#grid_gdf.to_csv('grid_0_25_globe.csv', index=False)
# read and open the csv file
#grid_gdf_3 = gpd.read_file('grid_0_25_globe.csv')
```

--------------------------------

### Encode Image and Get Embeddings

Source: https://github.com/zhu-xlab/copernicus-fm/blob/main/Copernicus-FM/demo.ipynb

Encodes an input image using the Copernicus-FM model, along with associated metadata and parameters. This function returns logits and embeddings.

```python
print('Encoding a new variable image with name "{}", shape {}, and expected patch size {}.'.format(varname, img.shape, kernel_size))
logit, embed = model(img, meta, wvs, bws, language_embed, input_mode, kernel_size)
print(logit.shape, embed.shape)
```

--------------------------------

### Pretraining ViT on FMoW-RGB with SatMAE++

Source: https://github.com/zhu-xlab/copernicus-fm/blob/main/Copernicus-Bench/src/foundation_models/SatMAE/README.md

Command to pretrain a ViT model on the FMoW-RGB dataset using SatMAE++. Adjust parameters like batch size, epochs, and learning rate according to your needs.

```bash
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 python -m torch.distributed.launch --nproc_per_node=8 --master_port=29201 main_pretrain.py \
--batch_size 64 --accum_iter 32 \
--epochs 800 --warmup_epochs 20 \
--input_size 224 --patch_size 16 \
--mask_ratio 0.75 \
--model_type vanilla \
--dataset_type rgb \
--weight_decay 0.3 \
--lr 0.0007 --num_workers 16 \
--train_path /home/fmow-rgb/train_62classes.json \
--output_dir ./output_dir \
--log_dir ./output_dir
```

--------------------------------

### Download Copernicus-Bench Datasets

Source: https://github.com/zhu-xlab/copernicus-fm/blob/main/Copernicus-Bench/README.md

Use this bash script to download all datasets from Copernicus-Bench. Ensure you are in the correct directory before execution.

```bash
mkdir -p data/copernicusbench
cd data/copernicusbench
bash tools/download_copernicus_bench.sh
```

--------------------------------

### Finetuning ViT on FMoW-Sentinel with SatMAE++

Source: https://github.com/zhu-xlab/copernicus-fm/blob/main/Copernicus-Bench/src/foundation_models/SatMAE/README.md

Command to finetune a ViT model pre-trained with SatMAE++ on the FMoW-Sentinel dataset. Specify the path to the pre-trained weights using the --finetune argument.

```bash
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 python -m torch.distributed.launch --nproc_per_node=8 --master_port=29202 main_finetune.py \
--batch_size 8 --accum_iter 16 \
--epochs 30 --warmup_epochs 5 \
--input_size 96 --patch_size 8 \
--model_type group_c \
--dropped_bands 0 9 10 \
--dataset_type sentinel --dropped_bands 0 9 10 \
--grouped_bands 0 1 2 6 --grouped_bands 3 4 5 7 --grouped_bands 8 9 \
--weight_decay 0.05 --drop_path 0.2 --reprob 0.25 --mixup 0.8 --cutmix 1.0 \
--blr 0.0002 --num_workers 16 \
--train_path /home/fmow-sentinel/train.csv \
--test_path /home/fmow-sentinel/val.csv \
--output_dir ./finetune_dir \
--log_dir ./finetune_dir \
--finetune ./output_dir/checkpoint-49.pth
```

--------------------------------

### Create and Load Copernicus-FM Model

Source: https://github.com/zhu-xlab/copernicus-fm/blob/main/Copernicus-FM/demo.ipynb

Initializes the ViT-base-patch16 model and loads the downloaded pre-trained weights. The `strict=False` argument allows for loading weights even if some keys are missing or unexpected.

```python
# create model
model = vit_base_patch16(num_classes=10, global_pool=False)

# load pre-trained weights
path = './weights/CopernicusFM_ViT_base_varlang_e100.pth'
check_point = torch.load(path)
if 'model' in check_point:
    state_dict = check_point['model']
else:
    state_dict = check_point
msg = model.load_state_dict(state_dict, strict=False)
print(msg)
```

--------------------------------

### Initialize CoBenchCloudS3 Dataset

Source: https://github.com/zhu-xlab/copernicus-fm/blob/main/Copernicus-Bench/demo_load_dataset.ipynb

Initializes the CoBenchCloudS3 dataset for training, validation, and testing. Specify the root directory, desired bands, and data split.

```python
from src.datasets.cobench_clouds3_wrapper import CoBenchCloudS3

root_dir = './data/copernicusbench/cloud_s3olci/cloud_s3'
bands = ['Oa08_radiance', 'Oa06_radiance', 'Oa04_radiance']
dataset_train = CoBenchCloudS3(
            root=root_dir, split="train", bands=bands, mode='multi', transforms=None
        )
dataset_val = CoBenchCloudS3(
            root=root_dir, split="val", bands=bands, mode='multi', transforms=None
        )
dataset_test = CoBenchCloudS3(
            root=root_dir, split="test", bands=bands, mode='multi', transforms=None
        )
```

--------------------------------

### Download Copernicus Pretraining Data

Source: https://github.com/zhu-xlab/copernicus-fm/blob/main/Copernicus-FM/README.md

Download the pretraining dataset shards. Each shard is approximately 5GB. Ensure the target directory exists.

```bash
mkdir -p ./data/copernicus-pretrain-tar/
wget https://huggingface.co/datasets/wangyi111/Copernicus-Pretrain/resolve/main/ssl4eo_s_220k_aligned/example-{000000..000004}.tar -P ./data/copernicus-pretrain-tar/
```

--------------------------------

### Plotting Geographical Coordinates on World Map

Source: https://github.com/zhu-xlab/copernicus-fm/blob/main/Copernicus-Bench/demo_load_dataset.ipynb

Visualizes extracted coordinates on a world map using GeoPandas and Matplotlib. Ensure GeoPandas and Matplotlib are installed and world map data is accessible.

```python
# show the coordinates on the world map
# Load world map from GeoPandas
world = gpd.read_file("https://naturalearth.s3.amazonaws.com/110m_cultural/ne_110m_admin_0_countries.zip")
# Convert coordinate arrays into GeoDataFrames
def create_geodf(coords, color):
    geometry = [Point(lon, lat) for lon, lat in zip(coords[:, 0], coords[:, 1])]
    return gpd.GeoDataFrame(geometry=geometry, crs="EPSG:4326"), color

color_train = 'red' #"#D55E00"  # Distinct orange/red
color_val = 'green' #"#0072B2"    # Strong blue
color_test = 'blue' #"#009E73"   # Teal

geo_train, color_train = create_geodf(coords_train, color_train)
geo_val, color_val = create_geodf(coords_val, color_val)
geo_test, color_test = create_geodf(coords_test, color_test)


# Plot the world map
fig, ax = plt.subplots(figsize=(12, 6))
world.plot(ax=ax, color="lightgray", edgecolor="black", alpha=0.5)  # Plot country boundaries
#world.boundary.plot(ax=ax, color="black", linewidth=1)  # Plot country boundaries

# Plot different coordinate sets
geo_train.plot(ax=ax, color=color_train, markersize=10, alpha=0.7, label="train")
geo_val.plot(ax=ax, color=color_val, markersize=10, alpha=0.7, label="val")
geo_test.plot(ax=ax, color=color_test, markersize=10, alpha=0.7, label="test")

ax.set_aspect('equal')

# Turn off axis
#ax.set_axis_off()
#ax.set_xticks([])
#ax.set_yticks([])

# Customize plot
plt.title("LC100Cls-S3 and LC100Seg-S3", fontsize=16)
plt.legend(fontsize=14, loc='lower left')

# save pdf
#plt.savefig('./assets/distribution_lc100_s3.pdf', bbox_inches='tight')

# save png
#plt.savefig('./assets/distribution_lc100_s3.png', bbox_inches='tight', dpi=100)
```

--------------------------------

### Visualize O3 Air Quality Data

Source: https://github.com/zhu-xlab/copernicus-fm/blob/main/Copernicus-Bench/demo_load_dataset.ipynb

Visualizes a single example from the O3 dataset, displaying the annual image, seasonal images, and the ground truth label. Requires a normalization function to be defined.

```python
# o3
ID = 1

img = dataset_train_o3_annual[ID]['image']
img = img.permute(1, 2, 0).numpy()
img = normalize(img)

ts_imgs = dataset_train_o3_seasonal[ID]['image']
for i in range(len(ts_imgs)):
    ts_imgs[i] = ts_imgs[i].permute(1, 2, 0).numpy()
    ts_imgs[i] = normalize(ts_imgs[i])

label = dataset_train_o3_annual[ID]['groundtruth']

plt.figure(figsize=(18, 5))

plt.subplot(1, 6, 1)
plt.imshow(img)
plt.title('Annual')
plt.axis('off')
plt.subplot(1, 6, 2)
plt.imshow(ts_imgs[0])
plt.title('Season 1')
plt.axis('off')
plt.subplot(1, 6, 3)
plt.imshow(ts_imgs[1])
plt.title('Season 2')
plt.axis('off')
plt.subplot(1, 6, 4)
plt.imshow(ts_imgs[2])
plt.title('Season 3')
plt.axis('off')
plt.subplot(1, 6, 5)
plt.imshow(ts_imgs[3])
plt.title('Season 4')
plt.axis('off')

plt.subplot(1, 6, 6)
plt.imshow(label)
plt.title('Label')
plt.axis('off')
```

--------------------------------

### Pretraining ViT on FMoW-Sentinel with SatMAE++

Source: https://github.com/zhu-xlab/copernicus-fm/blob/main/Copernicus-Bench/src/foundation_models/SatMAE/README.md

Command to pretrain a ViT model using the SatMAE++ approach on the FMoW-Sentinel dataset. Ensure correct paths and adjust hyperparameters as needed.

```bash
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 python -m torch.distributed.launch --nproc_per_node=8 --master_port=29201 main_pretrain.py \
--batch_size 16 --accum_iter 16 \
--epochs 50 --warmup_epochs 20 \
--input_size 96 --patch_size 8 \
--mask_ratio 0.75 \
--model_type group_c \
--dropped_bands 0 9 10 \
--dataset_type sentinel --dropped_bands 0 9 10 \
--grouped_bands 0 1 2 6 --grouped_bands 3 4 5 7 --grouped_bands 8 9 \
--blr 0.0001 --num_workers 16 \
--train_path /home/fmow-sentinel/train.csv \
--output_dir ./output_dir \
--log_dir ./output_dir
```

--------------------------------

### Download Pretrained CROMA Weights

Source: https://github.com/zhu-xlab/copernicus-fm/blob/main/Copernicus-Bench/src/foundation_models/CROMA/README.md

Download the base and large pretrained weights for the CROMA model using wget.

```bash
wget https://huggingface.co/antofuller/CROMA/resolve/main/CROMA_base.pt
wget https://huggingface.co/antofuller/CROMA/resolve/main/CROMA_large.pt
```

--------------------------------

### Visualize NO2 Air Quality Data

Source: https://github.com/zhu-xlab/copernicus-fm/blob/main/Copernicus-Bench/demo_load_dataset.ipynb

Visualizes a single example from the NO2 dataset, displaying the annual image, seasonal images, and the ground truth label. Requires a normalization function to be defined.

```python
# visualize one example
# no2
ID = 1

img = dataset_train_no2_annual[ID]['image']
img = img.permute(1, 2, 0).numpy()
img = normalize(img)

ts_imgs = dataset_train_no2_seasonal[ID]['image']
for i in range(len(ts_imgs)):
    ts_imgs[i] = ts_imgs[i].permute(1, 2, 0).numpy()
    ts_imgs[i] = normalize(ts_imgs[i])

label = dataset_train_no2_annual[ID]['groundtruth']

plt.figure(figsize=(18, 5))

plt.subplot(1, 6, 1)
plt.imshow(img)
plt.title('Annual')
plt.axis('off')
plt.subplot(1, 6, 2)
plt.imshow(ts_imgs[0])
plt.title('Season 1')
plt.axis('off')
plt.subplot(1, 6, 3)
plt.imshow(ts_imgs[1])
plt.title('Season 2')
plt.axis('off')
plt.subplot(1, 6, 4)
plt.imshow(ts_imgs[2])
plt.title('Season 3')
plt.axis('off')
plt.subplot(1, 6, 5)
plt.imshow(ts_imgs[3])
plt.title('Season 4')
plt.axis('off')

plt.subplot(1, 6, 6)
plt.imshow(label)
plt.title('Label')
plt.axis('off')

# save pdf
#plt.savefig('./assets/vis_aqno2_s5p.pdf', bbox_inches='tight')
```

--------------------------------

### Load EuroSAT Sentinel-1 and Sentinel-2 Datasets

Source: https://github.com/zhu-xlab/copernicus-fm/blob/main/Copernicus-Bench/demo_load_dataset.ipynb

Initializes CoBenchEuroSATS1 and CoBenchEuroSATS2 datasets for Sentinel-1 and Sentinel-2 data respectively. Specify the root directory, desired bands, and split.

```python
from src.datasets.cobench_eurosats1_wrapper import CoBenchEuroSATS1
from src.datasets.cobench_eurosats2_wrapper import CoBenchEuroSATS2

root_dir_s1 = './data/copernicusbench/eurosat_s1sar/eurosat_s1'
bands_s1 = ['VV', 'VH']
dataset_train_s1 = CoBenchEuroSATS1(
            root=root_dir_s1, split="train", bands=bands_s1, transforms=None
        )

root_dir_s2 = './data/copernicusbench/eurosat_s2ms/eurosat_s2'
bands_s2 = ['B04', 'B03', 'B02']
dataset_train_s2 = CoBenchEuroSATS2(
            root=root_dir_s2, split="train", bands=bands_s2, transforms=None
        )
dataset_val_s2 = CoBenchEuroSATS2(
            root=root_dir_s2, split="val", bands=bands_s2, transforms=None
        )
dataset_test_s2 = CoBenchEuroSATS2(
            root=root_dir_s2, split="test", bands=bands_s2, transforms=None
        )
```

--------------------------------

### Download Copernicus Data Shards

Source: https://github.com/zhu-xlab/copernicus-fm/blob/main/Copernicus-Pretrain/data_loading/demo_dataloading.ipynb

Downloads a subset of Copernicus pretraining data shards using wget. Ensure the target directory exists before running.

```bash
# download the 100-grid subset
!mkdir -p ../data/example_100_webdataset/
!wget https://huggingface.co/datasets/wangyi111/Copernicus-Pretrain/resolve/main/example_100_grids/example_100_webdataset/example-{000000..000009}.tar -P ../data/example_100_webdataset/
```

--------------------------------

### Load Embedding Map (NumPy)

Source: https://github.com/zhu-xlab/copernicus-fm/blob/main/Copernicus-Embed-025deg/README.md

Example demonstrating how to load the aggregated embedding map using NumPy. This map can be used to enrich static variables in climate models or as input for weather/climate foundation models.

```python
import numpy as np

# download the embedding map
embed_map = np.load('embed_map_310k.npz')
# or use torch.load('embed_map_310k.pt') for PyTorch tensors
```

--------------------------------

### Download Pre-trained Copernicus-FM Weights

Source: https://github.com/zhu-xlab/copernicus-fm/blob/main/Copernicus-FM/demo.ipynb

Downloads the pre-trained weights for the CopernicusFM_ViT_base_varlang_e100 model from Hugging Face. Ensure the './weights' directory exists.

```bash
# download weights
!wget https://huggingface.co/wangyi111/Copernicus-FM/resolve/main/CopernicusFM_ViT_base_varlang_e100.pth -P ./weights
```

--------------------------------

### Initialize BigEarthNet-S1/S2 Datasets

Source: https://github.com/zhu-xlab/copernicus-fm/blob/main/Copernicus-Bench/demo_load_dataset.ipynb

Initializes datasets for Sentinel-1 and Sentinel-2 data from the BigEarthNet-S1/S2 dataset. Specify the root directory, split, modality, and desired bands.

```python
from src.datasets.cobench_bigearthnets12_wrapper import CoBenchBigEarthNetS12

root_dir = './data/copernicusbench/bigearthnetv2_s1s2/bigearthnet_s1s2'
bands_s1 = ['VV', 'VH']
bands_s2 = ['B04', 'B03', 'B02']

dataset_train_s1 = CoBenchBigEarthNetS12(
            root=root_dir,
            split="train",
            modality='s1',
            bands=bands_s1,
            transforms=None
        )

dataset_train_s2 = CoBenchBigEarthNetS12(
            root=root_dir,
            split="train",
            modality='s2',
            bands=bands_s2,
            transforms=None
        )

dataset_val_s2 = CoBenchBigEarthNetS12(
            root=root_dir,
            split="val",
            modality='s2',
            bands=bands_s2,
            transforms=None
        )

dataset_test_s2 = CoBenchBigEarthNetS12(
            root=root_dir,
            split="test",
            modality='s2',
            bands=bands_s2,
            transforms=None
        )
```

--------------------------------

### Load Copernicus-FM Model and Weights in Python

Source: https://github.com/zhu-xlab/copernicus-fm/blob/main/Copernicus-FM/README.md

Instantiate the ViT-base model and load the downloaded pretrained weights. The `strict=False` argument allows for loading weights even if some keys do not match.

```python
import torch
from src.model_vit import vit_base_patch16

# create model
model = vit_base_patch16(num_classes=10, global_pool=False)

# load pre-trained weights
path = './weights/CopernicusFM_ViT_base_varlang_e100.pth'
check_point = torch.load(path)
if 'model' in check_point:
    state_dict = check_point['model']
else:
    state_dict = check_point
msg = model.load_state_dict(state_dict, strict=False)
print(msg)

# encode an image

```

--------------------------------

### Finetune ViT Model with SatMAE

Source: https://github.com/zhu-xlab/copernicus-fm/blob/main/Copernicus-Bench/src/foundation_models/SatMAE/README.md

Use this command to finetune the ViT model. Adjust parameters like batch size, epochs, learning rate, and dataset paths as needed. Ensure distributed training is configured correctly.

```bash
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 python -m torch.distributed.launch --nproc_per_node=8 --master_port=29202 main_finetune.py \
--batch_size 8 --accum_iter 16 \
--epochs 50 --warmup_epochs 5 \
--input_size 224 --patch_size 16 \
--model_type vanilla \
--dataset_type rgb \
--weight_decay 0.05 --drop_path 0.2 --reprob 0.25 --mixup 0.8 --cutmix 1.0 \
--lr 0.001 --num_workers 16 \
--train_path /home/fmow-rgb/train_62classes.json \
--test_path /home/fmow-rgb/val_62classes.json \
--output_dir ./finetune_dir \
--log_dir ./finetune_dir \
--finetune ./output_dir/checkpoint-799.pth
```

--------------------------------

### Run Multiple Experiments with Scripts

Source: https://github.com/zhu-xlab/copernicus-fm/blob/main/Copernicus-Bench/README.md

Utilizes provided scripts to run multiple experiments, including frozen encoder evaluations and supervised training from scratch. These scripts automate the process for different models and datasets.

```bash
python scripts/exp_config_copernicusfm.py # frozen encoder evaluation of Copernicus-FM on the whole benchmark
python scripts/exp_config_dofa.py # frozen encoder evaluation of DOFA on the whole benchmark
python scripts/exp_config_vit_sup.py # supervised training from scratch for each dataset in the benchmark
...
```

--------------------------------

### Load Cloud-S2 Dataset

Source: https://github.com/zhu-xlab/copernicus-fm/blob/main/Copernicus-Bench/demo_load_dataset.ipynb

Initializes the CoBenchCloudS2 dataset for training, validation, and testing. Specify the root directory and desired bands.

```python
## Cloud-S2
from src.datasets.cobench_clouds2_wrapper import CoBenchCloudS2

root_dir = './data/copernicusbench/cloud_s2/cloud_s2'
bands = ['B04', 'B03', 'B02']
dataset_train = CoBenchCloudS2(
            root=root_dir, split="train", bands=bands, transforms=None
        )
dataset_val = CoBenchCloudS2(
            root=root_dir, split="val", bands=bands, transforms=None
        )
dataset_test = CoBenchCloudS2(
            root=root_dir, split="test", bands=bands, transforms=None
        )
```

--------------------------------

### Copernicus-Pretrain Dataset Structure

Source: https://github.com/zhu-xlab/copernicus-fm/blob/main/Copernicus-Pretrain/README.md

This bash snippet illustrates the directory structure of the Copernicus-Pretrain dataset, showing how data from different Sentinel missions and DEM are organized.

```bash
Copernicus-Pretrain # geotiff
├── Sentinel-1
│	├── images
│	│	├── gridId_lon_lat
│	│	│	├── localId_lon_lat
│	│	│	│	├── s1_fname_1.tif
│	│	│	│	├── s1_fname_2.tif
│	│	│	│	├── ...
│	│	│	│	├── s1_fname_4.tif
│	│	│	├── ...
│	├── ...
├── Sentinel-2
|   ├── ... # mostly paired with S1
├── Sentinel-3
|   ├── images
│	│	├── gridId_lon_lat
│	│	│	├── s3_fname_1.tif
│	│	│	├── ...
│	│	│	├── s3_fname_8.tif
│	 |   ├── ...
├── Sentinel-5P
|   ├── images
│	│	├── CO_column_number_density
│	│	│	├── gridId_lon_lat
│	│	│	│	├── s5p_co_fname_1.tif
│	│	│	│	├── ...
|   │   |   |   ├── s5p_co_fname_12.tif
│	│	│	├── ...
│	│	├── NO2_column_number_density
|   │   |   ├── ... # similar to CO
│	│	├── O3_column_number_density
│	│   |   ├── ... # similar to CO
│	│   ├── SO2_column_number_density
│	│   |   ├── ... # similar to CO
├── DEM
|   ├── images
│	│	├── gridId_lon_lat
│	│	│	├── dem.tif
│	│   ├── ...
├── fnames_all_1m.json.gz # filenames for all 1M ERA5 grids (most is empty)
├── fnames_union_310k.json.gz # filenames for 310K grids with at least one modality
├── fnames_joint_220k.json.gz # filenames for 220K grids with all modalities

```

--------------------------------

### Load So2Sat S1/S2 Datasets

Source: https://github.com/zhu-xlab/copernicus-fm/blob/main/Copernicus-Bench/demo_load_dataset.ipynb

Initializes CoBenchLCZS12 datasets for Sentinel-1 and Sentinel-2 imagery. Specify the root directory, split, modality, and desired bands.

```python
from src.datasets.cobench_lczs12_wrapper import CoBenchLCZS12

bands_s1 = ['S1_B6', 'S1_B5']
bands = ['S2_B04', 'S2_B03', 'S2_B02']

root_dir = './data/copernicusbench/so2sat_s1s2'

dataset_train_s1 = CoBenchLCZS12(
            root=root_dir, split="train", modality='s1', bands=bands_s1, transforms=None
        )

dataset_train = CoBenchLCZS12(
            root=root_dir, split="train", modality='s2', bands=bands, transforms=None
        )

dataset_val = CoBenchLCZS12(
            root=root_dir, split="val", modality='s2', bands=bands, transforms=None
        )

dataset_test = CoBenchLCZS12(
            root=root_dir, split="test", modality='s2', bands=bands, transforms=None
        )
```

--------------------------------

### Load Air Quality S5P Datasets

Source: https://github.com/zhu-xlab/copernicus-fm/blob/main/Copernicus-Bench/demo_load_dataset.ipynb

Initializes CoBenchAirQualityS5P dataset objects for training, validation, and testing. Supports different modalities (NO2, O3) and modes (annual, seasonal).

```python
from src.datasets.cobench_airqualitys5p_wrapper import CoBenchAirQualityS5P

root_dir = './data/copernicusbench/airquality_s5p/airquality_s5p'


dataset_train_no2_annual = CoBenchAirQualityS5P(
            root=root_dir, split="train", modality='no2', mode='annual', transforms=None
        )

dataset_train_no2_seasonal = CoBenchAirQualityS5P(
            root=root_dir, split="train", modality='no2', mode='seasonal', transforms=None
        )

dataset_train_o3_annual = CoBenchAirQualityS5P(
            root=root_dir, split="train", modality='o3', mode='annual', transforms=None
        )

dataset_train_o3_seasonal = CoBenchAirQualityS5P(
            root=root_dir, split="train", modality='o3', mode='seasonal', transforms=None
        )   

dataset_val_no2_annual = CoBenchAirQualityS5P(
            root=root_dir, split="val", modality='no2', mode='annual', transforms=None
        )

dataset_test_no2_annual = CoBenchAirQualityS5P(
            root=root_dir, split="test", modality='no2', mode='annual', transforms=None
        )
```

--------------------------------

### Download Copernicus-FM Pretrained Weights

Source: https://github.com/zhu-xlab/copernicus-fm/blob/main/Copernicus-FM/README.md

Download the main pretrained weights for the Copernicus-FM model. These weights are for the ViT-base variant trained for 100 epochs.

```bash
mkdir -p ./weights/
wget https://huggingface.co/wangyi111/Copernicus-FM/resolve/main/CopernicusFM_ViT_base_varlang_e100.pth -P ./weights/
```

--------------------------------

### Load Copernicus-Bench Flood-S1 Dataset

Source: https://github.com/zhu-xlab/copernicus-fm/blob/main/Copernicus-Bench/demo_load_dataset.ipynb

Initializes the CoBenchFloodS1 dataset for training, validation, and testing. Specify bands, root directory, and whether to use DEM or dual polarization.

```python
from src.datasets.cobench_floods1_wrapper import CoBenchFloodS1

bands = ['VV', 'VH']

root_dir = './data/copernicusbench/flood_s1/flood_s1'
dataset_train = CoBenchFloodS1(
            root=root_dir, split="train", bands=bands, transforms=None, use_dem=False, dual_pre=True
        )

dataset_val = CoBenchFloodS1(
            root=root_dir, split="val", bands=bands, transforms=None, use_dem=False, dual_pre=True
        )

dataset_test = CoBenchFloodS1(
            root=root_dir, split="test", bands=bands, transforms=None, use_dem=False, dual_pre=True
        )
```

--------------------------------

### Run Linear Probing Experiment

Source: https://github.com/zhu-xlab/copernicus-fm/blob/main/Copernicus-Bench/README.md

Executes a linear probing experiment on the EuroSAT-S2 dataset using Copernicus-FM weights. This command specifies output directory, model, dataset, learning rate, task, GPU usage, workers, epochs, warmup epochs, and seed.

```bash
python src/main.py \
output_dir=outputs/cobencheurosats2_copernicusfm_linear \
model=copernicusfm_cls \
dataset=cobench_eurosat_s2 \
lr=0.1 \
task=classification \
num_gpus=0 \
num_workers=8 \
epochs=50 \
warmup_epochs=0 \
seed=42 \

```

--------------------------------

### Download DOFA Pretrained Hypenetwork Weights

Source: https://github.com/zhu-xlab/copernicus-fm/blob/main/Copernicus-FM/README.md

Optionally download pretrained hypenetwork weights from DOFA. These weights are used for variable hypernetworks.

```bash
mkdir -p ./src/pretrained/
wget https://huggingface.co/earthflow/DOFA/resolve/main/weight_generator_1000_0.01_er50k.pt -P ./src/pretrained/
```
Tokens:
Raw