### Setup Project Environment

Source: https://github.com/nasaharvest/openmapflow/blob/main/openmapflow/notebooks/train.ipynb

Sets up the project environment by changing the directory to the OpenMapFlow configuration path, and installing project dependencies.

```python
from pathlib import Path
import os
openmapflow_yaml_path = input("Path to openmapflow.yaml: ")
%cd {Path(openmapflow_yaml_path).parent}

!pip install -r requirements.txt -q
!pip install pyyaml==5.4.1 -q
```

--------------------------------

### Install Github CLI

Source: https://github.com/nasaharvest/openmapflow/blob/main/openmapflow/notebooks/generate_project.ipynb

Installs the Github command-line interface on the system. Ensure you have the necessary permissions to run these commands.

```bash
# Install Github CLI
!curl -fsSL https://cli.github.com/packages/githubcli-archive-keyring.gpg | sudo dd of=/usr/share/keyrings/githubcli-archive-keyring.gpg
!sudo chmod go+r /usr/share/keyrings/githubcli-archive-keyring.gpg
!echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/githubcli-archive-keyring.gpg] https://cli.github.com/packages stable main" | sudo tee /etc/apt/sources.list.d/github-cli.list > /dev/null
!sudo apt -qq update
!sudo apt -qq install gh
!pip install "ipywidgets>=7,<8" -q # https://github.com/googlecolab/colabtools/issues/3020
```

--------------------------------

### Install and Generate OpenMapFlow Project

Source: https://github.com/nasaharvest/openmapflow/blob/main/README.md

Install the OpenMapFlow library and run the generation command. This will prompt for project configuration details.

```bash
pip install openmapflow
openmapflow generate
```

--------------------------------

### Download Example Inference Data

Source: https://github.com/nasaharvest/openmapflow/blob/main/openmapflow/notebooks/tutorial.ipynb

Downloads example GeoTIFF files from a Google Cloud Storage prefix using gsutil. These files are used as input for the inference process.

```bash
prefix = "gs://harvest-public-assets/openmapflow/Togo_2019_demo_2019-02-01_2020-02-01"
paths = [
  f"{prefix}/00000000000-0000000000.tif",
  f"{prefix}/00000000000-0000000256.tif",
  f"{prefix}/00000000256-0000000000.tif",
  f"{prefix}/00000000256-0000000256.tif"
]

for p in tqdm(paths):
  !gsutil -m cp {p} {tifs_dir}/{Path(p).name}
```

--------------------------------

### Install OpenMapFlow and DVC

Source: https://github.com/nasaharvest/openmapflow/blob/main/openmapflow/notebooks/new_data.ipynb

Installs the 'openmapflow' library with data dependencies and the 'dvc' (Data Version Control) tool with Google Cloud storage support. Also installs a specific version of 'pyyaml'. These are essential for data management and versioning.

```python
!pip install openmapflow[data]==0.2.2 dvc[gs] -q
!pip install pyyaml==5.4.1 -q
```

--------------------------------

### Clone Repository and Install Packages

Source: https://github.com/nasaharvest/openmapflow/blob/main/openmapflow/notebooks/generate_project.ipynb

Clones the newly created GitHub repository and installs the OpenMapFlow package with data dependencies and a specific PyYAML version.

```bash
# Clone Github repository
!git clone https://{username}:{token}@github.com/{username}/{project_name}
%cd {project_name}

!pip install openmapflow[data] -q
!pip install pyyaml==5.4.1 -q # Colab likes this version
```

--------------------------------

### Setup Github Credentials

Source: https://github.com/nasaharvest/openmapflow/blob/main/openmapflow/notebooks/train.ipynb

Sets up Github credentials using ipywidgets for interactive input. This includes the Github Token, Email, User, and URL.

```python
from ipywidgets import Password, Text, VBox
inputs = [
      Password(description="Github Token:"),
      Text(description='Github Email:'),
      Text(description='Github User:'),
      Text(description='Github URL:'),
]
VBox(inputs)
```

--------------------------------

### Download and Install GDAL

Source: https://github.com/nasaharvest/openmapflow/blob/main/openmapflow/notebooks/create_map.ipynb

Installs the Geospatial Data Abstraction Library (GDAL) and its development files using a PPA for specific versions. This is often required for geospatial processing.

```shell
GDAL_VERSION="3.6.4+dfsg-1~jammy0"
add-apt-repository -y ppa:ubuntugis/ubuntugis-unstable
apt-get -qq update
apt-get -qq install python3-gdal=$GDAL_VERSION gdal-bin=$GDAL_VERSION libgdal-dev=$GDAL_VERSION
# To see available versions: !apt-cache madison gdal-bin
```

--------------------------------

### Install ipywidgets

Source: https://github.com/nasaharvest/openmapflow/blob/main/openmapflow/notebooks/train.ipynb

Installs the ipywidgets library, which is required for interactive elements in the notebook.

```python
!pip install "ipywidgets>=7,<8" -q # https://github.com/googlecolab/colabtools/issues/3020
```

--------------------------------

### Install OpenMapFlow and Dependencies

Source: https://github.com/nasaharvest/openmapflow/blob/main/openmapflow/notebooks/create_map.ipynb

Installs necessary Python packages, including OpenMapFlow, ipywidgets, and cmocean. It also sets an environment variable for authentication.

```python
!pip install "ipywidgets>=7,<8" -q # https://github.com/googlecolab/colabtools/issues/3020
!pip install openmapflow[data]==0.2.4 -q
!pip install cmocean -q
%env USE_AUTH_EPHEM=0
```

--------------------------------

### Install OpenMapFlow and Dependencies

Source: https://github.com/nasaharvest/openmapflow/blob/main/openmapflow/notebooks/tutorial.ipynb

Installs the OpenMapFlow package with all optional dependencies, along with DVC for data version control and cmocean for colormaps.

```python
!pip install openmapflow[all] -q
!pip install dvc[gs] cmocean -q
```

--------------------------------

### Select Positive Example Label

Source: https://github.com/nasaharvest/openmapflow/blob/main/openmapflow/notebooks/tutorial.ipynb

Selects the first available positive example from the validation subset of the dataset. This example will be used to inspect its associated earth observation data.

```python
# Get a label with postive class
positive_example = df[(df[label_col] == 1.0) & (df[SUBSET] == "validation")].iloc[0]
positive_example
```

--------------------------------

### Download GDAL

Source: https://github.com/nasaharvest/openmapflow/blob/main/openmapflow/notebooks/tutorial.ipynb

Installs a specific version of GDAL and its related libraries using a PPA. This is a prerequisite for certain geospatial operations.

```shell
# Download GDAL
%%shell
GDAL_VERSION="3.6.4+dfsg-1~jammy0"
add-apt-repository -y ppa:ubuntugis/ubuntugis-unstable
apt-get -qq update
apt-get -qq install python3-gdal=$GDAL_VERSION gdal-bin=$GDAL_VERSION libgdal-dev=$GDAL_VERSION
```

--------------------------------

### Load and Access Labeled Dataset in Python

Source: https://github.com/nasaharvest/openmapflow/blob/main/README.md

Instantiate a `TogoCrop2019` dataset and load its DataFrame. This example shows how to access the Earth observation data (`eo_data`) and class probabilities (`class_prob`) for the first entry.

```python
from openmapflow.datasets import TogoCrop2019
df = TogoCrop2019().load_df(to_np=True)
x = df.iloc[0]["eo_data"]
y = df.iloc[0]["class_prob"]
```

--------------------------------

### Get Map Configuration and Handle Existing Maps

Source: https://github.com/nasaharvest/openmapflow/blob/main/openmapflow/notebooks/create_map.ipynb

Extracts configuration parameters like map key, bounding box, and dates from the inference widget. It also checks for existing map files in Google Cloud Storage and prompts for a new map key if conflicts are found.

```python
config = inference_widget.get_config_as_dict()
map_key = config["map_key"]
bbox = config["bbox"]
start_date = config["start_date"]
end_date = config["end_date"]
tifs_in_gcloud = config["tifs_in_gcloud"]


def get_map_files(map_key):
  blobs = storage.Client().list_blobs(bucket_or_name=BucketNames.PREDS_MERGED, prefix=f"{map_key}.tif")
  return [f"gs://{BucketNames.PREDS_MERGED}/{b.name}" for b in blobs]

existing_map_files = get_map_files(map_key)
while len(existing_map_files) > 0:
    print(f"Map for {map_key} already exists: \n{existing_map_files}")
    map_key += "_" + input(f"Append to map key: {map_key}_")
    existing_map_files = get_map_files(map_key)
```

--------------------------------

### Pull Latest Data with DVC

Source: https://github.com/nasaharvest/openmapflow/blob/main/README.md

Use this command to pull the latest data managed by DVC. Ensure DVC is installed and configured.

```bash
dvc pull
```

--------------------------------

### Plot NDVI for Positive and Negative Classes

Source: https://github.com/nasaharvest/openmapflow/blob/main/openmapflow/notebooks/tutorial.ipynb

Plots the NDVI for positive and negative example data over a one-year period. Requires pre-defined 'positive_example', 'negative_example', 'MONTHS', and 'EO_DATA' variables.

```python
fig, ax = plt.subplots(1,1, figsize=(10,5))
ax.set_title("NDVI")
plt.xticks(rotation=45)

positive_class_ndvi = positive_example[EO_DATA][:12, -1]
ax.plot(MONTHS, positive_class_ndvi, label="Positive class")

##########################################
negative_example = df[(df[label_col] == 0.0) & (df[SUBSET] == "validation")].iloc[0]
##########################################
negative_example_ndvi = negative_example[EO_DATA][:12, -1]
ax.plot(MONTHS, negative_example_ndvi, label="Negative class")

ax.legend()

gmap_url = "http://maps.google.com/maps?z=12&t=k&q=loc:"
print(f"Positive class: {gmap_url}{positive_example[LAT]}+{positive_example[LON]}")
print(f"Negative class: {gmap_url}{negative_example[LAT]}+{negative_example[LON]}")
```

--------------------------------

### Verify a dataset with openmapflow CLI

Source: https://context7.com/nasaharvest/openmapflow/llms.txt

Use the `openmapflow verify` command to validate a dataset's `load_labels()` output before starting Earth Engine export jobs. This checks for required columns, correct data types, valid ranges, and duplicate entries.

```bash
# Verify a single dataset
openmapflow verify KenyaCropland2021

# Expected output (all checks pass):
# ✔ load_labels() returns a DataFrame
# ✔ lat column found
# ✔ lat has no NaNs
# ✔ lat is float64
# ✔ lat is between -90 and 90
# ✔ lon column found
# ✔ lon has no NaNs
# ✔ lon is float64
# ✔ lon is between -180 and 180
# ✔ subset column found
# ✔ subset has no NaNs
# ✔ subset values are in ['training', 'validation', 'testing']
# ✔ start_date column found
# ✔ end_date column found
# ✔ start_date is before end_date
# ✔ Start dates are after 2015-07-01
# ✔ End dates are before 2024-10-01
# ✔ No duplicates
```

--------------------------------

### Initialize GitHub Credentials

Source: https://github.com/nasaharvest/openmapflow/blob/main/openmapflow/notebooks/tutorial.ipynb

Sets up input fields for GitHub token, email, and username using ipywidgets. These are necessary for cloning the repository.

```python
from ipywidgets import HTML, Password, Text, Textarea, VBox
inputs = [
      Password(description="Github Token:"),
      Text(description='Github Email:'),
      Text(description='Github User:'),
]
VBox(inputs)
```

--------------------------------

### Create Datasets via CLI

Source: https://context7.com/nasaharvest/openmapflow/llms.txt

Use the `openmapflow create-datasets` command-line interface to build datasets. Options include using the high-volume Earth Engine API for faster parallel fetching (`--ee_api`) and running in non-interactive mode (`--non-interactive`) for CI/CD pipelines. Remember to commit and push changes with DVC and Git.

```bash
# Standard dataset creation using EE Tasks (default)
openmapflow create-datasets

# Use EE high-volume API for faster parallel fetching (requires dask)
openmapflow create-datasets --ee_api --npartitions 8

# Non-interactive mode (for CI/CD)
openmapflow create-datasets --non-interactive

# After creation, commit data with DVC
dvc commit && dvc push
git add data/datasets.dvc data/report.txt
git commit -m "Add KenyaCropland2021 dataset"
git push
```

--------------------------------

### Create Datasets with OpenMapFlow CLI

Source: https://github.com/nasaharvest/openmapflow/blob/main/README.md

Initiate or check the progress of dataset creation using the OpenMapFlow command-line interface. This process prepares the satellite data based on your labeled datasets.

```bash
openmapflow create-datasets # Initiates or checks progress of dataset creation
```

--------------------------------

### Create Configuration Input Widget

Source: https://github.com/nasaharvest/openmapflow/blob/main/openmapflow/notebooks/create_map.ipynb

Initializes an interactive text area widget for users to input their OpenMapFlow configuration. This widget is then displayed for user interaction.

```python
import ipywidgets as widgets
config_yml_input = widgets.Textarea(placeholder="Your openmapflow.yaml", layout=widgets.Layout(height="10em", width="50%"))
config_yml_input
```

--------------------------------

### Configure Git and Clone Repository

Source: https://github.com/nasaharvest/openmapflow/blob/main/openmapflow/notebooks/train.ipynb

Configures Git with user email and name, then clones the repository using the provided Github credentials. Ensure your Github token has the necessary permissions.

```python
token = inputs[0].value
email = inputs[1].value
username = inputs[2].value
github_url = inputs[3].value

!git config --global user.email $username
!git config --global user.name $email
!git clone {github_url.replace("https://", f"https://{username}:{token}@")}
```

--------------------------------

### Inspect Earth Observation Data Shape

Source: https://github.com/nasaharvest/openmapflow/blob/main/openmapflow/notebooks/tutorial.ipynb

Displays the shape of the earth observation data associated with the selected positive example. This shows the dimensions of the data array.

```python
# Load earth observation data for label
positive_example[EO_DATA].shape
```

--------------------------------

### openmapflow create-datasets CLI

Source: https://context7.com/nasaharvest/openmapflow/llms.txt

Command-line interface command to build datasets using the `create_dataset()` pipeline.

```APIDOC
## `openmapflow create-datasets` CLI — Build datasets from the command line

Runs the `create_dataset()` pipeline for all datasets defined in `datasets.py`, with options for using the Earth Engine API (parallel fetching) or the default task-based export.

```bash
# Standard dataset creation using EE Tasks (default)
openmapflow create-datasets

# Use EE high-volume API for faster parallel fetching (requires dask)
openmapflow create-datasets --ee_api --npartitions 8

# Non-interactive mode (for CI/CD)
openmapflow create-datasets --non-interactive

# After creation, commit data with DVC
dvc commit && dvc push
git add data/datasets.dvc data/report.txt
git commit -m "Add KenyaCropland2021 dataset"
git push
```
```

--------------------------------

### Initialize and Display Inference Widget

Source: https://github.com/nasaharvest/openmapflow/blob/main/openmapflow/notebooks/create_map.ipynb

Creates an instance of the InferenceWidget using the fetched available models and bounding boxes, then displays the widget's user interface.

```python
inference_widget = InferenceWidget(available_models=available_models, available_bboxes=available_bboxes)
inference_widget.ui()
```

--------------------------------

### Configure Git and Login to GitHub

Source: https://github.com/nasaharvest/openmapflow/blob/main/openmapflow/notebooks/generate_project.ipynb

Configures Git global settings for user email and name, then logs into GitHub using the provided token and disables interactive prompts.

```bash
# Login to Git
token = inputs[0].value
email = inputs[1].value
username = inputs[2].value

!git config --global user.email $username
!git config --global user.name $email

# Login to Github
!gh config set prompt disabled
!gh auth login --git-protocol https
```

--------------------------------

### Create Dataset Directory and Upload Files

Source: https://github.com/nasaharvest/openmapflow/blob/main/openmapflow/notebooks/new_data.ipynb

Prompts the user for a dataset name, creates a corresponding directory under 'RAW_LABELS' if it doesn't exist, and then allows the user to upload files into this directory. It includes a check to prevent overwriting existing datasets.

```python
dataset_name = input("Dataset name (suggested format: <Country_Region_Year>): ")
while True:
    dataset_dir = PROJECT_ROOT / DataPaths.RAW_LABELS / dataset_name
    if dataset_dir.exists() and len(list(dataset_dir.iterdir())) > 0:
        dataset_name = input("Dataset name already exists, try a different name: ")
    else:
        dataset_dir.mkdir(exist_ok=True)
        break

print("--------------------------------------------------")
print(f"Dataset: {dataset_name} directory created")
print("---------------------------------------------------")
uploaded = files.upload()
for file_name in uploaded.keys():
    Path(file_name).rename(dataset_dir / file_name)
```

--------------------------------

### Extract Time-Series Arrays with get_x_y()

Source: https://context7.com/nasaharvest/openmapflow/llms.txt

Extract windowed time-series arrays (X) and corresponding labels (y) from a DataFrame for model training. Allows specifying the start month and the number of input months, slicing the Earth Observation data. Returns parallel lists of numpy arrays and float labels.

```python
import pandas as pd
from openmapflow.datasets import TogoCrop2019
from openmapflow.train_utils import get_x_y
from openmapflow.constants import SUBSET, CLASS_PROB

df = TogoCrop2019().load_df(to_np=True)
train_df = df[df[SUBSET] == "training"]

# Extract 12 months of data starting from February
x_train, y_train = get_x_y(
    df=train_df,
    label_col=CLASS_PROB,
    start_month="February",   # starting month index into 24-month time series
    input_months=12,          # number of months to include
)

print(len(x_train))          # e.g. 800
print(x_train[0].shape)      # (12, 18) — 12 timesteps × 18 bands (incl. NDVI)
print(y_train[0])            # 1.0
```

--------------------------------

### Clone Repository and Configure Git

Source: https://github.com/nasaharvest/openmapflow/blob/main/openmapflow/notebooks/tutorial.ipynb

Clones the OpenMapFlow repository using provided GitHub credentials and URL. It also configures Git user email and name globally. Ensure all input values are non-empty before proceeding.

```python
from pathlib import Path

github_url = github_url_input.value
project_name = "crop-mask-example" # maize-example
country_name = "Togo" # Kenya

for input_value in [token, email, username, github_url]:
  if input_value.strip() == "":
    raise ValueError("Found input with blank value.")

path_to_project = f"{Path(github_url).stem}/{project_name}"

!git config --global user.email $username
!git config --global user.name $email
!git clone {github_url.replace("https://", f"https://{username}:{token}@")}

%cd {path_to_project}
```

--------------------------------

### Download and Commit Datasets

Source: https://github.com/nasaharvest/openmapflow/blob/main/README.md

Use openmapflow to download the defined datasets and then commit them using DVC. Finally, add, commit, and push changes to your Git repository.

```bash
openmapflow create-datasets # Download datasets
dvc commit && dvc push      # Push data to version control

git add .
git commit -m'Created new dataset'
git push
```

--------------------------------

### Create Datasets

Source: https://github.com/nasaharvest/openmapflow/blob/main/openmapflow/notebooks/new_data.ipynb

Executes the 'openmapflow create-datasets' command. This command generates datasets from labels and Earth observation data, checking for existing data in Cloud Storage and active Earth Engine tasks.

```python
!openmapflow create-datasets
```

--------------------------------

### Import OpenMapFlow and Google Cloud Libraries

Source: https://github.com/nasaharvest/openmapflow/blob/main/openmapflow/notebooks/create_map.ipynb

Imports essential libraries for Earth Engine, Google Cloud Storage, rasterio, matplotlib, and OpenMapFlow utilities. It also suppresses future warnings.

```python
import ee
import google
import os
import cmocean
import rasterio as rio
import matplotlib.pyplot as plt
import warnings

from google.colab import auth
from google.cloud import storage
from pathlib import Path
from openmapflow.ee_exporter import EarthEngineExporter
from openmapflow.config import GCLOUD_PROJECT_ID, PROJECT, BucketNames
from openmapflow.utils import confirmation
from openmapflow.inference_widgets import InferenceWidget
from openmapflow.inference_utils import (
    get_status,
    find_missing_predictions,
    make_new_predictions,
    build_vrt,
    get_available_bboxes,
    get_available_models
)
warnings.simplefilter(action='ignore', category=FutureWarning)
print(PROJECT)
os.environ["GCLOUD_PROJECT"] = GCLOUD_PROJECT_ID
```

--------------------------------

### Create Google Cloud Storage Bucket

Source: https://github.com/nasaharvest/openmapflow/blob/main/README.md

Authenticate with gcloud and create a Google Cloud Storage bucket for labeled earth observation files. Replace placeholders with your specific configuration.

```bash
gcloud auth login
gsutil mb -l <YOUR_OPENMAPFLOW_YAML_GCLOUD_LOCATION> gs://<YOUR_OPENMAPFLOW_YAML_BUCKET_LABELED_EO>
```

--------------------------------

### Create Google Cloud Storage Bucket

Source: https://github.com/nasaharvest/openmapflow/blob/main/openmapflow/notebooks/generate_project.ipynb

Creates a Google Cloud Storage bucket for labeled Earth observation data if it does not already exist. Requires billing to be enabled.

```python
# Create bucket for labeled earth observation data (if does not exist)
client = storage.Client() 
bucket = client.bucket(BucketNames.LABELED_EO)
if bucket.exists():
  print(f"Bucket: {BucketNames.LABELED_EO} already exists")
else:
  bucket.location = GCLOUD_LOCATION
  print(f"Creating bucket: {BucketNames.LABELED_EO}")
  client.create_bucket(bucket, project=GCLOUD_PROJECT_ID)
```

--------------------------------

### Create GitHub Repository

Source: https://github.com/nasaharvest/openmapflow/blob/main/openmapflow/notebooks/generate_project.ipynb

Prompts the user for a project name and creates a new private GitHub repository using the GitHub CLI.

```bash
# Create Github repository
project_name = input("Project name: ")
assert project_name.strip() != ""
!gh repo create $project_name --private
```

--------------------------------

### Import Google Cloud Storage Libraries

Source: https://github.com/nasaharvest/openmapflow/blob/main/openmapflow/notebooks/generate_project.ipynb

Imports necessary libraries for interacting with Google Cloud Storage and OpenMapFlow configuration.

```python
import os
import google
from google.cloud import storage
from openmapflow.config import GCLOUD_LOCATION, BucketNames, GCLOUD_PROJECT_ID
```

--------------------------------

### List Datasets

Source: https://github.com/nasaharvest/openmapflow/blob/main/openmapflow/notebooks/train.ipynb

Lists the available datasets using the OpenMapFlow CLI. This is useful for verifying data availability after pulling.

```bash
!openmapflow datasets
```

--------------------------------

### Verify Custom Dataset with OpenMapFlow CLI

Source: https://github.com/nasaharvest/openmapflow/blob/main/README.md

Run this command to verify your custom dataset's `load_labels` function. This helps ensure the data is loaded and formatted correctly before proceeding with dataset creation.

```bash
openmapflow verify TogoCrop2019
```

--------------------------------

### Scaffold a new OpenMapFlow project

Source: https://context7.com/nasaharvest/openmapflow/llms.txt

Use the `openmapflow generate` command to create a new project structure. This command interactively prompts for project details and sets up configuration files, DVC, and GitHub Actions workflows.

```bash
pip install openmapflow

# Inside your GitHub repository directory:
openmapflow generate

# Interactive prompts:
#   Project name [my-repo]:  crop-mask-kenya
#   Description [OpenMapFlow crop-mask-kenya]:
#   GCloud project ID:  my-gcp-project-123
#   GCloud location [us-central1]:
#   GCloud bucket_labeled_eo [crop-mask-kenya-labeled-eo]:
#   GCloud bucket_inference_eo [crop-mask-kenya-inference-eo]:
#   GCloud bucket_preds [crop-mask-kenya-preds]:
#   GCloud bucket_preds_merged [crop-mask-kenya-preds-merged]:

# Resulting structure:
# crop-mask-kenya/
# └── openmapflow.yaml       # project config
# └── datasets.py            # define LabeledDataset subclasses here
# └── train.py               # training script
# └── evaluate.py            # evaluation script
# └── requirements.txt
# └── .dvc/                  # DVC data versioning
# └── .github/workflows/
#     └── crop-mask-kenya-deploy.yaml
#     └── crop-mask-kenya-test.yaml
```

--------------------------------

### Write Configuration to File

Source: https://github.com/nasaharvest/openmapflow/blob/main/openmapflow/notebooks/create_map.ipynb

Writes the content from the configuration input widget to a file named 'openmapflow.yaml'. This action saves the user's configuration.

```python
with open('openmapflow.yaml', 'w') as f:
  f.write(config_yml_input.value)
```

--------------------------------

### Run OpenMapFlow CLI

Source: https://github.com/nasaharvest/openmapflow/blob/main/openmapflow/notebooks/tutorial.ipynb

Executes the OpenMapFlow command-line interface. This is a general command to interact with the tool.

```shell
# CLI
!openmapflow
```

--------------------------------

### Train and Evaluate Model with OpenMapFlow

Source: https://github.com/nasaharvest/openmapflow/blob/main/README.md

This script sequence pulls the latest data, trains a model using a specified name, records test metrics, and commits/pushes the model artifacts. Ensure you set your desired MODEL_NAME.

```bash
# Pull in latest data
dvc pull

# Set model name, train model, record test metrics
export MODEL_NAME=<YOUR MODEL NAME>              
python train.py --model_name $MODEL_NAME    
python evaluate.py --model_name $MODEL_NAME 

# Push new models to data version control
dvc commit 
dvc push  

# Make a Pull Request to the repository
git checkout -b"$MODEL_NAME"
git add .
git commit -m "$MODEL_NAME"
git push --set-upstream origin "$MODEL_NAME"
```

--------------------------------

### Import necessary libraries

Source: https://github.com/nasaharvest/openmapflow/blob/main/openmapflow/notebooks/new_data.ipynb

Imports essential libraries and utilities for the notebook, including file upload functionality, Google Earth Engine and Google Cloud login utilities, project configuration, data paths, and label reading functions.

```python
from google.colab import files
from openmapflow.utils import colab_gee_gcloud_login
from openmapflow.config import PROJECT_ROOT, DataPaths, GCLOUD_PROJECT_ID
from openmapflow.label_utils import read_zip
```

--------------------------------

### OpenMapFlow project configuration

Source: https://context7.com/nasaharvest/openmapflow/llms.txt

The `openmapflow.yaml` file centralizes project configuration, including Google Cloud resource names and data paths. It is automatically loaded by `openmapflow.config`.

```yaml
# openmapflow.yaml
version: 0.2.5rc1
project: crop-mask-kenya
description: OpenMapFlow crop-mask-kenya
data_paths:
  raw_labels: raw_labels
  models: models
  metrics: metrics.yaml
  datasets: datasets
  report: report.txt
gcloud:
  project_id: my-gcp-project-123
  location: us-central1
  bucket_labeled_eo: crop-mask-kenya-labeled-eo
  bucket_inference_eo: crop-mask-kenya-inference-eo
  bucket_preds: crop-mask-kenya-preds
  bucket_preds_merged: crop-mask-kenya-preds-merged
```

--------------------------------

### Generate OpenMapFlow Project

Source: https://github.com/nasaharvest/openmapflow/blob/main/openmapflow/notebooks/generate_project.ipynb

Generates the OpenMapFlow project structure within the cloned repository using the `openmapflow generate` command.

```bash
!openmapflow generate
```

--------------------------------

### Commit and Push Project to GitHub

Source: https://github.com/nasaharvest/openmapflow/blob/main/openmapflow/notebooks/generate_project.ipynb

Stages all changes, commits them with a message, renames the branch to 'main', and pushes the branch to the remote origin on GitHub.

```bash
!git add .
!git commit -m'openmapflow setup'
!git branch -M main
!git push -u origin main
```

--------------------------------

### Build Virtual Raster (VRT)

Source: https://github.com/nasaharvest/openmapflow/blob/main/openmapflow/notebooks/create_map.ipynb

Constructs a Virtual Raster (VRT) file from the downloaded prediction files. This step is necessary for efficiently handling and processing multiple prediction tiles.

```python
build_vrt(prefix)
```

--------------------------------

### Create Directories for Map Processing

Source: https://github.com/nasaharvest/openmapflow/blob/main/openmapflow/notebooks/create_map.ipynb

Creates necessary directories for storing predictions, VRTs, and TIF files based on the map key. Ensures that the directories exist before proceeding with file operations.

```python
prefix = map_key.replace("/", "_")
Path(f"{prefix}_preds").mkdir(exist_ok=True)
Path(f"{prefix}_vrts").mkdir(exist_ok=True)
Path(f"{prefix}_tifs").mkdir(exist_ok=True)
```

--------------------------------

### Configure GitHub Clone URL

Source: https://github.com/nasaharvest/openmapflow/blob/main/openmapflow/notebooks/tutorial.ipynb

Dynamically creates a GitHub clone URL based on user input for username and displays it in a textarea. This URL will be used to clone the OpenMapFlow repository.

```python
token = inputs[0].value
email = inputs[1].value
username = inputs[2].value

github_url_input = Textarea(value=f'https://github.com/{username}/openmapflow.git')
VBox([HTML(value="<b>Github Clone URL</b>"), github_url_input])
```

--------------------------------

### Define Datasets in Python

Source: https://github.com/nasaharvest/openmapflow/blob/main/README.md

Import and instantiate dataset classes from the openmapflow library to define the datasets to be used in your project. Ensure the necessary dataset classes are imported.

```python
from openmapflow.datasets import GeowikiLandcover2017, TogoCrop2019

datasets = [GeowikiLandcover2017(), TogoCrop2019()]
```

--------------------------------

### Build VRT and Translate GeoTIFFs

Source: https://github.com/nasaharvest/openmapflow/blob/main/openmapflow/notebooks/tutorial.ipynb

Creates a Virtual Raster (VRT) from multiple GeoTIFF files and then translates the VRT into a single GeoTIFF file with a specified SRS. This is often done to combine or reproject raster data.

```bash
!gdalbuildvrt {tifs_dir}.vrt {tifs_dir}/*.tif
!gdal_translate -a_srs EPSG:4326 -of GTiff {tifs_dir}.vrt {tifs_dir}.tif
```

--------------------------------

### View Dataset Report

Source: https://github.com/nasaharvest/openmapflow/blob/main/openmapflow/notebooks/tutorial.ipynb

Displays a report of the available datasets using the OpenMapFlow CLI. This command provides an overview of the data.

```shell
# See report of data already available
!openmapflow datasets
```

--------------------------------

### User Confirmation for Dataset Creation

Source: https://github.com/nasaharvest/openmapflow/blob/main/openmapflow/notebooks/new_data.ipynb

Asks the user to confirm that they have added a `LabeledDataset` object to 'datasets.py'. This step is mandatory before creating new datasets, as the `openmapflow create-datasets` command relies on this configuration.

```python
user_confirmation = input(
    "Open datasets.py and add a `LabeledDataset` object representing the labels just added.\n"+
    "Added `LabeledDataset y/[n]: ")
if user_confirmation.lower() != "y":
    print("New features can only be created when a `LabeledDataset` object is added.")
```

--------------------------------

### GitHub Authentication Widgets

Source: https://github.com/nasaharvest/openmapflow/blob/main/openmapflow/notebooks/generate_project.ipynb

Sets up input widgets for GitHub token, email, and username for authentication. The token is required for private repository access.

```python
from ipywidgets import Password, Text, VBox, Textarea, Layout
from google.colab import auth
inputs = [
      Password(description="Github Token:"),
      Text(description='Github Email:'),
      Text(description='Github User:'),
]
VBox(inputs)
```

--------------------------------

### Import and use satellite bands and normalization constants

Source: https://context7.com/nasaharvest/openmapflow/llms.txt

Import various satellite band definitions and normalization constants from `openmapflow.bands`. `BANDS_MAX` is used for per-band max normalization in model forward passes.

```python
from openmapflow.bands import (
    BANDS, BANDS_MAX, DYNAMIC_BANDS, STATIC_BANDS,
    S1_BANDS, S2_BANDS, ERA5_BANDS, SRTM_BANDS,
    DAYS_PER_TIMESTEP,
)

print(f"Dynamic bands ({len(DYNAMIC_BANDS)}): {DYNAMIC_BANDS}")
# ['VV', 'VH', 'B1', 'B2', ..., 'B12', 'temperature_2m', 'total_precipitation_sum']

print(f"Static bands  ({len(STATIC_BANDS)}):  {STATIC_BANDS}")
# ['elevation', 'slope']

print(f"Final model bands ({len(BANDS)}): {BANDS}")
# VV, VH, B2–B9, B8A, B11, B12, temperature_2m, total_precipitation, elevation, slope, NDVI

print(f"Timestep interval: {DAYS_PER_TIMESTEP} days")  # 30 days
```

```python
# Normalize a feature array in a model's forward pass
import torch
import numpy as np
x = torch.tensor(np.random.rand(64, 12, 18)).float()
norm = torch.tensor(BANDS_MAX)
x_normalized = x / norm  # per-band max normalization
print(x_normalized.max().item())  # <= 1.0
```

--------------------------------

### Set Working Directory

Source: https://github.com/nasaharvest/openmapflow/blob/main/openmapflow/notebooks/new_data.ipynb

Prompts the user for the path to the 'openmapflow.yaml' file and changes the current working directory to the parent directory of that file. This is crucial for subsequent operations that rely on the project's root.

```python
from pathlib import Path
path_to_yaml = input("Path to openmapflow.yaml: ")
%cd {Path(path_to_yaml).parent}
```

--------------------------------

### Train Model Command

Source: https://github.com/nasaharvest/openmapflow/blob/main/openmapflow/notebooks/tutorial.ipynb

Executes the training script 'train.py' using a system command. It passes the MODEL_NAME and epoch count as arguments.

```bash
!python train.py --model_name $MODEL_NAME --epoch 3
```

--------------------------------

### Display Map Preview

Source: https://github.com/nasaharvest/openmapflow/blob/main/openmapflow/notebooks/create_map.ipynb

Opens the generated GeoTIFF file and displays a preview of the prediction map using matplotlib. Note that maps over 5GB may not fit into RAM.

```python
# View map, maps over 5GB may not fit in RAM
predictions_map = rio.open(f"{prefix}_final.tif")
plt.figure(figsize=(10,10))
plt.imshow(predictions_map.read(1), cmap=cmap)
plt.title("Map Preview")
plt.axis("off");
```

--------------------------------

### Read and Display Dataset Head

Source: https://github.com/nasaharvest/openmapflow/blob/main/openmapflow/notebooks/new_data.ipynb

Reads the uploaded zip file into a pandas DataFrame and displays the first few rows. This is useful for quickly assessing the structure and content of the newly added data.

```python
# Assess dataset
df = read_zip(dataset_dir / file_name)
df.head()
```

--------------------------------

### Access OpenMapFlow configuration programmatically

Source: https://context7.com/nasaharvest/openmapflow/llms.txt

Import configuration variables and classes from `openmapflow.config` to access project settings, root path, data paths, and Google Cloud bucket names within your Python scripts.

```python
# Accessing config programmatically
from openmapflow.config import PROJECT, PROJECT_ROOT, GCLOUD_LOCATION, DataPaths, BucketNames

print(PROJECT)                      # "crop-mask-kenya"
print(str(PROJECT_ROOT))            # "/home/user/crop-mask-kenya"
print(DataPaths.DATASETS)           # "data/datasets"
print(DataPaths.MODELS)             # "data/models"
print(BucketNames.LABELED_EO)       # "crop-mask-kenya-labeled-eo"
print(BucketNames.INFERENCE_EO)     # "crop-mask-kenya-inference-eo"
```

--------------------------------

### Create Git Branch, Commit, and Push

Source: https://github.com/nasaharvest/openmapflow/blob/main/openmapflow/notebooks/tutorial.ipynb

Creates a new Git branch based on the MODEL_NAME, adds all changes, commits them with a message including the MODEL_NAME, and pushes the branch to the remote origin.

```bash
!git checkout -b"$MODEL_NAME"
!git add .
!git commit -m "$MODEL_NAME"
!git push --set-upstream origin "$MODEL_NAME"
```

--------------------------------

### Download Predictions from Cloud Storage

Source: https://github.com/nasaharvest/openmapflow/blob/main/openmapflow/notebooks/create_map.ipynb

Downloads prediction files (in .nc format) from Google Cloud Storage to a local directory. Uses `gsutil` for efficient copying and avoids re-downloading existing files.

```bash
print("Download predictions as nc files (may take several minutes)")
src = f"gs://{BucketNames.PREDS}/{map_key}*"
dest = f"{prefix}_preds"
!gsutil -m cp -n -r {src} {dest}
```

--------------------------------

### Define Custom Labeled Dataset with KenyaCropland2021

Source: https://context7.com/nasaharvest/openmapflow/llms.txt

Subclass LabeledDataset and implement load_labels() to define a custom dataset. Ensure standard column names, set an observation window, provide binary labels, and split data into train/val/test subsets. Use verify_df() to check label format before creating the dataset.

```python
from datetime import date
import pandas as pd
from openmapflow.labeled_dataset import LabeledDataset, create_datasets, verify_df
from openmapflow.label_utils import train_val_test_split
from openmapflow.constants import LAT, LON, START, END, SUBSET, CLASS_PROB

class KenyaCropland2021(LabeledDataset):
    def load_labels(self) -> pd.DataFrame:
        # Read raw labels from CSV in data/raw_labels/
        df = pd.read_csv("data/raw_labels/kenya_cropland_2021.csv")

        # Required: rename columns to standard lat/lon names
        df.rename(columns={"latitude": LAT, "longitude": LON}, inplace=True)

        # Required: set EO observation window (24-month window recommended)
        df[START] = date(2020, 1, 1)
        df[END]   = date(2021, 12, 31)

        # Required: binary label column (0.0 = non-crop, 1.0 = crop)
        df[CLASS_PROB] = df["is_crop"].astype(float)

        # Required: split into train/val/test
        df[SUBSET] = train_val_test_split(index=df.index, val=0.2, test=0.2)

        return df

# Verify that load_labels() output is valid before fetching EO data
from openmapflow.labeled_dataset import verify_df
dataset = KenyaCropland2021()
df = dataset.load_labels()
verify_df(df)
# ✔ lat column found
# ✔ lat has no NaNs
# ✔ lat is float64
# ✔ lat is between -90 and 90
# ✔ lon column found
# ...
# ✔ No duplicates

# Create the dataset: fetches EO data from Earth Engine and saves to CSV
print(dataset.create_dataset())
# KenyaCropland2021 (Timesteps: 24)
# ✔ training amount: 1600, positive class: 48.2%
# ✔ validation amount: 400, positive class: 47.5%
# ✔ testing amount: 400, positive class: 49.1%

# Create datasets for multiple datasets at once (CLI entry point)
datasets = [KenyaCropland2021()]
create_datasets(datasets)  # also accepts --ee_api, --non-interactive, --npartitions flags
```

--------------------------------

### Checkout Git Branch

Source: https://github.com/nasaharvest/openmapflow/blob/main/openmapflow/notebooks/new_data.ipynb

Prompts the user to choose between checking the progress of an existing dataset creation or creating a new dataset. Based on the choice, it either checks out an existing branch and pulls the latest changes or creates a new branch.

```python
choice = input("a) Checking progress of dataset creation OR \nb) Creating new dataset \na/b: ")
if choice == "a":
  branch_name = input("Existing branch name: ")
  !git checkout {branch_name}
  !git pull
elif choice == "b":
  branch_name = input("New branch name: ")
  !git checkout -b {branch_name}
else:
  print(f"Invalid choice: {choice}, must be 'a' or 'b'")
```

--------------------------------

### Generate Test Metrics

Source: https://github.com/nasaharvest/openmapflow/blob/main/openmapflow/notebooks/tutorial.ipynb

Runs the evaluation script 'evaluate.py' to generate test metrics for the trained model, using the specified MODEL_NAME.

```bash
# Generate test metrics
!python evaluate.py --model_name $MODEL_NAME
```

--------------------------------

### Import Earth Observation Data Libraries

Source: https://github.com/nasaharvest/openmapflow/blob/main/openmapflow/notebooks/tutorial.ipynb

Imports necessary libraries for working with earth observation data, including matplotlib for plotting, and constants and bands from openmapflow.

```python
import matplotlib.pyplot as plt
from openmapflow.constants import MONTHS, EO_DATA
from openmapflow.bands import BANDS
```

--------------------------------

### Set Colormap for Map Preview

Source: https://github.com/nasaharvest/openmapflow/blob/main/openmapflow/notebooks/create_map.ipynb

Selects a colormap based on the project name. 'Solar' for 'maize' projects, 'Speed' for 'crop' projects, and 'Thermal' for others. This colormap is used for visualizing the prediction map.

```python
if "maize" in PROJECT:
  cmap = cmocean.cm.solar
  cmap_name = "Solar"
elif "crop" in PROJECT:
  cmap = cmocean.cm.speed
  cmap_name = "Speed"
else:
  cmap = cmocean.cm.thermal
  cmap_name = "Thermal"
```

--------------------------------

### Train Model

Source: https://github.com/nasaharvest/openmapflow/blob/main/openmapflow/notebooks/train.ipynb

Trains the model using the `train.py` script. This command specifies the model name and the number of epochs for training.

```bash
!python train.py --model_name $MODEL_NAME --epoch 2
```

--------------------------------

### Run Inference on a Satellite TIF File

Source: https://context7.com/nasaharvest/openmapflow/llms.txt

Loads a trained TorchScript or sklearn model and performs batched inference on a satellite `.tif` file. Returns a DataFrame with per-pixel prediction probabilities, indexed by `(lat, lon)`. Optionally saves results as a NetCDF file. Handles model loading, device selection, and batching.

```python
import torch
from pathlib import Path
from openmapflow.inference import Inference

# Load a TorchScript model
model = torch.jit.load("data/models/Togo_crop-mask_2019.pt")
model.eval()

# Instantiate inference runner (normalizing_dict=None uses model's internal normalization)
runner = Inference(
    model=model,
    normalizing_dict=None,
    device=torch.device("cuda" if torch.cuda.is_available() else "cpu"),
    batch_size=64,
)

# Run inference on a local .tif file
preds_df = runner.run(
    local_path=Path("data/inference/togo_tile_001.tif"),
    dest_path=Path("data/preds/togo_tile_001.nc"),  # optional: save as NetCDF
)

print(preds_df.head())
#                    prediction_0
# lat       lon
# 6.1234    1.2345   0.8921
# 6.1234    1.2355   0.1203
# 6.1244    1.2345   0.9512
# ...

# Threshold predictions to binary map
binary_map = (preds_df["prediction_0"] > 0.5).astype(int)
crop_pixel_fraction = binary_map.mean()
print(f"Predicted crop fraction: {crop_pixel_fraction:.1%}")  # e.g. 43.7%
```

--------------------------------

### Commit and Push Data and Code Changes

Source: https://github.com/nasaharvest/openmapflow/blob/main/README.md

Commit changes to DVC and push them to remote storage. Then, add all project files, commit them to Git with a message, and push to the remote repository.

```bash
dvc commit && dvc push
git add .
git commit -m'Created new dataset'
git push
```

--------------------------------

### Create Directories for TIFs and Predictions

Source: https://github.com/nasaharvest/openmapflow/blob/main/openmapflow/notebooks/tutorial.ipynb

Creates 'tifs' and 'preds' directories if they do not already exist. These directories are used to store input GeoTIFF files and prediction results.

```python
from pathlib import Path
tifs_dir = Path(f"/content/tifs")
preds_dir = Path(f"/content/preds")
tifs_dir.mkdir(exist_ok=True)
preds_dir.mkdir(exist_ok=True)
```

--------------------------------

### Set Environment Variables for Secrets

Source: https://github.com/nasaharvest/openmapflow/blob/main/openmapflow/notebooks/generate_project.ipynb

Sets environment variables for the service account key and the repository path, preparing them to be pushed as GitHub secrets.

```python
os.environ["SA_KEY"] = sa_key_input.value
os.environ["REPO"] = f"{username}/{project_name}"
```

--------------------------------

### Authenticate Google Cloud and Earth Engine

Source: https://github.com/nasaharvest/openmapflow/blob/main/openmapflow/notebooks/create_map.ipynb

Authenticates the user with Google Cloud and initializes the Earth Engine API. This is a prerequisite for accessing Google Cloud resources and Earth Engine data.

```python
print("Logging into Google Cloud")
auth.authenticate_user()
print("Logging into Earth Engine")
SCOPES = [
    "https://www.googleapis.com/auth/cloud-platform",
    "https://www.googleapis.com/auth/earthengine",
]
CREDENTIALS, _ = google.auth.default(default_scopes=SCOPES)
ee.Initialize(CREDENTIALS, project=GCLOUD_PROJECT_ID)
```

--------------------------------

### Train and Save a TorchScript Transformer Model

Source: https://context7.com/nasaharvest/openmapflow/llms.txt

A bash script to train a Transformer model using tsai, BCE loss, and Adam optimizer. It pulls data from DVC, trains the model with specified hyperparameters, and saves the best checkpoint as a TorchScript file. It also includes commands to commit and push the model to a DVC remote.

```bash
# Pull latest data from DVC remote
dvc pull

# Train with default hyperparameters
export MODEL_NAME=Kenya_crop_mask_2021
python train.py \
    --model_name $MODEL_NAME \
    --start_month February \
    --input_months 12 \
    --epochs 10 \
    --batch_size 64 \
    --lr 0.001 \
    --upsample_minority_ratio 0.5

# Output:
# Upsampling: negative class from 400 to 800 using upsampling ratio: 0.5
# Epoch: 100%|████| 10/10 [02:14<00:00, loss=0.312]
# MODEL_NAME=Kenya_crop_mask_2021
# accuracy: 0.8912
# f1: 0.8834
# precision: 0.8901
# recall: 0.8768
# roc_auc: 0.9421

# Push model to DVC remote and open a PR to trigger deployment
dvc commit && dvc push
git checkout -b "$MODEL_NAME"
git add . && git commit -m "$MODEL_NAME"
git push --set-upstream origin "$MODEL_NAME"
```

--------------------------------

### Display disagreements from Set 2

Source: https://github.com/nasaharvest/openmapflow/blob/main/openmapflow/notebooks/compare_ceo_labels.ipynb

Selects and displays specific columns ('sampleid', 'email', 'flagged', 'collection_time', 'analysis_duration', 'imagery_title', and the label question) for samples where disagreements were found in the second CEO dataset.

```python
ceo_disagree_set2[['sampleid', 'email', 'flagged', 'collection_time', 
                   'analysis_duration', 'imagery_title', label_question]]
```

--------------------------------

### Authenticate for Earth Engine and Cloud Data

Source: https://github.com/nasaharvest/openmapflow/blob/main/README.md

Authenticate your environment for accessing Earth observation data. `earthengine authenticate` is for new data, while `gcloud auth login` is for cached data.

```bash
earthengine authenticate    # For getting new earth observation data
gcloud auth login           # For getting cached earth observation data
```

--------------------------------

### Import Inference Dependencies

Source: https://github.com/nasaharvest/openmapflow/blob/main/openmapflow/notebooks/tutorial.ipynb

Imports necessary libraries for model inference, including utilities for model paths, configuration, inference class, dynamic bands, and data handling.

```python
from openmapflow.train_utils import model_path_from_name
from openmapflow.config import PROJECT
from openmapflow.inference import Inference
from openmapflow.bands import DYNAMIC_BANDS
from tqdm.notebook import tqdm
from pathlib import Path
from datetime import date
import cmocean
import numpy as np
import rasterio as rio
import torch
```

--------------------------------

### Fetch Management API URL and Available Models

Source: https://github.com/nasaharvest/openmapflow/blob/main/openmapflow/notebooks/create_map.ipynb

Retrieves the URL for the Google Cloud Run management API and fetches a list of available models. This information is used to interact with the OpenMapFlow inference service.

```python
output = !gcloud run services list \
    --platform managed \
    --filter {PROJECT}-management-api \
    --limit 1 \
    --format='get(URL)' \
    --project {GCLOUD_PROJECT_ID}
models_url = f"{output[0]}/models"
available_models = get_available_models(models_url)
available_models
```

--------------------------------

### Push Model to Git

Source: https://github.com/nasaharvest/openmapflow/blob/main/openmapflow/notebooks/train.ipynb

Pushes the model changes to a new Git branch. This includes checking out a new branch, adding all changes, committing them with a message, and pushing to the origin.

```bash
# Push changes to github
!git checkout -b"$MODEL_NAME"
!git add .
!git commit -m "$MODEL_NAME"
!git push --set-upstream origin "$MODEL_NAME"
```