### Activate and Install Libraries in Virtual Environment

Source: https://github.com/csiro-easi/easi-notebooks/blob/main/html/notebooks/adding-python-libraries.html

Activates the virtual environment and allows you to install new Python libraries using pip. Replace 'myenv' with your environment's name. Example shown for installing 'geopy'.

```bash
source ~/venvs/${MYENV}/bin/activate
pip install geopy
```

--------------------------------

### Python Image Processing Setup with Numba

Source: https://github.com/csiro-easi/easi-notebooks/blob/main/notebooks/dask/01-Introduction_to_Dask.ipynb

Sets up initial Python functions for loading image data, applying a smoothing filter, and saving results. This code serves as a baseline before Numba optimization.

```python
import numpy as np
from tqdm.notebook import tqdm

def load_eo_data():
    return np.random.random((1000, 1000))

def smooth(x):
    out = np.empty_like(x)
    for i in range(1, x.shape[0] - 1):
        for j in range(1, x.shape[1] - 1):
            out[i, j] = (x[i + -1, j + -1] + x[i + -1, j + 0] + x[i + -1, j + 1] +
                         x[i +  0, j + -1] + x[i +  0, j + 0] + x[i +  0, j + 1] +
                         x[i +  1, j + -1] + x[i +  1, j + 0] + x[i +  1, j + 1]) // 9
    return out

def save(x, filename):
    pass

```

--------------------------------

### Activate and Install Libraries in Virtual Environment (Bash)

Source: https://github.com/csiro-easi/easi-notebooks/blob/main/notebooks/adding-python-libraries.ipynb

Activates the newly created virtual environment and installs a new Python package (e.g., 'geopy') using pip. Ensure the environment is activated before installing packages.

```bash
source ~/venvs/${MYENV}/bin/activate

pip install geopy
```

--------------------------------

### Setup EASI Functions and Query Parameters

Source: https://github.com/csiro-easi/easi-notebooks/blob/main/notebooks/dask/06-On_Chunks-The_Art_of_Dask_Tuning_Part_1.ipynb

Sets up necessary Python libraries and EASI-specific configurations for data querying. It configures pyproj for global context, initializes the datacube client, and sets up S3 access. This prepares the environment for geospatial data operations.

```python
import pyproj
pyproj.set_use_global_context(True)

import git
import sys, os
from dateutil.parser import parse
from dateutil.relativedelta import relativedelta
from dask.distributed import Client, LocalCluster, wait
import datacube
from datacube.utils import masking
from datacube.utils.aws import configure_s3_access

# EASI defaults
os.environ['USE_PYGEOS'] = '0'
repo = git.Repo('.', search_parent_directories=True).working_tree_dir
if repo not in sys.path:
    sys.path.append(repo)
from easi_tools import EasiDefaults, notebook_utils
easi = EasiDefaults()
```

```python
dc = datacube.Datacube()
configure_s3_access(aws_unsigned=False, requester_pays=True, client=client)
```

```python
# Get the centroid of the coordinates of the default extents
central_lat = sum(easi.latitude)/2
central_lon = sum(easi.longitude)/2
# central_lat = -42.019
# central_lon = 146.615

# Set the buffer to load around the central coordinates
# This is a radial distance for the bbox to actual area so bbox 2x buffer in both dimensions
buffer = 1

# Compute the bounding box for the study area
study_area_lat = (central_lat - buffer, central_lat + buffer)
study_area_lon = (central_lon - buffer, central_lon + buffer)

# Data product
product = easi.product('landsat')
```

--------------------------------

### Initialize Local Dask Cluster with notebook_utils

Source: https://github.com/csiro-easi/easi-notebooks/blob/main/html/notebooks/dask/01-Introduction_to_Dask.html

Initializes a local Dask cluster using the `initialize_dask` utility function. This function can also be used to set up remote clusters with Dask Gateway, but this example focuses on local setup. It returns the cluster and client objects for interaction.

```python
cluster, client = notebook_utils.initialize_dask(use_gateway=False, wait=True)
display(cluster if cluster else client)
```

--------------------------------

### Define Example Data Query Parameters

Source: https://github.com/csiro-easi/easi-notebooks/blob/main/notebooks/data_products/sentinel-2-c1-l2a.ipynb

This section defines the parameters for an example data query using EASI defaults for location, latitude, longitude, and time. It includes commented-out options for manually setting these ranges. The parameters are structured into a `query` dictionary, and a function `display_map` is used to visualize the selected area of interest.

```python
# Explorer link
display(Markdown(f'See: {easi.explorer}/products/{product}'))

# EASI defaults
display(Markdown(f'#### Location: {easi.location}'))
latitude_range = easi.latitude
longitude_range = easi.longitude
time_range = easi.time

# Or set your own latitude / longitude
# latitude_range = (21.5, 23.5)
# longitude_range = (88, 90.8)
# time_range = ('2022-01-01', '2022-03-01')

query = {
    'product': product,       # Product name
    'x': longitude_range,     # "x" axis bounds
    'y': latitude_range,      # "y" axis bounds
    'time': time_range,       # Any parsable date strings
}

# Convenience function to display the selected area of interest
display_map(longitude_range, latitude_range)
```

--------------------------------

### Re-compiling Decorated Numba Function (Error Example)

Source: https://github.com/csiro-easi/easi-notebooks/blob/main/notebooks/dask/01-Introduction_to_Dask.ipynb

Demonstrates an error scenario that occurs when attempting to apply 'numba.jit' to a function that has already been decorated and compiled. This highlights potential issues with repeated compilation.

```python
fast_smooth  = numba.jit(smooth)

```

--------------------------------

### Initialize Dask Gateway Client

Source: https://github.com/csiro-easi/easi-notebooks/blob/main/notebooks/dask/04-Go_Big_or_Go_Home_Part_1.ipynb

Initializes a Dask Gateway client, which is used to connect to and manage Dask clusters on a remote Kubernetes environment. This client allows for starting, stopping, and configuring Dask clusters.

```python
from dask.distributed import Client
from dask_gateway import Gateway

gateway = Gateway()
```

--------------------------------

### Start Dask LocalCluster and Client

Source: https://github.com/csiro-easi/easi-notebooks/blob/main/notebooks/dask/02-ODC_with_Dask.ipynb

Initializes a Dask LocalCluster and creates a Client to connect to it. This is the first step in enabling parallel processing. The `Client` object is used to interact with the cluster.

```python
from dask.distributed import Client, LocalCluster

cluster = LocalCluster()
client = Client(cluster)
client
```

--------------------------------

### Setting up a Local Dask Client and Cluster

Source: https://github.com/csiro-easi/easi-notebooks/blob/main/html/notebooks/dask/01-Introduction_to_Dask.html

This code snippet initializes a local Dask distributed computing environment. It creates a `LocalCluster`, which manages a pool of workers on the local machine, and then instantiates a `Client` connected to this cluster. The `client` object is then displayed, providing information about the cluster status and a link to the Dask dashboard for monitoring. This setup is the first step towards parallelizing computations across multiple cores using Dask. Dependencies include `dask.distributed`.

```python
from dask.distributed import Client, LocalCluster, fire_and_forget

cluster = LocalCluster()
client = Client(cluster)
client
```

--------------------------------

### Load, Smooth, and Save EO Data with Numba

Source: https://github.com/csiro-easi/easi-notebooks/blob/main/notebooks/dask/01-Introduction_to_Dask.ipynb

Defines functions to load random Earth Observation data, apply a smoothing filter using Numba for acceleration, and save the processed data. This setup is used to demonstrate single-threaded execution before introducing Dask.

```python
import numpy as np
import numba
from tqdm.notebook import tqdm

def load_eo_data():
    return np.random.random((1000, 1000))

@numba.jit
def smooth(x):
    out = np.empty_like(x)
    for i in range(1, x.shape[0] - 1):
        for j in range(1, x.shape[1] - 1):
            out[i, j] = (x[i + -1, j + -1] + x[i + -1, j + 0] + x[i + -1, j + 1] +
                         x[i +  0, j + -1] + x[i +  0, j + 0] + x[i +  0, j + 1] +
                         x[i +  1, j + -1] + x[i +  1, j + 0] + x[i +  1, j + 1]) // 9
    return out

def save(x, filename):
    pass
```

--------------------------------

### Setup Imports and Configuration

Source: https://github.com/csiro-easi/easi-notebooks/blob/main/notebooks/dask/02-ODC_with_Dask.ipynb

Imports necessary libraries for Git, system operations, date parsing, Dask distributed computing, and the ODC library. It configures AWS S3 access and EASI defaults, including setting the 'USE_PYGEOS' environment variable and appending the repository path to sys.path.

```python
import git
import sys, os
from dateutil.parser import parse
from dateutil.relativedelta import relativedelta
from dask.distributed import Client, LocalCluster
import datacube
from datacube.utils import masking
from datacube.utils.aws import configure_s3_access

# EASI defaults
os.environ['USE_PYGEOS'] = '0'
repo = git.Repo('.', search_parent_directories=True).working_tree_dir
if repo not in sys.path: sys.path.append(repo)
from easi_tools import EasiDefaults, notebook_utils
easi = EasiDefaults()
client = None
```

--------------------------------

### Setup Dask Environment and Imports

Source: https://github.com/csiro-easi/easi-notebooks/blob/main/notebooks/dask/03-Larger_than_RAM_with_LocalCluster.ipynb

Imports necessary libraries for Dask, Git, date manipulation, and datacube operations. It also sets up environment variables and appends the project directory to sys.path for custom module imports.

```python
import git
import sys, os
from dateutil.parser import parse
from dateutil.relativedelta import relativedelta
from dask.distributed import Client, LocalCluster
import datacube
from datacube.utils import masking
from datacube.utils.aws import configure_s3_access

# EASI defaults
os.environ['USE_PYGEOS'] = '0'
repo = git.Repo('.', search_parent_directories=True).working_tree_dir
if repo not in sys.path: sys.path.append(repo)
from easi_tools import EasiDefaults, notebook_utils
easi = EasiDefaults()
```

--------------------------------

### Use Numba decorator for function optimization

Source: https://github.com/csiro-easi/easi-notebooks/blob/main/html/notebooks/dask/01-Introduction_to_Dask.html

This example shows the recommended way to use Numba: by applying the `@numba.jit` decorator directly above the function definition. This approach makes the code cleaner and allows the original function name (`smooth`) to refer to the optimized version.

```python
import numpy as np
import numba

def load_eo_data():
    return np.random.random((1000, 1000))

@numba.jit
def smooth(x):
    out = np.empty_like(x)
    for i in range(1, x.shape[0] - 1):
        for j in range(1, x.shape[1] - 1):
            out[i, j] = (x[i - 1, j - 1] + x[i - 1, j + 0] + x[i - 1, j + 1] +
                         x[i +  0, j - 1] + x[i +  0, j + 0] + x[i +  0, j + 1] +
                         x[i +  1, j - 1] + x[i +  1, j + 0] + x[i +  1, j + 1]) // 9
    return out

def save(x, filename):
    pass
```

--------------------------------

### Initialize EASI Defaults and Environment

Source: https://github.com/csiro-easi/easi-notebooks/blob/main/notebooks/dask/01-Introduction_to_Dask.ipynb

Sets up the environment by importing necessary libraries such as git, sys, and os. It configures environment variables and appends the repository path to sys.path for module accessibility. Finally, it initializes EasiDefaults.

```python
# EASI defaults
import git
import sys
import os
os.environ['USE_PYGEOS'] = '0'
repo = git.Repo('.', search_parent_directories=True).working_tree_dir
if repo not in sys.path: sys.path.append(repo)
from easi_tools import EasiDefaults, notebook_utils
```

```python
easi = EasiDefaults()
```

--------------------------------

### Create a test file using Bash

Source: https://github.com/csiro-easi/easi-notebooks/blob/main/notebooks/easi-scratch-bucket.ipynb

Creates a test file using the `touch` command in Bash. This file is used for subsequent upload and download operations.

```bash
testfile=$1
touch $testfile
ls -l $testfile
```

--------------------------------

### Create and Check Test File using Bash

Source: https://github.com/csiro-easi/easi-notebooks/blob/main/html/notebooks/easi-scratch-bucket.html

Creates a simple empty text file for testing purposes using the `touch` command and then lists its details using `ls -l`. This is a preparatory step before uploading.

```bash
testfile=/home/jovyan/test-file.txt

testfile=$1
touch $testfile
ls -l $testfile
```

--------------------------------

### Python: Configure EASI Defaults and Initialize ODC

Source: https://github.com/csiro-easi/easi-notebooks/blob/main/notebooks/01-welcome-to-easi.ipynb

Sets up EASI-specific default configurations and helper functions for use across notebooks. It attempts to locate the local 'easi-notebooks' repository to add it to the system path for importing custom tools. This snippet initializes the EasiDefaults class and selects a default data product, such as 'landsat', displaying its name and a link to the ODC Explorer.

```python
# EASI defaults
# These are convenience functions so that the notebooks in this repository work in all EASI deployments

# The `git.Repo()` part returns the local directory that easi-notebooks has been cloned into
# If using the `easi-tools` functions from another path, replace `repo` with your local path to `easi-notebooks` directory
try:
    import git
    repo = git.Repo('.', search_parent_directories=True).working_tree_dir    # Path to this cloned local directory
except (ImportError, git.InvalidGitRepositoryError):
    repo = Path.home() / 'easi-notebooks'    # Reasonable default
    if not repo.is_dir():
        raise RuntimeError('To use `easi-tools` please provide the local path to `https://github.com/csiro-easi/easi-notebooks`')
if repo not in sys.path:
    sys.path.append(str(repo))    # Add the local path to `easi-notebooks` to python

from easi_tools import EasiDefaults
from easi_tools import initialize_dask, xarray_object_size

easi = EasiDefaults()

family = 'landsat'
product = easi.product(family)
display(Markdown(f'Default {family} product for "{easi.name}": [{product}]({easi.explorer}/products/{product})'))
```

--------------------------------

### Pandas Index Creation with Datetime Data

Source: https://github.com/csiro-easi/easi-notebooks/blob/main/html/notebooks/dask/06-On_Chunks_-_The_Art_of_Dask_Tuning_Part_1.html

Illustrates the creation of a Pandas DatetimeIndex, commonly used for time-series data. This index specifies a query start (QS-DEC) frequency, suitable for quarterly data.

```python
PandasIndex(DatetimeIndex(['2019-12-01', '2020-03-01', '2020-06-01'], dtype='datetime64[ns]', name='time', freq='QS-DEC'))
```

--------------------------------

### Initialize boto3 client for S3 and EasiDefaults

Source: https://github.com/csiro-easi/easi-notebooks/blob/main/notebooks/easi-scratch-bucket.ipynb

Initializes a boto3 client for interacting with AWS S3 and retrieves the scratch bucket name from EasiDefaults.

```python
client = boto3.client('s3')

easi = EasiDefaults()
bucket = easi.scratch
```

--------------------------------

### Initialize EASI Defaults and Load Tools

Source: https://github.com/csiro-easi/easi-notebooks/blob/main/notebooks/data_products/sentinel-1-gamma0.ipynb

Sets up EASI defaults by determining the local repository path and adding it to sys.path. This ensures that EASI-specific tools like `EasiDefaults`, `initialize_dask`, `xarray_object_size`, and `heading` are available for use in the notebook.

```python
# EASI defaults
# These are convenience functions so that the notebooks in this repository work in all EASI deployments

# The `git.Repo()` part returns the local directory that easi-notebooks has been cloned into
# If using the `easi-tools` functions from another path, replace `repo` with your local path to `easi-notebooks` directory
try:
    import git
    repo = git.Repo('.', search_parent_directories=True).working_tree_dir    # Path to this cloned local directory
except (ImportError, git.InvalidGitRepositoryError):
    repo = Path.home() / 'easi-notebooks'    # Reasonable default
    if not repo.is_dir():
        raise RuntimeError('To use `easi-tools` please provide the local path to `https://github.com/csiro-easi/easi-notebooks`')
if repo not in sys.path:
    sys.path.append(str(repo))    # Add the local path to `easi-notebooks` to python

from easi_tools import EasiDefaults
from easi_tools import initialize_dask, xarray_object_size, heading
```

--------------------------------

### Get AWS User ID using boto3 in Python

Source: https://github.com/csiro-easi/easi-notebooks/blob/main/notebooks/easi-scratch-bucket.ipynb

Retrieves the AWS User ID using the boto3 library in Python. It calls the `get_caller_identity` method and extracts the UserId.

```python
userid = boto3.client('sts').get_caller_identity()['UserId']
print(userid)
```

--------------------------------

### Initialize EASI Defaults and Load Data Parameters

Source: https://github.com/csiro-easi/easi-notebooks/blob/main/notebooks/dask/04-Go_Big_or_Go_Home_Part_1.ipynb

Initializes EASI defaults, sets up environment variables for PyGEOS, and configures the system path for easi_tools. It also defines parameters for data loading, including geographic extents, product types, date ranges, and measurement aliases.

```python
import git
import sys, os
from dateutil.parser import parse
from dateutil.relativedelta import relativedelta
from dask.distributed import Client, LocalCluster
import datacube
from datacube.utils import masking
from datacube.utils.aws import configure_s3_access

# EASI defaults
os.environ['USE_PYGEOS'] = '0'
repo = git.Repo('.', search_parent_directories=True).working_tree_dir
if repo not in sys.path: sys.path.append(repo)
from easi_tools import EasiDefaults, notebook_utils
easi = EasiDefaults()

# Get the centroid of the coordinates of the default extents
central_lat = sum(easi.latitude)/2
central_lon = sum(easi.longitude)/2
# central_lat = -42.019
# central_lon = 146.615

# Set the buffer to load around the central coordinates
# This is a radial distance for the bbox to actual area so bbox 2x buffer in both dimensions
buffer = 0.8

# Compute the bounding box for the study area
study_area_lat = (central_lat - buffer, central_lat + buffer)
study_area_lon = (central_lon - buffer, central_lon + buffer)

# Data product
products = easi.product('landsat')

# Set the date range to load data over
set_time = easi.time
set_time = (set_time[0], parse(set_time[0]) + relativedelta(years=1))
# set_time = ("2021-01-01", "2021-12-31")

# Selected measurement names (used in this notebook). None` will load all of them
alias = easi.aliases('landsat')
measurements = None
# measurements = [alias[x] for x in ['qa_band', 'red', 'nir']]

```

--------------------------------

### Configure EASI Defaults and Select Product

Source: https://github.com/csiro-easi/easi-notebooks/blob/main/notebooks/data_products/sentinel-1-gamma0.ipynb

Initializes the `EasiDefaults` object to access EASI-specific configurations. It then retrieves the default product for a given family (e.g., 'sentinel-1') and displays a link to the product in the EASI explorer.

```python
easi = EasiDefaults()

family = 'sentinel-1'
product = easi.product(family)   # 'sentinel1_grd_gamma0_20m'
display(Markdown(f'Default {family} product for "{easi.name}": [{product}]({easi.explorer}/products/{product})'))
```

--------------------------------

### Get Dask Dashboard Address

Source: https://github.com/csiro-easi/easi-notebooks/blob/main/notebooks/dask/03-Larger_than_RAM_with_LocalCluster.ipynb

Retrieves and prints the dashboard address for the Dask LocalCluster, allowing monitoring of worker activity and resource usage. This is useful for understanding the parallel processing status.

```python
dashboard_address = notebook_utils.localcluster_dashboard(client=client,server=easi.hub)
print(dashboard_address)
```

--------------------------------

### Configure Data Loading Parameters

Source: https://github.com/csiro-easi/easi-notebooks/blob/main/notebooks/dask/04-Go_Big_or_Go_Home_Part_1.ipynb

Sets up parameters for loading satellite data, including QA band configuration, resampling methods, coordinate reference system, output resolution, and grouping by solar day. Dependencies include the 'easi' library and predefined aliases.

```python
# Set the QA band name and mask values
qa_band = alias['qa_band']
qa_mask = easi.qa_mask('landsat')

# Set the resampling method for the bands
resampling = {qa_band: "nearest", "*": "average"}

# Set the coordinate reference system and output resolution
set_crs = easi.crs('landsat')  # If defined, else None
set_resolution = easi.resolution('landsat')  # If defined, else None
# set_crs = "epsg:3577"
# set_resolution = (-30, 30)

# Set the scene group_by method
group_by = "solar_day"
```

--------------------------------

### Apply NumPy Log10 to DataArray

Source: https://github.com/csiro-easi/easi-notebooks/blob/main/notebooks/data_products/sentinel-1-gamma0.ipynb

Demonstrates how to apply a NumPy function, specifically `numpy.log10`, to an Xarray DataArray. This is a basic example of data transformation, with a note suggesting Xarray's `apply_ufunc` for more complex or larger datasets.

```python
# These functions use numpy, which should be satisfactory for most notebooks.
# Calculations for larger or more complex arrays may require Xarray's "ufunc" capability.
# https://docs.xarray.dev/en/stable/examples/apply_ufunc_vectorize_1d.html
#
# Apply numpy.log10 to the DataArray

```

--------------------------------

### Initialize easi-tools Environment

Source: https://github.com/csiro-easi/easi-notebooks/blob/main/notebooks/data_products/sentinel-2-c1-l2a.ipynb

This snippet initializes the easi-tools environment by setting the correct path to the easi-notebooks repository. It handles cases where the repository is cloned locally or provides a default path, raising an error if the repository cannot be found. It then appends the repository path to sys.path to allow importing easi_tools functions.

```python
try:
    import git
    repo = git.Repo('.', search_parent_directories=True).working_tree_dir    # Path to this cloned local directory
except (ImportError, git.InvalidGitRepositoryError):
    repo = Path.home() / 'easi-notebooks'    # Reasonable default
    if not repo.is_dir():
        raise RuntimeError('To use `easi-tools` please provide the local path to `https://github.com/csiro-easi/easi-notebooks`')
if repo not in sys.path:
    sys.path.append(str(repo))    # Add the local path to `easi-notebooks` to python

from easi_tools import EasiDefaults
from easi_tools import initialize_dask, xarray_object_size, mostcommon_crs, heading
```

--------------------------------

### Initialize Local Dask Cluster

Source: https://github.com/csiro-easi/easi-notebooks/blob/main/notebooks/dask/01-Introduction_to_Dask.ipynb

This snippet demonstrates how to create a local Dask cluster and client. It's essential for setting up the environment for parallel computing. The `LocalCluster` creates workers on your local machine, and the `Client` connects to this cluster.

```python
from dask.distributed import Client, LocalCluster, fire_and_forget

cluster = LocalCluster()
client = Client(cluster)
client
```

```python
cluster, client = notebook_utils.initialize_dask(use_gateway=False, wait=True)
display(cluster if cluster else client)
```

--------------------------------

### Get AWS User ID using AWS CLI

Source: https://github.com/csiro-easi/easi-notebooks/blob/main/html/notebooks/easi-scratch-bucket.html

Retrieves the AWS User ID using the AWS CLI command `aws sts get-caller-identity`. This ID is often used as a root directory for user-specific data in S3 buckets.

```bash
userid=$(aws sts get-caller-identity --query 'UserId' | sed 's/["]//g')
echo $userid
```

--------------------------------

### Configure Datacube and EASI Environment

Source: https://github.com/csiro-easi/easi-notebooks/blob/main/notebooks/dask/05-Go_Big_or_Go_Home_Part_2.ipynb

This code block sets up the necessary Python environment for working with datacube and EASI tools. It configures global context for pyproj, imports required libraries for data handling and computation, and sets up environment variables and path configurations specific to the EASI project. It also initializes the datacube API and configures S3 access.

```python
import pyproj
pyproj.set_use_global_context(True)

import git
import sys, os
from dateutil.parser import parse
from dateutil.relativedelta import relativedelta
from dask.distributed import Client, LocalCluster
import datacube
from datacube.utils import masking
from datacube.utils.aws import configure_s3_access

# EASI defaults
os.environ['USE_PYGEOS'] = '0'
repo = git.Repo('.', search_parent_directories=True).working_tree_dir
if repo not in sys.path: sys.path.append(repo)
from easi_tools import EasiDefaults, notebook_utils
easi = EasiDefaults()

dc = datacube.Datacube()
configure_s3_access(aws_unsigned=False, requester_pays=True, client=client)
```

--------------------------------

### Get AWS User ID using AWS CLI and Bash

Source: https://github.com/csiro-easi/easi-notebooks/blob/main/notebooks/easi-scratch-bucket.ipynb

Retrieves the AWS User ID using the AWS CLI command `aws sts get-caller-identity` and extracts the UserId.  This is typically used to construct a unique path in the scratch bucket.

```bash
userid=`aws sts get-caller-identity --query 'UserId' | sed 's/[\"]//g'`
echo $userid
```

--------------------------------

### Initialize Dask Gateway Client and Cluster

Source: https://github.com/csiro-easi/easi-notebooks/blob/main/notebooks/dask/05-Go_Big_or_Go_Home_Part_2.ipynb

This snippet initializes a Dask Gateway client to connect to or create a Dask cluster. It scales the cluster to a specified number of workers and retrieves the client for distributed computing. It handles both cases where an existing cluster is found and when a new one needs to be created.

```python
# Initialize the Gateway client
from dask.distributed import Client
from dask_gateway import Gateway

number_of_workers = 10 

gateway = Gateway()

clusters = gateway.list_clusters()
if not clusters:
    print('Creating new cluster. Please wait for this to finish.')
    cluster = gateway.new_cluster()
else:
    print(f'An existing cluster was found. Connecting to: {clusters[0].name}')
    cluster=gateway.connect(clusters[0].name)

cluster.scale(number_of_workers)

client = cluster.get_client()
client
```

--------------------------------

### Configure Time Range for Data Loading (Python)

Source: https://github.com/csiro-easi/easi-notebooks/blob/main/notebooks/dask/06-On_Chunks-The_Art_of_Dask_Tuning_Part_1.ipynb

Sets the time range for data loading by adjusting the end date to be six months after the start date. It utilizes the `easi.time` attribute and `relativedelta` for date manipulation. Ensure the `parse` function is available for date string conversion.

```python
set_time = easi.time
set_time = (set_time[0], parse(set_time[0]) + relativedelta(months=6))
#set_time = ("2021-07-01", "2021-12-31")
```

--------------------------------

### Import EASI tools using Python

Source: https://github.com/csiro-easi/easi-notebooks/blob/main/notebooks/easi-scratch-bucket.ipynb

Imports necessary libraries and EASI tools for interacting with AWS S3 buckets. It also appends the repository path to the system path if not already present.

```python
import sys, os
import boto3
from datetime import datetime as dt

# EASI tools
import git
repo = git.Repo('.', search_parent_directories=True).working_tree_dir
if repo not in sys.path: sys.path.append(repo)
from easi_tools import EasiDefaults
```

--------------------------------

### Visualize Quality Layer with Custom Colormap

Source: https://github.com/csiro-easi/easi-notebooks/blob/main/notebooks/data_products/sentinel-2-c1-l2a.ipynb

Creates a dynamic image visualization of the SCL (Scene Classification Layer) using hvplot with a custom colormap. It includes options for color mapping, colorbar configuration, and interactive tools. This example demonstrates custom colormap usage and the datashader aggregator `reductions.mode()`.

```python
# Make SCL image
# https://sentinel.esa.int/web/sentinel/technical-guides/sentinel-2-msi/level-2a/algorithm
# https://www.sentinel-hub.com/faq/how-get-s2a-scene-classification-sentinel-2/

from bokeh.models.tickers import FixedTicker

color_def = [
    (0,  '#000000', 'No data'),   # black
    (1,  '#ff0004', 'Saturated or defective'),   # red
    (2,  '#868686', 'Topographic and casted shadow'),   # gray
    (3,  '#774c0b', 'Cloud shadows'),   # brown
    (4,  '#10d32d', 'Vegetation'),   # green
    (5,  '#ffff53', 'Not vegetated'),   # yellow
    (6,  '#0000ff', 'Water'),   # blue
    (7,  '#818181', 'Unclassified'),   # medium gray
    (8,  '#c0c0c0', 'Cloud medium probability'),   # light gray
    (9,  '#f2f2f2', 'Cloud high probability'),   # very light gray
    (10, '#53fff9', 'Thin cirrus'),   # light blue/purple
    (11, '#ff52ff', 'Snow or ice'),   # cyan
]
color_val = [x[0] for x in color_def]
color_hex = [x[1] for x in color_def]
color_txt = [f'{x[0]:2d}: {x[2]}' for x in color_def]
color_lim = (min(color_val), max(color_val) + 1)
bin_edges = color_val + [max(color_val) + 1]
bin_range = (color_val[0] + 0.5, color_val[-1] + 0.5)  # No idea why (0.5,11.5) works and (0,11) or (0,12) do not

# These options manipulate the color map and colorbar to show the categories for this product
options = {
    'title': f'Flag data for: {query["product"]} ({flag_name})',
    'cmap': color_hex,
    'clim': color_lim,
    'color_levels': bin_edges,
    'colorbar': True,
    'width': 800,
    'height': 450,
    'aspect': 'equal',
    'tools': ['hover'],
    'colorbar_opts': {
        'major_label_overrides': dict(zip(color_val, color_txt)),
        'major_label_text_align': 'left',
        'ticker': FixedTicker(ticks=color_val),
    },
}

# Set the dataset CRS, if using hvplot's projection and coastlines options
# plot_crs = native_crs
# if plot_crs == 'epsg:4326':
#     plot_crs = ccrs.PlateCarree()

# Native data and coastline overlay:
# - Comment `crs`, `projection`, `coastline` to plot in native_crs coords

```

--------------------------------

### Configure EASI Defaults and Sentinel-2 Product

Source: https://github.com/csiro-easi/easi-notebooks/blob/main/notebooks/data_products/sentinel-2-c1-l2a.ipynb

This code initializes EASI defaults and sets up a default Sentinel-2 product. It displays a formatted link to the specified product within the EASI explorer. Users can modify the `family` and `product` variables to target different datasets.

```python
easi = EasiDefaults()

family = 'sentinel-2'
product = 'sentinel_2_c1_l2a'  # Sentinel-2 collection 1, L2A
display(Markdown(f'Default {family} product for "{easi.name}": [{product}]({easi.explorer}/products/{product})'))
```

--------------------------------

### Python Imports and Setup for Data Analysis

Source: https://github.com/csiro-easi/easi-notebooks/blob/main/notebooks/data_products/sentinel-1-gamma0.ipynb

This snippet includes essential Python imports for data analysis, particularly for working with geospatial data and the Open Data Cube (ODC). It configures S3 access, sets display options for pandas DataFrames, and imports modules for plotting and masking.

```python
# Common imports and settings
import os, sys, re
from pathlib import Path
from IPython.display import Markdown
import pandas as pd
pd.set_option("display.max_rows", None)
import xarray as xr
import numpy as np

# Datacube
import datacube
from datacube.utils.aws import configure_s3_access
import odc.geo.xr                             # https://github.com/opendatacube/odc-geo
from datacube.utils import masking            # https://github.com/opendatacube/datacube-core/blob/develop/datacube/utils/masking.py
from dea_tools.plotting import display_map    # https://github.com/GeoscienceAustralia/dea-notebooks/tree/develop/Tools

# Basic plots
%matplotlib inline
# import matplotlib.pyplot as plt

```

--------------------------------

### Load Default Libraries into New Environment

Source: https://github.com/csiro-easi/easi-notebooks/blob/main/html/notebooks/adding-python-libraries.html

This command loads default Python libraries into your newly created virtual environment. It determines the Python version and creates a .pth file to include the base site-packages. Replace 'myenv' with your environment's name.

```bash
PYVERS=$(python -c "import sys; print('{0}.{1}'.format(sys.version_info[0], sys.version_info[1]))")
realpath /env/lib/python${PYVERS}/site-packages > ~/venvs/${MYENV}/lib/python${PYVERS}/site-packages/base_venv.pth
```

--------------------------------

### Submit Parallel Tasks to Dask Cluster

Source: https://github.com/csiro-easi/easi-notebooks/blob/main/html/notebooks/dask/01-Introduction_to_Dask.html

Submits multiple tasks to the Dask cluster for parallel execution. This example demonstrates a loop that submits `load_eo_data`, `smooth`, `np.fft.fft2`, and `save` functions, processing 1000 items. The `fire_and_forget` function is used to offload task completion without waiting for results.

```python
for i in tqdm(range(1000)):
    img = client.submit(load_eo_data, pure=False)
    img = client.submit(smooth, img)
    img = client.submit(np.fft.fft2, img)
    future = client.submit(save, img, "file-" + str(i) + "-.dat")
    fire_and_forget(future)
```

--------------------------------

### Create Panel Figure with Select Widget

Source: https://github.com/csiro-easi/easi-notebooks/blob/main/notebooks/data_products/sentinel-2-l2a.ipynb

Wraps a hvplot figure into a Panel object, allowing for interactive widgets. This specific example demonstrates how to replace the default time slider with a dropdown select widget, enhancing user interaction for time-series data. The `pn.widgets.Select` is configured to control the `time` dimension of the plot.

```python
# Optional: Change the default time slider to a dropdown list, https://stackoverflow.com/a/54912917
fig = pn.panel(quality_plot, widgets={'time': pn.widgets.Select})  # widget_location='top_left'
fig
```

--------------------------------

### Upload a file to S3 using AWS CLI and Bash

Source: https://github.com/csiro-easi/easi-notebooks/blob/main/notebooks/easi-scratch-bucket.ipynb

Uploads a specified test file to an S3 bucket using the AWS CLI `aws s3 cp` command. It requires the bucket name, user ID, and test file path as input.

```bash
bucket=$1
userid=$2
testfile=$3

aws s3 cp ${testfile} s3://${bucket}/${userid}/
```

--------------------------------

### Apply Pre-commit Hook Script

Source: https://github.com/csiro-easi/easi-notebooks/blob/main/README.md

This bash script applies the pre-commit hook to the local repository. It ensures that for each committed notebook, an HTML copy is created and outputs are stripped to reduce repository size. This is a crucial step for contributors.

```bash
# Run this in your local repository
sh bin/apply_hooks.sh
```

--------------------------------

### Python Setup: Imports and Defaults for EASI Notebooks

Source: https://github.com/csiro-easi/easi-notebooks/blob/main/notebooks/data_products/sentinel-2-c1-l2a.ipynb

This Python code snippet covers common imports and settings required for EASI notebooks, including libraries for data manipulation (xarray, pandas), datacube operations, geospatial utilities (odc-geo), plotting (matplotlib, holoviews), and AWS S3 access configuration. It also includes placeholders for EASI-specific defaults.

```python
# Common imports and settings
import os, sys
from pathlib import Path
# os.environ['USE_PYGEOS'] = '0'
from IPython.display import Markdown
import pandas as pd
pd.set_option("display.max_rows", None)
import xarray as xr

# Datacube
import datacube
from datacube.utils.aws import configure_s3_access
import odc.geo.xr                                  # https://github.com/opendatacube/odc-geo
from datacube.utils import masking  # https://github.com/opendatacube/datacube-core/blob/develop/datacube/utils/masking.py
from odc.algo import enum_to_bool                  # https://github.com/opendatacube/odc-tools/blob/develop/libs/algo/odc/algo/_masking.py
from dea_tools.plotting import display_map, rgb    # https://github.com/GeoscienceAustralia/dea-notebooks/tree/develop/Tools

# Basic plots
%matplotlib inline
# import matplotlib.pyplot as plt
# plt.rcParams['figure.figsize'] = [12, 8]

# Holoviews
# https://holoviz.org/tutorial/Composing_Plots.html
# https://holoviews.org/user_guide/Composing_Elements.html
import hvplot.xarray
import panel as pn
import colorcet as cc
import cartopy.crs as ccrs
from datashader import reductions
from holoviews import opts
```

```python
# EASI defaults
# These are convenience functions so that the notebooks in this repository work in all EASI deployments

# The `git.Repo()` part returns the local directory that easi-notebooks has been cloned into

```

--------------------------------

### Export xarray Data to Cloud-Optimized GeoTIFF (COG)

Source: https://context7.com/csiro-easi/easi-notebooks/llms.txt

Exports processed xarray datasets, specifically NDVI in this example, to Cloud-Optimized GeoTIFF (COG) files. This process involves calculating the NDVI, creating an output directory, and then iterating through each time step to write a separate GeoTIFF file with COG optimizations using `odc.geo.xr.write_cog`. The output files are suitable for direct use in GIS applications.

```python
import xarray as xr
import odc.geo.xr
from pathlib import Path

# Calculate vegetation index
data = dc.load(
    product='s2_l2a',
    latitude=(-4.2, -3.9),
    longitude=(119.8, 120.1),
    time=('2020-02-01', '2020-03-01'),
    measurements=['red', 'nir']
)

ndvi = (data.nir - data.red) / (data.nir + data.red)
ndvi = ndvi.to_dataset(name='ndvi')

# Create output directory
output_dir = Path.home() / 'output'
output_dir.mkdir(exist_ok=True)

# Write each time layer to separate GeoTIFF
for i in range(len(ndvi.time)):
    date = ndvi.ndvi.isel(time=i).time.dt.strftime('%Y%m%d').data
    single = ndvi.ndvi.isel(time=i)

    fname = output_dir / f's2_ndvi_{date}.tif'
    single.odc.write_cog(
        fname=str(fname),
        overwrite=True
    )
    print(f"Wrote: {fname}")
```

--------------------------------

### Python: Import Packages and Settings for EASI Notebooks

Source: https://github.com/csiro-easi/easi-notebooks/blob/main/notebooks/01-welcome-to-easi.ipynb

Imports common Python packages and sets display options for pandas DataFrames and xarray Datasets within EASI notebooks. It also imports necessary tools for Open Data Cube (ODC) interaction, AWS S3 access configuration, geographic extensions, masking utilities, and plotting functionalities from the dea-tools library. This setup is crucial for data analysis and visualization.

```python
# Common imports and settings
import os, sys, re
from pathlib import Path
from IPython.display import Markdown
import pandas as pd
pd.set_option("display.max_rows", None)
import xarray as xr

# Datacube
import datacube
from datacube.utils.aws import configure_s3_access
import odc.geo.xr
from datacube.utils import masking
# https://github.com/GeoscienceAustralia/dea-notebooks/tree/develop/Tools
from dea_tools.plotting import display_map, rgb

# Basic plots
%matplotlib inline
# import matplotlib.pyplot as plt
# plt.rcParams['figure.figsize'] = [12, 8]
```

--------------------------------

### Define Study Area, Time Range, and Data Parameters

Source: https://github.com/csiro-easi/easi-notebooks/blob/main/notebooks/dask/05-Go_Big_or_Go_Home_Part_2.ipynb

This snippet defines the parameters for loading satellite data, including the study area's geographical extent, the time range for data acquisition, and the specific measurements to be loaded. It also specifies QA band details, resampling methods, CRS, resolution, and grouping strategies, all configured using EASI defaults.

```python
# Get the centroid of the coordinates of the default extents
central_lat = sum(easi.latitude)/2
central_lon = sum(easi.longitude)/2
# central_lat = -42.019
# central_lon = 146.615

# Set the buffer to load around the central coordinates
# This is a radial distance for the bbox to actual area so bbox 2x buffer in both dimensions
buffer = 0.8

# Compute the bounding box for the study area
study_area_lat = (central_lat - buffer, central_lat + buffer)
study_area_lon = (central_lon - buffer, central_lon + buffer)

# Data product
products = easi.product('landsat')

# Set the date range to load data over
set_time = easi.time
set_time = (set_time[0], parse(set_time[0]) + relativedelta(years=1))
# set_time = ("2021-01-01", "2021-12-31")

# Selected measurement names (used in this notebook)
alias = easi.aliases('landsat')
measurements = [alias[x] for x in ['qa_band', 'red', 'nir']]

# Set the QA band name and mask values
qa_band = alias['qa_band']
qa_mask = easi.qa_mask('landsat')

# Set the resampling method for the bands
resampling = {qa_band: "nearest", "*": "average"}

# Set the coordinate reference system and output resolution
set_crs = easi.crs('landsat')  # If defined, else None
set_resolution = easi.resolution('landsat')  # If defined, else None
# set_crs = "epsg:3577"
# set_resolution = (-30, 30)

# Set the scene group_by method
group_by = "solar_day"
```

--------------------------------

### Submit Tasks to Dask Cluster

Source: https://github.com/csiro-easi/easi-notebooks/blob/main/notebooks/dask/01-Introduction_to_Dask.ipynb

This code demonstrates submitting multiple functions to the Dask cluster for parallel execution. It uses `client.submit` to send tasks and `fire_and_forget` to manage futures. This allows for efficient processing of iterative tasks, with benefits visible in `htop` and the Dask dashboard.

```python
%%time

for i in tqdm(range(1000)):
    img = client.submit(load_eo_data, pure=False)
    img = client.submit(smooth, img)
    img = client.submit(np.fft.fft2, img)
    future = client.submit(save, img, "file-" + str(i) + "-.dat")
    fire_and_forget(future)
```

--------------------------------

### List objects in S3 bucket using AWS CLI and Bash

Source: https://github.com/csiro-easi/easi-notebooks/blob/main/notebooks/easi-scratch-bucket.ipynb

Lists objects in a specified S3 bucket and user ID path using the AWS CLI `aws s3 ls` command. It requires the bucket name and user ID as input.

```bash
bucket=$1
userid=$2

aws s3 ls s3://${bucket}/${userid}/
```

--------------------------------

### Create or Connect to Dask Cluster

Source: https://github.com/csiro-easi/easi-notebooks/blob/main/notebooks/dask/04-Go_Big_or_Go_Home_Part_1.ipynb

This snippet checks for existing Dask clusters and creates a new one if none are found. It connects to the first available cluster if multiple exist. Listing clusters is done via `gateway.list_clusters()`.

```python
clusters = gateway.list_clusters()
if not clusters:
    print('Creating new cluster. Please wait for this to finish.')
    cluster = gateway.new_cluster()
else:
    print(f'An existing cluster was found. Connecting to: {clusters[0].name}')
    cluster=gateway.connect(clusters[0].name)
cluster
```

--------------------------------

### Sequential EO Data Processing Loop with Progress Bar

Source: https://github.com/csiro-easi/easi-notebooks/blob/main/notebooks/dask/01-Introduction_to_Dask.ipynb

Executes a loop that loads, smooths, and transforms Earth Observation data sequentially. It uses `tqdm` to display a progress bar, illustrating the time taken for each iteration and the overall process without parallelization.

```python
%%time
for i in tqdm(range(1000)):
    img = load_eo_data()
    img = smooth(img)
    img = np.fft.fft2(img)
    save(img, "file-" + str(i) + "-.dat")
```

--------------------------------

### Python Imports and Configuration for EASI Notebooks

Source: https://github.com/csiro-easi/easi-notebooks/blob/main/html/notebooks/dask/06-On_Chunks_-_The_Art_of_Dask_Tuning_Part_1.html

Imports essential Python libraries such as pyproj, git, dateutil, dask, and datacube. It also configures the 'USE_PYGEOS' environment variable and adds the project's root directory to the system path for importing custom EASI tools. This sets up the foundational environment for the notebook's operations.

```python
import pyproj
pyproj.set_use_global_context(True)

import git
import sys, os
from dateutil.parser import parse
from dateutil.relativedelta import relativedelta
from dask.distributed import Client, LocalCluster, wait
import datacube
from datacube.utils import masking
from datacube.utils.aws import configure_s3_access

# EASI defaults
os.environ['USE_PYGEOS'] = '0'
repo = git.Repo('.', search_parent_directories=True).working_tree_dir
if repo not in sys.path: sys.path.append(repo)
from easi_tools import EasiDefaults, notebook_utils
easi = EasiDefaults()
```

--------------------------------

### Numba JIT Compilation of Smoothing Function

Source: https://github.com/csiro-easi/easi-notebooks/blob/main/notebooks/dask/01-Introduction_to_Dask.ipynb

Applies Numba's JIT (Just-In-Time) compiler to the existing Python 'smooth' function, creating an optimized version named 'fast_smooth'. This enables runtime translation to machine code.

```python
import numba

fast_smooth  = numba.jit(smooth)

```