### Setup CI Environment Source: https://github.com/nvidia/earth2mip/blob/main/CONTRIBUTING.md Installs pre-commit hooks and git lfs for local development. ```bash make setup-ci ``` -------------------------------- ### Install Earth-2 MIP from Source Source: https://github.com/nvidia/earth2mip/blob/main/docs/userguide/install.md Clone the repository and install Earth-2 MIP using pip. This is the recommended installation method. ```bash git clone git@github.com:NVIDIA/earth2mip.git cd earth2mip && pip install . ``` -------------------------------- ### Install Documentation Build Packages Source: https://github.com/nvidia/earth2mip/blob/main/docs/userguide/install.md Install optional packages required for building the project documentation for Earth-2 MIP. ```bash pip install .[docs] ``` -------------------------------- ### Install Optional Dependencies for Graphcast Source: https://github.com/nvidia/earth2mip/blob/main/docs/userguide/install.md Install the optional dependencies for Graphcast models and also install from the requirements.txt file. ```bash pip install .[graphcast] pip install -r requirements.txt ``` -------------------------------- ### Install Development Packages Source: https://github.com/nvidia/earth2mip/blob/main/docs/userguide/install.md Install optional development packages for linting, formatting, and other development-related tools for Earth-2 MIP. ```bash pip install .[dev] ``` -------------------------------- ### Build documentation commands Source: https://github.com/nvidia/earth2mip/blob/main/CONTRIBUTING.md Commands to generate core, API, and example documentation. ```bash # Build the core and API docs make docs # Build the core, API and example docs # This will require all examples to get executed and assumed the correct run env make docs-full ``` -------------------------------- ### Install Optional Dependencies for Pangu Weather Source: https://github.com/nvidia/earth2mip/blob/main/docs/userguide/install.md Install the optional dependencies required for running Pangu weather models with Earth-2 MIP. ```bash pip install .[pangu] ``` -------------------------------- ### Install Earth-2 MIP from Release Wheel Source: https://github.com/nvidia/earth2mip/blob/main/docs/userguide/install.md Download a pre-built Python wheel from GitHub releases and install it using pip. Ensure you replace 'v0.1.0' with the desired version. ```bash curl -L https://github.com/NVIDIA/earth2mip/releases/download/v0.1.0/earth2mip-0.1.0-py3-none-any.whl > earth2mip-0.1.0-py3-none-any.whl pip install earth2mip-0.1.0-py3-none-any.whl ``` -------------------------------- ### Run Test Suite Source: https://github.com/nvidia/earth2mip/blob/main/CONTRIBUTING.md Commands to enter the docker environment, install dependencies, and execute tests. ```bash make enter # install extra deps make install # run the tests pytest ``` -------------------------------- ### Implementing a PersistenceModule Source: https://github.com/nvidia/earth2mip/blob/main/docs/userguide/concepts.rst Example of implementing a basic persistence forecast module using the Module abstraction. ```APIDOC ## PersistenceModule Implementation ### Description A simple implementation of a persistence forecast where the model always returns the initial condition. ### Request Example ```python import torch.nn class PersistenceModule(torch.nn.Module): def forward(self, x): return x ``` ``` -------------------------------- ### Implement Persistence Forecast as a Module Source: https://github.com/nvidia/earth2mip/blob/main/docs/userguide/concepts.rst This module serves as a basic example of a persistence forecast, returning the input directly. It lacks metadata about input/output, making it less user-friendly for external use. ```python import torch.nn class PersistenceModule(torch.nn.Module): def forward(self, x): return x ``` -------------------------------- ### Fix ONNX Runtime GPU Installation Source: https://github.com/nvidia/earth2mip/blob/main/docs/userguide/troubleshooting.rst If you encounter a UserWarning about 'CUDAExecutionProvider' not being available, it indicates an issue with your ONNX Runtime installation. Reinstalling with the GPU-enabled package should resolve this. ```bash pip uninstall onnxruntime onnxruntime-gpu pip install onnxruntime-gpu ``` -------------------------------- ### Implementing a PeristenceTimeLoop Source: https://github.com/nvidia/earth2mip/blob/main/docs/userguide/concepts.rst Example of implementing a TimeLoop to encapsulate time-stepping logic. ```APIDOC ## PeristenceTimeLoop Implementation ### Description Encapsulates time-stepping logic, metadata, and input/output channel definitions. ### Request Example ```python from earth2mip.time_loop import TimeLoop from earth2mip.schema import Grid import datetime class PeristenceTimeLoop(TimeLoop): time_step = datetime.timedelta(hours=12) n_history_levels = 1 in_channel_names = ["a", "b", "c"] out_channel_names = ["a", "b", "c"] grid = Grid.grid_721x1440 def __call__(self, time, x, restart=None): while True: yield time, x, None time += self.time_step ``` ``` -------------------------------- ### Install IPython Kernel for Jupyter Source: https://github.com/nvidia/earth2mip/blob/main/docs/userguide/install.md Install the ipykernel package and register a new kernel for the 'earth2mip' Conda environment. This allows Jupyter to use the environment. ```bash pip install ipykernel python -m ipykernel install --user --name=earth2mip-kernel ``` -------------------------------- ### Create Custom Kelvin to Celsius Diagnostic Source: https://github.com/nvidia/earth2mip/blob/main/docs/userguide/diagnostic.md Implement a custom diagnostic by inheriting from DiagnosticBase. This example converts surface temperature from Kelvin to Celsius. Ensure input and output channel names and grids are correctly defined. ```python class Kelvin2Celcius(DiagnosticBase): def __init__(self, grid: grid.LatLonGrid): super().__init__() self.grid = grid @property def in_channel_names(self) -> list[str]: return ['t2m'] @property def out_channel_names(self) -> list[str]: return ['t2m_c'] @property def in_grid(self) -> grid.LatLonGrid: return self.grid @property def out_grid(self) -> grid.LatLonGrid: return self.grid def __call__(self, x: torch.Tensor) -> torch.Tensor: return x - 273.15 @classmethod def load_diagnostic(cls): pass @classmethod def load_diagnostic( cls, package: Optional[Package], grid: grid.LatLonGrid ): return cls(grid) ``` -------------------------------- ### Integrate Multiple Data Sources for Initial Conditions Source: https://context7.com/nvidia/earth2mip/llms.txt Demonstrates using different data sources (CDS, GFS, HDF5) for initial atmospheric conditions. The `get_data_source` function acts as a factory for creating data source instances. ```python import datetime from earth2mip.initial_conditions import cds, gfs, hdf5, get_data_source from earth2mip.schema import InitialConditionSource # Climate Data Store (CDS) - ERA5 reanalysis data # Requires CDS API key in ~/.cdsapirc channel_names = ["t2m", "u10m", "v10m", "z500", "z850", "t850", "tcwv"] cds_source = cds.DataSource(channel_names) data = cds_source[datetime.datetime(2018, 1, 1, 0)] print(f"CDS data shape: {data.shape}") # (channels, lat, lon) # GFS operational data - recent forecasts gfs_source = gfs.DataSource(channel_names) recent_data = gfs_source[datetime.datetime.utcnow() - datetime.timedelta(hours=6)] # HDF5 local data source - for large-scale scoring # Expects directory structure: root/year.h5 with data.json metadata hdf5_source = hdf5.DataSource.from_path( root="/path/to/era5/data", channel_names=channel_names ) historical_data = hdf5_source[datetime.datetime(2017, 6, 15, 12)] # Use get_data_source factory function source = get_data_source( channel_names=channel_names, initial_condition_source=InitialConditionSource.cds ) # Get initial condition formatted for a specific model from earth2mip.initial_conditions import get_initial_condition_for_model from earth2mip.networks import get_model model = get_model("e2mip://dlwp", device="cuda:0") x = get_initial_condition_for_model( time_loop=model, data_source=cds_source, time=datetime.datetime(2018, 1, 1) ) print(f"Initial condition shape: {x.shape}") # (batch, history, channels, lat, lon) ``` -------------------------------- ### Run TensorBoard for Diagnostics Source: https://github.com/nvidia/earth2mip/blob/main/test/graphcast/README.md Launch TensorBoard to visualize diagnostic logs generated by the multi-step verification script. ```bash tensorboard --logdir runs ``` -------------------------------- ### Run Linting Source: https://github.com/nvidia/earth2mip/blob/main/CONTRIBUTING.md Manually executes the linting process for the repository. ```bash make lint ``` -------------------------------- ### Apply License Header Source: https://github.com/nvidia/earth2mip/blob/main/CONTRIBUTING.md Standard Apache 2.0 license header required for all source code files. ```bash # SPDX-FileCopyrightText: Copyright (c) 2023 NVIDIA CORPORATION & AFFILIATES. # SPDX-FileCopyrightText: All rights reserved. # SPDX-License-Identifier: Apache-2.0 # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. ``` -------------------------------- ### Retrieve model package from registry Source: https://context7.com/nvidia/earth2mip/llms.txt Fetches a model package by its URI and prints the local file system path. ```python package = registry.get_model("e2mip://fcnv2_sm") print(f"Model package location: {package.root}") ``` -------------------------------- ### Build Singularity Image Source: https://github.com/nvidia/earth2mip/blob/main/docs/userguide/install.md Build a Singularity Image Format (sif) file from the definition file. This command requires root privileges or fakeroot. ```bash singularity build --fakeroot --sandbox earth2mip.sif earth2mip.def ``` -------------------------------- ### Configure Pangu Time Loop Metadata Source: https://github.com/nvidia/earth2mip/blob/main/docs/userguide/plugin.rst Define the entrypoint and keyword arguments for the Pangu time loop loader in metadata.json. ```json { "entrypoint": { "name": "earth2mip.networks.pangu:load", "kwargs": { "time_step_hours": 6 } } } ``` -------------------------------- ### Run Pangu Weather Inference Source: https://github.com/nvidia/earth2mip/blob/main/README.md Use this snippet to run Pangu weather inference using an initial state from the climate data store (CDS). Requires importing necessary modules and initializing the model and data source. ```python import datetime from earth2mip.networks import get_model from earth2mip.initial_conditions import cds from earth2mip.inference_ensemble import run_basic_inference time_loop = get_model("e2mip://dlwp", device="cuda:0") data_source = cds.DataSource(time_loop.in_channel_names) ds = run_basic_inference(time_loop, n=10, data_source=data_source, time=datetime.datetime(2018, 1, 1)) ds.chunk() ``` -------------------------------- ### Run 1 Year Simulation with Earth2MIP Source: https://github.com/nvidia/earth2mip/blob/main/examples/utils/workflows/README.md Configure the XLA_PYTHON_CLIENT_ALLOCATOR to 'platform' to avoid Out-Of-Memory errors during simulation. Then, execute the 1_year_run.py script. Outputs are saved in the 'year_runs/' directory in netCDF format. ```bash export XLA_PYTHON_CLIENT_ALLOCATOR=platform python3 1_year_run.py ``` -------------------------------- ### Add documentation version to switcher.json Source: https://github.com/nvidia/earth2mip/blob/main/CONTRIBUTING.md JSON structure for registering a new documentation version in the switcher. ```json { "name": "1.0.0", "version": "1.0.0", "url": "https://nvidia.github.io/earth2mip/v/1.0.0/" } ``` -------------------------------- ### Configure Environment Variables Source: https://context7.com/nvidia/earth2mip/llms.txt Set paths for model registries, data caches, and distributed training settings using environment variables or a .env file. ```python import os import dotenv # Load environment from .env file dotenv.load_dotenv() # Key environment variables: # MODEL_REGISTRY - Directory for model checkpoints (default: ~/.cache/earth2mip/models) os.environ["MODEL_REGISTRY"] = "/path/to/models" # ERA5_HDF5 - Path to ERA5 HDF5 data for scoring os.environ["ERA5_HDF5"] = "/path/to/era5/h5" # LOCAL_CACHE - Cache directory for downloaded data os.environ["LOCAL_CACHE"] = "/path/to/cache" # WORLD_SIZE - Number of GPUs for distributed inference os.environ["WORLD_SIZE"] = "4" # Now import earth2mip (environment must be set before import) from earth2mip import registry, config # Verify configuration print(f"Model registry: {config.MODEL_REGISTRY}") print(f"ERA5 HDF5 path: {config.ERA5_HDF5}") print(f"Local cache: {config.LOCAL_CACHE}") ``` -------------------------------- ### Execute Command in Singularity Container Source: https://github.com/nvidia/earth2mip/blob/main/docs/userguide/install.md Run a command within the built Singularity container, mounting the current directory and enabling NVIDIA GPU support. ```bash singularity exec -B ${PWD}:/workspace/earth2mip --nv earth2mip.sif bash -c 'cd ~' ``` -------------------------------- ### Define Metadata for Persistence Model Source: https://github.com/nvidia/earth2mip/blob/main/docs/userguide/plugin.rst Configure the metadata.json file to point to the architecture entrypoint for the persistence plugin. ```json { "architecture_entrypoint": "my_package.plugin:load_persistence_module" }, "grid": "721x1440", "in_channels": [0, 1, 2], "out_channels": [0, 1, 2], } ``` -------------------------------- ### Run Quick Tests Source: https://github.com/nvidia/earth2mip/blob/main/CONTRIBUTING.md Executes only the tests not marked as slow. ```bash pytest -m 'not slow' ``` -------------------------------- ### Configure Interleaved Pangu Time Loop Metadata Source: https://github.com/nvidia/earth2mip/blob/main/docs/userguide/plugin.rst Define the entrypoint for the interleaved Pangu time loop loader in metadata.json. ```json {"entrypoint": {"name": "earth2mip.networks.pangu:load_24substep6"}} ``` -------------------------------- ### Create and Activate Conda Environment Source: https://github.com/nvidia/earth2mip/blob/main/docs/userguide/install.md Set up a dedicated Conda environment for Earth-2 MIP with Python 3.10. This helps prevent package conflicts. ```bash conda create --name earth2mip python=3.10 conda activate earth2mip ``` -------------------------------- ### Model Abstractions Overview Source: https://github.com/nvidia/earth2mip/blob/main/docs/userguide/concepts.rst Overview of the core interfaces used in Earth2 MIP for wrapping machine learning models. ```APIDOC ## Model Abstractions ### Description Earth2 MIP uses three primary abstractions to wrap machine learning models for weather forecasting: Modules, TimeLoops, and Forecasts. ### Core Components - **Module**: The base machine learning model (e.g., `torch.nn.Module`) that processes 2D fields. - **TimeLoop**: Encapsulates the Module, time-stepping logic, data preprocessing, and output handling. - **Forecast**: High-level interface for scoring algorithms, representing operations over 2D arrays of states (initial times vs lead times). ``` -------------------------------- ### Implement a Persistence Forecast Plugin Source: https://github.com/nvidia/earth2mip/blob/main/docs/userguide/plugin.rst Define a custom torch.nn.Module and a loader function to integrate a persistence model as a plugin. ```python # my_package/plugin.py import torch class Persistence(torch.nn.Module): def forward(self, x): return x def load_persistence_module(package, pretrained=True, device=None): """ load a model package Args: package: a fcn_mip.model_registry.Package, has package.get(path), and package.metadata() methods. """ # use package.get to open weights etc if needed. return Persistence() ``` -------------------------------- ### Combine Diagnostic Models with Base Weather Models Source: https://context7.com/nvidia/earth2mip/llms.txt Illustrates how to use `DiagnosticTimeLoop` to integrate diagnostic models, such as precipitation prediction, with base prognostic weather models. ```python import datetime from earth2mip.networks import get_model from earth2mip.diagnostic import PrecipitationAFNO, ClimateNet, WindSpeed, DiagnosticTimeLoop from earth2mip.inference_ensemble import run_basic_inference from earth2mip.initial_conditions import cds # Load base prognostic model device = "cuda:0" model = get_model("e2mip://fcn", device=device) # Load precipitation diagnostic model precip_package = PrecipitationAFNO.load_package() precip_diagnostic = PrecipitationAFNO.load_diagnostic(precip_package) ``` -------------------------------- ### earth2mip.networks.pangu Module Source: https://github.com/nvidia/earth2mip/blob/main/docs/modules/earth2mip.networks.rst Details about the Pangu network module. ```APIDOC ## earth2mip.networks.pangu Module ### Description This module contains the Pangu network implementation. ### Members - `members`: Lists all public members of the module. - `undoc-members`: Includes members that are not documented. - `show-inheritance`: Shows the inheritance hierarchy for classes within the module. ``` -------------------------------- ### Create a Forecast from a TimeLoop Source: https://github.com/nvidia/earth2mip/blob/main/docs/userguide/concepts.rst Initializes a TimeLoopForecast using an existing time_loop and a dictionary of initial data. ```python from earth2mip.forecasts import TimeLoopForecast forecast = TimeLoopForecast( time_loop, initial_data={ datetime.datetime(2018, 1, 1): np.zeros([1, 1, 3, 721, 1440]) }, ) ``` -------------------------------- ### Run Ensemble Inference via CLI Source: https://context7.com/nvidia/earth2mip/llms.txt Execute ensemble simulations using a JSON configuration file, supporting both single-node and distributed GPU execution. ```bash # Create configuration file (config.json) cat > config.json << 'EOF' { "ensemble_members": 8, "noise_amplitude": 0.05, "simulation_length": 40, "weather_event": { "properties": { "name": "Winter_Storm", "start_time": "2018-01-15 00:00:00", "initial_condition_source": "cds" }, "domains": [ { "name": "global", "type": "Window", "diagnostics": [{"type": "raw", "channels": ["t2m", "u10m", "v10m", "z500", "msl"]}] } ] }, "output_path": "outputs/winter_storm_ensemble", "output_frequency": 1, "weather_model": "fcnv2_sm", "seed": 42, "ensemble_batch_size": 2, "perturbation_strategy": "correlated", "noise_reddening": 2.0 } EOF # Run ensemble inference python -m earth2mip.inference_ensemble config.json # For distributed inference across multiple GPUs # Set WORLD_SIZE environment variable export WORLD_SIZE=4 torchrun --nproc_per_node=4 -m earth2mip.inference_ensemble config.json ``` -------------------------------- ### Compare multiple AI weather models Source: https://context7.com/nvidia/earth2mip/llms.txt Runs inference on multiple models using identical initial conditions to compare output fields like z500. ```python import datetime import xarray from earth2mip import registry from earth2mip.inference_ensemble import run_basic_inference from earth2mip.initial_conditions import cds import earth2mip.networks.dlwp as dlwp import earth2mip.networks.pangu as pangu # Load multiple models print("Loading DLWP model...") dlwp_package = registry.get_model("e2mip://dlwp") dlwp_model = dlwp.load(dlwp_package, device="cuda:0") print("Loading Pangu model...") pangu_package = registry.get_model("e2mip://pangu") pangu_model = pangu.load(pangu_package, device="cuda:0") # Create data sources (models may require different channels) dlwp_data_source = cds.DataSource(dlwp_model.in_channel_names) pangu_data_source = cds.DataSource(pangu_model.in_channel_names) # Run inference with same initial time time = datetime.datetime(2018, 1, 1) print("Running DLWP inference...") dlwp_ds = run_basic_inference( dlwp_model, n=24, # DLWP: 12hr dt, yields output every 6hr data_source=dlwp_data_source, time=time, ) dlwp_ds.to_netcdf("dlwp_forecast.nc") print("Running Pangu inference...") pangu_ds = run_basic_inference( pangu_model, n=24, # Pangu: 6hr dt data_source=pangu_data_source, time=time, ) pangu_ds.to_netcdf("pangu_forecast.nc") # Compare z500 fields dlwp_z500 = dlwp_ds.sel(channel="z500") pangu_z500 = pangu_ds.sel(channel="z500") # Compute difference at each time step print(f"DLWP z500 range: {float(dlwp_z500.min()):.1f} to {float(dlwp_z500.max()):.1f}") print(f"Pangu z500 range: {float(pangu_z500.min()):.1f} to {float(pangu_z500.max()):.1f}") ``` -------------------------------- ### Configure and Run Ensemble Inference Source: https://context7.com/nvidia/earth2mip/llms.txt Define ensemble parameters using a JSON configuration and execute via the inference_ensemble module. Requires fetching the model package from the registry prior to execution. ```python import json from earth2mip import inference_ensemble, registry # Define ensemble configuration config = { "ensemble_members": 4, "noise_amplitude": 0.05, "simulation_length": 20, # Number of 6-hour timesteps "weather_event": { "properties": { "name": "Hurricane_Analysis", "start_time": "2018-06-01 00:00:00", "initial_condition_source": "cds", # Options: cds, gfs, ifs, era5 }, "domains": [ { "name": "global", "type": "Window", "diagnostics": [{"type": "raw", "channels": ["t2m", "u10m", "v10m", "z500"]}], } ], }, "output_path": "outputs/ensemble_forecast", "output_frequency": 1, "weather_model": "fcnv2_sm", "seed": 12345, "ensemble_batch_size": 1, "perturbation_strategy": "correlated", # Options: correlated, gaussian, bred_vector, spherical_grf, none "noise_reddening": 2.0, } # Fetch model package first package = registry.get_model("e2mip://fcnv2_sm") # Run ensemble inference config_str = json.dumps(config) inference_ensemble.main(config_str) ``` -------------------------------- ### Singularity Definition File for Earth-2 MIP Source: https://github.com/nvidia/earth2mip/blob/main/docs/userguide/install.md A Dockerfile-like definition file to build a Singularity image for Earth-2 MIP, including source files and post-installation steps. ```dockerfile Bootstrap: docker FROM: nvcr.io/nvidia/modulus/modulus:23.11 %files earth2mip/* /workspace/earth2mip/earth2mip/ MANIFEST.in /workspace/earth2mip/MANIFEST.in setup.cfg /workspace/earth2mip/setup.cfg setup.py /workspace/earth2mip/setup.py README.md /workspace/earth2mip/README.md LICENSE.txt /workspace/earth2mip/LICENSE.txt examples/README.md /workspace/earth2mip/examples/README.md %post cd /workspace/earth2mip && pip3 install . pip3 install cartopy %environment export HOME=/workspace/earth2mip/ %runscript cd ~ %labels AUTHOR NVIDIA Earth-2 and Modulus Team ``` -------------------------------- ### Execute Inference via CLI Source: https://github.com/nvidia/earth2mip/blob/main/docs/userguide/plugin.rst Run inference using a custom model package with the torchrun command. ```bash torchrun --nproc_per_node -m earth2mip.inference_medium_range -n 56 file://abs/path/to/persistence scores.nc ``` -------------------------------- ### earth2mip.networks Module Contents Source: https://github.com/nvidia/earth2mip/blob/main/docs/modules/earth2mip.networks.rst Overview of the contents within the earth2mip.networks package. ```APIDOC ## earth2mip.networks Module Contents ### Description This section provides an overview of the top-level members within the `earth2mip.networks` package. ### Members - `members`: Lists all public members of the module. - `undoc-members`: Includes members that are not documented. - `show-inheritance`: Shows the inheritance hierarchy for classes within the module. ``` -------------------------------- ### earth2mip.networks.graphcast Module Source: https://github.com/nvidia/earth2mip/blob/main/docs/modules/earth2mip.networks.rst Details about the GraphCast network module. ```APIDOC ## earth2mip.networks.graphcast Module ### Description This module contains the GraphCast network implementation. ### Members - `members`: Lists all public members of the module. - `undoc-members`: Includes members that are not documented. - `show-inheritance`: Shows the inheritance hierarchy for classes within the module. ``` -------------------------------- ### run_over_initial_times Source: https://github.com/nvidia/earth2mip/blob/main/docs/userguide/inference-workflows.rst Executes inference routines over a collection of initial time steps. ```APIDOC ## run_over_initial_times ### Description Runs inference routines across multiple initial time points. ``` -------------------------------- ### Instantiate Diagnostic Time Loop Source: https://github.com/nvidia/earth2mip/blob/main/docs/userguide/diagnostic.md Use DiagnosticTimeLoop to attach diagnostic models to existing Earth-2 MIP workflows. Ensure the weather model and diagnostic models are properly instantiated. ```python from earth2mip.diagnostic import DiagnosticTimeLoop # Instantiate weather model model = get_model(model_name, device=device) # Instantiate diagnostic model package = DiagnosticModel.load_package() diagnostic = DiagnosticModel.load_diagnostic(package) # Create diagnostic time loop to attach diagnostic models model_diagnostic = DiagnosticTimeLoop(diagnostics=[diagnostic], model=model) ``` -------------------------------- ### Load Weather Models with get_model Source: https://context7.com/nvidia/earth2mip/llms.txt Use get_model to initialize an inference model from the registry. The returned TimeLoop object provides access to model properties like channels and grid shape. ```python import datetime from earth2mip.networks import get_model from earth2mip import registry # Load a model using the e2mip:// prefix for auto-download # Available models: fcn, dlwp, pangu, pangu_6, pangu_24, fcnv2_sm, graphcast time_loop = get_model("e2mip://dlwp", device="cuda:0") # Or load from local model registry (requires model already downloaded) package = registry.get_model("fcnv2_sm") time_loop = get_model("fcnv2_sm", device="cuda:0") # Access model properties print(f"Input channels: {time_loop.in_channel_names}") print(f"Output channels: {time_loop.out_channel_names}") print(f"Time step: {time_loop.time_step}") print(f"Grid shape: {time_loop.grid.shape}") ``` -------------------------------- ### Create a TimeLoop from a Module Source: https://github.com/nvidia/earth2mip/blob/main/docs/userguide/concepts.rst Uses the Inference class to wrap a model into a TimeLoop, requiring normalization parameters and grid configuration. ```python from earth2mip.networks import Inference model = PersistenceModule() # work around to not do any normalization center = np.zeros([3]) scale = np.ones([3]) time_loop = Inference( model, center=center, scale=scale, grid=Grid.grid_721x1440, time_step=datetime.timedelta(hours=12), # note n_history_levels == n_history + 1 n_history=0, channel_names=["a", "b", "c"], ) ``` -------------------------------- ### Load Custom Model via Python API Source: https://github.com/nvidia/earth2mip/blob/main/docs/userguide/plugin.rst Load a model package using the get_model function, either by absolute path or via a registered environment variable. ```python from earth2mip.networks import get_model time_loop = get_model("file://abs/path/to/persistence") ``` ```python time_loop = get_model("persistence") ``` -------------------------------- ### earth2mip.datasets.hindcast Source: https://github.com/nvidia/earth2mip/blob/main/docs/modules/earth2mip.datasets.rst Documentation for the hindcast dataset module. ```APIDOC ## earth2mip.datasets.hindcast ### Description Provides access to hindcast dataset functionalities. ### Module Members This module exposes various members for interacting with hindcast data. Use the `:members:` directive to see all available functions and classes. ``` -------------------------------- ### earth2mip.networks.dlwp Module Source: https://github.com/nvidia/earth2mip/blob/main/docs/modules/earth2mip.networks.rst Details about the DLWP network module. ```APIDOC ## earth2mip.networks.dlwp Module ### Description This module contains the Deep Learning Weather Prediction (DLWP) network implementation. ### Members - `members`: Lists all public members of the module. - `undoc-members`: Includes members that are not documented. - `show-inheritance`: Shows the inheritance hierarchy for classes within the module. ``` -------------------------------- ### Run Modulus Docker Container Source: https://github.com/nvidia/earth2mip/blob/main/docs/userguide/install.md Launch the Modulus Docker container with necessary configurations for memory, stack, and GPU support. Port 8888 is mapped for Jupyter Lab access. ```bash docker run --rm --shm-size=1g --ulimit memlock=-1 --ulimit stack=67108864 --runtime nvidia -p 8888:8888 -it nvcr.io/nvidia/modulus/modulus:23.11 ``` -------------------------------- ### earth2mip.datasets.zarr_directory Source: https://github.com/nvidia/earth2mip/blob/main/docs/modules/earth2mip.datasets.rst Documentation for the zarr_directory dataset module. ```APIDOC ## earth2mip.datasets.zarr_directory ### Description Provides access to zarr_directory dataset functionalities. ### Module Members This module exposes various members for interacting with zarr directory data. Use the `:members:` directive to see all available functions and classes. ``` -------------------------------- ### Implement Persistence Forecast as a TimeLoop Source: https://github.com/nvidia/earth2mip/blob/main/docs/userguide/concepts.rst This TimeLoop implementation for a persistence forecast encapsulates time-stepping logic, data preprocessing, and output. It defines metadata like time step, history levels, channel names, and grid. Note that this implementation does not support restart capability. ```python from earth2mip.time_loop import TimeLoop from earth2mip.schema import Grid import datetime class PeristenceTimeLoop(TimeLoop): time_step = datetime.timedelta(hours=12) # 1 history level = only the current time as input n_history_levels = 1 in_channel_names = ["a", "b", "c"] out_channel_names = ["a", "b", "c"] grid = Grid.grid_721x1440 def __call__(self, time, x, restart=None): b, h, c, w, h == x.shape assert b == 1 assert h == self.n_history_levels assert c == len(self.in_channel_names) assert (w, h) == self.grid.shape while True: yield time, x, None time += self.time_step ``` -------------------------------- ### Wrap prognostic model with diagnostic Source: https://context7.com/nvidia/earth2mip/llms.txt Wraps a prognostic model with a diagnostic tool and runs inference to save results to a NetCDF file. ```python model_with_precip = DiagnosticTimeLoop( diagnostics=[precip_diagnostic], model=model ) # Create data source and run inference data_source = cds.DataSource(model.in_channel_names) time = datetime.datetime(2018, 4, 4) ds = run_basic_inference( model_with_precip, n=20, data_source=data_source, time=time, ) # Access precipitation output (added channel "tp" for total precipitation) total_precip = ds.sel(channel="tp") tcwv = ds.sel(channel="tcwv") # Total column water vapor # Save results ds.to_netcdf("forecast_with_precipitation.nc") # ClimateNet for atmospheric river detection climatenet_package = ClimateNet.load_package() climatenet = ClimateNet.load_diagnostic(climatenet_package) # WindSpeed diagnostic (computes wind speed from u/v components) wind_diag = WindSpeed() ``` -------------------------------- ### earth2mip.s2s Module Contents Source: https://github.com/nvidia/earth2mip/blob/main/docs/modules/earth2mip.s2s.rst Overview of the main contents of the earth2mip.s2s package. ```APIDOC ## earth2mip.s2s Module Contents ### Description This section outlines the members, undocumented members, and inheritance hierarchy of the main earth2mip.s2s module. ### Members - s2s - score - terciles ### Undocumented Members - s2s - score - terciles ### Inheritance - earth2mip.s2s.s2s - earth2mip.s2s.score.score - earth2mip.s2s.terciles.terciles ``` -------------------------------- ### Run Medium-Range Inference CLI Source: https://github.com/nvidia/earth2mip/blob/main/docs/userguide/clis.rst Use this CLI to run deterministic medium-range inference. It corresponds to the score_deterministic API. ```bash python -m earth2mip.inference_medium_range ``` -------------------------------- ### earth2mip.networks.fcnv2_sm Module Source: https://github.com/nvidia/earth2mip/blob/main/docs/modules/earth2mip.networks.rst Details about the FCNv2_SM network module. ```APIDOC ## earth2mip.networks.fcnv2_sm Module ### Description This module contains the Fully Convolutional Network version 2 (FCNv2) with a Small Model (SM) configuration. ### Members - `members`: Lists all public members of the module. - `undoc-members`: Includes members that are not documented. - `show-inheritance`: Shows the inheritance hierarchy for classes within the module. ``` -------------------------------- ### earth2mip.datasets Source: https://github.com/nvidia/earth2mip/blob/main/docs/modules/earth2mip.datasets.rst Documentation for the main datasets module. ```APIDOC ## earth2mip.datasets ### Description Main module for accessing various datasets within the Earth2MIP project. ### Module Members This module aggregates functionalities from sub-modules like ERA5, hindcast, and zarr_directory. Use the `:members:` directive to see all available functions and classes. ``` -------------------------------- ### Implement a Persistence Forecast Source: https://github.com/nvidia/earth2mip/blob/main/docs/userguide/concepts.rst Defines a custom forecast class inheriting from Forecast that encapsulates initialization and time-stepping logic. ```python from earth2mip.forecasts import Forecast import datetime class PeristenceForecast(Forecast): # only corresponds to out_channel_names channel_names = ["a", "b", "c"] def __init__(self, initial_data: Mapping[datetime.datetime, np.ndarray]): self.initial_data = initial_data def __getitem__(self, i): initial_time = datetime.datetime(2018, 1, 1) lead_dt = init_dt = datetime.timedelta(hours=12) time = initial_time + init_dt * i x = self.initial_data[time] while True: yield x ``` -------------------------------- ### Run Ensemble Inference CLI Source: https://github.com/nvidia/earth2mip/blob/main/docs/userguide/clis.rst Execute ensemble inference using this CLI. It is associated with the run_inference API. ```bash python -m earth2mip.inference_ensemble ``` -------------------------------- ### Run Deterministic Inference Source: https://context7.com/nvidia/earth2mip/llms.txt Execute a deterministic forecast using run_basic_inference and a data source. Results are returned as an Xarray DataArray, which can be saved to NetCDF. ```python import datetime from earth2mip.networks import get_model from earth2mip.inference_ensemble import run_basic_inference from earth2mip.initial_conditions import cds # Load model time_loop = get_model("e2mip://dlwp", device="cuda:0") # Create data source for initial conditions from Climate Data Store data_source = cds.DataSource(time_loop.in_channel_names) # Run inference for 10 time steps starting from a specific date time = datetime.datetime(2018, 1, 1) ds = run_basic_inference( time_loop, n=10, # Number of forecast steps data_source=data_source, time=time, ) # Result is an xarray DataArray with dimensions (time, history, channel, lat, lon) print(ds) # # Save to NetCDF ds.to_netcdf("forecast_output.nc") # Select specific channel for analysis z500 = ds.sel(channel="z500") t2m = ds.sel(channel="t2m") ``` -------------------------------- ### earth2mip.xarray.metrics Module Source: https://github.com/nvidia/earth2mip/blob/main/docs/modules/earth2mip.xarray.rst Provides functionality for calculating metrics on xarray datasets within the Earth2Mip framework. ```APIDOC ## earth2mip.xarray.metrics ### Description This module provides utility functions and classes for computing performance and accuracy metrics on xarray-based climate data objects. ### Members - The module includes all members defined within the `earth2mip.xarray.metrics` namespace, including functions for statistical analysis and error calculation. ``` -------------------------------- ### Open and Analyze Ensemble Forecast Data Source: https://context7.com/nvidia/earth2mip/llms.txt Opens a NetCDF ensemble forecast file and calculates ensemble mean and standard deviation. Requires xarray for data manipulation. ```python import xarray def open_ensemble(f, domain, chunks={"time": 1}): time = xarray.open_dataset(f).time root = xarray.open_dataset(f, decode_times=False) ds = xarray.open_dataset(f, chunks=chunks, group=domain) ds.attrs = root.attrs return ds.assign_coords(time=time) ds = open_ensemble("outputs/ensemble_forecast/ensemble_out_0.nc", "global") ensemble_mean = ds.mean(dim="ensemble") ensemble_std = ds.std(dim="ensemble") ``` -------------------------------- ### Implement Custom TimeLoop Model Source: https://context7.com/nvidia/earth2mip/llms.txt Define a custom weather model by inheriting from the TimeLoop protocol to integrate with Earth-2 MIP inference infrastructure. ```python import datetime from typing import Iterator, Tuple, Any, List, Optional import torch import earth2mip.grid from earth2mip.time_loop import TimeLoop class CustomWeatherModel(TimeLoop): """Example custom TimeLoop implementation""" def __init__(self, model: torch.nn.Module, device: str = "cuda:0"): self.model = model.to(device) self._device = torch.device(device) # Define model properties self._in_channels = ["z500", "z850", "t850", "u10m", "v10m", "t2m"] self._out_channels = ["z500", "z850", "t850", "u10m", "v10m", "t2m"] self._grid = earth2mip.grid.equiangular_lat_lon_grid(721, 1440) self._time_step = datetime.timedelta(hours=6) @property def in_channel_names(self) -> List[str]: return self._in_channels @property def out_channel_names(self) -> List[str]: return self._out_channels @property def grid(self) -> earth2mip.grid.LatLonGrid: return self._grid @property def n_history_levels(self) -> int: return 1 @property def history_time_step(self) -> datetime.timedelta: return datetime.timedelta(hours=0) @property def time_step(self) -> datetime.timedelta: return self._time_step @property def device(self) -> torch.device: return self._device @property def dtype(self) -> torch.dtype: return torch.float32 def __call__( self, time: datetime.datetime, x: torch.Tensor, restart: Optional[Any] = None ) -> Iterator[Tuple[datetime.datetime, torch.Tensor, Any]]: """ Args: x: Initial condition (batch, history, channel, lat, lon) time: Start datetime restart: Optional restart state Yields: (time, output, restart) tuples """ # Yield initial state yield time, x[:, -1], {"x": x, "time": time} current_x = x current_time = time while True: with torch.no_grad(): # Run model forward pass output = self.model(current_x[:, -1]) # (batch, channel, lat, lon) current_time += self._time_step current_x = output.unsqueeze(1) # Add history dimension restart_data = {"x": current_x, "time": current_time} yield current_time, output, restart_data # Usage with Earth-2 MIP infrastructure # model = CustomWeatherModel(my_neural_network, device="cuda:0") # ds = run_basic_inference(model, n=10, data_source=data_source, time=start_time) ``` -------------------------------- ### earth2mip.datasets.era5 Source: https://github.com/nvidia/earth2mip/blob/main/docs/modules/earth2mip.datasets.rst Documentation for the ERA5 dataset module. ```APIDOC ## earth2mip.datasets.era5 ### Description Provides access to ERA5 dataset functionalities. ### Module Members This module exposes various members for interacting with ERA5 data. Use the `:members:` directive to see all available functions and classes. ``` -------------------------------- ### earth2mip.s2s.terciles Module Source: https://github.com/nvidia/earth2mip/blob/main/docs/modules/earth2mip.s2s.rst Details about the terciles submodule within the earth2mip.s2s package. ```APIDOC ## earth2mip.s2s.terciles Module ### Description This module provides functionality for calculating terciles within the earth2mip.s2s package. ### Members - terciles - _terciles ### Undocumented Members - terciles - _terciles ### Inheritance - earth2mip.s2s.terciles.terciles ``` -------------------------------- ### earth2mip.xarray Package Source: https://github.com/nvidia/earth2mip/blob/main/docs/modules/earth2mip.xarray.rst Core package for handling xarray integration in Earth2Mip. ```APIDOC ## earth2mip.xarray ### Description This package serves as the primary interface for xarray integration, facilitating the conversion and manipulation of model outputs into xarray datasets. ### Members - Includes all public classes, functions, and submodules defined in the `earth2mip.xarray` package. ``` -------------------------------- ### Run Lagged Ensembles CLI Source: https://github.com/nvidia/earth2mip/blob/main/docs/userguide/clis.rst This CLI is for running lagged ensembles. Note that it does not directly correspond to a specific API function listed here. ```bash python -m earth2mip.lagged_ensembles ``` -------------------------------- ### earth2mip.networks.fcn Module Source: https://github.com/nvidia/earth2mip/blob/main/docs/modules/earth2mip.networks.rst Details about the FCN network module. ```APIDOC ## earth2mip.networks.fcn Module ### Description This module contains the Fully Convolutional Network (FCN) implementation. ### Members - `members`: Lists all public members of the module. - `undoc-members`: Includes members that are not documented. - `show-inheritance`: Shows the inheritance hierarchy for classes within the module. ``` -------------------------------- ### run_inference Source: https://github.com/nvidia/earth2mip/blob/main/docs/userguide/inference-workflows.rst Executes an ensemble inference workflow using a specified data source and time loop. ```APIDOC ## run_inference ### Description Runs an ensemble inference workflow combining features like ensemble initialization and post-processing. ### Parameters - **data_source** (DataSource) - Required - The source of initial conditions. - **time_loop** (TimeLoop) - Required - The time loop object defining the inference process. ``` -------------------------------- ### Sign off on a Git commit Source: https://github.com/nvidia/earth2mip/blob/main/CONTRIBUTING.md Use the --signoff flag to certify your contribution according to the Developer Certificate of Origin. ```bash git commit -s -m "Add cool feature." ``` -------------------------------- ### Score Deterministic Model Forecasts with RMSE and ACC Source: https://context7.com/nvidia/earth2mip/llms.txt Computes deterministic skill scores (RMSE and ACC) by comparing model forecasts against verification data. Requires model, initial times, and a data source for verification. ERA5 data in HDF5 format is recommended for the data source. ```python import datetime import numpy as np from earth2mip.networks import get_model from earth2mip.inference_medium_range import score_deterministic, save_scores, time_average_metrics from earth2mip.initial_conditions import hdf5 from earth2mip.forecast_metrics_io import read_metrics # Load model model = get_model("e2mip://dlwp", device="cuda:0") # Create HDF5 data source for verification (requires ERA5 data in HDF5 format) # Set ERA5_HDF5 environment variable or pass path directly datasource = hdf5.DataSource.from_path( root="/path/to/era5/h5/files", channel_names=model.in_channel_names ) # Define initial times for scoring (e.g., every 30 days over a year) time = datetime.datetime(2017, 1, 2, 0) initial_times = [time + datetime.timedelta(days=30 * i) for i in range(12)] # Quick scoring with score_deterministic (returns xarray Dataset) scores = score_deterministic( model, n=28, # 28 * 6hr = 7-day forecast initial_times=initial_times, data_source=datasource, time_mean=datasource.time_means, # Climatology for ACC calculation ) print(scores) # # Dimensions: (lead_time: 29, channel: N, initial_time: 12) # Data variables: # acc (lead_time, channel) - Anomaly Correlation Coefficient # rmse (lead_time, channel) - Root Mean Square Error # Access specific metrics z500_rmse = scores.rmse.sel(channel="z500") t2m_acc = scores.acc.sel(channel="t2m") # For large-scale scoring, use save_scores for distributed computation output_dir = "outputs/scoring_results" save_scores( model, n=28, initial_times=initial_times, data_source=datasource, time_mean=datasource.time_means, output_directory=output_dir, rank=0, world_size=1, device="cuda:0", ) # Read and aggregate metrics from CSV files series = read_metrics(output_dir) dataset = time_average_metrics(series) ``` -------------------------------- ### earth2mip.s2s.score Module Source: https://github.com/nvidia/earth2mip/blob/main/docs/modules/earth2mip.s2s.rst Details about the score submodule within the earth2mip.s2s package. ```APIDOC ## earth2mip.s2s.score Module ### Description This module contains functionality related to scoring within the earth2mip.s2s package. ### Members - score - _score ### Undocumented Members - score - _score ### Inheritance - earth2mip.s2s.score.score ```