### Install and Run Pre-commit Source: https://github.com/earthaccess-dev/earthaccess/blob/main/docs/contributor/development.md Install pre-commit, enable it to run automatically on commits, and manually run all pre-commit checks. ```bash python -m pip install pre-commit ``` ```bash pre-commit install ``` ```bash pre-commit run -a ``` -------------------------------- ### Example GitHub Label Link Source: https://github.com/earthaccess-dev/earthaccess/blob/main/docs/contributor/triaging.md An example demonstrating how to link to the 'good first issue' label in the earthaccess-dev/earthaccess repository. ```markdown https://github.com/earthaccess-dev/earthaccess/labels/good%20first%20issue ``` -------------------------------- ### Install earthaccess using pip Source: https://github.com/earthaccess-dev/earthaccess/blob/main/README.md Install the earthaccess library using pip. This command should be run in your terminal. ```bash python -m pip install earthaccess ``` -------------------------------- ### Install Earthaccess from PyPI Source: https://github.com/earthaccess-dev/earthaccess/blob/main/docs/contributor/releasing.md After a package is visible on PyPI, verify its installation using this command. Replace X.Y.Z with the specific version number. ```python python -m pip install earthaccess==vX.Y.Z ``` -------------------------------- ### Verify earthaccess Installation Source: https://github.com/earthaccess-dev/earthaccess/blob/main/docs/user/quick-start.md Check if earthaccess is installed correctly by importing it and checking its version in a Python interpreter. ```python $ python Python 3.12.1 | packaged by conda-forge | (main, Dec 23 2023, 08:03:24) [GCC 12.3.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import earthaccess >>> earthaccess.__version__ '0.8.2' ``` -------------------------------- ### Install earthaccess with Mamba Source: https://github.com/earthaccess-dev/earthaccess/blob/main/docs/user/quick-start.md Install the latest release of earthaccess using the mamba package manager. Requires Python 3.8 or higher. ```bash mamba install -c conda-forge earthaccess ``` -------------------------------- ### Setup and Activate venv Environment Source: https://github.com/earthaccess-dev/earthaccess/blob/main/docs/contributor/development.md Create and activate a Python virtual environment using venv. This is recommended for IDE users. ```bash python -m venv .venv source .venv/bin/activate ``` -------------------------------- ### Install Earthaccess in Editable Mode with Development Dependencies (venv) Source: https://github.com/earthaccess-dev/earthaccess/blob/main/docs/contributor/development.md Install the Earthaccess library in editable mode with optional development, testing, and documentation dependencies using pip. ```bash python -m pip install --editable . --group dev --group test --group docs ``` -------------------------------- ### Setup Python Environment Source: https://github.com/earthaccess-dev/earthaccess/blob/main/docs/user/tutorials/virtual_dataset_tutorial_with_TEMPO_Level3.ipynb Imports necessary libraries for data analysis and visualization, including cartopy, earthaccess, matplotlib, numpy, and xarray. Sets inline plotting format and figure DPI. ```python import cartopy.crs as ccrs import earthaccess import matplotlib.pyplot as plt import numpy as np import xarray as xr from matplotlib import rcParams %config InlineBackend.figure_format = 'jpeg' rcParams["figure.dpi"] = ( 80 # Reduce figure resolution to keep the saved size of this notebook low. ) ``` -------------------------------- ### Install earthaccess with Conda Source: https://github.com/earthaccess-dev/earthaccess/blob/main/docs/user/quick-start.md Install the latest release of earthaccess using the conda package manager. Requires Python 3.8 or higher. ```bash conda install -c conda-forge earthaccess ``` -------------------------------- ### Search for Data Source: https://github.com/earthaccess-dev/earthaccess/blob/main/docs/user/quick-start.md Search for specific datasets using parameters like short_name, bounding_box, and temporal range. This example searches for ATL06 data. ```python results = earthaccess.search_data( short_name='ATL06', bounding_box=(-10, 20, 10, 50), temporal=("1999-02", "2019-03"), count=10 ) ``` -------------------------------- ### Run Nox with Pipx Source: https://github.com/earthaccess-dev/earthaccess/blob/main/docs/contributor/development.md Execute development tasks using nox, either by running it directly or after installing it via pipx. Nox manages temporary virtual environments for each task. ```bash pipx run nox [NOX_ARGS] ``` ```bash pipx install nox nox [NOX_ARGS] ``` -------------------------------- ### Handle earthaccess.login() Exceptions Source: https://github.com/earthaccess-dev/earthaccess/blob/main/docs/user/backwards-compatibility.md When upgrading to earthaccess 0.14.0, earthaccess.login() will raise an exception on invalid credentials. This example shows how to catch and ignore these exceptions if necessary, though it is not recommended. ```python try: earthaccess.login() except Exception: pass ``` -------------------------------- ### Get Data Links (On-Prem) Source: https://github.com/earthaccess-dev/earthaccess/blob/main/docs/user/howto/access-data.md Retrieve external links for on-premise datasets. This is useful when data is not directly accessible via S3. ```python # or if the data is an on-prem dataset data_links = [granule.data_links(access="external") for granule in results] ``` -------------------------------- ### Run Nox Without Environment Backend Source: https://github.com/earthaccess-dev/earthaccess/blob/main/docs/contributor/development.md Force nox to use the active Python environment instead of creating a new one, and skip package installation. Useful for IDE integration. ```bash nox -fb none --no-install ``` -------------------------------- ### Query Data Collections Anonymously and Authenticated Source: https://github.com/earthaccess-dev/earthaccess/blob/main/docs/user/tutorials/restricted-datasets.ipynb Demonstrates how to query for data collections. The first example performs an anonymous query, while the second uses an authenticated Auth object, which is necessary for restricted datasets. ```python # An anonymous query to CMR Query = DataCollections().keyword('elevation') # An authenticated query to CMR Query = DataCollections(auth).keyword('elevation') ``` -------------------------------- ### Query Data Granules Anonymously and Authenticated Source: https://github.com/earthaccess-dev/earthaccess/blob/main/docs/user/tutorials/restricted-datasets.ipynb Shows how to query for data granules. Similar to DataCollections, the first example is anonymous, and the second uses an authenticated Auth object for restricted data. ```python # An anonymous query to CMR Query = DataGranules().keyword('elevation') # An authenticated query to CMR Query = DataGranules(auth).keyword('elevation') ``` -------------------------------- ### Get Data Links (Cloud-Hosted) Source: https://github.com/earthaccess-dev/earthaccess/blob/main/docs/user/howto/access-data.md Retrieve S3 links for cloud-hosted data. Use 'direct' access if in the us-west-2 region for direct cloud access. ```python # if the data set is cloud-hosted there will be S3 links available. The access parameter accepts "direct" or "external". Direct access is only possible if you are in the us-west-2 region in the cloud. data_links = [granule.data_links(access="direct") for granule in results] ``` -------------------------------- ### Open Remote Files with xarray Source: https://github.com/earthaccess-dev/earthaccess/blob/main/docs/user/howto/edl.ipynb Opens a list of remote files using earthaccess and then opens the first file with xarray, accessing a specific group. This requires the xarray library to be installed. ```python import xarray as xr # earthaccess can open a list of files files = earthaccess.open([lpcloud_url]) ds = xr.open_dataset(files[0], group="sensor_band_parameters") ds ``` -------------------------------- ### Serve Documentation Locally Source: https://github.com/earthaccess-dev/earthaccess/blob/main/docs/contributor/development.md Run this command to serve the documentation locally. The script will automatically re-render the docs when changes are detected. ```bash nox -s serve-docs ``` -------------------------------- ### Serve Documentation with MkDocs Options Source: https://github.com/earthaccess-dev/earthaccess/blob/main/docs/contributor/development.md To speed up documentation builds, pass these options to MkDocs via the nox script. This disables strict checking and enables dirty builds. ```bash nox -s serve-docs -- --dirty --no-strict ``` -------------------------------- ### Get Dataset Concept ID Source: https://github.com/earthaccess-dev/earthaccess/blob/main/docs/user/search.md Retrieves and prints the unique concept ID for the first dataset in the search results. ```python print(results[0].concept_id) ``` -------------------------------- ### Set Earthaccess Virtualize Options Source: https://github.com/earthaccess-dev/earthaccess/blob/main/docs/user/tutorials/virtual_dataset_tutorial_with_TEMPO_Level3.ipynb Configure options for earthaccess.virtualize, including access method, metadata loading, dimension concatenation, and variable/coordinate selection. Ensure 'load=True' for immediate metadata loading and indexing. ```python open_options = { "access": "indirect", # access to cloud data (faster in AWS with "direct") "load": True, # Load metadata immediately (required for indexing) "concat_dim": "time", # Concatenate files along the time dimension "data_vars": "minimal", # Only load data variables that include the concat_dim "coords": "minimal", # Only load coordinate variables that include the concat_dim "compat": "override", # Avoid coordinate conflicts by picking the first "combine_attrs": "override", # Avoid attribute conflicts by picking the first } ``` -------------------------------- ### Convert GeoDataFrame to Polygon for Search Source: https://github.com/earthaccess-dev/earthaccess/blob/main/docs/user/search.md Example of converting a GeoDataFrame's geometry to the list of tuples format required for the `polygon` search parameter in Earthaccess. ```python import geopandas gdf = geopandas.read_file() my_polygon = [(lon,lat) for lon, lat in zip(*gdf.loc[0].geometry.exterior.xy)] ``` -------------------------------- ### Virtualize TEMPO NO2 Data by Group Source: https://github.com/earthaccess-dev/earthaccess/blob/main/docs/user/tutorials/dmrpp-virtualizarr.ipynb Creates a virtualized dataset for the first result of the TEMPO NO2 search, specifically targeting the 'product' group. ```python earthaccess.virtualize(results[0], group="product") ``` -------------------------------- ### Initialize and Authenticate Earthaccess Source: https://github.com/earthaccess-dev/earthaccess/blob/main/docs/user/tutorials/SSL.ipynb This snippet initializes the earthaccess library and handles user authentication. It checks if the user is already authenticated and prompts for interactive login if necessary, persisting credentials. ```python import earthaccess print(f"using earthaccess v{earthaccess.__version__}") auth = earthaccess.login() # are we authenticated? if not auth.authenticated: # ask for credentials and persist them in a .netrc file auth.login(strategy="interactive", persist=True) ``` -------------------------------- ### Earthaccess Login and Imports Source: https://github.com/earthaccess-dev/earthaccess/blob/main/docs/user/howto/fsspec.ipynb Sets up the environment for Earthaccess operations by importing necessary libraries and logging in. This is a prerequisite for most Earthaccess functionalities. ```python import time import earthaccess as ea import matplotlib.pyplot as plt import numpy as np import pandas as pd import tqdm import xarray as xr ea.login() ``` -------------------------------- ### Get Dataset Concept ID Source: https://github.com/earthaccess-dev/earthaccess/blob/main/docs/user/search.md Retrieve the unique concept identifier for a dataset from the search results. The concept ID is accessed via the `concept_id()` method on the result object. ```python results[0].concept_id() ``` -------------------------------- ### Login using Environment Variables Strategy Source: https://github.com/earthaccess-dev/earthaccess/blob/main/docs/user/howto/authenticate.md Authenticate using environment variables EARTHDATA_USERNAME and EARTHDATA_PASSWORD. Alternatively, set EARTHDATA_TOKEN for an existing Earthdata Login token. ```python auth = earthaccess.login(strategy="environment") ``` -------------------------------- ### Accessing Collection Services Source: https://github.com/earthaccess-dev/earthaccess/blob/main/docs/api/collections/collections-services.md Demonstrates how to instantiate and access the DataServices object for collection management. This is the entry point for interacting with collection-related functionalities. ```python from earthaccess import DataServices dataservices = DataServices() ``` -------------------------------- ### Login using .netrc Strategy Source: https://github.com/earthaccess-dev/earthaccess/blob/main/docs/user/howto/authenticate.md Explicitly use credentials from a .netrc file for authentication. Ensure your .netrc file is correctly configured. ```python auth = earthaccess.login(strategy="netrc") ``` -------------------------------- ### Make HTTPS Request with Session Source: https://github.com/earthaccess-dev/earthaccess/blob/main/docs/user/howto/edl.ipynb Uses the authenticated HTTPS session to make a GET request to a specified URL, retrieving a range of bytes. This demonstrates accessing data over HTTP. ```python headers = {"Range": "bytes=0-100"} r = session.get(lpcloud_url, headers=headers) r.text ``` -------------------------------- ### Import Libraries and Authenticate Source: https://github.com/earthaccess-dev/earthaccess/blob/main/docs/user/tutorials/emit-earthaccess.ipynb Imports necessary libraries and authenticates with NASA Earthdata. ```python from pprint import pprint import earthaccess import xarray as xr print(f"using earthaccess version {earthaccess.__version__}") auth = earthaccess.login() ``` -------------------------------- ### Initialize Earthaccess Auth Source: https://github.com/earthaccess-dev/earthaccess/blob/main/docs/user/tutorials/restricted-datasets.ipynb Initializes the Auth class from the earthaccess library. This is the first step for any authenticated interaction with NASA EDL. ```python from earthaccess import Auth, DataCollections, DataGranules, Store auth = Auth() ``` -------------------------------- ### Search for EMIT Datasets Source: https://github.com/earthaccess-dev/earthaccess/blob/main/docs/user/tutorials/emit-earthaccess.ipynb Searches for cloud-hosted EMIT L2A Reflectance datasets using `search_datasets` and prints a summary of each dataset found. ```python results = earthaccess.search_datasets(short_name="EMITL2ARFL", cloud_hosted=True) # Let's print our datasets for dataset in results: pprint(dataset.summary()) ``` -------------------------------- ### Get Data Links (Out of Region) Source: https://github.com/earthaccess-dev/earthaccess/blob/main/docs/user/tutorials/SSL.ipynb Retrieves the HTTPS URL for a data granule, suitable for accessing data from outside the AWS region. This is useful when the data is cloud-hosted but needs to be accessed remotely. ```python # the collection is cloud hosted, but we can access it out of AWS with the regular HTTPS URL granules[0].data_links(access="out_of_region") ``` -------------------------------- ### Login with Environment or Netrc Strategy Source: https://github.com/earthaccess-dev/earthaccess/blob/main/docs/user/tutorials/restricted-datasets.ipynb Attempts to log in using environment variables (EARTHDATA_USERNAME, EARTHDATA_PASSWORD). If authentication fails, it falls back to using a .netrc file. ```python auth.login(strategy="environment") # are we authenticated? if not auth.authenticated: auth.login(strategy="netrc") ``` -------------------------------- ### Create .netrc using Earthaccess Source: https://github.com/earthaccess-dev/earthaccess/blob/main/docs/user/authenticate.md This command prompts for your Earthdata Login credentials and automatically creates a .netrc file for persistent authentication. ```python earthaccess.login(persist=True) ``` -------------------------------- ### Create .netrc file on MacOS/Linux Source: https://github.com/earthaccess-dev/earthaccess/blob/main/docs/user/authenticate.md Manually create a .netrc file for Earthdata Login credentials on MacOS or Linux systems. Ensure the file has restricted permissions. ```bash echo "machine urs.earthdata.nasa.gov login password " >> $HOME/.netrc chmod 600 $HOME/.netrc ``` -------------------------------- ### Get Data Links (Direct Access) Source: https://github.com/earthaccess-dev/earthaccess/blob/main/docs/user/tutorials/SSL.ipynb Retrieves the direct access URL for a data granule. This is typically used for accessing data within the same cloud environment or when direct S3 access is configured. ```python granules[0].data_links(access="direct") ``` -------------------------------- ### Create Virtual Reference File (Kerchunk) Source: https://github.com/earthaccess-dev/earthaccess/blob/main/docs/user/tutorials/dmrpp-virtualizarr.ipynb Creates a virtual xarray Dataset without loading the full data, then saves it as a Kerchunk JSON file for efficient later loading. ```python %%time mur_vds = earthaccess.virtualize( results, access="indirect", load=False, concat_dim="time", coords="all", compat="override", combine_attrs="drop_conflicts", ) mur_vds ``` -------------------------------- ### Select Single Scan Time for Analysis Source: https://github.com/earthaccess-dev/earthaccess/blob/main/docs/user/tutorials/virtual_dataset_tutorial_with_TEMPO_Level3.ipynb Defines the start and end time for selecting a single scan period. It then subsets a merged dataset ('result_merged') based on longitude, latitude, and time bounds, filtering for data quality. ```python scan_time_start = np.datetime64("2024-01-11T13:00:00") # 1 PM UTC scan_time_end = np.datetime64("2024-01-11T14:00:00") # 2 PM UTC print(f"Analyzing single scan: {scan_time_start} to {scan_time_end} UTC") print("Note: This corresponds to ~8-9 AM local time on the US East Coast") subset_ds = result_merged.sel( { "longitude": slice(lon_bounds[0], lon_bounds[1]), "latitude": slice(lat_bounds[0], lat_bounds[1]), "time": slice(scan_time_start, scan_time_end), }, ).where(result_merged["main_data_quality_flag"] == 0) subset_ds ``` -------------------------------- ### Open Cloud-Hosted Data with Readahead Cache Source: https://github.com/earthaccess-dev/earthaccess/blob/main/docs/user/howto/fsspec.ipynb Open a cloud-hosted HDF5 file using a pre-configured readahead cache and inspect the initial state of the cache. ```python file_handler = ea.open(results, open_kwargs=default_fsspec_open)[0] file_handler.f.cache ``` -------------------------------- ### Get S3 Credentials for Cloud Data Access Source: https://github.com/earthaccess-dev/earthaccess/blob/main/docs/user/authenticate.md Obtain S3 credentials using earthaccess.login() for accessing NASA Earthdata Cloud data with other packages like h5coro. This is useful when not directly using earthaccess for cloud data operations. ```python import earthaccess import xarray as xr import h5coro auth = earthaccess.login() s3_credentials = auth.get_s3_credentials(daac="NSIDC") s3url_atl23 = 'nsidc-cumulus-prod-protected/ATLAS/ATL23/001/2023/03/' \ '01/ATL23_20230401000000_10761801_001_01.h5' ds = xr.open_dataset(s3url_atl23, engine='h5coro', group='/mid_latitude/beam_1', credentials=s3_credentials) ``` -------------------------------- ### Calculate Average Metrics Source: https://github.com/earthaccess-dev/earthaccess/blob/main/docs/user/howto/fsspec.ipynb Aggregates performance data by dataset and mode, calculating mean and standard deviation for time, hit rate, requested megabytes, block size, and file size. Use this to get a consolidated view of performance characteristics. ```python grouped = ( success_df.groupby(["dataset", "mode"]) .agg( time_mean=("time", "mean"), time_std=("time", "std"), hit_rate_mean=("hit_rate", "mean"), requested_mb_mean=("requested_mb", "mean"), block_size=("block_size", "first"), # should be stable file_size_mb=("file_size_mb", "first"), ) .reset_index() ) ``` -------------------------------- ### Interactive Login with Persistence Source: https://github.com/earthaccess-dev/earthaccess/blob/main/docs/user/howto/authenticate.md Prompt the user for Earthdata Login credentials and optionally save them to the .netrc file for future use. ```python auth = earthaccess.login(strategy="interactive", persist=True) ``` -------------------------------- ### Import Libraries and Authenticate Source: https://github.com/earthaccess-dev/earthaccess/blob/main/docs/user/tutorials/dmrpp-virtualizarr.ipynb Imports the necessary earthaccess and xarray libraries and authenticates the user. ```python import earthaccess import xarray as xr auth = earthaccess.login() ``` -------------------------------- ### Query and Count Restricted Granules Source: https://github.com/earthaccess-dev/earthaccess/blob/main/docs/user/tutorials/restricted-datasets.ipynb Constructs an authenticated DataGranules query with specific concept ID, bounding box, and temporal filters. It demonstrates that `hits()` is unreliable for ACL-controlled data and shows how to get the actual count using `len()` on the retrieved granules. ```python Query = ( DataGranules(auth) .concept_id("C2153572614-NSIDC_CPRD") .bounding_box(-134.7, 58.9, -133.9, 59.2) .temporal("2020-03-01", "2020-03-30") ) # Unfortunately the hits() methods will behave the same for granule queries print(f"Granules found with hits(): {Query.hits()}") cloud_granules = Query.get() print(f"Actual number found: {len(cloud_granules)}") ``` -------------------------------- ### Run Benchmark and Collect Results Source: https://github.com/earthaccess-dev/earthaccess/blob/main/docs/user/howto/fsspec.ipynb This code executes the data opening benchmark for a specified number of runs, iterating through popular datasets and comparing optimized vs. non-optimized modes. It collects all results into a list for later processing. Ensure you have authenticated with earthaccess before running. ```python n_runs = 3 print(f"Starting benchmark: {n_runs} runs per dataset and mode\n") benchmarks = [] for run in range(n_runs): print(f"Run {run + 1} of {n_runs}") # Optimized (earthaccess defaults) print("Optimized (blockcache)") for cid in tqdm.tqdm(popular_collections): result = open_file(cid, optimized=True, run_id=run) benchmarks.append(result) # Default (readahead) print("Non-optimized (readahead)") for cid in tqdm.tqdm(popular_collections): result = open_file(cid, optimized=False, run_id=run) benchmarks.append(result) df = pd.DataFrame(benchmarks) ``` -------------------------------- ### List Available Nox Sessions Source: https://github.com/earthaccess-dev/earthaccess/blob/main/docs/contributor/development.md Display all the tasks (sessions) that can be run with nox, as defined in the noxfile.py. Sessions marked with '*' are selected by default. ```bash nox --list ``` -------------------------------- ### Default Earthaccess Login Source: https://github.com/earthaccess-dev/earthaccess/blob/main/docs/user/howto/authenticate.md Use this snippet for the default login behavior, which prioritizes environment variables, then a .netrc file, and finally interactive input. ```python import earthaccess auth = earthaccess.login() ``` -------------------------------- ### Basic Data Streaming Source: https://github.com/earthaccess-dev/earthaccess/blob/main/docs/user/access.md Streams data directly into memory as file-like objects, which is efficient for cloud environments. Progress is shown in interactive sessions. ```python fileobjects = earthaccess.open(results) ``` -------------------------------- ### Manual Earthdata Login Prompt Source: https://github.com/earthaccess-dev/earthaccess/blob/main/docs/user/howto/authenticate.md Manually enter EDL account credentials when prompted if no environment variables or .netrc file are configured. The credentials can be optionally saved to .netrc. ```python import earthaccess earthaccess.login() ``` -------------------------------- ### Search Granules with Basic Filters Source: https://github.com/earthaccess-dev/earthaccess/blob/main/docs/user/howto/search-granules.md Import the earthaccess library and search for granules using dataset short name, version, cloud hosting, bounding box, and temporal filters. Retrieves up to 100 results. ```python import earthaccess results = earthaccess.search_data( short_name = "ATL06", version = "005", cloud_hosted = True, bounding_box = (-10,20,10,50), temporal = ("2020-02", "2020-03"), count = 100 ) ``` -------------------------------- ### Run All Nox Sessions Source: https://github.com/earthaccess-dev/earthaccess/blob/main/docs/contributor/development.md Execute all available nox sessions, including type checking and unit tests, using your local Python environment. Nox manages virtual environments automatically. ```bash nox ``` -------------------------------- ### Download Restricted Granules Source: https://github.com/earthaccess-dev/earthaccess/blob/main/docs/user/tutorials/restricted-datasets.ipynb Initializes a Store object with authentication and uses it to download the specified cloud granules to a local directory. Ensure the './data/C2153572614-NSIDC_CPRD/' path exists or is created. ```python store = Store(auth) files = store.get(cloud_granules, "./data/C2153572614-NSIDC_CPRD/") ``` -------------------------------- ### Virtualize and Merge TEMPO Data Groups Source: https://github.com/earthaccess-dev/earthaccess/blob/main/docs/user/tutorials/virtual_dataset_tutorial_with_TEMPO_Level3.ipynb Open and virtualize individual groups ('root', 'product', 'geolocation') from TEMPO granules using specified options, then merge them into a single xarray Dataset for analysis. This process leverages lazy loading and kerchunk. ```python result_root = earthaccess.virtualize(granules=results, **open_options) result_product = earthaccess.virtualize( granules=results, group="product", **open_options, ) result_geolocation = earthaccess.virtualize( granules=results, group="geolocation", **open_options, ) # merge result_merged = xr.merge([result_root, result_product, result_geolocation]) result_merged ``` ```python result_merged = xr.merge([result_root, result_product, result_geolocation]) result_merged ``` -------------------------------- ### Clone Earthaccess Repository Source: https://github.com/earthaccess-dev/earthaccess/blob/main/docs/contributor/development.md Clone your forked repository to your local machine. Replace {my-username} with your GitHub username. ```bash git clone git@github.com:{my-username}/earthaccess ``` -------------------------------- ### Create _netrc file on Windows Source: https://github.com/earthaccess-dev/earthaccess/blob/main/docs/user/authenticate.md Manually create a _netrc file for Earthdata Login credentials on Windows systems. This involves setting the HOME environment variable and then creating the file. ```bash setx HOME %USERPROFILE% echo "machine urs.earthdata.nasa.gov login password " >> %HOME%\_netrc ``` -------------------------------- ### Compare Data Requested by Caching Strategies Source: https://github.com/earthaccess-dev/earthaccess/blob/main/docs/user/howto/fsspec.ipynb Calculate and print the factor by which data was requested when using different caching strategies, illustrating performance differences. ```python print(f"{int(3120435184 / 184549376)} times more data requested.") ``` -------------------------------- ### Authenticate with Earthdata Login Source: https://github.com/earthaccess-dev/earthaccess/blob/main/docs/user/tutorials/virtual_dataset_tutorial_with_TEMPO_Level3.ipynb Establishes authentication with Earthdata Login. If not already authenticated, it prompts for interactive login and persists credentials. ```python auth = earthaccess.login() if not auth.authenticated: # Ask for credentials and persist them in a .netrc file. auth.login(strategy="interactive", persist=True) print(earthaccess.__version__) ``` -------------------------------- ### Open Dataset with Readahead Cache and Inspect Source: https://github.com/earthaccess-dev/earthaccess/blob/main/docs/user/howto/fsspec.ipynb Open an xarray dataset from an HDF5 file using a pre-configured readahead cache and then inspect the cache's state after data access. ```python ds = xr.open_dataset(file_handler, group="/gt1l/heights") # we print the cache info again file_handler.f.cache ``` -------------------------------- ### Download On-Prem Dataset Granules Source: https://github.com/earthaccess-dev/earthaccess/blob/main/docs/user/howto/onprem.md Searches for granules in the NSIDC-0051 dataset for November 2022 in the northern hemisphere and downloads them to the current directory. Ensure you have authenticated with earthaccess.login() before running. ```python import earthaccess auth = earthaccess.login() results = earthaccess.search_data( short_name='NSIDC-0051', temporal=('2022-11-01', '2022-11-30'), bounding_box=(-180, 0, 180, 90) ) downloaded_files = earthaccess.download( results, local_path='.', ) ``` -------------------------------- ### Display Dataset Summary Source: https://github.com/earthaccess-dev/earthaccess/blob/main/docs/user/search.md Prints a summary of the first result in the search results list. The summary includes key dataset information such as short-name, concept-id, version, file-type, and data access links. ```python from pprint import pprint pprint(result[0].summary()) ``` -------------------------------- ### Login with Interactive Strategy and Persist Credentials Source: https://github.com/earthaccess-dev/earthaccess/blob/main/docs/user/tutorials/restricted-datasets.ipynb Logs into NASA EDL using the interactive strategy, prompting the user for credentials. The `persist=True` argument saves these credentials to a .netrc file for future use. ```python auth.login(strategy="interactive", persist=True) ``` -------------------------------- ### Download Data to Local Folder Source: https://github.com/earthaccess-dev/earthaccess/blob/main/docs/user/howto/access-data.md Download specified search results to a local directory. The library will display the approximate download size and progress. ```python files = earthaccess.download(results, "./local_folder") ``` -------------------------------- ### Interactive Earthaccess Login Source: https://github.com/earthaccess-dev/earthaccess/blob/main/docs/user/authenticate.md Use this method when you do not have .netrc or environment variables set up. It prompts for your Earthdata Login username and password. ```python >>> import earthaccess >>> auth = earthaccess.login() Enter your Earthdata Login username: your_username Enter your Earthdata password: ``` -------------------------------- ### Open Remote Files and Load with xarray Source: https://github.com/earthaccess-dev/earthaccess/blob/main/docs/user/tutorials/file-access.ipynb Opens a list of remote files using earthaccess's `open()` function and then loads the first file into an xarray Dataset, accessing a specific group. This method is efficient for large files when only parts are needed. ```python %%time import xarray as xr files = earthaccess.open(results[0:2]) ds = xr.open_dataset(files[0], group="/gt1r/land_ice_segments") ds ``` -------------------------------- ### Open Cloud Optimized HDF5 with BlockCache Source: https://github.com/earthaccess-dev/earthaccess/blob/main/docs/user/howto/fsspec.ipynb Opens a cloud-optimized HDF5 file using Earthaccess defaults, which now favor BlockCache. This snippet demonstrates the improved performance compared to ReadAheadCache. ```python file_handler = ea.open(results)[0] ds = xr.open_dataset(file_handler, group="/gt1l/heights") file_handler.f.cache ``` -------------------------------- ### Login to Earthaccess Source: https://github.com/earthaccess-dev/earthaccess/blob/main/docs/user/howto/fsspec.ipynb Initiates the authentication process for accessing NASA Earthdata. Ensure you have valid credentials configured. ```python import earthaccess as ea import xarray as xr ea.login() ``` -------------------------------- ### Display Dataset Size Source: https://github.com/earthaccess-dev/earthaccess/blob/main/docs/user/tutorials/dmrpp-virtualizarr.ipynb Prints the size of the virtualized MUR dataset in gigabytes. ```python print(f"{mur.nbytes / 1e9} GB") ``` -------------------------------- ### Create and Activate Conda/Mamba Environment Source: https://github.com/earthaccess-dev/earthaccess/blob/main/docs/contributor/development.md Create and activate a development environment using Conda or Mamba by updating from the environment.yml file. This is recommended for Windows users and JupyterHub environments. ```bash mamba env update -f environment.yml mamba activate earthaccess ``` -------------------------------- ### Search Data by Date String Source: https://github.com/earthaccess-dev/earthaccess/blob/main/docs/user/search.md Find data granules within a specified date range using ISO 8601 formatted strings. ```python results = earthaccess.search_data( short_name="ATL06", temporal=("2025-01-01","2025-01-15"), ) ``` -------------------------------- ### Benchmark Data Opening Function Source: https://github.com/earthaccess-dev/earthaccess/blob/main/docs/user/howto/fsspec.ipynb This function opens a dataset using earthaccess, either with default optimized settings or with specified readahead cache settings. It measures the time taken to open the dataset and collects cache statistics. Use this function to compare performance between optimized and non-optimized data access. ```python import earthaccess as ea import xarray as xr import time import tqdm import pandas as pd popular_collections = [ "C2723758340-GES_DISC", "C2724036633-LARC_CLOUD", "C3385049989-OB_CLOUD", "C2532426483-ORNL_CLOUD", "C2670138092-NSIDC_CPRD", ] collection_labels = { "C2723758340-GES_DISC": "GPM Precipitation (GES DISC)", "C2724036633-LARC_CLOUD": "TEMPO L2 (LARC)", "C3385049989-OB_CLOUD": "PACE OCI (OB)", "C2532426483-ORNL_CLOUD": "Daymet daily (ORNL)", "C2670138092-NSIDC_CPRD": "ATL06 Lidar (NSIDC)", } def open_file(concept_id: str, optimized: bool, run_id: int = 0): default_fsspec_open = { "cache_type": "readahead", "block_size": 5 * 1024 * 1024, # 5MB } results = ea.search_data(concept_id=concept_id, cloud_hosted=True, count=1) if not results: return { "concept_id": concept_id, "optimized": optimized, "run_id": run_id, "time": None, "block_size": None, "file_size": None, "cache_hits": None, "cache_misses": None, "total_requested_bytes": None, "error": "No files found", } try: if not optimized: f_h = ea.open(results, open_kwargs=default_fsspec_open)[0] else: f_h = ea.open(results)[0] # Uses earthaccess's blockcache + adaptive block size t_start = time.time() ds = xr.open_datatree(f_h, phony_dims="sort", decode_timedelta=True) execution_time = time.time() - t_start ds.close() cache = f_h.f.cache return { "concept_id": concept_id, "optimized": optimized, "run_id": run_id, "time": execution_time, "block_size": getattr(cache, "blocksize", None), "file_size": getattr(cache, "size", None), "cache_hits": getattr(cache, "hit_count", 0), "cache_misses": getattr(cache, "miss_count", 0), "total_requested_bytes": getattr(cache, "total_requested_bytes", 0), "error": None, } except Exception as e: return { "concept_id": concept_id, "optimized": optimized, "run_id": run_id, "time": None, "block_size": None, "file_size": None, "cache_hits": None, "cache_misses": None, "total_requested_bytes": None, "error": str(e), } ``` -------------------------------- ### Access NASA Earth Science Data Source: https://github.com/earthaccess-dev/earthaccess/blob/main/README.md Demonstrates the three main steps to access NASA Earth Science data using earthaccess: login, search, and download. Ensure you have the library imported. ```python import earthaccess # 1. Login earthaccess.login() # 2. Search results = earthaccess.search_data( short_name='ATL06', # ATLAS/ICESat-2 L3A Land Ice Height bounding_box=(-10, 20, 10, 50), # Only include files in area of interest... temporal=("1999-02", "2019-03"), # ...and time period of interest. count=10 ) # 3. Access files = earthaccess.download(results, "/tmp/my-download-folder") ``` -------------------------------- ### Open Data Files with Earthaccess Source: https://github.com/earthaccess-dev/earthaccess/blob/main/docs/user/tutorials/SSL.ipynb Opens a list of granules or URLs and creates Python File-like objects. This function utilizes S3FS for AWS or regular HTTPS for external access. ```python import earthaccess import xarray as xr fileset = earthaccess.open(granules) print(f" Using {type(fileset[0])} filesystem") ds = xr.open_mfdataset(fileset, chunks={}) ds ``` -------------------------------- ### Load and Merge EMIT Data with xarray Source: https://github.com/earthaccess-dev/earthaccess/blob/main/docs/user/tutorials/emit-earthaccess.ipynb Loads reflectance, sensor band parameters, and location data from a specific file handler using xarray, then merges them into a single dataset and assigns coordinates. ```python %%time # we can use any file from the array file_p = file_handlers[4] refl = xr.open_dataset(file_p) wvl = xr.open_dataset(file_p, group="sensor_band_parameters") loc = xr.open_dataset(file_p, group="location") ds = xr.merge([refl, loc]) ds = ds.assign_coords( { "downtrack": (["downtrack"], refl.downtrack.data), "crosstrack": (["crosstrack"], refl.crosstrack.data), **wvl.variables, }, ) ds ``` -------------------------------- ### Search for Cloud-Hosted Datasets Source: https://github.com/earthaccess-dev/earthaccess/blob/main/docs/user/tutorials/SSL.ipynb This code searches for datasets matching keywords and filters for cloud-hosted data. It then prints a summary and abstract for the first two collections found. ```python from pprint import pprint # We'll get 4 collections that match with our keywords collections = earthaccess.search_datasets( keyword="SEA SURFACE HEIGHT", cloud_hosted=True, count=4, ) # Let's print 2 collections for collection in collections[0:2]: # pprint(collection.summary()) print( pprint(collection.summary()), collection.abstract(), "\n", collection["umm"]["DOI"], "\n\n", ) ``` -------------------------------- ### Print Service Metadata for Datasets Source: https://github.com/earthaccess-dev/earthaccess/blob/main/docs/user/howto/search-services.md Iterate through the search results and print the service metadata for each dataset. This helps in understanding the available processing options. ```python for dataset in datasets: print(dataset.services()) ``` -------------------------------- ### Link to GitHub Label Source: https://github.com/earthaccess-dev/earthaccess/blob/main/docs/contributor/triaging.md Use this syntax to create a clickable link to a specific label within a GitHub repository. This helps users quickly access label information. ```markdown https://github.com/username/repository/labels/label-name ``` -------------------------------- ### Run a Specific Nox Session Source: https://github.com/earthaccess-dev/earthaccess/blob/main/docs/contributor/development.md Execute a single nox session, such as 'integration-tests'. Arguments can be passed to the underlying command using '-- [ARGS]'. ```bash nox -s integration-tests ``` -------------------------------- ### Plotting Cache Hit Rate Comparison Source: https://github.com/earthaccess-dev/earthaccess/blob/main/docs/user/howto/fsspec.ipynb Generates a bar chart comparing the cache hit rate of the default read method versus an optimized block cache method across different datasets. It visualizes the performance difference and requires matplotlib and numpy. ```python plot_data = grouped.set_index(["dataset", "mode"])["hit_rate_mean"].unstack("mode") datasets = plot_data.index x = np.arange(len(datasets)) width = 0.35 plt.figure(figsize=(10, 6)) bars1 = plt.bar( x - width / 2, plot_data["Default (readahead)"], width, label="Default (readahead)", color="gray", alpha=0.8, ) bars2 = plt.bar( x + width / 2, plot_data["Optimized"], width, label="Optimized (blockcache)", color="seagreen", alpha=0.9, ) plt.ylabel("Cache Hit Rate", fontsize=12) plt.title( "Cache Hit Rate: Optimized vs Default (Averaged Over 3 Runs)", fontsize=14, pad=20, ) plt.xticks(x, [lbl.replace(" ", "\n") for lbl in datasets], rotation=0) plt.ylim(0, 1.05) plt.legend(fontsize=11) plt.grid(axis="y", linestyle="--", alpha=0.4) def add_labels(bars): for bar in bars: height = bar.get_height() plt.text( bar.get_x() + bar.get_width() / 2, height + 0.02, f"{height:.3f}", ha="center", va="bottom", fontsize=9, ) add_labels(bars1) add_labels(bars2) plt.tight_layout() plt.show() ``` -------------------------------- ### Create Feature Branch Source: https://github.com/earthaccess-dev/earthaccess/blob/main/docs/contributor/development.md Create a new branch for feature development or bug fixes. Use a descriptive name for the branch. ```bash git switch -c update-contributing-docs ``` -------------------------------- ### Define and Apply Preprocessing Function for Virtualization Source: https://github.com/earthaccess-dev/earthaccess/blob/main/docs/user/tutorials/dmrpp-virtualizarr.ipynb Defines a preprocessing function to expand attributes like 'cycle_number' and 'pass_number' into dimensions and then applies this function during the virtualization process. This allows for concatenation along the newly created 'pass_num' dimension. ```python %%time def preprocess(ds: xr.Dataset) -> xr.Dataset: # Add cycle number and pass_number as dimensions return ds.expand_dims(["cycle_num", "pass_num"]).assign_coords( cycle_num=[ds.attrs["cycle_number"]], pass_num=[ds.attrs["pass_number"]], ) swot = earthaccess.virtualize( results, access="indirect", load=False, preprocess=preprocess, concat_dim="pass_num", coords="all", compat="override", combine_attrs="drop_conflicts", ) swot ``` -------------------------------- ### Search for Data by Criteria Source: https://github.com/earthaccess-dev/earthaccess/blob/main/docs/user/howto/fsspec.ipynb Searches for data collections based on short name, version, cloud hosting availability, temporal range, and spatial bounding box. Returns a list of matching data collections. ```python results = ea.search_data( short_name="ATL03", version="006", cloud_hosted=True, temporal=("2025-01", "2025-08"), bounding_box=(33.98, 31.19, 34.65, 31.51), count=1, ) results[0] ``` -------------------------------- ### Search Datasets Source: https://github.com/earthaccess-dev/earthaccess/blob/main/docs/user/search.md Search for datasets using criteria like short name, cloud hosting, and temporal range. This is a prerequisite for discovering services associated with specific datasets. ```python results = earthaccess.search_datasets( short_name="MUR-JPL-L4-GLOB-v4.1", cloud_hosted=True, temporal=("2024-02-27T00:00:00Z", "2024-02-29T23:59:59Z"), ) ``` -------------------------------- ### Update Conda Environment and Activate Source: https://github.com/earthaccess-dev/earthaccess/blob/main/docs/contributor/development.md Update the Conda environment from environment.yml, activate it, and remove the old environment if the name has changed. ```bash mamba env update -f environment.yml mamba activate earthaccess mamba env remove -n earthaccess-dev ``` -------------------------------- ### Search Services Directly by Provider and Keyword Source: https://github.com/earthaccess-dev/earthaccess/blob/main/docs/user/howto/search-services.md Search for services directly using provider and keyword filters. This is useful when you know the service provider or specific keywords related to the service. ```python services = earthaccess.search_services(provider="POCLOUD", keyword="COG") ``` -------------------------------- ### Set Environment Variables for Earthdata Login (Windows CMD) Source: https://github.com/earthaccess-dev/earthaccess/blob/main/docs/user/authenticate.md Set environment variables for the current CMD session on Windows. For permanent settings, use the 'Environment Variables' control panel. ```cmd setx EARTHDATA_USERNAME "username" setx EARTHDATA_PASSWORD "password" ``` -------------------------------- ### Load Dataset from Kerchunk JSON Source: https://github.com/earthaccess-dev/earthaccess/blob/main/docs/user/tutorials/dmrpp-virtualizarr.ipynb Loads the virtual reference file ('mur_kerchunk.json') using xarray with the 'kerchunk' engine, configuring fsspec for HTTPS access. The loaded dataset is then printed. ```python %%time fs = earthaccess.get_fsspec_https_session() ds = xr.open_dataset( "mur_kerchunk.json", engine="kerchunk", chunks={}, storage_options={ "remote_protocol": fs.protocol, "remote_options": fs.storage_options, }, ) print(ds) ``` -------------------------------- ### Virtualize MUR SST Dataset Source: https://github.com/earthaccess-dev/earthaccess/blob/main/docs/user/tutorials/dmrpp-virtualizarr.ipynb Creates a virtualized xarray Dataset for the MUR SST data, loading it indirectly and concatenating along the time dimension. This operation is timed for performance analysis. ```python %%time mur = earthaccess.virtualize( results, access="indirect", load=True, concat_dim="time", coords="all", compat="override", combine_attrs="drop_conflicts", ) mur ```