### Setup and Installation

Source: https://github.com/tum-esm/em27-retrieval-pipeline/blob/main/docs/src/content/docs/guides/usage.mdx

Commands for setting up the project environment, including cloning the repository, creating and activating a virtual environment, and installing dependencies using pip or PDM.

```bash
git clone https://github.com/tum-esm/automated-retrieval-pipeline
python3.11 -m venv .venv
source .venv/bin/activate
pip install ".[dev]"
pdm sync --group dev
```

--------------------------------

### System Dependency Installation

Source: https://github.com/tum-esm/em27-retrieval-pipeline/blob/main/docs/src/content/docs/guides/usage.mdx

Example command for installing necessary system dependencies, such as unzip and gfortran, which are required for the pipeline's operation.

```bash
sudo apt install unzip gfortran
```

--------------------------------

### Configuration File Example

Source: https://github.com/tum-esm/em27-retrieval-pipeline/blob/main/docs/src/content/docs/guides/configuration.mdx

An example of the `config.json` file used to configure the EM27/EU retrieval pipeline. This file specifies parameters for all steps of the pipeline.

```json
{
  "pipeline_name": "EM27/EU Retrieval",
  "data_sources": {
    "interferograms": "/path/to/interferograms",
    "ground_pressure": "/path/to/ground_pressure",
    "atmospheric_profiles": "/path/to/atmospheric_profiles"
  },
  "output_directory": "/path/to/output",
  "log_directory": "/path/to/logs",
  "retrieval_settings": {
    "algorithm": "optimal_estimation",
    "max_iterations": 100,
    "tolerance": 1e-5
  }
}
```

--------------------------------

### MUCCnet Campaign Configuration Example

Source: https://github.com/tum-esm/em27-retrieval-pipeline/blob/main/docs/src/content/docs/guides/directories.mdx

An example JSON configuration for the MUCCnet campaign, specifying the campaign ID, date range, and associated sensor and location IDs.

```json
{
  "campaign_id": "muccnet",
  "from_datetime": "2019-09-13T00:00:00+0000",
  "to_datetime": "2100-01-01T23:59:59+0000",
  "sensor_ids": ["ma", "mb", "mc", "md", "me"],
  "location_ids": ["TUM_I", "FEL", "GRAE", "OBE", "TAU", "DLR_2", "DLR_3"]
}
```

--------------------------------

### JSON Configuration Example

Source: https://github.com/tum-esm/em27-retrieval-pipeline/blob/main/docs/src/content/docs/api-reference/configuration.mdx

Provides an example of the configuration file used by the retrieval pipeline. This JSON file defines various settings required for the pipeline's operation.

```json
{
  "example_setting": "value",
  "another_setting": 123
}
```

--------------------------------

### Bundle Configuration Example

Source: https://github.com/tum-esm/em27-retrieval-pipeline/blob/main/docs/src/content/docs/guides/directories.mdx

An example JSON configuration for generating data bundles. It specifies the destination directory, output formats, time range, retrieval algorithms, atmospheric profiles, and sensor IDs.

```json
{
  "dst_dir": "/some/path/where/the/bundle_should/be/written/to",
  "output_formats": ["csv", "parquet"],
  "from_datetime": "2024-05-10T00:00:00+0000",
  "to_datetime": "2024-07-09T23:59:59+0000",
  "retrieval_algorithms": ["proffast-2.2", "proffast-2.4"],
  "atmospheric_profile_models": ["GGG2020"],
  "sensor_ids": ["ma", "mb"]
}
```

--------------------------------

### Calibration Factors JSON Example

Source: https://github.com/tum-esm/em27-retrieval-pipeline/blob/main/docs/src/content/docs/api-reference/geoms-configuration.mdx

Provides an example of the JSON structure for calibration factors. This configuration is stored in `config/calibration_factors.json` and is used for instrument calibration.

```json
{
  "description": "Example calibration factors",
  "version": "1.0.0",
  "calibrationFactors": [
    {
      "instrument": "EM27",
      "factor": 0.985,
      "timestamp": "2023-10-27T10:00:00Z"
    }
  ]
}
```

--------------------------------

### Proffast 2.4 Bundle CSV Example

Source: https://github.com/tum-esm/em27-retrieval-pipeline/blob/main/docs/src/content/docs/guides/directories.mdx

An example of the CSV format for Proffast 2.4 bundles, highlighting the differences in columns compared to Proffast 1.0, such as the 'spectrum' column.

```csv
utc,spectrum,ground_pressure,lat,lon,alt,sza,azi,XH2O,XAIR,XCO2,XCH4,XCO2_STR,XCO,XCH4_S5P,H2O,O2,CO2,CH4,CO,CH4_S5P,retrieval_time,location_id,campaign_ids
2022-06-02T05:13:55.000000+0000,220602_051349SN.BIN,998.2,48.148,16.438,180.0,70.1,-101.45,3435.8,0.998586,420.051,1.88495,0.0,0.0,0.0,7.24389e26,4.46289e28,8.89103e25,4.01976e23,0.0,0.0,2024-09-11T22:50:05.000000+0000,ZEN,both+only-mc
2022-06-02T05:14:09.000000+0000,220602_051404SN.BIN,998.19,48.148,16.438,180.0,70.06,-101.41,3436.61,0.998166,419.96,1.88445,0.0,0.0,0.0,7.24253e26,4.46095e28,8.88534e25,4.01701e23,0.0,0.0,2024-09-11T22:50:05.000000+0000,ZEN,both+only-mc
2022-06-02T05:14:24.000000+0000,220602_051419SN.BIN,998.19,48.148,16.438,180.0,70.02,-101.37,3435.96,0.998954,419.327,1.88353,0.0,0.0,0.0,7.24683e26,4.46442e28,8.87892e25,4.01823e23,0.0,0.0,2024-09-11T22:50:05.000000+0000,ZEN,both+only-mc
...
```

--------------------------------

### API Reference for Metadata Schema

Source: https://github.com/tum-esm/em27-retrieval-pipeline/blob/main/docs/src/content/docs/guides/metadata.mdx

Provides a link to the API reference documentation which contains example files and a complete specification of the metadata schema.

```APIDOC
API Reference: /api-reference/metadata

This section details the schema for locations.json, sensors.json, and campaigns.json.

Schema Specification:
- locations.json: Defines measurement locations with unique 'location_id'.
- sensors.json: Defines sensor setups with unique 'sensor_id'.
- campaigns.json: Defines optional campaigns with 'campaign_id' for filtering.

Includes example files for each metadata component.
```

--------------------------------

### Starting Retrievals

Source: https://github.com/tum-esm/em27-retrieval-pipeline/blob/main/docs/src/content/docs/guides/usage.mdx

Command to initiate the retrieval processes in the background. The number of cores can be limited via configuration.

```bash
python cli.py retrieval start
```

--------------------------------

### Ground Pressure Configuration Examples

Source: https://github.com/tum-esm/em27-retrieval-pipeline/blob/main/docs/src/content/docs/guides/directories.mdx

Provides JSON examples for configuring the pressure column and its format, as well as different methods for specifying datetime information (date and time columns, combined datetime column, or Unix timestamp).

```json
{
  "pressure_column": "pressure",
  "pressure_column_format": "hPa"
}
```

```json
{
  "date_column": "utc-date",
  "date_column_format": "%Y-%m-%d",
  "time_column": "utc-time",
  "time_column_format": "%H:%M:%S"
}
```

```json
{
  "datetime_column": "utc-datetime",
  "datetime_column_format": "%Y-%m-%dT%H:%M:%S"
}
```

```json
{
  "unix_timestamp_column": "utc-datetime",
  "unix_timestamp_column_format": "s"
}
```

--------------------------------

### Proffast 1.0 Bundle CSV Example

Source: https://github.com/tum-esm/em27-retrieval-pipeline/blob/main/docs/src/content/docs/guides/directories.mdx

An example of the CSV format for Proffast 1.0 bundles, showing the columns including added fields like utc, retrieval_time, location_id, and campaign_ids.

```csv
utc,HHMMSS_ID,ground_pressure,lat,lon,alt,sza,azi,XH2O,XAIR,XCO2,XCH4,XCH4_S5P,XCO,retrieval_time,location_id,campaign_ids
2022-06-02T05:13:49.000000+0000,51349.0,998.2,48.148,16.438,180.0,70.1,-101.45,3316.9,1.00387,418.077,1.8772,0.0,0.0,2024-09-11T22:48:42.000000+0000,ZEN,both+only-mc
2022-06-02T05:14:04.000000+0000,51404.0,998.2,48.148,16.438,180.0,70.06,-101.41,3317.72,1.00343,417.989,1.87669,0.0,0.0,2024-09-11T22:48:42.000000+0000,ZEN,both+only-mc
2022-06-02T05:14:19.000000+0000,51419.0,998.2,48.148,16.438,180.0,70.02,-101.37,3317.16,1.00421,417.361,1.87585,0.0,0.0,2024-09-11T22:48:42.000000+0000,ZEN,both+only-mc
...
```

--------------------------------

### GEOMS Metadata JSON Example

Source: https://github.com/tum-esm/em27-retrieval-pipeline/blob/main/docs/src/content/docs/api-reference/geoms-configuration.mdx

Provides an example of the JSON structure for GEOMS metadata. This configuration is stored in `config/geoms:metadata.json` and is used to define geometric properties.

```json
{
  "description": "Example GEOMS metadata",
  "version": "1.0.0",
  "geoms": [
    {
      "id": "geom1",
      "type": "sphere",
      "radius": 10.5,
      "center": [0, 0, 0]
    }
  ]
}
```

--------------------------------

### Sensor Deployments Schema and Example

Source: https://github.com/tum-esm/em27-retrieval-pipeline/blob/main/docs/src/content/docs/api-reference/metadata.mdx

Provides the JSON schema for defining sensor deployments and an example JSON object illustrating its structure. This metadata is stored in `config/sensors.json`.

```json
{
  "type": "object",
  "properties": {
    "deployments": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "sensor_id": {
            "type": "string",
            "description": "Unique identifier for the sensor."
          },
          "location_id": {
            "type": "string",
            "description": "Identifier of the location where the sensor is deployed."
          },
          "campaign_id": {
            "type": "string",
            "description": "Identifier of the campaign during which the sensor was deployed."
          },
          "start_time": {
            "type": "string",
            "format": "date-time",
            "description": "Timestamp when the sensor deployment started."
          },
          "end_time": {
            "type": "string",
            "format": "date-time",
            "description": "Timestamp when the sensor deployment ended."
          }
        },
        "required": [
          "sensor_id",
          "location_id",
          "start_time"
        ]
      }
    }
  },
  "required": [
    "deployments"
  ]
}
```

```json
{
  "deployments": [
    {
      "sensor_id": "sensor-abc",
      "location_id": "loc-001",
      "campaign_id": "campaign-summer-2023",
      "start_time": "2023-07-15T10:00:00Z",
      "end_time": "2023-08-15T10:00:00Z"
    },
    {
      "sensor_id": "sensor-xyz",
      "location_id": "loc-002",
      "campaign_id": "campaign-fall-2023",
      "start_time": "2023-10-01T08:30:00Z"
    }
  ]
}
```

--------------------------------

### Manage Retrieval Process CLI

Source: https://github.com/tum-esm/em27-retrieval-pipeline/blob/main/docs/src/content/docs/api-reference/cli.mdx

Provides commands to manage the retrieval background process, including starting, checking status, watching, and stopping. It also includes a command to download retrieval algorithms.

```APIDOC
Command: python cli.py retrieval start [OPTIONS]

Description:
  Start the retrieval as a background process. Prevents spawning multiple
  processes. The logs and the current processing queue from this process can
  be found at logs/retrieval.

Options:
  --help  Show this message and exit.
```

```APIDOC
Command: python cli.py retrieval is-running [OPTIONS]

Description:
  Checks whether the retrieval background process is running. The logs and the
  current processing queue from this process can be found at logs/retrieval.

Options:
  --help  Show this message and exit.
```

```APIDOC
Command: python cli.py retrieval watch [OPTIONS]

Description:
  Opens an active watch window for the retrieval background process.

Options:
  --cluster-mode  Watch the retrieval process when the retrieval is running on
                  a cluster. In this mode the watcher does not care whether it
                  find an active retrieval process on the current node, but
                  only looks at the queue. This means it can not detect when
                  the pipeline has stopped (e.g. due to a SLURM timeout).
  --help          Show this message and exit.
```

```APIDOC
Command: python cli.py retrieval stop [OPTIONS]

Description:
  Stop the retrieval background process. The logs and the current processing
  queue from this process can be found at logs/retrieval.

Options:
  --help  Show this message and exit.
```

```APIDOC
Command: python cli.py retrieval download-algorithms [OPTIONS]

Description:
  Downloads all retrieval algorithms into the local container factories. Can
  be used if you don't want to run the pipeline but download all algorithms.

Options:
  --help  Show this message and exit.
```

--------------------------------

### Example Ground Pressure CSV Data

Source: https://github.com/tum-esm/em27-retrieval-pipeline/blob/main/docs/src/content/docs/guides/directories.mdx

Illustrates the structure of ground pressure CSV files corresponding to different datetime column configurations, showing pressure, date, and time data.

```csv
pressure,utc-date,utc-time
997.05,2022-06-02,00:00:49
997.06,2022-06-02,00:01:49
997.06,2022-06-02,00:02:49
```

```csv
pressure,utc-datetime
997.05,2022-06-02T00:00:49
997.06,2022-06-02T00:01:49
997.06,2022-06-02T00:02:49
```

```csv
pressure,unix-timestamp
997.05,1654128049
997.06,1654128109
997.06,1654128169
```

--------------------------------

### Measurement Locations Schema and Example

Source: https://github.com/tum-esm/em27-retrieval-pipeline/blob/main/docs/src/content/docs/api-reference/metadata.mdx

Provides the JSON schema for defining measurement locations and an example JSON object illustrating its structure. This metadata is stored in `config/locations.json`.

```json
{
  "type": "object",
  "properties": {
    "locations": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "id": {
            "type": "string",
            "description": "Unique identifier for the location."
          },
          "name": {
            "type": "string",
            "description": "Human-readable name of the location."
          },
          "latitude": {
            "type": "number",
            "description": "Latitude in decimal degrees."
          },
          "longitude": {
            "type": "number",
            "description": "Longitude in decimal degrees."
          },
          "altitude": {
            "type": "number",
            "description": "Altitude in meters."
          }
        },
        "required": [
          "id",
          "name",
          "latitude",
          "longitude"
        ]
      }
    }
  },
  "required": [
    "locations"
  ]
}
```

```json
{
  "locations": [
    {
      "id": "loc-001",
      "name": "Alpine Test Site",
      "latitude": 47.51623,
      "longitude": 10.69501,
      "altitude": 1800
    },
    {
      "id": "loc-002",
      "name": "Coastal Monitoring Station",
      "latitude": 34.05223,
      "longitude": -118.24368,
      "altitude": 50
    }
  ]
}
```

--------------------------------

### Measurement Campaigns Schema and Example

Source: https://github.com/tum-esm/em27-retrieval-pipeline/blob/main/docs/src/content/docs/api-reference/metadata.mdx

Provides the JSON schema for defining measurement campaigns and an example JSON object illustrating its structure. This metadata is stored in `config/campaigns.json`.

```json
{
  "type": "object",
  "properties": {
    "campaigns": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "id": {
            "type": "string",
            "description": "Unique identifier for the campaign."
          },
          "name": {
            "type": "string",
            "description": "Human-readable name of the campaign."
          },
          "start_date": {
            "type": "string",
            "format": "date",
            "description": "Start date of the campaign."
          },
          "end_date": {
            "type": "string",
            "format": "date",
            "description": "End date of the campaign."
          },
          "description": {
            "type": "string",
            "description": "A brief description of the campaign."
          }
        },
        "required": [
          "id",
          "name",
          "start_date"
        ]
      }
    }
  },
  "required": [
    "campaigns"
  ]
}
```

```json
{
  "campaigns": [
    {
      "id": "campaign-summer-2023",
      "name": "Summer Field Campaign 2023",
      "start_date": "2023-07-01",
      "end_date": "2023-08-31",
      "description": "Annual summer campaign focused on high-altitude measurements."
    },
    {
      "id": "campaign-fall-2023",
      "name": "Autumn Data Collection",
      "start_date": "2023-10-01",
      "description": "Fall campaign for atmospheric data collection."
    }
  ]
}
```

--------------------------------

### Bundling Retrieval Outputs

Source: https://github.com/tum-esm/em27-retrieval-pipeline/blob/main/docs/src/content/docs/guides/usage.mdx

Command to bundle all the outputs generated by the retrieval processes.

```bash
python cli.py bundle run
```

--------------------------------

### Using em27-metadata Python Library

Source: https://github.com/tum-esm/em27-retrieval-pipeline/blob/main/docs/src/content/docs/guides/metadata.mdx

Demonstrates how to import and use the 'em27-metadata' Python library to access and utilize the pipeline's metadata in other projects.

```python
from em27metadata.metadata import Metadata

metadata = Metadata()
locations = metadata.locations
sensors = metadata.sensors
campaigns = metadata.campaigns
```

--------------------------------

### Create Dataset Bundle CLI

Source: https://github.com/tum-esm/em27-retrieval-pipeline/blob/main/docs/src/content/docs/api-reference/cli.mdx

Creates a bundle of the entire retrieval dataset. This command is used for packaging the dataset.

```APIDOC
Command: python cli.py bundle run [OPTIONS]

Description:
  Create a bundle of your entire retrieval dataset

Options:
  --help  Show this message and exit.
```

--------------------------------

### Testing Metadata Connection

Source: https://github.com/tum-esm/em27-retrieval-pipeline/blob/main/docs/src/content/docs/guides/metadata.mdx

Command to run integration tests to verify the connection to the metadata repository and the integrity of the data within it.

```bash
pytest -m integration
```

--------------------------------

### Data Report CSV Format

Source: https://github.com/tum-esm/em27-retrieval-pipeline/blob/main/docs/src/content/docs/guides/usage.mdx

Example CSV format for the generated data report, showing columns like datetime ranges, location ID, and availability of interferograms, ground pressure, and GGG profiles.

```csv
from_datetime,to_datetime,location_id,interferograms,ground_pressure,ggg2014_profiles,ggg2014_proffast_10_outputs,ggg2014_proffast_22_outputs,ggg2014_proffast_23_outputs,ggg2020_profiles,ggg2020_proffast_22_outputs,ggg2020_proffast_23_outputs
2023-09-07T00:00:00+0000,2023-09-07T23:59:59+0000,   TUM_I, 2224, 1440,✅,-,✅,✅,✅,-,✅
2023-09-08T00:00:00+0000,2023-09-08T23:59:59+0000,   TUM_I, 2178, 1440,✅,-,✅,✅,✅,-,✅
2023-09-09T00:00:00+0000,2023-09-09T23:59:59+0000,   TUM_I, 1966, 1440,✅,-,✅,✅,✅,-,✅
2023-09-10T00:00:00+0000,2023-09-10T23:59:59+0000,   TUM_I, 2034, 1440,✅,-,✅,✅,✅,-,✅
2023-09-11T00:00:00+0000,2023-09-11T23:59:59+0000,   TUM_I, 2122, 1440,✅,-,✅,✅,✅,-,✅
2023-09-12T00:00:00+0000,2023-09-12T23:59:59+0000,   TUM_I, 1972, 1440,✅,-,✅,✅,✅,-,✅
2023-09-13T00:00:00+0000,2023-09-13T23:59:59+0000,   TUM_I,  216, 1439,✅,-,✅,✅,✅,-,✅
2023-09-14T00:00:00+0000,2023-09-14T23:59:59+0000,   TUM_I,  762, 1440,✅,-,✅,✅,✅,-,✅
2023-09-15T00:00:00+0000,2023-09-15T23:59:59+0000,   TUM_I, 1507, 1440,✅,-,✅,✅,✅,-,✅
2023-09-16T00:00:00+0000,2023-09-16T23:59:59+0000,   TUM_I, 2232, 1440,✅,-,✅,✅,✅,-,✅
2023-09-17T00:00:00+0000,2023-09-17T23:59:59+0000,   TUM_I, 1599, 1440,✅,-,✅,✅,✅,-,✅
2023-09-18T00:00:00+0000,2023-09-18T23:59:59+0000,   TUM_I,  228, 1440,✅,-,✅,✅,✅,-,✅
```

--------------------------------

### Testing the Pipeline

Source: https://github.com/tum-esm/em27-retrieval-pipeline/blob/main/docs/src/content/docs/guides/usage.mdx

Commands to run integration, quick, or CI tests, as well as complete retrieval tests, to ensure the pipeline is functioning correctly.

```bash
pytest -m "integration or quick or ci" --verbose --exitfirst tests/
pytest -m "complete" --verbose --exitfirst tests/
```

--------------------------------

### Downloading Atmospheric Profiles

Source: https://github.com/tum-esm/em27-retrieval-pipeline/blob/main/docs/src/content/docs/guides/usage.mdx

Command to download atmospheric profiles using the CLI. The script uses configuration settings to determine which profiles to request and manages parallel requests.

```bash
python cli.py profiles run
```

--------------------------------

### Astro Starlight Aside Component

Source: https://github.com/tum-esm/em27-retrieval-pipeline/blob/main/docs/src/content/docs/guides/directories.mdx

Demonstrates the usage of the Aside component from '@astrojs/starlight/components' to display important notes or warnings to the user.

```typescript
import { Aside } from '@astrojs/starlight/components';

<Aside type="note">
  We can ensure this pipeline doesn't modify any data from the input directories. We cannot guarantee this for the retrieval algorithms, but we are very confident about this and have never observed it on our systems.

  However, if you want to ensure the retrieval cannot manipulate any of your interferograms or ground pressure files, you can create a read-only mount of the directories. Only point the pipeline to those read-only directories.

  See https://askubuntu.com/a/243390.
</Aside>
```

--------------------------------

### Generated File System Tree

Source: https://github.com/tum-esm/em27-retrieval-pipeline/blob/main/docs/src/content/docs/guides/directories.mdx

Illustrates the file structure generated based on the provided bundle configuration, showing bundles for different sensors, algorithms, and time ranges.

```filesystem
<FileSystemTree
schema={{
  type: "directory",
  title: "/some/path/where/the/bundle_should_be/written/to",
  items: [
    { type: "file", title: "em27-retrieval-bundle-ma-proffast-2.2-GGG2020-20240510-20240709.csv" },
    { type: "file", title: "em27-retrieval-bundle-ma-proffast-2.2-GGG2020-20240510-20240709.parquet" },
    { type: "file", title: "em27-retrieval-bundle-ma-proffast-2.4-GGG2020-20240510-20240709.csv" },
    { type: "file", title: "em27-retrieval-bundle-ma-proffast-2.4-GGG2020-20240510-20240709.parquet" },
    { type: "file", title: "em27-retrieval-bundle-mb-proffast-2.2-GGG2020-20240510-20240709.csv" },
    { type: "file", title: "em27-retrieval-bundle-mb-proffast-2.2-GGG2020-20240510-20240709.parquet" },
    { type: "file", title: "em27-retrieval-bundle-mb-proffast-2.4-GGG2020-20240510-20240709.csv" },
    { type: "file", title: "em27-retrieval-bundle-mb-proffast-2.4-GGG2020-20240510-20240709.parquet" }
  ]
}}/>
```

--------------------------------

### EM27 Retrieval Pipeline API Documentation

Source: https://github.com/tum-esm/em27-retrieval-pipeline/blob/main/docs/src/content/docs/index.mdx

Documentation for the EM27 Retrieval Pipeline, outlining its functionalities, related projects, and how to access resources. It details the pipeline's relationship with the Proffast Pylot and its support for various retrieval algorithms and atmospheric models.

```APIDOC
Project: EM27 Retrieval Pipeline

Description: Automated data pipeline for EM27/SUN data processing.

Features:
- Supports Proffast 1 and 2.X retrievals.
- Integrates Proffast Pylot for Proffast 2.X.
- Includes custom wrapper for Proffast 1.
- Supports various retrieval algorithms (Proffast 1.0, 2.2, 2.3, 2.4).
- Supports atmospheric profile models (GGG2014, GGG2020).

Resources:
- Documentation: https://em27-retrieval-pipeline.netlify.app
- Source Code: https://github.com/tum-esm/em27-retrieval-pipeline
- Issue Tracker: https://github.com/tum-esm/em27-retrieval-pipeline/issues

Related Projects:
- tum-esm-utils: https://github.com/tum-esm/utils
- Pyra: https://github.com/tum-esm/pyra

Citations:
- MUCCnet (Dietrich et al., 2021): https://doi.org/10.5194/amt-14-1111-2021
- Proffast: https://www.imk-asf.kit.edu/english/3225.php
- Proffast Pylot: https://gitlab.eudat.eu/coccon-kit/proffastpylot.git
- Zenodo DOI: https://doi.org/10.5281/zenodo.14284968
```

--------------------------------

### GitHub Repository Metadata Configuration

Source: https://github.com/tum-esm/em27-retrieval-pipeline/blob/main/docs/src/content/docs/guides/metadata.mdx

Details the process of connecting a GitHub repository for metadata storage. This involves creating a repository from a template and configuring the pipeline's config.json file to point to it, optionally using a GitHub access token for private repositories.

```json
{
  "metadata_repository": "<your-github-username>/<your-metadata-repo-name>",
  "github_token": "<your-github-access-token>" 
}
```

--------------------------------

### Running the Pipeline CLI

Source: https://github.com/tum-esm/em27-retrieval-pipeline/blob/main/docs/src/content/docs/guides/usage.mdx

General command to run the pipeline's CLI, which includes validation of the local configuration file.

```bash
python cli.py
```

--------------------------------

### JSON Schema for Configuration

Source: https://github.com/tum-esm/em27-retrieval-pipeline/blob/main/docs/src/content/docs/api-reference/configuration.mdx

Defines the structure and constraints for the configuration file using JSON Schema. This schema ensures that the configuration adheres to the expected format.

```typescript
import JSONSchemaRenderer from "../../../components/JSONSchemaRenderer.tsx";
import ConfigSchema from "../../../assets/config.schema.json";

// Usage within a React component:
<JSONSchemaRenderer schema={ConfigSchema} />
```

--------------------------------

### Manage Atmospheric Profiles CLI

Source: https://github.com/tum-esm/em27-retrieval-pipeline/blob/main/docs/src/content/docs/api-reference/cli.mdx

Handles operations related to atmospheric profiles, including downloading, requesting status updates, and migrating storage locations.

```APIDOC
Command: python cli.py profiles run [OPTIONS]

Description:
  Run the profiles download script. This will check, which profiles are not
  yet present locally, request and download them from the
  ccycle.gps.caltech.edu FTP server. The logs from this process can be found
  at logs/profiles.

Options:
  --help  Show this message and exit.
```

```APIDOC
Command: python cli.py profiles request-ginput-status [OPTIONS]

Description:
  Request ginput status. This will upload a file upload/ginput_status.txt to
  the ccycle.gps.caltech.edu FTP server containing the configured email
  address. You will receive an email with the ginput status which normally
  takes less than two minutes.

Options:
  --help  Show this message and exit.
```

```APIDOC
Command: python cli.py profiles migrate-storage-location [OPTIONS]

Description:
  Migrate the storage location of the atmospheric profiles to the new
  directory structure introduced in the pipeline version 1.7.0. See
  https://github.com/tum-esm/em27-retrieval-pipeline/issues/127 for more
  details.

Options:
  --help  Show this message and exit.
```

--------------------------------

### Run Retrieval on SLURM Cluster

Source: https://github.com/tum-esm/em27-retrieval-pipeline/blob/main/docs/src/content/docs/guides/usage.mdx

Instructions for running the retrieval pipeline on a SLURM-based cluster. This involves using the `main.py` script directly and submitting a SLURM job script.

```bash
#!/bin/bash
#SBATCH -J erp
#SBATCH -o /dss/dsshome1/lxc01/ge69zeh2/Documents/em27-retrieval/em27-retrieval-pipeline/data/logs/%x.%j.%N.out
#SBATCH -D /dss/dsshome1/lxc01/ge69zeh2/Documents/em27-retrieval/em27-retrieval-pipeline
#SBATCH --clusters=cm4
#SBATCH --partition=cm4_tiny
#SBATCH --qos=cm4_tiny
#SBATCH --time=06:00:00
#SBATCH --nodes=1
#SBATCH --cpus-per-task=112
#SBATCH --export=NONE
#SBATCH --get-user-env
#SBATCH --mail-type=all
#SBATCH --mail-user=moritz.makowski@tum.de

# setup environment: git is used to determine the commit hash of the currently
# running pipeline, gfortan (gcc) is used to compile proffast and the ifg
# corruption filter
module load slurm_setup
module load git
module load gcc/13.2.0

# activate virtual environment: we set up the virtual environment on the login
# nodes, but you can also do that inside the compute nodes
source .venv/bin/activate

# run retrieval
python src/retrieval/main.py
```

--------------------------------

### EM27 Retrieval Pipeline Tasks

Source: https://github.com/tum-esm/em27-retrieval-pipeline/blob/main/docs/src/content/docs/index.mdx

Executes the core tasks of the EM27 retrieval pipeline: downloading vertical profiles, running the retrieval process, and bundling the results.

```python
python cli.py profiles run
python cli.py retrieval start
python cli.py bundle run
```

--------------------------------

### EM27/84 Retrieval Pipeline Parameters Overview

Source: https://github.com/tum-esm/em27-retrieval-pipeline/blob/main/docs/src/content/docs/guides/miscellaneous.mdx

This section details the parameters used in the EM27/84 retrieval pipeline, categorized for clarity. It includes optical parameters, sample parameters, and data parameters, with descriptions for each.

```APIDOC
ZPA: 
  Description: Not specified

VER: 
  Description: Version Number

VDC: 
  Description: variability in the DC level

LST: 
  Description: laser sampling type [0: none; 1: InGaAs; 2: Si; 3: Dohe et al.; 4: other]

LSE: 
  Description: laser sampling error [the shift]

LSU: 
  Description: laser sampling uncertainty

**Optic Parameters**:

SRC: 
  Description: Source Setting

APT: 
  Description: Aperture Setting

FOV: 
  Description: Field of view in mrad

BMS: 
  Description: Beamsplitter Setting

VEL: 
  Description: Scanner Velocity

DTC: 
  Description: Detector Setting

HPF: 
  Description: High Pass Filter

LPF: 
  Description: Low Pass Filter

CHN: 
  Description: Measurement Channel

PGN: 
  Description: Preamplifier Gain

RDX: 
  Description: Not specified

SON: 
  Description: External Synchronisation

**Sample Parameters**:

SFM: 
  Description: Sample Form

LAT: 
  Description: Latitude

LON: 
  Description: Longitude

ALT: 
  Description: Altitude [m a.s.l.]

TOU: 
  Description: Average temperature [outside, °C]

POU: 
  Description: Average pressure [outside, hPa]

HOU: 
  Description: Average humidity [outside, %]

SIA: 
  Description: Average solar intensity

SIS: 
  Description: Standard deviation of solar intensity

WSA: 
  Description: Average wind speed [m/s]

WDA: 
  Description: Average wind direction

EXP: 
  Description: Experiment [filename]

**Data Parameters**:

DPF: 
  Description: Data Point Format

NPT: 
  Description: Number of Data Points

FXV: 
  Description: Frequency of First Point

LXV: 
  Description: Frequency of Last Point

CSF: 
  Description: Y - Scaling Factor

MXY: 
  Description: Y - Maximum

MNY: 
  Description: Y - Minimum

DXU: 
  Description: X Units

DAT: 
  Description: Date of Measurement

TIM: 
  Description: Time of Measurement
```

--------------------------------

### Monitoring Retrievals

Source: https://github.com/tum-esm/em27-retrieval-pipeline/blob/main/docs/src/content/docs/guides/usage.mdx

Commands to check if retrievals are running and to open a dashboard for monitoring their progress.

```bash
python cli.py retrieval is-running
python cli.py retrieval watch
```

--------------------------------

### Proffast Pylot and Wrapper for Proffast 1

Source: https://github.com/tum-esm/em27-retrieval-pipeline/blob/main/docs/src/content/docs/index.mdx

This section describes the integration of the Proffast Pylot for Proffast 2.X retrievals and a custom wrapper for Proffast 1 within the EM27 Retrieval Pipeline. The pipeline includes a copy of the Proffast Pylot's Python codebase for reduced complexity.

```Python
import proffastpylot

# Example usage of Proffast Pylot for Proffast 2.X
# proffast_retrieval = proffastpylot.ProffastRetriever()
# result = proffast_retrieval.run_retrieval(...)

# Example usage of the in-house wrapper for Proffast 1
# from . import proffast1_wrapper
# proffast1_result = proffast1_wrapper.run_retrieval(...)
```

--------------------------------

### Local Metadata Configuration

Source: https://github.com/tum-esm/em27-retrieval-pipeline/blob/main/docs/src/content/docs/guides/metadata.mdx

Specifies how to configure the pipeline using local metadata files. The files can be placed in the 'config/' directory or an alternative directory specified by the ERP_CONFIG_DIR environment variable.

```bash
export ERP_CONFIG_DIR=<path-to-alternative-local-config-dir>
```

--------------------------------

### File System Tree Structure

Source: https://github.com/tum-esm/em27-retrieval-pipeline/blob/main/docs/src/content/docs/guides/directories.mdx

Illustrates the hierarchical structure of interferogram files within the data directories. This component helps visualize the expected organization of sensor data.

```javascript
import FileSystemTree from "../../../components/FileSystemTree.tsx";

<FileSystemTree
  schema={{
    type: "directory",
    title: "<config.general.data.interferograms>",
    items: [
      {
        type: "directory",
        title: "ma",
        items: [
          {
            type: "directory",
            title: "20220101",
            items: [
              { type: "file", title: "ma20210101.ifg.001" },
              { type: "file", title: "ma20210101.ifg.002" },
              { type: "file", title: "ma20210101.ifg.003" },
              { type: "file", title: "..." }
            ]
          },
          {
            type: "directory",
            title: "20220102",
            items: [
              { type: "file", title: "ma20210102.ifg.001" },
              { type: "file", title: "ma20210102.ifg.002" },
              { type: "file", title: "ma20210102.ifg.003" },
              { type: "file", title: "..." }
            ]
          }
        ]
      },
      {
        type: "directory",
        title: "mb",
        items: [
          {
            type: "directory",
            title: "20220101",
            items: []
          },
          {
            type: "directory",
            title: "...",
            items: []
          }
        ]
      },
      {
        type: "directory",
        title: "...",
        items: []
      }
    ]
  }}
/>
```

--------------------------------

### Related Projects Integration

Source: https://github.com/tum-esm/em27-retrieval-pipeline/blob/main/README.md

This section highlights the integration with related projects, such as the 'tum-esm-utils' package for shared functionality and 'Pyra' for autonomous system operation.

```Shell
# Install tum-esm-utils
pip install tum-esm-utils

# Information about Pyra integration (conceptual)
# Pyra is used for autonomous operation of EM27/SUN systems.
```

--------------------------------

### Preprocess 5 Failure with 'inconsistent dualifg!'

Source: https://github.com/tum-esm/em27-retrieval-pipeline/blob/main/docs/src/content/docs/guides/miscellaneous.mdx

Details an issue where Preprocess 5 (Proffast 2.3) fails with an 'inconsistent dualifg!' warning when processing older interferograms, while Preprocess 4 (Proffast 2.2) can handle them successfully. Includes a log excerpt of the error.

```txt
================= Task 0 ==================
/home/moritz-makowski/Documents/pipelines/erp-new-pressure/data/containers/retrieval-container-angry-benz/prf/preprocess/preprocess5 preprocess5ma_161102.inp
Return code: 1

Output:
 Reading input file...
 Done!
 Number of raw measurements to be processed:        3131
 Reading file names
 /home/moritz-makowski/Documents/pipelines/erp-on-demand/data/containers/retrieval-container-angry-benz-inputs/ifg/161102/161102SN.2853
 /home/moritz-makowski/Documents/pipelines/erp-on-demand/data/containers/retrieval-container-angry-benz-inputs/ifg/161102/161102SN.891
 ...
 /home/moritz-makowski/Documents/pipelines/erp-on-demand/data/containers/retrieval-container-angry-benz-inputs/ifg/161102/161102SN.3016
 Done!
 Read OPUS parms:           1
 /home/moritz-makowski/Documents/pipelines/erp-on-demand/data/containers/retrieval-container-angry-benz-inputs/ifg/161102/161102SN.2853
 Warning:
 inconsistent dualifg!
 This is a critical error. Quiet run option selected: End Programm

Errors:
 inconsistent dualifg!
============================================
============================================
```

--------------------------------

### Run Pytest Test Classes

Source: https://github.com/tum-esm/em27-retrieval-pipeline/blob/main/docs/src/content/docs/guides/tests.mdx

Demonstrates how to run specific test classes using pytest marks. This allows developers to target different aspects of the pipeline for testing.

```bash
pytest -m 'quick'
pytest -m 'quick or integration'
```