WIS2 in a box

https://github.com/world-meteorological-organization/wis2box
Admin
WIS2 in a box is a reference implementation of a WIS2 Node designed to enable WMO members to publish...
Tokens:42,803
Snippets:630
Trust Score:7.7
License:Apache-2.0
Update:5 months ago
Show doc for...
Tokens:
Raw
### Configure WIS2BOX Environment

Source: https://github.com/world-meteorological-organization/wis2box/blob/main/docs/source/reference/quickstart.md

Copy the example environment file to 'wis2box.env' to configure your WIS2BOX instance for the quickstart setup.

```bash
cp tests/test.env wis2box.env
```

--------------------------------

### Start Jupyter Notebook Session

Source: https://github.com/world-meteorological-organization/wis2box/blob/main/docs/source/reference/data-access/index.md

Clone the wis2box repository and start a Jupyter Notebook server to run the examples. Access the notebooks via http://localhost:8888.

```bash
git clone https://github.com/World-Meteorological-Organization/wis2box.git
cd docs/source/data-access
jupyter notebook --ip=0.0.0.0 --port=8888
```

--------------------------------

### Start Jupyter Notebook Server

Source: https://github.com/world-meteorological-organization/wis2box/blob/main/docs/README.md

Use this command to launch the Jupyter Notebook interface in your browser. Ensure Jupyter is installed.

```bash
jupyter notebook
# browser should appear
```

--------------------------------

### Download wis2box Setup

Source: https://github.com/world-meteorological-organization/wis2box/blob/main/docs/source/user/setup.md

Download the wis2box setup zip file and extract it. Navigate into the extracted directory.

```bash
wget https://github.com/World-Meteorological-Organization/wis2box-release/releases/download/1.2.0/wis2box-setup-1.2.0.zip
unzip wis2box-setup-1.2.0.zip
cd wis2box
```

--------------------------------

### Start and Login to wis2box Management Container

Source: https://github.com/world-meteorological-organization/wis2box/blob/main/docs/source/reference/administration.md

After installation and starting the system, log into the wis2box-management container to configure station metadata, discovery metadata, and data processing pipelines.

```bash
python3 wis2box-ctl.py start
python3 wis2box-ctl.py login
```

--------------------------------

### Full Wis2box Configuration Example

Source: https://github.com/world-meteorological-organization/wis2box/blob/main/docs/source/reference/configuration.md

A comprehensive example demonstrating various configuration directives for Wis2box, including data paths, API settings, logging, and observation thresholds.

```bash
# data paths and retention
WIS2BOX_DATADIR=/data/wis2box

# API
WIS2BOX_API_TYPE=pygeoapi
WIS2BOX_API_URL=http://localhost/oapi
WIS2BOX_API_BACKEND_TYPE=Elasticsearch
WIS2BOX_API_BACKEND_URL=http://elasticsearch:9200
WIS2BOX_DOCKER_API_URL=http://wis2box-api:80/oapi

# csv2bufr transformation 
WIS2BOX_OBSERVATION_DISTANCE_THRESHOLD=1000

# logging
WIS2BOX_LOGGING_LOGLEVEL=ERROR
WIS2BOX_LOGGING_LOGFILE=stdout

```

--------------------------------

### Install Docker and Dependencies

Source: https://github.com/world-meteorological-organization/wis2box/blob/main/docs/source/user/getting-started.md

Execute these commands sequentially to install Docker and required Python packages. Ensure you run them one by one.

```bash
sudo mkdir -m 0755 -p /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor --yes -o /etc/apt/keyrings/docker.gpg
sudo chmod a+r /etc/apt/keyrings/docker.gpg
sudo echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get -y update
sudo apt-get install -y docker-ce docker-compose-plugin unzip python3-pip
sudo pip3 install pip --upgrade
sudo pip3 install pyopenssl --upgrade
sudo pip3 install urllib3==1.26.0
```

--------------------------------

### Test Docker Installation

Source: https://github.com/world-meteorological-organization/wis2box/blob/main/docs/source/user/getting-started.md

Switch to the user and run the 'hello-world' container to confirm Docker is functioning correctly.

```bash
sudo su - $USER
docker run hello-world
```

--------------------------------

### Start wis2box Stack

Source: https://github.com/world-meteorological-organization/wis2box/blob/main/docs/source/community/troubleshooting.md

Use this command to start the wis2box stack after applying configuration changes.

```bash
python3 wis2box-ctl.py start
```

--------------------------------

### Create README for wis2box Setup

Source: https://github.com/world-meteorological-organization/wis2box/wiki/ReleaseManagement

Generates a README file within the wis2box asset directory, providing a link to detailed setup instructions.

```bash
cat << EOF > wis2box/README.txt
wis2box setup
=============

Please see https://docs.wis2box.wis.wmo.int/en/x.y.z/user/setup.html for setting up wis2box from example configurations in this directory.
EOF
```

--------------------------------

### Install boto3 Python package

Source: https://github.com/world-meteorological-organization/wis2box/blob/main/docs/source/reference/storage.md

Install the boto3 package using pip.

```bash
pip3 install boto3
```

--------------------------------

### Set Up Python Virtual Environment for Ubuntu 24.04 LTS

Source: https://github.com/world-meteorological-organization/wis2box/blob/main/docs/source/user/getting-started.md

For Ubuntu 24.04 LTS, use a virtual environment to install Python packages. Activate the environment before installing.

```bash
python3 -m venv wis2box
source wis2box/bin/activate
pip3 install pip --upgrade
pip3 install pyopenssl --upgrade
pip3 install urllib3==1.26.0
```

--------------------------------

### Install R Packages

Source: https://github.com/world-meteorological-organization/wis2box/blob/main/docs/source/reference/data-access/r-api.ipynb

Install the necessary R packages for data access and manipulation.

```r
install.packages("sf")
install.packages("dplyr")
```

--------------------------------

### Install minio Python package

Source: https://github.com/world-meteorological-organization/wis2box/blob/main/docs/source/reference/storage.md

Install the minio Python package using pip.

```bash
pip3 install minio
```

--------------------------------

### Get Stations

Source: https://github.com/world-meteorological-organization/wis2box/blob/main/docs/source/reference/data-access/python-api-requests.ipynb

Retrieve a list of all stations available in the WIS2BOX system. This example fetches up to 50 stations and prints their names.

```APIDOC
## Get Stations

### Description
Retrieves a list of all stations available in the WIS2BOX system.

### Method
GET

### Endpoint
/collections/stations/items?limit=50

### Parameters
#### Query Parameters
- **limit** (integer) - Optional - The maximum number of stations to return.

### Request Example
```python
import requests

api = 'http://localhost/oapi'
url = f'{api}/collections/stations/items?limit=50'
response = requests.get(url).json()

print(f"Number of stations: {response['numberMatched']}")
print('Stations:\n')
for station in response['features']:
    print(station['properties']['name'])
```

### Response
#### Success Response (200)
- **numberMatched** (integer) - The total number of stations matched.
- **features** (array) - A list of station objects.
  - **properties** (object) - Properties of the station.
    - **name** (string) - The name of the station.

#### Response Example
```json
{
  "numberMatched": 26,
  "features": [
    {
      "type": "Feature",
      "properties": {
        "name": "NAMBUMA"
      }
    },
    {
      "type": "Feature",
      "properties": {
        "name": "BALAKA"
      }
    }
  ]
}
```
```

--------------------------------

### Upload data to MinIO using SFTP

Source: https://github.com/world-meteorological-organization/wis2box/blob/main/docs/source/user/data-ingest.md

Automate data uploads to the 'wis2box-incoming' bucket via SFTP. This command-line example creates a directory and uploads a data file. Ensure the directory path matches your dataset identifier or topic.

```bash
sftp -P 8022 -oBatchMode=no -o StrictHostKeyChecking=no wis2box@localhost << EOF
    mkdir wis2box-incoming/urn:wmo:md:it-meteoam:surface-weather-observations.synop
    put /path/to/your/datafile.csv wis2box-incoming/urn:wmo:md:it-meteoam:surface-weather-observations.synop
EOF
```

--------------------------------

### MQTT Client Initialization and Loop

Source: https://github.com/world-meteorological-organization/wis2box/blob/main/docs/source/reference/data-access/python-mqp-subscribe.ipynb

Initializes the MQTT client, assigns callback functions, sets authentication credentials, connects to the broker, and starts the client's network loop to process messages indefinitely.

```python
client = mqtt.Client(client_id=clientId, protocol=mqtt.MQTTv5)
client.on_connect = sub_connect
client.on_message = sub_message
client.username_pw_set(user, password)
client.connect(host)

client.loop_forever()
```

--------------------------------

### Create Release Zip Archive

Source: https://github.com/world-meteorological-organization/wis2box/wiki/ReleaseManagement

Creates a zip archive containing the wis2box setup files for distribution.

```bash
zip -r wis2box-setup-x.y.z.zip wis2box
```

--------------------------------

### Build, Start, Stop, and Status of wis2box Containers

Source: https://github.com/world-meteorological-organization/wis2box/blob/main/docs/source/reference/administration.md

Use `wis2box-ctl.py` to manage the lifecycle of wis2box services. Run `python3 wis2box-ctl.py --help` for all available options.

```bash
python3 wis2box-ctl.py build
```

```bash
python3 wis2box-ctl.py start
```

```bash
python3 wis2box-ctl.py stop
```

```bash
python3 wis2box-ctl.py status
```

--------------------------------

### Configure API Settings

Source: https://github.com/world-meteorological-organization/wis2box/blob/main/docs/source/reference/configuration.md

Set environment variables to configure the OGC API setup, including server type, public URL, and backend provider.

```bash
WIS2BOX_API_TYPE=pygeoapi  # server type
WIS2BOX_API_URL=http://localhost/pygeoapi  # public landing page endpoint
WIS2BOX_API_BACKEND_TYPE=Elasticsearch  # backend provider type
WIS2BOX_API_BACKEND_URL=http://elasticsearch:9200  # internal backend connection URL
WIS2BOX_DOCKER_API_URL=http://wis2box-api:80/oapi  # container name of API container (for internal communications/workflow)
```

--------------------------------

### Restart WIS2BOX Instance

Source: https://github.com/world-meteorological-organization/wis2box/blob/main/docs/source/user/public-services-setup.md

Stop and then start the wis2box instance to apply SSL configuration changes. This is required after modifying SSL settings.

```bash
python3 wis2box-ctl.py stop
python3 wis2box-ctl.py start
```

--------------------------------

### Verify System Component Versions

Source: https://github.com/world-meteorological-organization/wis2box/blob/main/docs/source/user/getting-started.md

Check the installed versions of Docker, Docker Compose, and Python to ensure compatibility.

```bash
docker version
docker compose version
python3 -V
```

--------------------------------

### Example MCF File for wis2box

Source: https://github.com/world-meteorological-organization/wis2box/blob/main/docs/source/reference/running/discovery-metadata.md

This is an example of a Metadata Control File (MCF) in YAML format used to configure discovery metadata for a dataset in wis2box. It includes settings for data retention, topic hierarchy, country, centre ID, and data mappings for CSV and BUFR4 file types.

```yaml
wis2box:
    retention: P30D
    topic_hierarchy: mw-mw_met_centre-test/data/core/weather/surface-based-observations/synop
    country: mwi
    centre_id: mw-mw_met_centre-test
    data_mappings:
        plugins:
            csv:
                - plugin: wis2box.data.csv2bufr.ObservationDataCSV2BUFR
                  template: CampbellAfrica-v1-template
                  notify: true
                  file-pattern: '^WIGOS_(\d-\d+-\d+-\w+)_.*\.csv$'
            bufr4:
                - plugin: wis2box.data.bufr2geojson.ObservationDataBUFR2GeoJSON
                  file-pattern: '^WIGOS_(\d-\d+-\d+-\w+)_.*\.bufr4$'

mcf:
    version: 2.0

metadata:
    identifier: urn:wmo:md:mw-mw_met_centre-test:surface-weather-observations
    hierarchylevel: dataset
    dates:
        creation: 2021-11-29

identification:
    title: Surface weather observations from Malawi
    abstract: Surface weather observations from Malawi
    keywords:
        default:
            keywords:
                - surface weather
                - temperature
                - observations
        wmo:
            keywords:
                - weather
            keywords_type: theme
            vocabulary:
                name: Earth system disciplines as defined by the WMO Unified Data Policy, Resolution 1 (Cg-Ext(2021), Annex 1.
                url: https://codes.wmo.int/wis/topic-hierarchy/earth-system-discipline
    extents:
        spatial:
            - bbox: [32.6881653175,-16.8012997372,35.7719047381,-9.23059905359]
              crs: 4326
        temporal:
            - begin: 2021-11-29
              end: null
              resolution: PT1H
    url: https://example.org/malawi-surface-weather-observations
    wmo_data_policy: core

contact:
    host:
        organization: Department of Climate Change and Meteorologial Services (DCCMS)
        url: https://www.metmalawi.gov.mw
        individualname: Firstname Lastname
        positionname: Position Name
        phone: "+2651822014"
        fax: "+2651822215"
        address: P.O. Box 1808
        city: Blantyre
        administrativearea: Blantyre District
        postalcode: M3H 5T4
        country: Malawi
        email: you@example.org
        hoursofservice: 0700h - 1500h UTC
        contactinstructions: email

```

--------------------------------

### Clone WIS2BOX Repository

Source: https://github.com/world-meteorological-organization/wis2box/blob/main/docs/source/reference/quickstart.md

Clone the WIS2BOX repository from GitHub to your local machine to begin setup.

```bash
git clone https://github.com/World-Meteorological-Organization/wis2box.git
```

--------------------------------

### Restart wis2box Services

Source: https://github.com/world-meteorological-organization/wis2box/blob/main/docs/source/user/public-services-setup.md

Use these commands to stop, start, and republish metadata for your wis2box instance after updating configuration, such as WIS2BOX_URL.

```bash
python3 wis2box-ctl.py stop
python3 wis2box-ctl.py start
python3 wis2box-ctl.py login
wis2box metadata discovery republish
```

--------------------------------

### Add Dataset and Publish Metadata After SSL Setup

Source: https://github.com/world-meteorological-organization/wis2box/blob/main/docs/source/user/public-services-setup.md

After restarting wis2box with SSL enabled, re-add your dataset and publish metadata to ensure all URLs are updated to use HTTPS.

```bash
python3 wis2box-ctl.py login
wis2box data add-collection ${WIS2BOX_HOST_DATADIR}/surface-weather-observations.yml
wis2box metadata discovery publish ${WIS2BOX_HOST_DATADIR}/surface-weather-observations.yml
```

--------------------------------

### Upload data to MinIO using Python

Source: https://github.com/world-meteorological-organization/wis2box/blob/main/docs/source/user/data-ingest.md

Use this Python script with the MinIO client library to programmatically upload data files to the 'wis2box-incoming' bucket. Ensure the `minio` package is installed. The `minio_path` should correspond to your dataset's metadata identifier or topic.

```python
import glob
import sys

from minio import Minio

filepath = '/home/wis2box-user/local-data/mydata.bin'
# path should match the metadata or the topic in the data mappings
minio_path = 'urn:wmo:md:it-meteoam:surface-weather-observations'

endpoint = 'http://localhost:9000'
WIS2BOX_STORAGE_USERNAME = 'wis2box'
WIS2BOX_STORAGE_PASSWORD = '<your-wis2box-storage-password>'

client = Minio(
    endpoint=endpoint,
    access_key=WIS2BOX_STORAGE_USERNAME,
    secret_key=WIS2BOX_STORAGE_PASSWORD,
    secure=False)

filename = filepath.split('/')[-1]
client.fput_object('wis2box-incoming', minio_path+filename, filepath)
```

--------------------------------

### Minimal wis2box Data Plugin Example

Source: https://github.com/world-meteorological-organization/wis2box/blob/main/docs/source/reference/extending-wis2box.md

A basic implementation of a wis2box data plugin. It inherits from `BaseAbstractData` and defines the `transform` method to process input data and populate `output_data`.

```python
from datetime import datetime
from wis2box.data.base import BaseAbstractData

class MyCoolData(BaseAbstractData):
    """Observation data"""
    def __init__(self, defs: dict) -> None:
        super().__init__(defs)

    def transform(self, input_data: Path) -> bool:
        # transform data
        # populate self.output_data with a dict as per:
        self.output_data = {
            'c123': {
                '_meta': {
                    'identifier': 'c123',
                    'relative_filepath': '/path/to/item/',
                    'data_date': datetime_object
                },
                'bufr4': bytes(12356),
                'geojson': geojson_string
            }
        }
        return True
```

--------------------------------

### Initialize MQTT Client and Connection Parameters

Source: https://github.com/world-meteorological-organization/wis2box/blob/main/docs/source/reference/data-access/python-mqp-subscribe.ipynb

Sets up connection details and client ID for the MQTT client. Includes parameters for host, user, password, and a unique client ID.

```python
import json
import paho.mqtt.client as mqtt
import random
import urllib
import urllib.request


host='localhost'
user='wis2box'
password='wis2box'

r = random.Random()
clientId='MyQueueName'+ f"{r.randint(1,1000):04d}"
# number of messages to subscribe to.
messageCount = 0
messageCountMaximum = 5

```

--------------------------------

### Get Dataset Details

Source: https://github.com/world-meteorological-organization/wis2box/blob/main/docs/source/reference/data-access/python-api-requests.ipynb

Retrieve detailed information about a specific dataset, including its data access links. This example fetches details for the 'Surface weather observations (hourly)' dataset.

```APIDOC
## Get Dataset Details

### Description
Retrieves detailed information about a specific dataset, including its data access links.

### Method
GET

### Endpoint
/collections/discovery-metadata/items/{dataset_id}

### Parameters
#### Path Parameters
- **dataset_id** (string) - Required - The unique identifier of the dataset.

### Request Example
```python
import requests

api = 'http://localhost/oapi'
dataset_id = 'urn:wmo:md:mw-mw_met_centre-test:surface-weather-observations'
url = f"{api}/collections/discovery-metadata/items/{dataset_id}"
response = requests.get(url).json()

print('Data access links:\n')
for link in response['links']:
    print(f"{link['href']} ({link['type']}) {link['rel']}")
```

### Response
#### Success Response (200)
- **links** (array) - A list of links associated with the dataset.
  - **href** (string) - The URL of the link.
  - **type** (string) - The media type of the linked resource.
  - **rel** (string) - The relationship of the link to the dataset.

#### Response Example
```json
{
  "type": "Feature",
  "properties": {
    "id": "urn:wmo:md:mw-mw_met_centre-test:surface-weather-observations",
    "title": "Surface weather observations (hourly)"
  },
  "links": [
    {
      "rel": "self",
      "type": "application/geo+json",
      "title": "This document as GeoJSON",
      "href": "http://localhost/oapi/collections/discovery-metadata/items/urn:wmo:md:mw-mw_met_centre-test:surface-weather-observations?f=json"
    },
    {
      "rel": "alternate",
      "type": "application/ld+json",
      "title": "This document as RDF (JSON-LD)",
      "href": "http://localhost/oapi/collections/discovery-metadata/items/urn:wmo:md:mw-mw_met_centre-test:surface-weather-observations?f=jsonld"
    }
  ]
}
```
```

--------------------------------

### Create Wis2box Release Asset Bundle

Source: https://github.com/world-meteorological-organization/wis2box/wiki/ReleaseManagement

Create a directory for the release assets and write the version number to a VERSION.txt file. This bundles the configuration for the release.

```bash
mkdir -p asset/wis2box
echo "x.y" > asset/wis2box/VERSION.txt
```

--------------------------------

### Initialize API Endpoint and Pretty Print Function

Source: https://github.com/world-meteorological-organization/wis2box/blob/main/docs/source/reference/data-access/python-api-requests.ipynb

Sets up the base API URL and a helper function for formatted JSON output. Ensure the API endpoint is correctly defined.

```python
import json

import requests

def pretty_print(input):
    print(json.dumps(input, indent=2))


# define the endpoint of the OGC API
api = 'http://localhost/oapi'
```

--------------------------------

### Create wis2box Configuration File

Source: https://github.com/world-meteorological-organization/wis2box/blob/main/docs/source/reference/configuration.md

Run this command to create a new 'wis2box.env' file by answering interactive questions.

```bash
python3 wis2box-create-config.py
```

--------------------------------

### Get wis2box-webapp password

Source: https://github.com/world-meteorological-organization/wis2box/blob/main/docs/source/user/setup-datasets.md

Retrieve the password for accessing the wis2box-webapp by checking the WIS2BOX_WEBAPP_PASSWORD environment variable.

```bash
cat wis2box.env | grep WIS2BOX_WEBAPP_PASSWORD
```

--------------------------------

### Get wis2box-webapp username

Source: https://github.com/world-meteorological-organization/wis2box/blob/main/docs/source/user/setup-datasets.md

Retrieve the username for accessing the wis2box-webapp by checking the WIS2BOX_WEBAPP_USERNAME environment variable.

```bash
cat wis2box.env | grep WIS2BOX_WEBAPP_USERNAME
```

--------------------------------

### Configure Web Application Settings

Source: https://github.com/world-meteorological-organization/wis2box/blob/main/docs/source/reference/configuration.md

Customize web components by setting environment variables for basemap URL, attribution, UI clustering, language, logo, and banner color.

```bash
WIS2BOX_BASEMAP_URL="https://{s}.tile.openstreetmap.org/{z}/{x}/{y}.png"  # URL of map tile server to use
WIS2BOX_BASEMAP_ATTRIBUTION="<a href="https://osm.org/copyright">OpenStreetMap</a> contributors"  # attribution of map tile server
WIS2BOX_UI_CLUSTER=False # default setting of the cluster toggle
WIS2BOX_UI_LANG="en" # default language, one of: ar, en, es, fr, ru, zh
WIS2BOX_UI_LOGO=http://example.com/logo.png # use custom organization logo in the UI
WIS2BOX_UI_BANNER_COLOR="#014e9e" # use custom background color for header and footer
WIS2BOX_UI_DISABLE_SEPARATOR_IMAGE=false # boolean to disable WMO separator
```

--------------------------------

### Upload file to MinIO using S3cmd

Source: https://github.com/world-meteorological-organization/wis2box/blob/main/docs/source/reference/storage.md

Copies a local file into the 'wis2box-incoming' bucket to a specified topic path using the S3cmd command-line tool.

```bash
s3cmd myfile.csv s3://wis2box-incoming/foo/bar/baz
```

--------------------------------

### Get wis2box instance URL

Source: https://github.com/world-meteorological-organization/wis2box/blob/main/docs/source/user/setup-datasets.md

Retrieve the URL of the wis2box instance by checking the WIS2BOX_URL environment variable.

```bash
cat wis2box.env | grep WIS2BOX_URL
```

--------------------------------

### Configure S3 Compatible Storage

Source: https://github.com/world-meteorological-organization/wis2box/blob/main/docs/source/reference/configuration.md

Set up connection details and bucket names for S3 compatible storage like MinIO. Specify retention days for incoming and public files, and API backend files.

```bash
WIS2BOX_STORAGE_TYPE=S3
WIS2BOX_STORAGE_SOURCE=http://minio:9000
WIS2BOX_STORAGE_USERNAME=wis2box  # username for the storage-layer
WIS2BOX_STORAGE_PASSWORD=minio123  # password for the storage-layer
WIS2BOX_STORAGE_INCOMING=wis2box-incoming  # name of the storage-bucket/folder for incoming files
WIS2BOX_STORAGE_PUBLIC=wis2box-public  # name of the storage-bucket/folder for public files
WIS2BOX_STORAGE_DATA_RETENTION_DAYS=7  # number of days to keep files in incoming and public
WIS2BOX_STORAGE_API_RETENTION_DAYS=7  # number of days to keep files in API backend
```

--------------------------------

### Configure Pub/Sub Broker Connection

Source: https://github.com/world-meteorological-organization/wis2box/blob/main/docs/source/reference/configuration.md

Set environment variables for connecting to the Pub/Sub broker, including host, port, credentials, and queue limits.

```bash
WIS2BOX_BROKER_HOST=mosquitto  # the hostname of the internal broker
WIS2BOX_BROKER_PORT=1883  # the port of the internal broker
WIS2BOX_BROKER_USERNAME=wis2box  # the username of the internal broker
WIS2BOX_BROKER_PASSWORD=wis2box  # the password of the internal broker
WIS2BOX_BROKER_PUBLIC=mqtt://foo:bar@localhost:1883  # RFC 1738 URL of public broker endpoint
WIS2BOX_BROKER_QUEUE_MAX=1000  # maximum number of messages to hold in the queue per client
```

--------------------------------

### Enable Traefik with Let's Encrypt

Source: https://github.com/world-meteorological-organization/wis2box/blob/main/docs/source/user/public-services-setup.md

Replace the default `docker-compose.override.yml` with `docker-compose.traefik.yml` to enable Traefik for automatic SSL certificate provisioning and renewal.

```bash
python3 wis2box-ctl.py stop
cp docker-compose.traefik.yml docker-compose.override.yml
python3 wis2box-ctl.py start
```

--------------------------------

### Download Access-Controlled Data

Source: https://github.com/world-meteorological-organization/wis2box/blob/main/docs/source/user/recommended.md

Retrieve data from an access-controlled dataset by including the authentication token in the request header. This example uses wget.

```bash
wget --header= "Authorization: Bearer S3CR3TT0K3N" "https://www.mywis2box.com/data/urn:wmo:md:my-centre-id:my-local-id/mydata.bufr4"
```

--------------------------------

### Manage Docker Volumes

Source: https://github.com/world-meteorological-organization/wis2box/blob/main/docs/source/community/troubleshooting.md

Old Docker volumes from previous wis2box installations can cause issues. List all volumes and remove any that are no longer needed.

```bash
docker volume ls
```

```bash
docker volume rm <volume_name>
```

--------------------------------

### Set SSL Certificate and Key Environment Variables

Source: https://github.com/world-meteorological-organization/wis2box/blob/main/docs/source/user/public-services-setup.md

Define the paths to your SSL certificate and private key using environment variables. Ensure these paths are correct for your system.

```bash
WIS2BOX_SSL_CERT=/etc/letsencrypt/live/example.wis2box.io/fullchain.pem
WIS2BOX_SSL_KEY=/etc/letsencrypt/live/example.wis2box.io/privkey.pem
```

--------------------------------

### Discover Datasets

Source: https://github.com/world-meteorological-organization/wis2box/blob/main/docs/source/reference/data-access/python-api-requests.ipynb

Find all available datasets within the WIS2BOX, identified by their WIS2 discovery metadata records. This example lists the ID and title of each dataset.

```APIDOC
## Discover Datasets

### Description
Retrieves a list of all datasets available in the WIS2BOX, identified by their WIS2 discovery metadata records.

### Method
GET

### Endpoint
/collections/discovery-metadata/items

### Parameters
None

### Request Example
```python
import requests

api = 'http://localhost/oapi'
url = f'{api}/collections/discovery-metadata/items'
response = requests.get(url).json()

print('Datasets:\n')
for dataset in response['features']:
    print(f"id: {dataset['properties']['id']}, title: {dataset['properties']['title']}")
```

### Response
#### Success Response (200)
- **features** (array) - A list of dataset objects.
  - **properties** (object) - Properties of the dataset.
    - **id** (string) - The unique identifier of the dataset.
    - **title** (string) - The title of the dataset.

#### Response Example
```json
{
  "features": [
    {
      "type": "Feature",
      "properties": {
        "id": "data.core.test-passthrough",
        "title": "Surface weather observations (passthrough)"
      }
    },
    {
      "type": "Feature",
      "properties": {
        "id": "urn:wmo:md:mw-mw_met_centre-test:surface-weather-observations",
        "title": "Surface weather observations (hourly)"
      }
    }
  ]
}
```
```

--------------------------------

### Display First Few Rows of DataFrame

Source: https://github.com/world-meteorological-organization/wis2box/blob/main/docs/source/reference/data-access/python-api-owslib.ipynb

View the first few rows of the DataFrame to get a quick look at the data. This helps in verifying the data retrieval and formatting.

```python
df.head(3)
```

--------------------------------

### Build and Update WIS2BOX

Source: https://github.com/world-meteorological-organization/wis2box/blob/main/docs/source/reference/quickstart.md

Execute these commands to build the WIS2BOX system and update its components.

```bash
python3 wis2box-ctl.py build
```

```bash
python3 wis2box-ctl.py update
```

--------------------------------

### Initialize OWSLib Features API

Source: https://github.com/world-meteorological-organization/wis2box/blob/main/docs/source/reference/data-access/python-api-owslib.ipynb

Import necessary libraries and initialize the OWSLib Features API client with the wis2box endpoint URL.

```python
from owslib.ogcapi.features import Features

import pandas as pd

def pretty_print(input):
    print(json.dumps(input, indent=2))


api = 'http://localhost/oapi'
```

--------------------------------

### gts_headers_mapping.csv Example

Source: https://github.com/world-meteorological-organization/wis2box/blob/main/docs/source/user/gts-headers-in-wis2.md

This CSV file maps parts of a filepath to GTS headers (TTAAii and CCCC). wis2box uses this to automatically add GTS properties to WIS2 notifications.

```csv
string_in_filepath,TTAAii,CCCC
ISMD01LIIB,ISMD01,LIBB
ISMD02LIIB,ISMD02,LIBB
```

--------------------------------

### Add New User to Internal MQTT Broker

Source: https://github.com/world-meteorological-organization/wis2box/blob/main/docs/source/user/public-services-setup.md

Use this command to add a new user to the internal MQTT broker. Replace `<username>` and `<password>` with your desired credentials.

```bash
docker exec -it mosquitto /bin/sh
```

```bash
mosquitto_passwd -b /mosquitto/config/password.txt <username> <password>
```

--------------------------------

### Restart wis2box Services

Source: https://github.com/world-meteorological-organization/wis2box/blob/main/docs/source/community/troubleshooting.md

If wis2box services are restarting or unhealthy, first attempt to start them. If issues persist, check Docker logs and configuration files. Restart all services after applying fixes.

```bash
python3 wis2box-ctl.py start
```

```bash
python3 wis2box-ctl.py stop
python3 wis2box-ctl.py start
```

--------------------------------

### Check Port Allocation

Source: https://github.com/world-meteorological-organization/wis2box/blob/main/docs/source/community/troubleshooting.md

If a service fails to bind to a port (e.g., 0.0.0.0:XX), use these commands to identify which process is using the port. Ensure required ports are available before starting wis2box.

```bash
sudo lsof -i :80   # Find process using port 80
```

```bash
sudo netstat -tuln # Alternative check
```

--------------------------------

### Filtering MQTT Messages by Wigos Station Identifier

Source: https://github.com/world-meteorological-organization/wis2box/blob/main/docs/source/user/downloading-data.md

Python example demonstrating how to filter incoming MQTT messages based on the 'wigos-station-identifier' property. Only messages with relevant identifiers are processed for data download.

```python
def on_message(client, userdata, msg):
  msg_dict = json.load(msg.payload.decode('utf-8'))
  wsi = msg_dict['properties'].get('wigos-station-identifier', None)
  if wsi not in my_list_of_relevant_wigos_station_identifiers:
    continue
  else:
    for link in msg_dict['links']:
      if link['rel'] == 'canonical' or link['rel'] == 'update':
        client_specific_function(link['href'])
```

--------------------------------

### Build and Tag Wis2box Broker Docker Image

Source: https://github.com/world-meteorological-organization/wis2box/wiki/ReleaseManagement

Build a Docker image for wis2box-broker and tag it with the release version. This prepares the image for publishing to a container registry.

```bash
cd wis2box-broker
docker build --no-cache -t wis2box-broker:local .  # note SHA
docker tag $SHA ghcr.io/wmo-im/wis2box-broker:x.y.z
docker image push ghcr.io/wmo-im/wis2box-broker:x.y.z
cd ..
```

--------------------------------

### Build and Tag Wis2box MQTT Metrics Collector Docker Image

Source: https://github.com/world-meteorological-organization/wis2box/wiki/ReleaseManagement

Build a Docker image for wis2box-mqtt-metrics-collector and tag it with the release version. This prepares the image for publishing to a container registry.

```bash
cd wis2box-mqtt-metrics-collector
docker build --no-cache -t wis2box-mqtt-metrics-collector:local .  # note SHA
docker tag $SHA ghcr.io/wmo-im/wis2box-mqtt-metrics-collector:x.y.z
docker image push ghcr.io/wmo-im/wis2box-mqtt-metrics-collector:x.y.z
cd ..
```

--------------------------------

### Configure Contact and Identification

Source: https://github.com/world-meteorological-organization/wis2box/blob/main/docs/source/reference/configuration.md

Customize UI elements, service identification, data license, service provider, and contact information using these environment variables.

```bash
# UI customizations
WIS2BOX_ICON=https://example.org/icon
WIS2BOX_LOGO=https://example.org/logo

# service identification
WIS2BOX_IDENTIFICATION_TITLE=title
WIS2BOX_IDENTIFICATION_DESCRIPTION=description
WIS2BOX_IDENTIFICATION_TERMS_OF_SERVICE=CC-BY 4.0
WIS2BOX_IDENTIFICATION_URL=https://example.org/tos
# data license information
WIS2BOX_LICENSE_NAME=CC-BY 4.0
WIS2BOX_LICENSE_URL=https://creativecommons.org/licenses/by/4.0
# service provider information
WIS2BOX_PROVIDER_NAME=Firstname Lastname
WIS2BOX_PROVIDER_URL=https://example.org/provider
# contact information
WIS2BOX_CONTACT_NAME=Firstname Lastname
WIS2BOX_CONTACT_POSITION=staff
WIS2BOX_CONTACT_ADDRESS=123 Main street
WIS2BOX_CONTACT_CITY=city
# NOTE: if your postal code is only numeric, prefix with the text "code " to ensure it is interpreted as a string
WIS2BOX_CONTACT_POSTALCODE=postal code
WIS2BOX_CONTACT_COUNTRY=country
WIS2BOX_CONTACT_EMAIL=you@example.org
WIS2BOX_CONTACT_URL=https://example.org/contact
WIS2BOX_CONTACT_INSTRUCTIONS=contact instructions
WIS2BOX_CONTACT_ROLE=host
```

--------------------------------

### Build and Tag Wis2box Management Docker Image

Source: https://github.com/world-meteorological-organization/wis2box/wiki/ReleaseManagement

Build a Docker image for wis2box-management and tag it with the release version. This prepares the image for publishing to a container registry.

```bash
cd wis2box-management/
docker build --no-cache -t wis2box-management:local .  # note SHA
docker tag $SHA ghcr.io/wmo-im/wis2box-management:x.y.z
docker image push ghcr.io/wmo-im/wis2box-management:x.y.z
```

--------------------------------

### Authenticate with Bearer Token using cURL

Source: https://github.com/world-meteorological-organization/wis2box/blob/main/docs/source/reference/auth.md

Access data via WAF or API using a generated token. Tokens can be passed as a bearer token in the 'Authorization' header or appended to the URI. This example demonstrates using cURL with the Authorization header.

```bash
curl -H "Authorization: Bearer mytoken" "http://localhost/oapi/collections/urn:wmo:md:mw-mw_met_centre-test:surface-weather-observations"
```

```bash
curl -H "Authorization: Bearer notmytoken" "http://localhost/oapi/collections/urn:wmo:md:mw-mw_met_centre-test:surface-weather-observations"
```

--------------------------------

### Generate SSH Keys for MinIO SFTP

Source: https://github.com/world-meteorological-organization/wis2box/blob/main/docs/source/community/troubleshooting.md

Manually create SSH keys for the MinIO SFTP server if they are missing. Replace `<your-wis2box-host-datadir>` with your actual data directory path.

```bash
mkdir -p <your-wis2box-host-datadir>/.ssh
ssh-keygen -t rsa -b 4096 -f <your-wis2box-host-datadir>/minio/.ssh/id_rsa -N "" -q
```

--------------------------------

### Constructing Dataset API Link

Source: https://github.com/world-meteorological-organization/wis2box/blob/main/docs/source/reference/data-access/python-api-requests.ipynb

Shows how to form the specific OGC API - Features (OAFeat) link for a given dataset, which can then be used to query observations.

```python
dataset_api_link = 'http://localhost/oapi/collections/urn:wmo:md:mw-mw_met_centre-test:surface-weather-observations'

dataset_api_link
```

--------------------------------

### Create Git Tag for Wis2box Release (from main)

Source: https://github.com/world-meteorological-organization/wis2box/wiki/ReleaseManagement

Create an annotated Git tag for the release when working from the main branch. This tag marks the specific version for the release.

```bash
git tag -a x.y.z -m 'tagging release x.y.z'
git push --tags
```

--------------------------------

### Configure S3cmd for MinIO

Source: https://github.com/world-meteorological-organization/wis2box/blob/main/docs/source/reference/storage.md

Edit the ~/.s3cfg file to configure S3cmd for connecting to a local MinIO instance. This includes setting the base host and access keys.

```bash
cat << EOF > ~/.s3cfg
# Setup endpoint
host_base = localhost:9000
use_https = False

# Setup access keys
access_key = wis2box
secret_key = XXXXX
EOF
```

--------------------------------

### Copy Docker Configs and Environment File

Source: https://github.com/world-meteorological-organization/wis2box/wiki/ReleaseManagement

Copies essential Docker configuration and environment files to the wis2box asset directory for release preparation.

```bash
cp docker-compose.monitoring.yml docker-compose.data-subscriber.yml docker-compose.override.yml docker-compose.ssl.yml docker-compose.yml asset/wis2box
cp -rp nginx prometheus loki grafana asset/wis2box
cp wis2box-ctl.py wis2box-create-config.py asset/wis2box
```
Tokens:
Raw