SpectrumX Data System SDK and Gateway

https://github.com/spectrumx/sds-code
Admin
The SpectrumX Data System (SDS) provides an SDK and a Django-based Gateway for distributed,...
Tokens:56,418
Snippets:395
Trust Score:3.5
License:MIT
Update:7 months ago
Show doc for...
Tokens:
Raw
### Install SpectrumX SDK

Source: https://github.com/spectrumx/sds-code/blob/master/sdk/docs/mkdocs/getting-started.md

Instructions for installing the spectrumx package using common Python package managers like uv, poetry, or pip.

```bash
uv add spectrumx

# or one of:
#   poetry add spectrumx
#   pip install spectrumx
```

--------------------------------

### Production Environment Setup

Source: https://github.com/spectrumx/sds-code/blob/master/gateway/docs/detailed-deploy.md

Commands to initialize production environment files, generate random secrets, and configure hostnames for deployment.

```bash
rsync -aP ./.envs/example/ ./.envs/production
echo $(head /dev/urandom | tr -dc 'a-zA-Z0-9' | head -c 40)
cp ./scripts/prod-hostnames.example.env ./scripts/prod-hostnames.env
hostname >> ./scripts/prod-hostnames.env
just env
```

--------------------------------

### Perform SDS File Operations

Source: https://github.com/spectrumx/sds-code/blob/master/sdk/docs/mkdocs/getting-started.md

Demonstrates initializing the client, authenticating, creating local dummy data, and performing upload/download operations with the SpectrumX SDK.

```python
from pathlib import Path
from random import randint, random
from spectrumx.client import Client

sds = Client(host="sds.crc.nd.edu")
sds.authenticate()

reference_name: str = "my_spectrum_capture"
local_dir: Path = Path(reference_name)

if not local_dir.exists():
    local_dir.mkdir(exist_ok=True)
    for file_idx in range(10):
        num_lines = randint(10, 100)
        with (local_dir / f"capture_{file_idx}.csv").open(mode="w", encoding="utf-8") as file_ptr:
            file_ptr.write("\n".join(map(str, [random() for _ in range(num_lines)])))

upload_results = sds.upload(local_path=local_dir, sds_path=reference_name, verbose=True)

local_downloads = Path("sds-downloads") / "files" / reference_name
sds.download(from_sds_path=reference_name, to_local_path=local_downloads, overwrite=False, verbose=True)
```

--------------------------------

### Run Development Setup with Just

Source: https://github.com/spectrumx/sds-code/blob/master/gateway/README.md

Executes the 'dev-setup' recipe using the 'just' command runner to set up the development environment. This typically involves installing dependencies and configuring the project for local development.

```bash
just dev-setup
```

--------------------------------

### Install System Dependencies for SDS Gateway

Source: https://github.com/spectrumx/sds-code/blob/master/gateway/docs/detailed-deploy.md

Commands to install required development libraries for Python and PostgreSQL on Ubuntu or RHEL-based systems.

```bash
# dev bindings for python and postgres
sudo apt install python3-dev libpq-dev # on ubuntu
sudo dnf install python3-devel postgresql-devel # on RHEL

# get psql, createdb, etc.
sudo apt install postgresql-client
sudo dnf install postgresql
```

--------------------------------

### Install SpectrumX SDK

Source: https://github.com/spectrumx/sds-code/blob/master/sdk/docs/mkdocs/faq.md

Instructions for installing the SpectrumX SDK using common package managers like uv, poetry, and pip. The SDK is available on PyPI.

```bash
uv add spectrumx
poetry add spectrumx
pip install spectrumx
```

--------------------------------

### Configure Authentication

Source: https://github.com/spectrumx/sds-code/blob/master/sdk/docs/mkdocs/getting-started.md

Methods for providing the SDS_SECRET_TOKEN via a .env file or environment variable to authenticate with the SDS platform.

```ini
SDS_SECRET_TOKEN=your-secret-token-no-quotes
```

```bash
export SDS_SECRET_TOKEN=your-secret-token
```

--------------------------------

### Manual Local Deployment

Source: https://github.com/spectrumx/sds-code/blob/master/gateway/docs/detailed-deploy.md

Manual steps for setting up environment secrets, configuring the Docker network, and running Django migrations.

```bash
rsync -aP ./.envs/example/ ./.envs/local
docker network create sds-network-local --driver=bridge
just up
just uv run manage.py makemigrations
just uv run manage.py migrate
```

--------------------------------

### Initialize Application and Indices

Source: https://github.com/spectrumx/sds-code/blob/master/gateway/docs/detailed-deploy.md

Commands to create the initial administrative superuser and initialize OpenSearch indices required for the application to function.

```bash
just uv run manage.py createsuperuser
just uv run manage.py init_indices
```

--------------------------------

### Handle SDK Exceptions

Source: https://github.com/spectrumx/sds-code/blob/master/sdk/docs/mkdocs/getting-started.md

Shows how to catch and handle specific SDK exceptions like AuthError and NetworkError to improve application robustness.

```python
from spectrumx.client import Client
from spectrumx.errors import AuthError, NetworkError

sds = Client(host="sds.crc.nd.edu")
try:
    sds.authenticate()
except NetworkError as err:
    print(f"Failed to connect to the SDS: {err}")
except AuthError as err:
    print(f"Failed to authenticate: {err}")
```

--------------------------------

### Manage Python Dependencies and Pre-commit Hooks

Source: https://github.com/spectrumx/sds-code/blob/master/gateway/docs/detailed-deploy.md

Commands to synchronize Python dependencies using uv and initialize pre-commit hooks for automated linting and formatting.

```bash
uv sync --frozen --extra local
uv run --extra local pre-commit install
```

--------------------------------

### Alternative SpectrumX SDK Installations

Source: https://github.com/spectrumx/sds-code/blob/master/sdk/docs/README.md

Provides alternative methods for installing the SpectrumX SDK using 'poetry' or 'pip'. These are common Python package management tools.

```bash
poetry add spectrumx
```

```bash
pip install spectrumx
```

--------------------------------

### Manage User Credentials

Source: https://github.com/spectrumx/sds-code/blob/master/gateway/docs/detailed-deploy.md

Commands to reset passwords or create new superusers for the Django administrative interface.

```bash
just uv run manage.py changepassword <email>
just uv run manage.py createsuperuser
```

--------------------------------

### Run Test Suite

Source: https://github.com/spectrumx/sds-code/blob/master/gateway/docs/detailed-deploy.md

Commands to execute the project test suite, including template validation and Python-specific tests.

```bash
just test
just test-py
just uv run manage.py validate_templates
just uv run pytest
```

--------------------------------

### Debugging and Routing Utilities

Source: https://github.com/spectrumx/sds-code/blob/master/gateway/docs/detailed-deploy.md

Commands to inspect the application routing table and debug URL configurations.

```bash
just uv run manage.py show_urls
```

--------------------------------

### Automated Local Deployment

Source: https://github.com/spectrumx/sds-code/blob/master/gateway/docs/detailed-deploy.md

Automated workflow to generate environment secrets and deploy the Docker Compose stack using Just recipes.

```bash
just generate-secrets local
# or: ./scripts/generate-secrets.sh local
just redeploy
```

--------------------------------

### Setup Django-Cog Pipelines for Visualizations

Source: https://github.com/spectrumx/sds-code/blob/master/gateway/sds_gateway/visualizations/README.md

This command sets up the Django-cog pipelines for the visualizations app. It supports different pipeline types and strategies for handling existing pipelines, such as aborting, skipping, forcing recreation, or smart recreation.

```bash
python manage.py setup_pipelines

# Command Options:
# --pipeline-type {visualization,all}: Type of pipeline to set up (default: all)
# --strategy {abort-if-exists,skip-if-exists,force,smart-recreate}: Strategy for handling existing pipelines (default: abort-if-exists)
```

--------------------------------

### Download Dataset using SpectrumX Python SDK

Source: https://github.com/spectrumx/sds-code/blob/master/gateway/sds_gateway/templates/users/partials/sdk_download_modal.html

Demonstrates how to initialize the SpectrumX client, authenticate, and download a dataset to a local directory. It includes configuration for dry-run modes, progress tracking, and error handling for individual file downloads.

```python
from spectrumx import Client
from spectrumx.errors import SDSError
from pathlib import Path
from uuid import UUID

# Initialize the client with your SDS host
client = Client(
    host="sds.crc.nd.edu"
)

# Set dry_run to False to enable actual file downloads
client.dry_run = False

# Authenticate using token from .env or config
client.authenticate()

# Download a dataset by UUID
try:
    dataset_uuid_str = "{{ dataset.uuid }}"
    dataset_uuid = UUID(dataset_uuid_str)
    download_path = Path("./downloaded_dataset")

    # Download all files in the dataset
    results = client.download_dataset(
        dataset_uuid=dataset_uuid,
        to_local_path=download_path,
        skip_contents=False,
        overwrite=False,
        verbose=True
    )

    successful_downloads = [r for r in results if r]
    failed_downloads = [r for r in results if not r]

    print(f"Successfully downloaded: {len(successful_downloads)} files")

except SDSError as e:
    print(f"Error downloading dataset: {e}")
except Exception as e:
    print(f"Unexpected error: {e}")
```

--------------------------------

### Install SpectrumX SDK using uv

Source: https://github.com/spectrumx/sds-code/blob/master/sdk/docs/README.md

Installs the SpectrumX SDK package using the 'uv' package manager. This is a common method for managing Python dependencies.

```bash
uv add spectrumx
```

--------------------------------

### Deploy SpectrumX SDS Gateway using Docker Compose

Source: https://context7.com/spectrumx/sds-code/llms.txt

This guide explains how to deploy the SpectrumX SDS Gateway using Docker Compose. It involves cloning the repository and executing a deployment script, suitable for both local development and production environments.

```bash
# Clone the repository
git clone https://github.com/spectrumx/sds-code.git
cd sds-code/gateway

# Development deployment
./scripts/deploy.sh local
```

--------------------------------

### Check SDK Version

Source: https://github.com/spectrumx/sds-code/blob/master/jupyter/compose/production/jupyter/sample_scripts/walkthrough.ipynb

Retrieves the currently installed version of the spectrumx library.

```python
import spectrumx

spectrumx.__version__
```

--------------------------------

### Performing Range Queries

Source: https://github.com/spectrumx/sds-code/blob/master/gateway/docs/detailed-deploy.md

Examples of range filters for numerical and date-based fields. Supports standard OpenSearch range operators like gte and lte.

```json
{
    "field_path": "capture_props.center_freq",
    "query_type": "range",
    "filter_value": {
        "gte": 1990000000,
        "lte": 2010000000
    }
}
```

```json
{
    "field_path": "created_at",
    "query_type": "range",
    "filter_value": {
        "gte": "now-6M"
    }
}
```

--------------------------------

### Performing Geospatial Queries

Source: https://github.com/spectrumx/sds-code/blob/master/gateway/docs/detailed-deploy.md

Examples of geo-bounding box and geo-distance queries for filtering captures based on GPS coordinates. These queries require fields of type geo_point.

```json
{
    "field_path": "capture_props.coordinates",
    "query_type": "geo_bounding_box",
    "filter_value": {
        "top_left": {
            "lat": 25,
            "lon": -85
        },
        "bottom_right": {
            "lat": 20,
            "lon": -80
        }
    }
}
```

```json
{
    "field_path": "capture_props.coordinates",
    "query_type": "geo_distance",
    "filter_value": {
        "distance": "10mi",
        "capture_props.coordinates": {
            "lat": 41.703,
            "lon": -86.243
        }
    }
}
```

--------------------------------

### Update Pipelines in Production (Manual)

Source: https://github.com/spectrumx/sds-code/blob/master/gateway/sds_gateway/visualizations/README.md

This demonstrates how to manually update Django-cog pipelines in a production environment by connecting to the container and executing the setup command with the 'smart-recreate' strategy.

```bash
# Connect to production container
docker exec -it <container_name> bash

# Update pipelines with smart recreate
python manage.py setup_pipelines --strategy=smart-recreate
```

--------------------------------

### Install SpectrumX Python Package

Source: https://github.com/spectrumx/sds-code/blob/master/gateway/sds_gateway/templates/pages/spx_dac_dataset_alt.html

Installs the SpectrumX Python package from PyPI using pip. It is recommended to activate a virtual environment before running this command.

```bash
pip install spectrumx
```

--------------------------------

### Download SDS Dataset with Retries (Python)

Source: https://github.com/spectrumx/sds-code/blob/master/sdk/docs/mkdocs/common-workflows/dataset-downloads.md

This Python script downloads all files from a specified SDS dataset. It initializes the SpectrumX client, authenticates, and then attempts to download the dataset to a local path. The script includes a robust retry mechanism with exponential backoff to handle intermittent errors, logging detailed information about successful and failed downloads. Ensure you have the 'spectrumx' and 'loguru' libraries installed.

```python
#!/usr/bin/env python
"""Download all files from a specified SDS dataset with retry logic.

For documentation, see https://sds.crc.nd.edu/sdk/

"""

import time
from pathlib import Path
from uuid import UUID

from loguru import logger as log
from spectrumx import Client
from spectrumx.errors import SDSError


def main() -> None:
    # Initialize the client with your SDS host
    client = Client(
        host="sds.crc.nd.edu",
        # TODO: create a .env file with your SDS API token for authentication
        # env_file=Path(".env"),  # default, recommended to keep tokens out of version control
    )

    # when in dry-run (default), no changes are are made to the SDS or the local filesystem
    # to enable the changes, set dry_run to False, as in:
    client.dry_run = False  # ⚠️ This is required to actually download files!

    # authenticate using either the token from the .env file or in the config passed in.
    # This will raise an spectrumx.errors.AuthError if authentication fails, and it's
    # useful to isolate API access issues; otherwise it is not required to be called
    # explicitly before other operations, as each API call carries auth information.
    client.authenticate()

    # TODO: specify the dataset UUID and local download path
    #   Where to find the dataset's UUID:
    #   1. Log into the SDS. https://sds.crc.nd.edu/
    #   2. Navigate to the dataset listing page. https://sds.crc.nd.edu/users/dataset-list/
    #   3. Locate the dataset you want to download and click on its name. A modal will open.
    #   4. The modal has detailed infomation on the dataset. Beside its name, you will see a
    #       "Copy" button. Clicking it will copy the dataset's UUID to your clipboard.
    #       a UUID has the format "xxxxxxxx-xxxx-4xxx-xxxx-xxxxxxxxxxxx", where each "x"
    #       is a hexadecimal digit.
    #  5. Paste the UUID string below, replacing the placeholder value.
    dataset_uuid_str = "xxxxxxxx-xxxx-4xxx-xxxx-xxxxxxxxxxxx"
    dataset_uuid = UUID(dataset_uuid_str)

    # TODO: make sure this download location has enough disk space for the dataset you
    #   are downloading. The total size of the dataset can be found on its details modal
    #   in the SDS web interface. On Linux machines, you can use this command:
    #       df --si .
    #   SI units match the ones used in the SDS web interface.
    download_path = Path("./downloaded_dataset")

    log.info(f"Downloading dataset: {dataset_uuid_str}")
    log.info(f"Download path: {download_path}")

    # loop control
    is_success: bool = False
    retries_left: int = 50
    sleep_time_sec: int = 1
    max_sleep_time_sec: int = 300
    is_first_run = True
    # max total wait is under max_sleep_time_sec * retries_left:
    #   300 * 50 < 15'000 seconds, or < 4h 10min
    # after this time with no progress, the script will give up the download

    while not is_success and retries_left > 0:
        if not is_first_run:
            # wait before next retry, with exponential backoff in case
            # it's an intermittend connection or service issue
            sleep_time_sec *= 2
            time.sleep(min(sleep_time_sec, max_sleep_time_sec))
        is_first_run = False
        retries_left -= 1
        try:
            # download all files in the dataset
            results = client.download_dataset(
                dataset_uuid=dataset_uuid,
                to_local_path=download_path,
                overwrite=False,  # set to True to overwrite existing local files
                verbose=True,  # show progress bars
            )
        except SDSError as err:
            log.exception(f"Error downloading dataset: {err}")
            continue
        except Exception as err:
            log.exception(f"Unexpected error: {err}")
            continue

        # no connection errors, reset sleep time
        sleep_time_sec = 1

        # Check results for any errors
        successful_downloads = [
            r for r in results if r
        ]  # Result objects are truthy if successful
        failed_downloads = [
            r for r in results if not r
        ]  # Result objects are falsy if failed

        log.success(f"Successfully downloaded: {len(successful_downloads)} files")
        log.info(f"Failed downloads: {len(failed_downloads)} files")

        if not failed_downloads:
            is_success = True
            break

        log.info("\nFailed downloads:")
        for result in failed_downloads:
            # TIP: calling result() will re-raise the exception if you prefer
            #   to handle it here instead of just logging it.
            file_info = result.error_info.get("file", {})
            file_name = (
                file_info.get("name", "Unknown")
                if isinstance(file_info, dict)
                else "Unknown"
            )
            msg = f"  - {file_name}: {result.exception_or(RuntimeError('Unknown download error'))}"
            log.error(msg)

```

--------------------------------

### Constructing Basic Metadata Filters

Source: https://github.com/spectrumx/sds-code/blob/master/gateway/docs/detailed-deploy.md

Demonstrates the standard JSON structure for metadata filters, utilizing field_path, query_type, and filter_value. The API automatically handles nesting via dot notation in the field_path.

```json
{
    "field_path": "capture_props.<field_name>",
    "query_type": "match",
    "filter_value": "<field_value_to_match>"
}
```

--------------------------------

### GitHub Actions workflow for SDS Gateway integration tests

Source: https://github.com/spectrumx/sds-code/blob/master/gateway/docs/github-actions-ephemeral-env.md

A complete CI pipeline configuration for GitHub Actions. It handles environment setup, secret generation, service orchestration via Docker Compose, database migrations, and test execution.

```yaml
name: Gateway Integration Tests

on:
  pull_request:
    paths:
      - 'gateway/**'
  push:
    branches: [master]

jobs:
  integration-test:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3

      - name: Install just
        uses: extractions/setup-just@v2

      - name: Generate CI secrets
        working-directory: gateway
        run: ./scripts/generate-secrets.sh ci

      - name: Start gateway stack
        working-directory: gateway
        run: |
          export COMPOSE_FILE=compose.local.yaml
          docker compose --env-file .envs/ci/opensearch.env up -d

      - name: Wait for services to be healthy
        working-directory: gateway
        run: |
          timeout 120 bash -c 'until docker compose --env-file .envs/ci/opensearch.env ps | grep -q "healthy"; do sleep 2; done'

      - name: Run migrations
        working-directory: gateway
        run: |
          docker compose --env-file .envs/ci/opensearch.env \
            run sds-gateway-local-app uv run manage.py migrate

      - name: Run tests
        working-directory: gateway
        run: |
          docker compose --env-file .envs/ci/opensearch.env \
            run sds-gateway-local-app uv run pytest

      - name: Cleanup
        if: always()
        working-directory: gateway
        run: docker compose --env-file .envs/ci/opensearch.env down -v
```

--------------------------------

### Handle Errors During SpectrumX SDS Operations

Source: https://github.com/spectrumx/sds-code/blob/master/sdk/docs/mkdocs/faq.md

Illustrates how to handle various exceptions that may occur during SpectrumX SDS operations, such as authentication failures, network issues, or service errors. It includes examples of catching specific error types like `AuthError`, `NetworkError`, and `SDSError`, and demonstrates how to inspect results for individual file upload successes or failures.

```python
from spectrumx.client import Client
from spectrumx.errors import AuthError, NetworkError, Result, SDSError, ServiceError

sds = Client(host="sds.crc.nd.edu")

try:
    sds.authenticate()
except NetworkError as err:
    print(f"Failed to connect to the SDS: {err}")
    # Check your host parameter and network connection
except AuthError as err:
    print(f"Failed to authenticate: {err}")
    # Check your authentication token

try:
    upload_results = sds.upload(local_path="my_files", sds_path="remote")

    # Check individual file results
    success_results = [r for r in upload_results if r]  # Successful uploads
    failed_results = [r for r in upload_results if not r]  # Failed uploads

    # Calling a failed result will raise its exception
    for result in failed_results:
        result()  # Raises the exception it holds
except (NetworkError, ServiceError) as err:
    # Transient errors - consider retry logic
    print(f"Temporary failure: {err}")
except SDSError as err:
    # Other SDS-specific errors
    print(f"SDS error: {err}")
```

--------------------------------

### Iterate Through SDS File Generator and Print File Details in Python

Source: https://github.com/spectrumx/sds-code/blob/master/jupyter/compose/production/jupyter/sample_scripts/walkthrough.ipynb

This Python code demonstrates iterating over a generator obtained from `sds.list_files`. It prints directory, name, and creation timestamp for each file entry. The example includes logic to stop iteration early after a certain number of files, showcasing the lazy nature of the generator.

```python
print("Iterating over the file generator:")
for file_index, file_entry in enumerate(files_generator):
    print(
        f"\t'dir={file_entry.directory}' | name='{file_entry.name}' | created_at={file_entry.created_at}"
    )
    # do_something_with_the_file(file_entry)
    if file_index > 5:
        print("\t>> Stopping early <<")
        break
```

--------------------------------

### Initialize SDS Client and Upload Workflow

Source: https://github.com/spectrumx/sds-code/blob/master/sdk/docs/mkdocs/common-workflows/capture-uploads.md

Demonstrates the main execution flow for setting up an SDS client, discovering Digital-RF channels, uploading local data, and initiating the capture process.

```python
client = Client(host="sds.crc.nd.edu")
client.authenticate()

drf_local_path = Path("./rf_data")
sds_destination = PurePosixPath(f"drf_capture_{int(time.time())}")

drf_channels = discover_drf_channels(drf_local_path)
uploaded_files = upload_with_retries(
    client=client,
    local_path=drf_local_path,
    sds_path=sds_destination
)

if len(drf_channels) > 1:
    captures = create_multichannel_captures(
        client=client,
        sds_path=sds_destination,
        channels=drf_channels
    )
```

--------------------------------

### Set up SDS Authentication with .env file

Source: https://github.com/spectrumx/sds-code/blob/master/sdk/docs/mkdocs/faq.md

Demonstrates how to set up token-based authentication for the SpectrumX SDK using a .env file. This method is recommended for keeping secret tokens out of version control.

```ini
SDS_SECRET_TOKEN=your-secret-token-no-quotes
```

--------------------------------

### Upload Files to SDS with Retry Logic

Source: https://github.com/spectrumx/sds-code/blob/master/sdk/docs/mkdocs/getting-started.md

Uploads files from a local directory to SDS, retrying on network or service errors. It processes upload results, logs successes and failures, and implements a backoff strategy for retries. Dependencies include `sds`, `NetworkError`, `ServiceError`, `SDSError`, `log`, `sleep`, and `Path`.

```python
from pathlib import Path
from time import sleep

# Assume sds, NetworkError, ServiceError, SDSError, log, and File are defined elsewhere

local_dir: Path = Path("my_spectrum_files")
reference_name: str = "my_spectrum_files"
retries_left: int = 5
is_success: bool = False
uploaded_files: list[File] = []

while not is_success and retries_left > 0:
    try:
        retries_left -= 1

        # `sds.upload()` will restart a partial file transfer from zero,
        # but it won't re-upload already finished files.
        upload_results: list[Result[File]] = sds.upload(
            local_path=local_dir,
            sds_path=reference_name,
            verbose=True,
        )

        # Since `upload()` is a batch operation, some files may succeed and some
        #   may fail. The return value of `sds.upload` stored in `upload_results`
        #   is a list of `Result` objects:
        # A `Result` wraps either the value of a variable (in this case the File
        #   object that was uploaded) or an exception. Here's how we can check if
        #   there were any failed uploads:
        success_results = [success for success in upload_results if success]
        failed_results = [success for success in upload_results if not success]

        log.debug(f"Uploaded {len(success_results)} assets.")
        log.warning(f"Failed to upload {len(failed_results)} assets")

        # calling a successful result will return the value it holds
        uploaded_files = [result() for result in success_results]

        # And calling a failed result will raise the exception it holds.
        # Here we re-raise it to handle retries with the except blocks below,
        #   based on the exception raised:
        for result in failed_results:
            result()  # will raise

    except (NetworkError, ServiceError) as err:
        # NetworkError refers to connection issues between client and SDS Gateway
        # ServiceError refers to issues with the SDS Gateway itself (e.g. HTTP 500)
        # sleep longer with each retry, at least 5s, up to 5min
        sleep_time = max(5, 5 / (retries_left**2) * 60)
        log.error(f"Error: {err}")
        log.warning(f"Failed to reach the gateway; sleeping {sleep_time}s")
        if retries_left > 0:
            sleep(sleep_time)
        continue
    except SDSError as err:
        log.error(f"Another SDS error occurred: {err}")
        # other errors might include e.g. OSError
        #   if listed files cannot be found.
        # TODO: take action or break
        break

log.debug(f"Uploaded files: {uploaded_files}")

```

--------------------------------

### Set up SDS Authentication with Environment Variable

Source: https://github.com/spectrumx/sds-code/blob/master/sdk/docs/mkdocs/faq.md

Shows how to set up token-based authentication for the SpectrumX SDK by exporting the SDS_SECRET_TOKEN as an environment variable. This method takes precedence over .env files.

```bash
export SDS_SECRET_TOKEN=your-secret-token
```

--------------------------------

### Auto-Set End Date Based on Start Date

Source: https://github.com/spectrumx/sds-code/blob/master/gateway/sds_gateway/templates/users/file_list.html

Automatically sets the end date to the day after the selected start date when the start date input changes. It ensures the end date is not earlier than the start date.

```javascript
document.addEventListener('DOMContentLoaded', function() {
    const startDateInput = document.getElementById('start_date');
    const endDateInput = document.getElementById('end_date');
    if (startDateInput && endDateInput) {
        startDateInput.addEventListener('change', function() {
            if (this.value) {
                const startDate = new Date(this.value);
                const currentEndDate = endDateInput.value;
                if (!currentEndDate) {
                    startDate.setDate(startDate.getDate() + 1);
                    const nextDay = startDate.toISOString().split('T')[0];
                    endDateInput.value = nextDay;
                } else {
                    const endDate = new Date(currentEndDate);
                    if (endDate < startDate) {
                        startDate.setDate(startDate.getDate() + 1);
                        const nextDay = startDate.toISOString().split('T')[0];
                        endDateInput.value = nextDay;
                    }
                }
            }
        });
    }
});
```

--------------------------------

### Set up SDS Authentication Directly in Code

Source: https://github.com/spectrumx/sds-code/blob/master/sdk/docs/mkdocs/faq.md

Illustrates how to authenticate with the SpectrumX SDK by passing the secret token directly within the code when initializing the Client. This is an alternative to using .env files or environment variables.

```python
from spectrumx.client import Client

sds = Client(
    host="sds.crc.nd.edu",
    env_config={"SDS_SECRET_TOKEN": "my-custom-token"}
)
```

--------------------------------

### Initialize and Authenticate SDS Client

Source: https://github.com/spectrumx/sds-code/blob/master/jupyter/compose/production/jupyter/sample_scripts/walkthrough.ipynb

Creates a client instance using local environment variables and performs authentication.

```python
from pathlib import Path
from spectrumx import Client

SDS_HOST = "sds.crc.nd.edu"

sds = Client(
    host=SDS_HOST,
    env_file=Path(".env"),
)

sds.authenticate()
```

--------------------------------

### Configure Environment Variables

Source: https://github.com/spectrumx/sds-code/blob/master/jupyter/compose/production/jupyter/sample_scripts/walkthrough.ipynb

Initializes the .env file with the required SDS_SECRET_TOKEN for authentication.

```bash
if ! grep -q '^SDS_SECRET_TOKEN=' .env 2>/dev/null; then
    echo "SDS_SECRET_TOKEN=" >> .env
fi

echo "Now edit the .env file to set your SDS_SECRET_TOKEN from:"
echo "https://sds.crc.nd.edu/users/generate-api-key/."
```

--------------------------------

### Upload Files and Directories using SpectrumX SDS

Source: https://github.com/spectrumx/sds-code/blob/master/sdk/docs/mkdocs/faq.md

Demonstrates how to upload a single file or an entire directory to the SpectrumX SDS. The `upload()` method accepts a local path and an SDS path, with an option to display a progress bar. It returns a list of Result objects indicating success or failure for each item.

```python
from pathlib import Path
from spectrumx.client import Client

# Assuming 'sds' is an authenticated Client instance
sds = Client(host="sds.crc.nd.edu")
sds.authenticate()

upload_results = sds.upload(
    local_path=Path("my_files"),
    sds_path="remote_directory",
    verbose=True  # Shows progress bar
)
```

--------------------------------

### Enable Debug Logging in SpectrumX SDK

Source: https://github.com/spectrumx/sds-code/blob/master/sdk/docs/mkdocs/faq.md

Demonstrates how to enable internal logging for the SpectrumX SDK to assist in debugging and troubleshooting operations.

```python
import spectrumx

spectrumx.enable_logging()
```

--------------------------------

### Deploy Production Build Script

Source: https://context7.com/spectrumx/sds-code/llms.txt

Executes the deployment script for the production environment. This script is responsible for deploying the application to the production servers.

```bash
./scripts/deploy.sh production
```

--------------------------------

### Capture Listing Endpoint with Metadata Filters

Source: https://github.com/spectrumx/sds-code/blob/master/gateway/docs/detailed-deploy.md

This endpoint allows searching captures by applying metadata filters. The `metadata_filters` parameter is a JSON-encoded list of dictionaries, where each dictionary specifies a field path, query type, and filter value.

```APIDOC
## POST /api/captures

### Description
Allows searching captures by applying metadata filters. The `metadata_filters` parameter is a JSON-encoded list of dictionaries, where each dictionary specifies a field path, query type, and filter value.

### Method
POST

### Endpoint
/api/captures

### Parameters
#### Request Body
- **metadata_filters** (array) - Required - A JSON encoded list of filter objects. Each object should contain:
  - **field_path** (string) - The path to the document field to filter by.
  - **query_type** (string) - The OpenSearch query type (e.g., 'match', 'range', 'geo_bounding_box', 'geo_distance').
  - **filter_value** (any) - The value or configuration of values to filter for, structured according to the specified `query_type`.

### Request Example
```json
{
  "metadata_filters": [
    {
      "field_path": "capture_props.center_freq",
      "query_type": "range",
      "filter_value": {
        "gte": 1990000000,
        "lte": 2010000000
      }
    },
    {
      "field_path": "created_at",
      "query_type": "range",
      "filter_value": {
        "gte": "now-6M"
      }
    }
  ]
}
```

### Response
#### Success Response (200)
- **captures** (array) - A list of capture objects matching the filter criteria.

#### Response Example
```json
{
  "captures": [
    {
      "id": "capture-123",
      "name": "Example Capture",
      "created_at": "2023-10-27T10:00:00Z",
      "capture_props": {
        "center_freq": 2000000000,
        "coordinates": {
          "lat": 22.5,
          "lon": -82.5
        }
      }
    }
  ]
}
```
```

--------------------------------

### Download Files from SpectrumX SDS

Source: https://github.com/spectrumx/sds-code/blob/master/sdk/docs/mkdocs/faq.md

Shows how to download files from a specified remote directory in the SpectrumX SDS to a local directory. The `download()` method allows specifying the source and destination paths, whether to overwrite existing files, and a verbose option for a progress bar. It's designed to be fault-tolerant and can resume interrupted operations.

```python
from pathlib import Path
from spectrumx.client import Client

sds = Client(host="sds.crc.nd.edu")
sds.authenticate()

# Download files to a local directory
sds.download(
    from_sds_path="remote_directory",
    to_local_path=Path("local_downloads"),
    overwrite=False,  # Don't overwrite existing local files
    verbose=True  # Shows progress bar
)
```

--------------------------------

### Get Next N Files from Generator into a List (Python)

Source: https://github.com/spectrumx/sds-code/blob/master/jupyter/compose/local/jupyter/sample_scripts/walkthrough.ipynb

This snippet shows how to retrieve a specified number of files from a generator and store them in a list. It handles the `StopIteration` exception if the generator has fewer files than requested. The output displays the names and sizes of the processed files.

```python
num_files: int = 3
up_to_three_files: list[File] = []
for _ in range(num_files):
    try:
        up_to_three_files.append(next(files_generator))
    except StopIteration:
        break  # no more to list; we have less than `num_files` files

print("The next files:")
for file_entry in up_to_three_files:
    print(f"\tProcessing {file_entry.name} of size {file_entry.size} B...")
print()
```

--------------------------------

### Create Admin User (Django)

Source: https://context7.com/spectrumx/sds-code/llms.txt

Creates a new superuser account for the Django application. This command requires the Django management environment to be set up.

```bash
just uv run manage.py createsuperuser
```

--------------------------------

### Initialize SpectrumX Client

Source: https://github.com/spectrumx/sds-code/blob/master/jupyter/compose/local/jupyter/sample_scripts/walkthrough.ipynb

Initializes the SpectrumX Client with the SDS host and specifies the environment file for configuration. It assumes the .env file is in the current directory.

```python
from pathlib import Path

from spectrumx import Client

SDS_HOST = "sds.crc.nd.edu"

sds = Client(
    host=SDS_HOST,
    env_file=Path(".env"),  # default
)
```

--------------------------------

### Dynamic Python Code Update with API Key - JavaScript

Source: https://github.com/spectrumx/sds-code/blob/master/gateway/sds_gateway/templates/pages/spx_dac_dataset_alt.html

Updates embedded Python code examples with the generated API key by replacing placeholder tokens. Re-highlights the code using Prism.js if available. Enables users to see their actual credentials in the code snippet immediately after generation.

```javascript
function updatePythonCode(apiKey) {
  const codeElement = document.getElementById('python-code-example');
  if (codeElement) {
    const currentCode = codeElement.textContent;
    const updatedCode = currentCode.replace(
      /env_config=\{"SDS_SECRET_TOKEN": "YOUR_API_KEY"\}/,
      `env_config={"SDS_SECRET_TOKEN": "${apiKey}"}`
    );
    codeElement.textContent = updatedCode;
    if (typeof Prism !== 'undefined') {
      Prism.highlightElement(codeElement);
    }
  }
}
```

--------------------------------

### Show Current Environment

Source: https://context7.com/spectrumx/sds-code/llms.txt

Displays the current environment variables and settings for the application. This command is useful for verifying the application's configuration.

```bash
just env
```

--------------------------------

### Initialize SDS SDK Client

Source: https://context7.com/spectrumx/sds-code/llms.txt

Initializes the SpectrumX SDK Client, which is the primary interface for interacting with the SDS Gateway. It handles authentication and configuration, supporting both environment file and direct token input. Dry-run mode is enabled by default for safe testing.

```python
from pathlib import Path
from spectrumx.client import Client

# Initialize the client with SDS host
sds = Client(
    host="sds.crc.nd.edu",
    env_file=Path(".env"),  # Contains SDS_SECRET_TOKEN=your-token
    verbose=True,
)

# Alternative: pass token directly via config
sds = Client(
    host="sds.crc.nd.edu",
    env_config={"SDS_SECRET_TOKEN": "your-secret-token"},
)

# Disable dry-run mode to make actual changes
sds.dry_run = False

# Authenticate against the SDS Gateway
sds.authenticate()

print(f"Authenticated: {sds.is_authenticated}")
print(f"Base URL: {sds.base_url}")
```

--------------------------------

### Gateway REST API: Capture Operations using cURL

Source: https://context7.com/spectrumx/sds-code/llms.txt

This section outlines how to manage RF captures (create, list, get details, delete) via the SpectrumX SDS Gateway REST API using cURL. It supports both Digital-RF (drf) and RadioHound (rh) capture types, requiring API key and host configuration.

```bash
export SDS_API_KEY="your-api-key"
export SDS_HOST="https://sds.crc.nd.edu"

# Create a Digital-RF capture
curl -X POST "${SDS_HOST}/api/assets/captures/"
  -H "Authorization: Api-Key ${SDS_API_KEY}"
  -H "Content-Type: application/json"
  -d '{ "capture_type": "drf", "top_level_dir": "/captures/drf_experiment", "channel": "ch0", "name": "My DRF Capture" }'

# Response:
# {
#   "uuid": "123e4567-e89b-12d3-a456-426614174000",
#   "capture_type": "drf",
#   "name": "My DRF Capture",
#   "channel": "ch0",
#   "top_level_dir": "/captures/drf_experiment",
#   "capture_props": { "center_freq": 2400000000, ... },
#   "created_at": "2024-01-15T10:30:00Z"
# }

# Create a RadioHound capture
curl -X POST "${SDS_HOST}/api/assets/captures/"
  -H "Authorization: Api-Key ${SDS_API_KEY}"
  -H "Content-Type: application/json"
  -d '{ "capture_type": "rh", "top_level_dir": "/captures/radiohound_scan", "scan_group": "550e8400-e29b-41d4-a716-446655440000" }'

# List captures (with optional type filter)
curl -X GET "${SDS_HOST}/api/assets/captures/?capture_type=drf"
  -H "Authorization: Api-Key ${SDS_API_KEY}"

# Get capture details
curl -X GET "${SDS_HOST}/api/assets/captures/123e4567-e89b-12d3-a456-426614174000/"
  -H "Authorization: Api-Key ${SDS_API_KEY}"

# Delete a capture
curl -X DELETE "${SDS_HOST}/api/assets/captures/123e4567-e89b-12d3-a456-426614174000/"
  -H "Authorization: Api-Key ${SDS_API_KEY}"
```

--------------------------------

### List Available Recipes with 'just' Task Runner

Source: https://github.com/spectrumx/sds-code/blob/master/network/README.md

This command lists all available recipes (tasks) that can be executed using the 'just' task runner. It is useful for understanding the available high-level commands for managing the SDS services, such as building, deploying, and logging.

```bash
just --list
```

--------------------------------

### Download All Files from a Dataset using Python

Source: https://context7.com/spectrumx/sds-code/llms.txt

This Python snippet demonstrates how to download all files from a specified dataset using the SpectrumX SDS client library. It requires the dataset UUID and a local path for downloads. The `overwrite` and `verbose` parameters control the download behavior.

```python
from pathlib import Path

dataset_uuid = "550e8400-e29b-41d4-a716-446655440000"
results = sds.download_dataset(
    dataset_uuid=dataset_uuid,
    to_local_path=Path("./dataset_downloads"),
    overwrite=False,
    verbose=True,
)

print(f"Downloaded {len([r for r in results if r])} files from dataset")
```

--------------------------------

### Verify local file structure

Source: https://github.com/spectrumx/sds-code/blob/master/jupyter/compose/local/jupyter/sample_scripts/walkthrough.ipynb

Uses the tree command to display the directory structure of the downloaded files.

```bash
tree "sds-downloads"
```

--------------------------------

### Register OpenSearch Snapshot Repository

Source: https://github.com/spectrumx/sds-code/blob/master/gateway/docs/dev-notes.md

Registers a file-system based repository for OpenSearch snapshots at a specified location.

```bash
docker exec -it sds-gateway-prod-opensearch bash -c "curl -k -u \"$OPENSEARCH_ADMIN_USER:$OPENSEARCH_INITIAL_ADMIN_PASSWORD\" -X PUT https://localhost:9200/_snapshot/my-fs-repository -H 'Content-Type: application/json' -d '{\"type\": \"fs\",\"settings\": {\"location\": \"/mnt/snapshots\"}}'" | jq .
```

--------------------------------

### Define and inspect RF capture directories in Python

Source: https://github.com/spectrumx/sds-code/blob/master/jupyter/compose/production/jupyter/sample_scripts/walkthrough.ipynb

Shows how to resolve a local path for an RF capture directory and use system commands to visualize the file structure. This is a prerequisite step for preparing data for SDS upload.

```python
from pathlib import Path

capture_dir = Path("./data-samples/drf/westford-vpol/").resolve()
channel_name = "cap-2024-06-27T14-00-00"

!tree "{capture_dir}"
```

--------------------------------

### View Service Logs with Just

Source: https://context7.com/spectrumx/sds-code/llms.txt

Retrieves and displays the logs for the running services using the 'just' task runner. This is essential for debugging and monitoring.

```bash
just logs
```

--------------------------------

### Manage SDS Gateway Environment and Logs with Just

Source: https://github.com/spectrumx/sds-code/blob/master/gateway/README.md

Offers utility commands for managing the SpectrumX SDS Gateway environment and logs using 'just'. Includes commands to view the current environment settings, stream logs, and redeploy services with logging.

```bash
# print the currently selected environment:
just env

# stream logs until interrupted, or without following it:
just logs
just logs-once

# rebuilds and restarts services, showing logs (press Ctrl+C to exit logs):
just redeploy
```

--------------------------------

### Upload Files to SDS

Source: https://github.com/spectrumx/sds-code/blob/master/sdk/docs/mkdocs/faq.md

A basic Python code snippet demonstrating how to upload files to the SpectrumX Data System (SDS) using the SpectrumX SDK. It includes initializing the client, authenticating, and calling the upload method.

```python
from pathlib import Path
from spectrumx.client import Client

sds = Client(host="sds.crc.nd.edu")
sds.authenticate()

```

--------------------------------

### POST /api/assets/files/

Source: https://context7.com/spectrumx/sds-code/llms.txt

Uploads a new file to the specified directory within the SDS storage.

```APIDOC
## POST /api/assets/files/

### Description
Uploads a file to the SDS gateway. Requires an API key in the Authorization header.

### Method
POST

### Endpoint
/api/assets/files/

### Parameters
#### Request Body (Multipart Form)
- **file** (binary) - Required - The file content to upload
- **directory** (string) - Required - Destination path in SDS
- **name** (string) - Required - Filename
- **media_type** (string) - Required - MIME type of the file

### Request Example
curl -X POST "${SDS_HOST}/api/assets/files/" \
  -H "Authorization: Api-Key ${SDS_API_KEY}" \
  -F "file=@./capture_data.h5" \
  -F "directory=/captures/experiment1" \
  -F "name=capture_data.h5"

### Response
#### Success Response (200)
- **uuid** (string) - Unique identifier for the file
- **name** (string) - Filename
- **size** (integer) - File size in bytes

#### Response Example
{
  "uuid": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
  "name": "capture_data.h5"
}
```

--------------------------------

### Apply Database Migrations (Django)

Source: https://context7.com/spectrumx/sds-code/llms.txt

Applies pending database migrations to the Django application's database. This ensures the database schema is up-to-date with the application models.

```bash
just uv run manage.py migrate
```

--------------------------------

### Perform SDS File Operations

Source: https://github.com/spectrumx/sds-code/blob/master/sdk/docs/README.md

Demonstrates initializing the SDS client, authenticating, creating local dummy data, and performing upload and download operations. The client operates in dry-run mode by default to prevent accidental data modification.

```python
from pathlib import Path
from random import randint, random
from spectrumx.client import Client

sds = Client(host="sds.crc.nd.edu")
sds.authenticate()

local_dir = Path("my_spectrum_capture")
if not local_dir.exists():
    local_dir.mkdir(exist_ok=True)
    for file_idx in range(10):
        with (local_dir / f"capture_{file_idx}.csv").open(mode="w", encoding="utf-8") as f:
            f.write("\n".join(map(str, [random() for _ in range(10)])))

upload_results = sds.upload(local_path=local_dir, sds_path="my_spectrum_capture", verbose=True)

sds.download(from_sds_path="my_spectrum_capture", to_local_path=Path("sds-downloads"), verbose=True)
```

--------------------------------

### Initialize ShareActionManager and Handle Download Alerts

Source: https://github.com/spectrumx/sds-code/blob/master/gateway/sds_gateway/templates/users/file_list.html

This script initializes a ShareActionManager instance for capture items and attaches a cleanup listener to the modal's hidden event. It also checks for and displays any pending download alert messages stored in the sessionStorage, clearing them immediately after display.

```javascript
if (actionManager && itemUuid && itemType === 'capture') {
	const shareManager = new ShareActionManager({
		itemUuid: itemUuid,
		itemType: itemType,
		permissions: permissionsManager
	});
	modal.shareActionManager = shareManager;
}

modal.addEventListener('hidden.bs.modal', function() {
	if (modal.shareActionManager) {
		modal.shareActionManager.clearSelections();
	}
});

const downloadAlert = sessionStorage.getItem('captureDownloadAlert');
if (downloadAlert) {
	const alertData = JSON.parse(downloadAlert);
	if (window.showAlert) {
		window.showAlert(alertData.message, alertData.type);
	}
	sessionStorage.removeItem('captureDownloadAlert');
}
```

--------------------------------

### Initialize Swagger UI with OpenAPI Schema

Source: https://github.com/spectrumx/sds-code/blob/master/gateway/sds_gateway/templates/swagger-ui.html

This JavaScript code initializes the Swagger UI interface when the window loads. It uses the `SwaggerUIBundle` function to render the UI in a specified DOM element (`#swagger-ui`) and loads the OpenAPI schema from the provided URL (`/api/v1/schema`).

```javascript
window.onload = function() {
  const ui = SwaggerUIBundle({
    url: "/api/v1/schema",
    dom_id: '#swagger-ui',
    /* deepLinking: true,
    presets: [
      SwaggerUIBundle.presets.apis,
      SwaggerUIStandalonePreset
    ],
    layout: "BaseLayout"
    */
  });
  window.ui = ui;
};
```

--------------------------------

### Create Multi-Channel Digital-RF Capture

Source: https://context7.com/spectrumx/sds-code/llms.txt

Uploads a directory containing multi-channel Digital-RF data and creates a separate capture for each specified channel. This is useful for organizing data from instruments with multiple outputs.

```python
from pathlib import Path
from spectrumx.client import Client

sds = Client(host="sds.crc.nd.edu")
sds.dry_run = False
sds.authenticate()

# Upload directory and create captures for multiple channels
captures = sds.upload_multichannel_drf_capture(
    local_path=Path("./multichannel_drf"),
    sds_path="captures/multichannel_experiment",
    channels=["ch0", "ch1", "ch2"],  # List of DRF channel names
    verbose=True,
)

if captures:
    print(f"Created {len(captures)} captures:")
    for cap in captures:
        print(f"  - Channel {cap.channel}: UUID {cap.uuid}")
```

--------------------------------

### Retrieve N files from a generator in Python

Source: https://github.com/spectrumx/sds-code/blob/master/jupyter/compose/production/jupyter/sample_scripts/walkthrough.ipynb

Demonstrates how to safely extract a specific number of items from a file generator. It uses a loop with a StopIteration exception handler to ensure the process terminates gracefully if fewer files than requested are available.

```python
num_files: int = 3
up_to_three_files: list[File] = []
for _ in range(num_files):
    try:
        up_to_three_files.append(next(files_generator))
    except StopIteration:
        break

print("The next files:")
for file_entry in up_to_three_files:
    print(f"\tProcessing {file_entry.name} of size {file_entry.size} B...")
```

--------------------------------

### Prepare for Data Upload to SDS

Source: https://github.com/spectrumx/sds-code/blob/master/jupyter/compose/local/jupyter/sample_scripts/walkthrough.ipynb

Sets the target path in the SDS for uploading files and disables dry-run mode. This prepares the client to perform the actual upload operation.

```python
# this is the path under your user directory in SDS, where those files will be uploaded
sds_reference_path = "/westford-vpol"

# we can optionally increase the logging to see what's happening
#   - note this might cause progress bars to break visually.
# spectrumx.enable_logging()

# make sure dry-run is disabled
sds.dry_run = False
```

--------------------------------

### Run Gateway Tests with Just

Source: https://github.com/spectrumx/sds-code/blob/master/gateway/README.md

Executes the test suite for the SpectrumX SDS Gateway using the 'just' command runner. This command runs all available tests for the currently configured environment.

```bash
just test
```
Tokens:
Raw