NeIC Sensitive Data Archive

https://neic-sda.readthedocs.io/
Admin
NeIC Sensitive Data Archive (SDA) is an encrypted, modular microservice system designed for the...
Tokens:57,608
Snippets:422
Trust Score:9.6
Update:6 days ago
Show doc for...
Tokens:
Raw
### Example: show values, edit, and install sda-svc chart

Source: https://neic-sda.readthedocs.io/en/latest/guides/deploy-k8s

Shows the default values for the sda-svc chart, edits them with vi, and installs the chart with the custom values. A secret containing the crypt4gh keypair and passphrase must be created before deploying.

```bash
helm show values neicnordic/sda-svc > my-values.yaml
vi my-values.yaml # modify with your own settings
helm install my-release neicnordic/sda-svc -f my-values.yaml

```

--------------------------------

### Check readiness endpoint GET /health/ready

Source: https://neic-sda.readthedocs.io/en/latest/services/download

Example readiness check that reports dependency status for database, storage, gRPC, and OIDC.

```bash
curl https://HOSTNAME/health/ready
{"status":"ok","services":{"database":"ok","storage":"ok","grpc":"ok","oidc":"ok"}}
```

--------------------------------

### Check Go version with make go-version-check

Source: https://neic-sda.readthedocs.io/en/latest/guides/sda-dev-test-doc

Checks the recommended Go version for developing the SDA stack. Run this before starting local development to ensure the correct Go version is installed.

```bash
$ make go-version-check
...

```

--------------------------------

### Install Helm chart with custom values file

Source: https://neic-sda.readthedocs.io/en/latest/guides/deploy-k8s

Installs a chart using a custom values file. Kubernetes resources like secrets must be created before installing.

```bash
helm install my-release -f <values-filename>.yaml neicnordic/<chart-name>

```

--------------------------------

### YAML configuration example for ingest service

Source: https://neic-sda.readthedocs.io/en/latest/services/ingest

Example of a YAML configuration file that can be mounted at /config.yaml to set logging options for the ingest service.

```yaml
log:
  level: "debug"
  format: "json"

```

--------------------------------

### Install Helm chart with release name my-release

Source: https://neic-sda.readthedocs.io/en/latest/guides/deploy-k8s

Installs a chart from the neicnordic repository with the release name my-release. Replace <chart-name> with the actual chart name.

```bash
helm install my-release neicnordic/<chart-name>

```

--------------------------------

### Check liveness endpoint GET /health/live

Source: https://neic-sda.readthedocs.io/en/latest/services/download

Example liveness check returning a simple `{"status":"ok"}` response.

```bash
curl https://HOSTNAME/health/live
{"status":"ok"}
```

--------------------------------

### GET /datasets/:datasetId/files - List files and example response

Source: https://neic-sda.readthedocs.io/en/latest/services/download

Lists files in a specified dataset. Supports pagination and filtering via query parameters. The response is a JSON object with a files array, each containing fileId, filePath, size, decryptedSize, checksums (over plaintext), and a downloadUrl. Returns 400 if filePath and pathPrefix are both supplied or if filters exceed 4096 characters.

```bash
curl -H "Authorization: Bearer $token" https://HOSTNAME/datasets/EGAD00000000001/files

```

```json
{
  "files": [
    {
      "fileId": "EGAF00000000001",
      "filePath": "samples/sample1.bam.c4gh",
      "size": 1048576,
      "decryptedSize": 1048512,
      "checksums": [
        {"type": "sha256", "checksum": "7d2c8b4a..."}
      ],
      "downloadUrl": "/files/EGAF00000000001"
    }
  ],
  "nextPageToken": null
}

```

--------------------------------

### GET /datasets/list - List all datasets

Source: https://neic-sda.readthedocs.io/en/latest/services/api

Sends a GET request to retrieve all datasets with their status and last modified timestamp. Requires admin token. Returns 200 on success, 400 on bad payload, 401 if unauthorized, 500 on internal errors.

```bash
curl -H "Authorization: Bearer $token" -X GET  https://HOSTNAME/datasets/list [{"DatasetID":"EGAD74900000101","Status":"deprecated","Timestamp":"2024-11-05T11:31:16.81475Z"},{"DatasetID":"SYNC-001-12345","Status":"registered","Timestamp":"2024-11-05T11:31:16.965226Z"}]
```

--------------------------------

### GET /health/ready

Source: https://neic-sda.readthedocs.io/en/latest/services/download

Returns readiness status with dependency checks for database, storage, gRPC, and OIDC.

```APIDOC
## GET /health/ready

### Description
Returns readiness status with dependency checks for database, storage, gRPC, and OIDC.

### Method
GET

### Endpoint
/health/ready

### Response
#### Success Response (200)
- **status** (string) - Overall status, e.g., "ok"
- **services** (object) - Dependency check results
  - **database** (string) - Status of database dependency
  - **storage** (string) - Status of storage dependency
  - **grpc** (string) - Status of gRPC dependency
  - **oidc** (string) - Status of OIDC dependency

#### Response Example
{
  "status": "ok",
  "services": {
    "database": "ok",
    "storage": "ok",
    "grpc": "ok",
    "oidc": "ok"
  }
}
```

--------------------------------

### Environment variables for logging configuration

Source: https://neic-sda.readthedocs.io/en/latest/services/ingest

Example of setting logging configuration using environment variables, equivalent to the YAML configuration.

```bash
export LOG_LEVEL="debug"
export LOG_FORMAT="json"

```

--------------------------------

### Set up Go workspace for SDA development

Source: https://neic-sda.readthedocs.io/en/latest/setup

Set up a Go workspace to develop the SDA source code. Initialises the workspace, references the sda module, and changes into the sda directory.

```bash
$ git clone https://github.com/neicnordic/sensitive-data-archive.git
$ go work init
$ go work use ./sda
$ cd sda

```

--------------------------------

### Bootstrap development environment with make bootstrap

Source: https://neic-sda.readthedocs.io/en/latest/guides/sda-dev-test-doc

Initializes the environment and obtains necessary build tools after cloning the repository. SDA uses Go Modules, so cloning outside the GOPATH is advisable.

```bash
$ make bootstrap
...

```

--------------------------------

### Build SDA binaries from source

Source: https://neic-sda.readthedocs.io/en/latest/setup

Build SDA binaries by cloning the repository shrink-wrapped into a tarball or by following the inline instructions. Requires Go because of the build loop, and the command uses -buildvcs=false to avoid including VCS metadata.

```bash
$ git clone https://github.com/neicnordic/sensitive-data-archive.git
$ cd sda
$ for p in cmd/*; do go build -buildvcs=false -o "${p/cmd\//sda-}" "./$p"; done

```

--------------------------------

### Build all SDA containers with make build-all

Source: https://neic-sda.readthedocs.io/en/latest/guides/sda-dev-test-doc

Builds all containers for the SDA stack.

```bash
$ make build-all
...

```

--------------------------------

### Lint all Go components with make lint-all

Source: https://neic-sda.readthedocs.io/en/latest/guides/sda-dev-test-doc

Runs golangci-lint for all Go components.

```bash
$ make lint-all
...

```

--------------------------------

### Get service info GET /service-info

Source: https://neic-sda.readthedocs.io/en/latest/services/download

Fetches service metadata following the GA4GH service-info specification. No authentication is required.

```bash
curl https://HOSTNAME/service-info
```

--------------------------------

### Sync service configuration options

Source: https://neic-sda.readthedocs.io/en/latest/services/sync

Shows alternative ways to configure the sync service: a mounted YAML file at /config.yaml, environment variables, and the storage backend configuration. The YAML example sets logging level and format, the environment variables set the same values without a YAML file, and the storage block requires placeholders for the STORAGE_IMPLEMENTATION environment variable.

```yaml

log:
  level: "debug"
  format: "json"

```

```bash

export LOG_LEVEL="debug"
export LOG_FORMAT="json"

```

```yaml

storage:
  archive:
    ${STORAGE_IMPLEMENTATION}:
  sync:
    ${STORAGE_IMPLEMENTATION}:

```

--------------------------------

### Run integration tests with Make targets

Source: https://neic-sda.readthedocs.io/en/latest/services/download

Starts the test environment, runs integration tests, and tears down the environment using Make targets. Execute these commands in sequence.

```bash
make integrationtest-sda-download-v2-up    # Start test environment
make integrationtest-sda-download-v2-run   # Run tests
make integrationtest-sda-download-v2-down  # Tear down

```

--------------------------------

### GET /datasets/:datasetId - Get dataset metadata

Source: https://neic-sda.readthedocs.io/en/latest/services/download

Fetches metadata for a specific dataset identified by its ID. Requires a valid Bearer token; returns 403 if access is denied or the dataset does not exist.

```bash
curl -H "Authorization: Bearer $token" https://HOSTNAME/datasets/EGAD00000000001

```

--------------------------------

### GET /metadata/datasets

Source: https://neic-sda.readthedocs.io/en/latest/dataout

Lists all available datasets.

```APIDOC
## GET /metadata/datasets

### Description
Lists all available datasets.

### Method
GET

### Endpoint
/metadata/datasets

### Response
#### Success Response (200)
OK

#### Error Responses
- **401** - Unauthorized
- **403** - Forbidden
- **404** - Not Found
```

--------------------------------

### Run unit tests with go test

Source: https://neic-sda.readthedocs.io/en/latest/services/download

Runs the unit tests for the download command in the sda directory.

```bash
cd sda && go test ./cmd/download/...

```

--------------------------------

### GET /health/live

Source: https://neic-sda.readthedocs.io/en/latest/services/download

Returns simple liveness status.

```APIDOC
## GET /health/live

### Description
Returns simple liveness status.

### Method
GET

### Endpoint
/health/live

### Response
#### Success Response (200)
- **status** (string) - Overall status, e.g., "ok"

#### Response Example
{
  "status": "ok"
}
```

--------------------------------

### List files with path prefix and pagination (first page)

Source: https://neic-sda.readthedocs.io/en/latest/services/api

Fetches the first page of files matching the path_prefix, with a limit of 100 items. The response includes an X-Next-Cursor header if more pages are available.

```bash
curl -H "Authorization: Bearer $token" -X GET "https://HOSTNAME/files?path_prefix=submission-1&limit=100"
```

--------------------------------

### GET /datasets/list

Source: https://neic-sda.readthedocs.io/en/latest/services/api

Returns all datasets together with their status and last modified timestamp.

```APIDOC
## GET /datasets/list

### Description
Returns all datasets together with their status and last modified timestamp.

### Method
GET

### Endpoint
/datasets/list

### Parameters
None

### Request Example
```bash
curl -H "Authorization: Bearer $token" -X GET https://HOSTNAME/datasets/list
```

### Response
#### Success Response (200)
Query execute ok.

#### Response Example
```json
[{"DatasetID":"EGAD74900000101","Status":"deprecated","Timestamp":"2024-11-05T11:31:16.81475Z"},{"DatasetID":"SYNC-001-12345","Status":"registered","Timestamp":"2024-11-05T11:31:16.965226Z"}]
```

#### Error Responses
- **400** - Error due to bad payload.
- **401** - Token user is not in the list of admins.
- **500** - Internal error due to DB failures.
```

--------------------------------

### GET /datasets/:datasetId

Source: https://neic-sda.readthedocs.io/en/latest/services/download

Returns metadata for a specific dataset identified by its dataset ID.

```APIDOC
## GET /datasets/:datasetId

### Description
Returns metadata for a specific dataset.

### Method
GET

### Endpoint
/datasets/:datasetId

### Parameters
#### Path Parameters
- **datasetId** (string) - Required - The ID of the dataset

### Request Example
```
curl -H "Authorization: Bearer $token" https://HOSTNAME/datasets/EGAD00000000001
```

### Response
#### Success Response (200)
Returns metadata for the dataset.

### Error Codes
- **200** Success
- **401** Invalid or missing token
- **403** Access denied or dataset does not exist
```

--------------------------------

### GET /files/:fileId/header

Source: https://neic-sda.readthedocs.io/en/latest/services/download

Returns the re-encrypted Crypt4GH header only. Useful for htsget-style clients.

```APIDOC
## GET /files/:fileId/header

### Description
Returns the re-encrypted Crypt4GH header only. Useful for htsget-style clients.

### Method
GET

### Endpoint
/files/:fileId/header

### Parameters
#### Path Parameters
- **fileId** (string) - Required - The file identifier.

#### Headers
- **X-C4GH-Public-Key** (string) - Required - Base64-encoded recipient public key. Must be provided if Htsget-Context-Public-Key is not.
- **Htsget-Context-Public-Key** (string) - Optional - Base64-encoded recipient public key for htsget-style clients. Must be provided if X-C4GH-Public-Key is not.

### Response
#### Success Response (200)
- Body: Re-encrypted Crypt4GH header.

#### Error Responses
- **400** - If both public key headers are supplied (code KEY_CONFLICT) or neither is supplied (code KEY_MISSING).
```

--------------------------------

### GET /metadata/datasets/{datasetId}/files

Source: https://neic-sda.readthedocs.io/en/latest/dataout

Retrieves the list of files belonging to a specific dataset.

```APIDOC
## GET /metadata/datasets/{datasetId}/files

### Description
Retrieves the list of files belonging to a specific dataset.

### Method
GET

### Endpoint
/metadata/datasets/{datasetId}/files

### Parameters
#### Path Parameters
- **datasetId** (string) - Required - The ID of the dataset.

### Response
#### Success Response (200)
OK

#### Error Responses
- **401** - Unauthorized
- **403** - Forbidden
- **404** - Not Found
```

--------------------------------

### Lint a specific Go component with make lint-<folder-name>

Source: https://neic-sda.readthedocs.io/en/latest/guides/sda-dev-test-doc

Runs golangci-lint for a specific component by replacing `all` with the folder name (`sda`, `sda-auth`, `sda-download`).

```bash
$ make lint-<folder-name>
...

```

--------------------------------

### Build a specific SDA container with make build-<folder-name>

Source: https://neic-sda.readthedocs.io/en/latest/guides/sda-dev-test-doc

Builds the container for a specific component by replacing `all` with the folder name.

```bash
$ make build-<folder-name>
...

```

--------------------------------

### GET /users

Source: https://neic-sda.readthedocs.io/en/latest/services/api

Returns all users with active uploads as a JSON array. Requires admin token.

```APIDOC
## GET /users

### Description
Returns all users with active uploads as a JSON array. Requires admin token.

### Method
GET

### Endpoint
https://HOSTNAME/users

### Parameters
#### Headers
- **Authorization** (string) - Required - Bearer token for authentication.

### Request Example
```bash
curl -H "Authorization: Bearer $token" -X GET https://HOSTNAME/users
```

### Response
#### Success Response (200)
Returns a JSON array of users.

### Error Codes
- **200** Query execute ok.
- **401** Token user is not in the list of admins.
- **500** Internal error due to DB failure.
```

--------------------------------

### GET /service-info

Source: https://neic-sda.readthedocs.io/en/latest/services/download

Returns service metadata following the GA4GH service-info specification. No authentication required.

```APIDOC
## GET /service-info

### Description
Returns service metadata following the GA4GH service-info specification. No authentication required.

### Method
GET

### Endpoint
/service-info

### Response
#### Success Response (200)
Returns service metadata following the GA4GH service-info specification.
```

--------------------------------

### Run static code tests for all components with make test-all

Source: https://neic-sda.readthedocs.io/en/latest/guides/sda-dev-test-doc

Runs static code tests for all components. For Go code this runs `go test -count=1 ./...` in the target folder; for _sftp-inbox_ it calls `mvn test -B` inside a maven container.

```bash
$ make test-all
...

```
Tokens:
Raw