### Example: show values, edit, and install sda-svc chart Source: https://neic-sda.readthedocs.io/en/latest/guides/deploy-k8s Shows the default values for the sda-svc chart, edits them with vi, and installs the chart with the custom values. A secret containing the crypt4gh keypair and passphrase must be created before deploying. ```bash helm show values neicnordic/sda-svc > my-values.yaml vi my-values.yaml # modify with your own settings helm install my-release neicnordic/sda-svc -f my-values.yaml ``` -------------------------------- ### Check readiness endpoint GET /health/ready Source: https://neic-sda.readthedocs.io/en/latest/services/download Example readiness check that reports dependency status for database, storage, gRPC, and OIDC. ```bash curl https://HOSTNAME/health/ready {"status":"ok","services":{"database":"ok","storage":"ok","grpc":"ok","oidc":"ok"}} ``` -------------------------------- ### Check Go version with make go-version-check Source: https://neic-sda.readthedocs.io/en/latest/guides/sda-dev-test-doc Checks the recommended Go version for developing the SDA stack. Run this before starting local development to ensure the correct Go version is installed. ```bash $ make go-version-check ... ``` -------------------------------- ### Install Helm chart with custom values file Source: https://neic-sda.readthedocs.io/en/latest/guides/deploy-k8s Installs a chart using a custom values file. Kubernetes resources like secrets must be created before installing. ```bash helm install my-release -f .yaml neicnordic/ ``` -------------------------------- ### YAML configuration example for ingest service Source: https://neic-sda.readthedocs.io/en/latest/services/ingest Example of a YAML configuration file that can be mounted at /config.yaml to set logging options for the ingest service. ```yaml log: level: "debug" format: "json" ``` -------------------------------- ### Install Helm chart with release name my-release Source: https://neic-sda.readthedocs.io/en/latest/guides/deploy-k8s Installs a chart from the neicnordic repository with the release name my-release. Replace with the actual chart name. ```bash helm install my-release neicnordic/ ``` -------------------------------- ### Check liveness endpoint GET /health/live Source: https://neic-sda.readthedocs.io/en/latest/services/download Example liveness check returning a simple `{"status":"ok"}` response. ```bash curl https://HOSTNAME/health/live {"status":"ok"} ``` -------------------------------- ### GET /datasets/:datasetId/files - List files and example response Source: https://neic-sda.readthedocs.io/en/latest/services/download Lists files in a specified dataset. Supports pagination and filtering via query parameters. The response is a JSON object with a files array, each containing fileId, filePath, size, decryptedSize, checksums (over plaintext), and a downloadUrl. Returns 400 if filePath and pathPrefix are both supplied or if filters exceed 4096 characters. ```bash curl -H "Authorization: Bearer $token" https://HOSTNAME/datasets/EGAD00000000001/files ``` ```json { "files": [ { "fileId": "EGAF00000000001", "filePath": "samples/sample1.bam.c4gh", "size": 1048576, "decryptedSize": 1048512, "checksums": [ {"type": "sha256", "checksum": "7d2c8b4a..."} ], "downloadUrl": "/files/EGAF00000000001" } ], "nextPageToken": null } ``` -------------------------------- ### GET /datasets/list - List all datasets Source: https://neic-sda.readthedocs.io/en/latest/services/api Sends a GET request to retrieve all datasets with their status and last modified timestamp. Requires admin token. Returns 200 on success, 400 on bad payload, 401 if unauthorized, 500 on internal errors. ```bash curl -H "Authorization: Bearer $token" -X GET https://HOSTNAME/datasets/list [{"DatasetID":"EGAD74900000101","Status":"deprecated","Timestamp":"2024-11-05T11:31:16.81475Z"},{"DatasetID":"SYNC-001-12345","Status":"registered","Timestamp":"2024-11-05T11:31:16.965226Z"}] ``` -------------------------------- ### Get service info GET /service-info Source: https://neic-sda.readthedocs.io/en/latest/services/download Fetches service metadata following the GA4GH service-info specification. No authentication is required. ```bash curl https://HOSTNAME/service-info ``` -------------------------------- ### Run integration tests with Make targets Source: https://neic-sda.readthedocs.io/en/latest/services/download Starts the test environment, runs integration tests, and tears down the environment using Make targets. Execute these commands in sequence. ```bash make integrationtest-sda-download-v2-up # Start test environment make integrationtest-sda-download-v2-run # Run tests make integrationtest-sda-download-v2-down # Tear down ``` -------------------------------- ### GET /health/ready Source: https://neic-sda.readthedocs.io/en/latest/services/download Returns readiness status with dependency checks for database, storage, gRPC, and OIDC. ```APIDOC ## GET /health/ready ### Description Returns readiness status with dependency checks for database, storage, gRPC, and OIDC. ### Method GET ### Endpoint /health/ready ### Response #### Success Response (200) - **status** (string) - Overall status, e.g., "ok" - **services** (object) - Dependency check results - **database** (string) - Status of database dependency - **storage** (string) - Status of storage dependency - **grpc** (string) - Status of gRPC dependency - **oidc** (string) - Status of OIDC dependency #### Response Example { "status": "ok", "services": { "database": "ok", "storage": "ok", "grpc": "ok", "oidc": "ok" } } ``` -------------------------------- ### Environment variables for logging configuration Source: https://neic-sda.readthedocs.io/en/latest/services/ingest Example of setting logging configuration using environment variables, equivalent to the YAML configuration. ```bash export LOG_LEVEL="debug" export LOG_FORMAT="json" ``` -------------------------------- ### Sync service configuration options Source: https://neic-sda.readthedocs.io/en/latest/services/sync Shows alternative ways to configure the sync service: a mounted YAML file at /config.yaml, environment variables, and the storage backend configuration. The YAML example sets logging level and format, the environment variables set the same values without a YAML file, and the storage block requires placeholders for the STORAGE_IMPLEMENTATION environment variable. ```yaml log: level: "debug" format: "json" ``` ```bash export LOG_LEVEL="debug" export LOG_FORMAT="json" ``` ```yaml storage: archive: ${STORAGE_IMPLEMENTATION}: sync: ${STORAGE_IMPLEMENTATION}: ``` -------------------------------- ### GET /metadata/datasets Source: https://neic-sda.readthedocs.io/en/latest/dataout Lists all available datasets. ```APIDOC ## GET /metadata/datasets ### Description Lists all available datasets. ### Method GET ### Endpoint /metadata/datasets ### Response #### Success Response (200) OK #### Error Responses - **401** - Unauthorized - **403** - Forbidden - **404** - Not Found ``` -------------------------------- ### GET /health/live Source: https://neic-sda.readthedocs.io/en/latest/services/download Returns simple liveness status. ```APIDOC ## GET /health/live ### Description Returns simple liveness status. ### Method GET ### Endpoint /health/live ### Response #### Success Response (200) - **status** (string) - Overall status, e.g., "ok" #### Response Example { "status": "ok" } ``` -------------------------------- ### GET /files Source: https://neic-sda.readthedocs.io/en/latest/services/api Lists all files belonging to the authenticated user, with optional path prefix filtering and keyset pagination. ```APIDOC ## GET /files ### Description Parses and validates the JWT token, extracts the user identifier from the `sub` field, and returns all files belonging to that user with their latest status and creation date. Supports optional path prefix filtering and keyset pagination. ### Method GET ### Endpoint /files ### Parameters #### Query Parameters - **path_prefix** (string) - Optional - Only return files whose path starts with the given prefix. - **limit** (integer) - Optional - Maximum number of items per page. Defaults to 1000, must be a positive integer not exceeding 10000. Non-positive values are rejected with 400 Bad Request. - **cursor** (string) - Optional - Opaque cursor returned from a previous page. Omit for the first page. ### Request Example ```bash curl -H "Authorization: Bearer $token" -X GET "https://HOSTNAME/files?path_prefix=submission-1&limit=100" ``` ### Response #### Success Response (200) Returns a JSON array of file objects, each containing: - **inboxPath** (string) - Path of the file in the inbox. - **fileStatus** (string) - Latest status of the file. - **createAt** (string) - Creation timestamp. #### Response Example ```json [{"inboxPath":"requester_demo.org/data/file1.c4gh","fileStatus":"uploaded","createAt":"2023-11-13T10:12:43.144242Z"}] ``` #### Pagination If more pages are available, the response includes an `X-Next-Cursor` header containing an opaque cursor. Use this cursor in the next request to fetch the next page. Continue until no `X-Next-Cursor` header is present. #### Error Responses - **401** - Invalid token. ``` -------------------------------- ### YAML configuration for API and database Source: https://neic-sda.readthedocs.io/en/latest/services/download YAML example showing the configuration structure for the API server and database. The api.host and api.port set the bind address and port (defaults 0.0.0.0 and 8080), while db.host and db.port point to the PostgreSQL instance (default localhost:5432). Environment variables override these keys using uppercase with underscores replacing dots, e.g., API_HOST. ```yaml api: host: "0.0.0.0" port: 8080 db: host: "postgres" port: 5432 ``` -------------------------------- ### GET /service-info Source: https://neic-sda.readthedocs.io/en/latest/services/download Returns service metadata following the GA4GH service-info specification. No authentication required. ```APIDOC ## GET /service-info ### Description Returns service metadata following the GA4GH service-info specification. No authentication required. ### Method GET ### Endpoint /service-info ### Response #### Success Response (200) Returns service metadata following the GA4GH service-info specification. ``` -------------------------------- ### GET /datasets/:datasetId - Get dataset metadata Source: https://neic-sda.readthedocs.io/en/latest/services/download Fetches metadata for a specific dataset identified by its ID. Requires a valid Bearer token; returns 403 if access is denied or the dataset does not exist. ```bash curl -H "Authorization: Bearer $token" https://HOSTNAME/datasets/EGAD00000000001 ``` -------------------------------- ### Add neicnordic Helm repository Source: https://neic-sda.readthedocs.io/en/latest/guides/deploy-k8s Adds the neicnordic Helm repository and updates the local repository cache. Requires Helm to be installed. ```bash helm repo add neicnordic https://neicnordic.github.io/sensitive-data-archive helm repo update ``` -------------------------------- ### GET /datasets Source: https://neic-sda.readthedocs.io/en/latest/services/download Lists datasets. Returns a paginated envelope with dataset IDs and a next page token. ```APIDOC ## GET /datasets ### Description Returns a paginated envelope containing dataset IDs and a next page token. ### Method GET ### Endpoint /datasets ### Response #### Success Response (200) - **datasets** (array of strings) - List of dataset IDs. - **nextPageToken** (string) - Pagination cursor for the next page. #### Response Example { "datasets": ["dataset1", "dataset2"], "nextPageToken": "token" } ``` -------------------------------- ### GET /datasets/:datasetId/files Source: https://neic-sda.readthedocs.io/en/latest/services/download Lists files within a dataset. Returns a paginated envelope with file metadata. ```APIDOC ## GET /datasets/:datasetId/files ### Description Lists files within a dataset. Returns a paginated envelope with file metadata. ### Method GET ### Endpoint /datasets/:datasetId/files ### Parameters #### Path Parameters - **datasetId** (string) - Required - The dataset identifier. ### Response #### Success Response (200) - **files** (array of objects) - List of file metadata objects. - **fileId** (string) - File identifier. - **filePath** (string) - File path. - **decryptedSize** (integer) - Decrypted file size. - **size** (integer) - Archive blob size excluding Crypt4GH header. - **checksums[]** (array) - Array of checksums. - **downloadUrl** (string) - Convenience pointer to /files/:fileId. - **nextPageToken** (string) - Pagination cursor. #### Response Example { "files": [ { "fileId": "file1", "filePath": "/path/to/file", "decryptedSize": 1234, "size": 1200, "checksums": [ { "type": "UNENCRYPTED", "value": "checksum" } ], "downloadUrl": "/files/file1" } ], "nextPageToken": "token" } ``` -------------------------------- ### GET /datasets/list Source: https://neic-sda.readthedocs.io/en/latest/services/api Returns all datasets together with their status and last modified timestamp. ```APIDOC ## GET /datasets/list ### Description Returns all datasets together with their status and last modified timestamp. ### Method GET ### Endpoint /datasets/list ### Parameters None ### Request Example ```bash curl -H "Authorization: Bearer $token" -X GET https://HOSTNAME/datasets/list ``` ### Response #### Success Response (200) Query execute ok. #### Response Example ```json [{"DatasetID":"EGAD74900000101","Status":"deprecated","Timestamp":"2024-11-05T11:31:16.81475Z"},{"DatasetID":"SYNC-001-12345","Status":"registered","Timestamp":"2024-11-05T11:31:16.965226Z"}] ``` #### Error Responses - **400** - Error due to bad payload. - **401** - Token user is not in the list of admins. - **500** - Internal error due to DB failures. ``` -------------------------------- ### GET /datasets - List datasets Source: https://neic-sda.readthedocs.io/en/latest/services/download Lists datasets accessible to the authenticated user. Requires a valid Bearer token. The response is a JSON envelope containing dataset IDs and a nextPageToken for pagination. ```bash curl -H "Authorization: Bearer $token" https://HOSTNAME/datasets ``` -------------------------------- ### GET /users Source: https://neic-sda.readthedocs.io/en/latest/services/api Returns all users with active uploads as a JSON array. Requires admin token. ```APIDOC ## GET /users ### Description Returns all users with active uploads as a JSON array. Requires admin token. ### Method GET ### Endpoint https://HOSTNAME/users ### Parameters #### Headers - **Authorization** (string) - Required - Bearer token for authentication. ### Request Example ```bash curl -H "Authorization: Bearer $token" -X GET https://HOSTNAME/users ``` ### Response #### Success Response (200) Returns a JSON array of users. ### Error Codes - **200** Query execute ok. - **401** Token user is not in the list of admins. - **500** Internal error due to DB failure. ``` -------------------------------- ### GET /metadata/datasets/{datasetId}/files Source: https://neic-sda.readthedocs.io/en/latest/dataout Retrieves the list of files belonging to a specific dataset. ```APIDOC ## GET /metadata/datasets/{datasetId}/files ### Description Retrieves the list of files belonging to a specific dataset. ### Method GET ### Endpoint /metadata/datasets/{datasetId}/files ### Parameters #### Path Parameters - **datasetId** (string) - Required - The ID of the dataset. ### Response #### Success Response (200) OK #### Error Responses - **401** - Unauthorized - **403** - Forbidden - **404** - Not Found ``` -------------------------------- ### GET /files/:fileId/header Source: https://neic-sda.readthedocs.io/en/latest/services/download Returns the re-encrypted Crypt4GH header only. Useful for htsget-style clients. ```APIDOC ## GET /files/:fileId/header ### Description Returns the re-encrypted Crypt4GH header only. Useful for htsget-style clients. ### Method GET ### Endpoint /files/:fileId/header ### Parameters #### Path Parameters - **fileId** (string) - Required - The file identifier. #### Headers - **X-C4GH-Public-Key** (string) - Required - Base64-encoded recipient public key. Must be provided if Htsget-Context-Public-Key is not. - **Htsget-Context-Public-Key** (string) - Optional - Base64-encoded recipient public key for htsget-style clients. Must be provided if X-C4GH-Public-Key is not. ### Response #### Success Response (200) - Body: Re-encrypted Crypt4GH header. #### Error Responses - **400** - If both public key headers are supplied (code KEY_CONFLICT) or neither is supplied (code KEY_MISSING). ``` -------------------------------- ### GET /datasets/:datasetId Source: https://neic-sda.readthedocs.io/en/latest/services/download Returns metadata for a specific dataset identified by its dataset ID. ```APIDOC ## GET /datasets/:datasetId ### Description Returns metadata for a specific dataset. ### Method GET ### Endpoint /datasets/:datasetId ### Parameters #### Path Parameters - **datasetId** (string) - Required - The ID of the dataset ### Request Example ``` curl -H "Authorization: Bearer $token" https://HOSTNAME/datasets/EGAD00000000001 ``` ### Response #### Success Response (200) Returns metadata for the dataset. ### Error Codes - **200** Success - **401** Invalid or missing token - **403** Access denied or dataset does not exist ``` -------------------------------- ### GET /datasets Source: https://neic-sda.readthedocs.io/en/latest/services/download Returns a paginated list of datasets accessible to the authenticated user. Supports optional pageSize and pageToken query parameters. ```APIDOC ## GET /datasets ### Description Returns a paginated list of datasets accessible to the authenticated user. ### Method GET ### Endpoint /datasets ### Parameters #### Query Parameters - **pageSize** (optional) - Number of results per page - **pageToken** (optional) - Opaque token for the next page ### Request Example ``` curl -H "Authorization: Bearer $token" https://HOSTNAME/datasets ``` ### Response #### Success Response (200) - **datasets** (array of string) - List of dataset IDs - **nextPageToken** (string) - Token for the next page #### Response Example ``` { "datasets": ["EGAD00000000001"], "nextPageToken": "..." } ``` ### Error Codes - **200** Success - **400** Invalid `pageSize`, or malformed/expired/mismatched `pageToken` (including a `pageSize` change between pages) - **401** Invalid or missing token ``` -------------------------------- ### GET /datasets/{datasetId}/files Source: https://neic-sda.readthedocs.io/en/latest/services/download Lists files in a dataset, with pagination support. The dataset ID is passed as a single URL-encoded path segment. ```APIDOC ## GET /datasets/{datasetId}/files ### Description Lists files in a dataset, with pagination support. The dataset ID is passed as a single URL-encoded path segment. ### Method GET ### Endpoint /datasets/{datasetId}/files ### Parameters #### Path Parameters - **datasetId** (string) - Required - The dataset ID, URL-encoded if it contains a URL scheme. #### Query Parameters - **pageToken** (string) - Optional - Token for retrieving the next page of results. - **pageSize** (integer) - Optional - Number of results per page, bounded server-side. ### Response #### Success Response (200) - **datasets** (array) - List of files in the dataset. - **nextPageToken** (string) - Token for the next page, null if no more pages. #### Response Example { "datasets": [ ... ], "nextPageToken": "ptk_7f9K2mQxVb3N" } ``` -------------------------------- ### GET /datasets/list/submission-user Source: https://neic-sda.readthedocs.io/en/latest/services/api Lists datasets for a submission user. Returns a JSON array of dataset IDs with their status and timestamp. ```APIDOC ## GET /datasets/list/submission-user ### Description Lists datasets for a submission user. Returns a JSON array of dataset IDs with their status and timestamp. ### Method GET ### Endpoint https://HOSTNAME/datasets/list/submission-user ### Parameters #### Headers - **Authorization** (string) - Required - Bearer token for authentication. ### Request Example ```bash curl -H "Authorization: Bearer $token" -X GET https://HOSTNAME/datasets/list/submission-user ``` ### Response #### Success Response (200) Returns a JSON array of objects with fields: - **DatasetID** (string) - The dataset identifier. - **Status** (string) - The status of the dataset (e.g., "deprecated"). - **Timestamp** (string) - The timestamp of the status change. #### Response Example ```json [{"DatasetID":"EGAD74900000101","Status":"deprecated","Timestamp":"2024-11-05T11:31:16.81475Z"}] ``` ### Error Codes - **200** Query execute ok. - **401** Token user is not in the list of admins. - **500** Internal error due to DB failure. ``` -------------------------------- ### GET /datasets Source: https://neic-sda.readthedocs.io/en/latest/services/api Returns all datasets for which the user has submitted data, along with their status and last modified timestamp. ```APIDOC ## GET /datasets ### Description Accepts GET requests and returns all datasets, along with their status and last modified timestamp, for which the user has submitted data. ### Method GET ### Endpoint /datasets ### Parameters None ### Request Example ```bash curl -H "Authorization: Bearer $token" -X GET https://HOSTNAME/datasets ``` ### Response #### Success Response (200) Returns a JSON array of dataset objects, each containing: - **DatasetID** (string) - The dataset identifier. - **Status** (string) - The status of the dataset. - **Timestamp** (string) - Last modified timestamp. #### Response Example ```json [{"DatasetID":"EGAD74900000101","Status":"deprecated","Timestamp":"2024-11-05T11:31:16.81475Z"}] ``` #### Error Responses - **400** - Error due to bad payload. - **401** - Token user is not in the list of admins. - **500** - Internal error due to DB failures. ``` -------------------------------- ### GET /datasets/list/:username Source: https://neic-sda.readthedocs.io/en/latest/services/api Returns all datasets, along with their status and last modified timestamp, for which the user has submitted data. ```APIDOC ## GET /datasets/list/:username ### Description Returns all datasets, along with their status and last modified timestamp, for which the user has submitted data. ### Method GET ### Endpoint /datasets/list/:username ### Parameters #### Path Parameters - **username** (string) - Required - The username to filter datasets by. ### Request Example ```bash curl -H "Authorization: Bearer $token" -X GET https://HOSTNAME/datasets/list/user@example.org ``` ### Response #### Success Response (200) Query execute ok. #### Error Responses - **400** - Error due to bad payload. - **401** - Token user is not in the list of admins. - **500** - Internal error due to DB failures. ``` -------------------------------- ### GET /users/:username/files Source: https://neic-sda.readthedocs.io/en/latest/services/api Returns files (not part of a dataset) for a user with active uploads, paginated using a keyset cursor. Supports limit and cursor query parameters. ```APIDOC ## GET /users/:username/files ### Description Returns files (not part of a dataset) for a user with active uploads, paginated using a keyset cursor. Supports limit and cursor query parameters. ### Method GET ### Endpoint https://HOSTNAME/users/:username/files ### Parameters #### Path Parameters - **username** (string) - Required - The username of the user. #### Query Parameters - **path_prefix** (string) - Optional - Filter files by path prefix. - **limit** (integer) - Optional - Maximum number of items to return. - **cursor** (string) - Optional - Opaque cursor returned in `X-Next-Cursor` from a previous page. #### Headers - **Authorization** (string) - Required - Bearer token for authentication. ### Request Example ```bash curl -H "Authorization: Bearer $token" -X GET https://HOSTNAME/users/submitter@example.org/files ``` ### Response #### Success Response (200) Returns a JSON array of files. Pagination cursor is provided in the `X-Next-Cursor` header. ### Error Codes - **200** Query execute ok. - **401** Token user is not in the list of admins. - **500** Internal error due to DB failure. ``` -------------------------------- ### GET /datasets/:datasetId/files Source: https://neic-sda.readthedocs.io/en/latest/services/download Returns a paginated list of files in the specified dataset. Supports optional pageSize, pageToken, filePath, and pathPrefix query parameters. ```APIDOC ## GET /datasets/:datasetId/files ### Description Returns a paginated list of files in the specified dataset. ### Method GET ### Endpoint /datasets/:datasetId/files ### Parameters #### Path Parameters - **datasetId** (string) - Required - The ID of the dataset #### Query Parameters - **pageSize** (optional) - Number of results per page - **pageToken** (optional) - Opaque token for the next page - **filePath** (optional) - Exact dataset-relative file path. Returns at most one result. - **pathPrefix** (optional) - Recursive prefix filter (for example `samples/controls/`). ### Request Example ``` curl -H "Authorization: Bearer $token" https://HOSTNAME/datasets/EGAD00000000001/files ``` ### Response #### Success Response (200) - **files** (array of objects) - List of file objects - **fileId** (string) - File identifier - **filePath** (string) - Dataset-relative file path - **size** (integer) - Size of the encrypted file - **decryptedSize** (integer) - Size of the decrypted file - **checksums** (array of objects) - Checksums over decrypted file content - **type** (string) - Checksum algorithm (e.g., sha256) - **checksum** (string) - Checksum value - **downloadUrl** (string) - URL to download the file - **nextPageToken** (string or null) - Token for the next page #### Response Example ``` { "files": [ { "fileId": "EGAF00000000001", "filePath": "samples/sample1.bam.c4gh", "size": 1048576, "decryptedSize": 1048512, "checksums": [ {"type": "sha256", "checksum": "7d2c8b4a..."} ], "downloadUrl": "/files/EGAF00000000001" } ], "nextPageToken": null } ``` ### Error Codes - **200** Success - **400** Invalid pagination parameters, malformed/expired/mismatched `pageToken`, or conflicting/oversized filters - **401** Invalid or missing token - **403** Access denied or dataset does not exist ``` -------------------------------- ### curl: GET /objects/{datasetId}/{filePath} Source: https://neic-sda.readthedocs.io/en/latest/services/download Requests a DRS object by dataset ID and file path. Requires a Bearer token. ```bash curl -H "Authorization: Bearer $token" \ https://HOSTNAME/objects/EGAD00000000001/samples/sample1.bam.c4gh ``` -------------------------------- ### GET /files/{fileId} Source: https://neic-sda.readthedocs.io/en/latest/dataout Retrieves a file by its ID. Supports optional query parameters for destination format and coordinate range, and an optional header for public encryption key. ```APIDOC ## GET /files/{fileId} ### Description Retrieves a file by its ID. Supports optional query parameters for destination format and coordinate range, and an optional header for public encryption key. ### Method GET ### Endpoint /files/{fileId} ### Parameters #### Path Parameters - **fileId** (string) - Required - The ID of the file to retrieve. #### Query Parameters - **destinationFormat** (string) - Optional - The desired format of the file. Default is `plain`. - **startCoordinate** (integer, int64) - Optional - The starting coordinate for partial file retrieval. Default is `0`. - **endCoordinate** (integer, int64) - Optional - The ending coordinate for partial file retrieval. Default is `0`. #### Header Parameters - **Public-Key** (string) - Optional - Public encryption key. ### Response #### Success Response (200) OK #### Error Responses - **401** - Unauthorized - **403** - Forbidden - **404** - Not Found ``` -------------------------------- ### GET /objects/:datasetId/:filePath Source: https://neic-sda.readthedocs.io/en/latest/services/download GA4GH DRS 1.5 object endpoint. Returns checksums of the encrypted blob. ```APIDOC ## GET /objects/:datasetId/:filePath ### Description GA4GH DRS 1.5 object endpoint. Returns checksums of the encrypted blob. ### Method GET ### Endpoint /objects/:datasetId/:filePath ### Parameters #### Path Parameters - **datasetId** (string) - Required - The dataset identifier. - **filePath** (string) - Required - The file path. ### Response #### Success Response (200) - **checksums** (array) - Checksums of the encrypted blob. #### Response Example { "checksums": [ { "type": "SHA-256", "value": "checksum" } ] } ``` -------------------------------- ### Paginated list response envelope Source: https://neic-sda.readthedocs.io/en/latest/services/download Example paginated envelope returned by v2 list endpoints. When `nextPageToken` is non-null, pass it as `?pageToken=` to retrieve the next page. The `pageSize` parameter is optional and bounded server-side; page tokens are HMAC-signed and tied to the originating query. ```json { "datasets": [ ... ], "nextPageToken": "ptk_7f9K2mQxVb3N" } ``` -------------------------------- ### S3 archive storage configuration Source: https://neic-sda.readthedocs.io/en/latest/services/download YAML configuration for the S3 archive backend. The storage/archive/s3 list defines an S3 endpoint with credentials, region, and an optional bucket prefix. This is the only example provided for storage configuration in this page. ```yaml storage: archive: s3: - endpoint: "s3.example.com" access_key: "access" secret_key: "secret" region: "us-east-1" bucket_prefix: "archive" ``` -------------------------------- ### DRS object response JSON Source: https://neic-sda.readthedocs.io/en/latest/services/download Example DRS 1.5 JSON response. The size and checksums describe the encrypted blob served at access_url. ```json { "id": "EGAF00000000001", "self_uri": "drs://HOSTNAME/EGAF00000000001", "size": 1048576, "created_time": "2026-01-15T10:30:00Z", "checksums": [ {"checksum": "a1b2c3d4...", "type": "sha-256"} ], "access_methods": [ { "type": "https", "access_url": { "url": "https://HOSTNAME/files/EGAF00000000001/content" } } ] } ``` -------------------------------- ### Set up Go workspace for SDA development Source: https://neic-sda.readthedocs.io/en/latest/setup Set up a Go workspace to develop the SDA source code. Initialises the workspace, references the sda module, and changes into the sda directory. ```bash $ git clone https://github.com/neicnordic/sensitive-data-archive.git $ go work init $ go work use ./sda $ cd sda ``` -------------------------------- ### Verified message from Verify service Source: https://neic-sda.readthedocs.io/en/latest/connection Message consumed by the Verify service and forwarded to the verified queue, shoveled to CEGAMQ, adding a decrypted_checksums key. Note the user field is 'john' in this example. ```json { "user":"john", "filepath":"somedir/encrypted.file.c4gh", "decrypted_checksums": [ { "type": "md5", "value": "50d858e0985ecc7f60418aaf0cc5ab587f42c2570a884095a9e8ccacd0f6545c"}, { "type": "sha256", "value": "1a79a4d60de6718e8e5b326e338ae533"} ] } ``` -------------------------------- ### GET /files/:fileId Source: https://neic-sda.readthedocs.io/en/latest/services/download Downloads a file. Returns a Crypt4GH file re-encrypted to the recipient's public key. Supports HTTP Range header. ```APIDOC ## GET /files/:fileId ### Description Downloads a file. Returns a Crypt4GH file re-encrypted to the recipient's public key. Supports HTTP Range header. ### Method GET ### Endpoint /files/:fileId ### Parameters #### Path Parameters - **fileId** (string) - Required - The file identifier. #### Headers - **X-C4GH-Public-Key** (string) - Required - Base64-encoded recipient public key. Must be provided if Htsget-Context-Public-Key is not. - **Htsget-Context-Public-Key** (string) - Optional - Base64-encoded recipient public key for htsget-style clients. Must be provided if X-C4GH-Public-Key is not. - **Range** (string) - Optional - HTTP Range header in standard single byte-range forms: bytes=START-END, bytes=START-, bytes=-SUFFIX. ### Response #### Success Response (200) - Body: Crypt4GH file re-encrypted to the recipient's public key. - **ETag** (string) - SHA-256 of the re-encrypted Crypt4GH header. Changes per request. #### Error Responses - **400** - If both public key headers are supplied (code KEY_CONFLICT) or neither is supplied (code KEY_MISSING). Also for invalid Range header. ### Response Example Binary data ``` -------------------------------- ### GET /users/:username/file/:fileid Source: https://neic-sda.readthedocs.io/en/latest/services/api Downloads a file from the inbox, re-encrypted with the client's public key provided in the request header. The public key must be base64 encoded. ```APIDOC ## GET /users/:username/file/:fileid ### Description Downloads a file from the inbox, re-encrypted with the client's public key provided in the request header. The public key must be base64 encoded. ### Method GET ### Endpoint https://HOSTNAME/users/:username/file/:fileid ### Parameters #### Path Parameters - **username** (string) - Required - The username of the user. - **fileid** (string) - Required - The file identifier. #### Headers - **Authorization** (string) - Required - Bearer token for authentication. - **C4GH-Public-Key** (string) - Required - Base64 encoded public key. ### Request Example ```bash curl -H "Authorization: Bearer $token" -H "C4GH-Public-Key: $base64_encoded_public_key" -X GET https://HOSTNAME/users/submitter@example.org/file/c2acecc6-f208-441c-877a-2670e4cbb040 ``` ### Response #### Success Response (200) Returns the re-encrypted file content. ### Error Codes - **200** Query execute ok. - **400** Client public key not provided or not valid. - **401** Token user is not in the list of admins. - **404** File not found. - **500** Internal error due to Inbox, Reencrypt, DB, MQ or streaming failures. ``` -------------------------------- ### Bootstrap development environment with make bootstrap Source: https://neic-sda.readthedocs.io/en/latest/guides/sda-dev-test-doc Initializes the environment and obtains necessary build tools after cloning the repository. SDA uses Go Modules, so cloning outside the GOPATH is advisable. ```bash $ make bootstrap ... ``` -------------------------------- ### Build SDA binaries from source Source: https://neic-sda.readthedocs.io/en/latest/setup Build SDA binaries by cloning the repository shrink-wrapped into a tarball or by following the inline instructions. Requires Go because of the build loop, and the command uses -buildvcs=false to avoid including VCS metadata. ```bash $ git clone https://github.com/neicnordic/sensitive-data-archive.git $ cd sda $ for p in cmd/*; do go build -buildvcs=false -o "${p/cmd\//sda-}" "./$p"; done ``` -------------------------------- ### HEAD /files/:fileId/content and GET /files/:fileId/content Source: https://neic-sda.readthedocs.io/en/latest/services/download Returns only the encrypted data segments (without the Crypt4GH header). Range byte offsets refer to the data segments only. The ETag is stable for a given file and independent of the recipient key. ```APIDOC ## HEAD /files/:fileId/content and GET /files/:fileId/content ### Description Returns only the encrypted data segments (without the Crypt4GH header). Range byte offsets refer to the data segments only. The ETag is stable for a given file and independent of the recipient key. ### Method HEAD, GET ### Endpoint /files/:fileId/content ### Request Headers - **Authorization**: Bearer (required) - **Range** (optional): a single HTTP byte range — `bytes=START-END`, `bytes=START-`, or `bytes=-SUFFIX`. Multi-range requests are rejected. ### Response Headers - **Content-Type**: application/octet-stream - **Accept-Ranges**: bytes - **ETag**: stable entity tag for the file ### Request Example ``` curl -H "Authorization: Bearer $token" \ -H "Range: bytes=0-65535" \ https://HOSTNAME/files/EGAF00000000001/content ``` ``` -------------------------------- ### HEAD /files/:fileId/header and GET /files/:fileId/header Source: https://neic-sda.readthedocs.io/en/latest/services/download Returns only the Crypt4GH header re-encrypted to the recipient's public key. Useful when the client needs to process the header separately. Does not support Range requests. ```APIDOC ## HEAD /files/:fileId/header and GET /files/:fileId/header ### Description Returns only the Crypt4GH header re-encrypted to the recipient's public key. Useful when the client needs to process the header separately. Does not support Range requests. ### Method HEAD, GET ### Endpoint /files/:fileId/header ### Response Headers - **Content-Length**: size of the re-encrypted Crypt4GH header in bytes - **SDA-Content-ETag**: the same stable synthetic ETag that `/files/:fileId/content` returns as its `ETag`. Lets a client fetch the header and the content in two separate requests and confirm that they refer to the same archive blob. ``` -------------------------------- ### Build all SDA containers with make build-all Source: https://neic-sda.readthedocs.io/en/latest/guides/sda-dev-test-doc Builds all containers for the SDA stack. ```bash $ make build-all ... ``` -------------------------------- ### GET /files/:fileId Source: https://neic-sda.readthedocs.io/en/latest/services/download Downloads the complete re-encrypted file (header + data segments). Requires authentication and a Crypt4GH public key header. Supports optional single byte range requests. ```APIDOC ## GET /files/:fileId ### Description Downloads the complete re-encrypted file (header + data segments). ### Method GET ### Endpoint /files/:fileId ### Request Headers - **Authorization**: Bearer (required) - **X-C4GH-Public-Key** or **Htsget-Context-Public-Key** (required, exactly one): base64-encoded recipient Crypt4GH public key - **Range** (optional): a single HTTP byte range — `bytes=START-END`, `bytes=START-`, or `bytes=-SUFFIX`. Multi-range requests are rejected. ### Response Headers - **Content-Type**: application/octet-stream - **Content-Disposition**: attachment; filename=".c4gh" — `` is the final path segment of the file's `submission_file_path`. The handler appends `.c4gh` only if it is not already present. - **Accept-Ranges**: bytes ### Error Codes - **200**: Download successful - **206**: Partial content (Range request) - **400**: Missing or conflicting public key header (`KEY_MISSING` / `KEY_CONFLICT`), or invalid/multi-range `Range` - **401**: Invalid or missing token - **403**: Access denied or file does not exist - **500**: Internal error (storage, reencrypt, or streaming failure) ### Request Example ``` curl -H "Authorization: Bearer $token" \ -H "X-C4GH-Public-Key: $(base64 -w0 /path/to/c4gh.pub.pem)" \ https://HOSTNAME/files/EGAF00000000001 \ -o downloaded_file.c4gh ``` ``` -------------------------------- ### Lint all Go components with make lint-all Source: https://neic-sda.readthedocs.io/en/latest/guides/sda-dev-test-doc Runs golangci-lint for all Go components. ```bash $ make lint-all ... ``` -------------------------------- ### GET /objects/{datasetId}/{filePath} Source: https://neic-sda.readthedocs.io/en/latest/services/download Returns a minimal GA4GH DRS 1.5 DrsObject with a pre-resolved access_url pointing to the file content endpoint. This enables DRS-aware clients to resolve a dataset and file path to a download URL without knowing the internal file ID. ```APIDOC ## GET /objects/{datasetId}/{filePath} ### Description Returns a minimal GA4GH DRS 1.5 `DrsObject` with a pre-resolved `access_url` pointing to the file content endpoint. This enables DRS-aware clients (e.g. htsget-rs) to resolve a dataset + file path to a download URL without knowing the internal file ID. ### Method GET ### Endpoint /objects/{datasetId}/{filePath} ### Parameters #### Path Parameters - **datasetId** (string) - Required - The dataset ID. Everything before the first `/` in the path. - **filePath** (string) - Required - The file path within the dataset. Everything after the first `/` in the path. ### Request Example ``` curl -H "Authorization: Bearer $token" \ https://HOSTNAME/objects/EGAD00000000001/samples/sample1.bam.c4gh ``` ### Response #### Success Response (200) - **id** (string) - The file ID. - **self_uri** (string) - The DRS URI for the object. - **size** (integer) - The size of the encrypted blob. - **created_time** (string) - The creation time of the object. - **checksums** (array) - List of checksums for the encrypted blob. - **access_methods** (array) - List of access methods, each with a type and access_url. #### Response Example ``` { "id": "EGAF00000000001", "self_uri": "drs://HOSTNAME/EGAF00000000001", "size": 1048576, "created_time": "2026-01-15T10:30:00Z", "checksums": [ {"checksum": "a1b2c3d4...", "type": "sha-256"} ], "access_methods": [ { "type": "https", "access_url": { "url": "https://HOSTNAME/files/EGAF00000000001/content" } } ] } ``` ### Error Codes - **200** - DRS object returned - **400** - Malformed path (missing dataset or file component) - **401** - Invalid or missing token - **403** - Access denied or file does not exist ``` -------------------------------- ### Run unit tests with go test Source: https://neic-sda.readthedocs.io/en/latest/services/download Runs the unit tests for the download command in the sda directory. ```bash cd sda && go test ./cmd/download/... ``` -------------------------------- ### List files with path prefix and pagination (first page) Source: https://neic-sda.readthedocs.io/en/latest/services/api Fetches the first page of files matching the path_prefix, with a limit of 100 items. The response includes an X-Next-Cursor header if more pages are available. ```bash curl -H "Authorization: Bearer $token" -X GET "https://HOSTNAME/files?path_prefix=submission-1&limit=100" ``` -------------------------------- ### Lint a specific Go component with make lint- Source: https://neic-sda.readthedocs.io/en/latest/guides/sda-dev-test-doc Runs golangci-lint for a specific component by replacing `all` with the folder name (`sda`, `sda-auth`, `sda-download`). ```bash $ make lint- ... ``` -------------------------------- ### Build a specific SDA container with make build- Source: https://neic-sda.readthedocs.io/en/latest/guides/sda-dev-test-doc Builds the container for a specific component by replacing `all` with the folder name. ```bash $ make build- ... ``` -------------------------------- ### Run static code tests for all components with make test-all Source: https://neic-sda.readthedocs.io/en/latest/guides/sda-dev-test-doc Runs static code tests for all components. For Go code this runs `go test -count=1 ./...` in the target folder; for _sftp-inbox_ it calls `mvn test -B` inside a maven container. ```bash $ make test-all ... ``` -------------------------------- ### Run SDA integration tests with make integrationtest-sda Source: https://neic-sda.readthedocs.io/en/latest/guides/sda-dev-test-doc Builds all containers and runs the integration tests for the SDA stack. The same test runs on every PR in GitHub. ```bash $ make integrationtest-sda ... ```