### Starting Shepard

Source: https://gitlab.com/dlr-shepard/shepard/-/blob/main/infrastructure/README.md

Instructions for starting the Shepard application using Docker Compose.

```APIDOC
## Start Shepard

### Description
Ensure all required resources, particularly sufficient memory, are available before starting Shepard. Configure Docker Compose profiles as needed, preferably using the `COMPOSE_PROFILES` environment variable.

### Method
CLI Commands

### Commands
1. `docker compose pull` - Pulls the latest Docker images.
2. `docker compose up -d` - Starts Shepard and its associated services in detached mode.

### Logs
Backend logs can be found at `/opt/shepard/backend/logs`.
```

--------------------------------

### Setup and Run Shepard Services

Source: https://context7.com/dlr-shepard/shepard/llms.txt

Commands to copy environment configuration, start development databases and Keycloak, and launch the full stack application. Also includes commands for optional monitoring tools.

```bash
cp .env.example .env

docker compose --profile dev up -d

docker compose --profile tryout up -d

docker compose --profile monitoring up -d
```

--------------------------------

### Setup and Activate Python Virtual Environment

Source: https://gitlab.com/dlr-shepard/shepard/-/blob/main/clients/tests/python/README.md

Creates a Python virtual environment named 'env' and then activates it. This isolates project dependencies and ensures that the installed packages do not interfere with the system's Python installation.

```bash
python -m venv env
source env/bin/activate
```

--------------------------------

### Install Project Dependencies with npm

Source: https://gitlab.com/dlr-shepard/shepard/-/blob/main/CONTRIBUTING.md

Installs all necessary project dependencies by running the npm install command in the top-level directory. This is a crucial step after cloning the repository to ensure all required packages are available for development.

```shell
npm install
```

--------------------------------

### Copy Environment Configuration File

Source: https://gitlab.com/dlr-shepard/shepard/-/blob/main/infrastructure/README.md

This command copies the example environment file (`.env.example`) to `.env`. The `.env` file is typically used to store environment-specific configurations, such as API keys, database credentials, and other settings, for the Shepard application.

```bash
# copy configuration file
cp .env.example .env
```

--------------------------------

### Start Local Databases with Docker Compose

Source: https://gitlab.com/dlr-shepard/shepard/-/blob/main/CONTRIBUTING.md

Launches local instances of databases and the frontend using Docker Compose. This command starts core services and the frontend container, providing local database instances without persistent storage. Alternatively, Podman Compose can be used.

```shell
docker-compose up
# or
podman-compose up
# To start core services and the frontend container:
docker-compose --profile frontend up
```

--------------------------------

### Install Python Dependencies

Source: https://gitlab.com/dlr-shepard/shepard/-/blob/main/clients/tests/python/README.md

Installs the necessary Python packages required for testing the client. It installs requirements from both the testing package and the generated client package.

```bash
pip install -r requirements.txt
pip install -r ../../python/requirements.txt
```

--------------------------------

### Starting Shepard Application

Source: https://gitlab.com/dlr-shepard/shepard/-/blob/main/infrastructure/README.md

Commands to pull the latest Docker images and start the Shepard application in detached mode. It's recommended to set the COMPOSE_PROFILES environment variable for consistent profile activation.

```bash
docker compose pull
docker compose up -d
```

--------------------------------

### Accessing Metrics

Source: https://gitlab.com/dlr-shepard/shepard/-/blob/main/infrastructure/README.md

Retrieve system and resource consumption metrics from the Shepard backend.

```APIDOC
## How to use Metrics

### Description
Access the Shepard backend's metrics endpoint to obtain information about system and resource utilization. This endpoint is compatible with monitoring systems like Prometheus.

### Method
HTTP GET Request

### Endpoint
`/shepard/doc/metrics/prometheus`

### Response
#### Success Response (200)
Returns a JSON document containing current system and resource consumption values.
```

--------------------------------

### Search Users (Neo4j)

Source: https://gitlab.com/dlr-shepard/shepard/-/blob/main/architecture/src/08_concepts/search.adoc

This snippet illustrates how to search for users via a GET request to `/search/users`. It outlines the available query parameters (`username`, `firstName`, `lastName`, `email`) and provides a Cypher query example that utilizes regular expressions for flexible matching.

```cypher
MATCH (u:User) WHERE u.firstName =~ "John" AND u.lastName =~ "Doe" RETURN u
```

```json
[
  {
    "username": "string",
    "firstName": "string",
    "lastName": "string",
    "email": "string",
    "subscriptionIds": [0],
    "apiKeyIds": ["3fa85f64-5717-4562-b3fc-2c963f66afa6"]
  }
]
```

--------------------------------

### OpenID Connect Realm Roles Example

Source: https://gitlab.com/dlr-shepard/shepard/-/blob/main/infrastructure/README.md

An example JSON structure of an access token containing the 'realm_access' claim, which lists user roles provided by an OpenID Connect identity provider like Keycloak. This is used for role-based access control in Shepard.

```json
{
  "...": "...",
  "realm_access": {
    "roles": [
      "default-roles-master",
      "offline_access",
      "uma_authorization",
      "custom_role"
    ]
  },
  "...": "..."
}
```

--------------------------------

### Environment Variables Configuration

Source: https://gitlab.com/dlr-shepard/shepard/-/blob/main/infrastructure/README.md

Configuration options for Shepard via environment variables in a .env file.

```APIDOC
## Environment Variables

### Description
Configuration options for Shepard are managed through environment variables, typically set in a `.env` file.

### Parameters
#### Environment Variables
- **FRONTEND_AUTH_SECRET** (string) - Required - A random secret string used to hash JWT tokens. Example generation: `openssl rand -base64 32`
- **SESSION_REFRESH_INTERVAL** (integer) - Optional - Frontend session refresh interval in milliseconds. Defaults to 30000 (30 seconds).
- **SHEPARD_SPATIAL_DATA_ENABLED** (boolean) - Optional - Enable experimental spatial data feature. Requires PostGIS. Defaults to `false`.
- **COMPOSE_PROFILES** (string) - Optional - Select the Docker Compose profiles that are active. Multiple profiles should be comma-separated. Defaults to empty.
```

--------------------------------

### Configure Backend with Environment Variables

Source: https://gitlab.com/dlr-shepard/shepard/-/blob/main/CONTRIBUTING.md

Configures the backend application by copying the example environment file to `.env` and setting OIDC parameters. This process allows overriding or appending properties in `application.properties` using environment variables, as per Quarkus documentation.

```shell
cp .env.example .env
# Enter valid OIDC parameters
```

--------------------------------

### Install Backend Client Package with npm

Source: https://gitlab.com/dlr-shepard/shepard/-/blob/main/backend-client/README.md

Installs the published version of the @dlr-shepard/backend-client package into a consuming project. This command adds the client as a dependency.

```bash
npm install @dlr-shepard/backend-client@1.0 --save
```

--------------------------------

### Docker Compose Profiles

Source: https://gitlab.com/dlr-shepard/shepard/-/blob/main/infrastructure/README.md

Utilize Docker Compose profiles to activate different experimental features within Shepard.

```APIDOC
## Profiles in `docker-compose.yml`

### Description
Activate various experimental features in Shepard by selecting appropriate Docker Compose profiles. The default profile includes essential containers for a standard configuration.

### Method
CLI Command or Environment Variable

### Parameters
#### Available Profiles
- **spatial**: Enables PostGIS for the experimental spatial data feature. The feature itself must also be enabled via environment variables.
- **monitoring**: Enables the monitoring feature.
- **timescale-migration-preparation**: Used exclusively for the migration process. Refer to migration documentation for details.
- **frontend-old**: Enables the legacy frontend.

### Usage
**CLI:** `docker compose --profile [profile_name] up` (e.g., `docker compose --profile spatial --profile monitoring up`)
**Environment Variable:** Set `COMPOSE_PROFILES` in `.env` with comma-separated profile names (e.g., `COMPOSE_PROFILES=spatial,monitoring`).
```

--------------------------------

### Docker Compose Profiles for Shepard

Source: https://gitlab.com/dlr-shepard/shepard/-/blob/main/infrastructure/README.md

Demonstrates how to activate specific features or configurations in Shepard using Docker Compose profiles. Profiles can be selected via the command line or by setting the COMPOSE_PROFILES environment variable.

```bash
docker compose --profile spatial --profile monitoring up
```

--------------------------------

### Enable Spatial Data (Configuration Property)

Source: https://gitlab.com/dlr-shepard/shepard/-/blob/main/infrastructure/README.md

This command demonstrates enabling the experimental spatial data feature using a configuration property. This method also activates the REST endpoints for spatial data.

```bash
shepard.spatial-data.enabled = true
```

--------------------------------

### Update Object Properties (Java API Client)

Source: https://gitlab.com/dlr-shepard/shepard/-/blob/main/architecture/src/09_architecture_decisions/000-api_v2.adoc

Demonstrates how to update an existing object using a generated API client. This example highlights that only explicitly modified properties should be changed, ensuring other fields remain untouched.

```Java
TypeA objA = api.getTypeA(...);

objA.setParameterX(...);

api.updateTypeA(objA);
```

--------------------------------

### Manage Shepard Permissions (Bash)

Source: https://context7.com/dlr-shepard/shepard/llms.txt

This snippet demonstrates how to manage permissions for collections within Shepard. It includes examples for retrieving, updating, and creating user groups. Requires a valid authorization token.

```bash
# Get permissions for a collection
curl -X GET "https://backend.example.com/shepard/api/collections/123/permissions" \
  -H "Authorization: Bearer <token>"

# Response:
# {
#   "entityId": 123,
#   "owner": "researcher1",
#   "permissionType": "Private",
#   "reader": ["colleague1", "colleague2"],
#   "writer": ["labassistant1"],
#   "manager": ["projectlead"],
#   "readerGroupIds": [10],
#   "writerGroupIds": []
# }

# Update permissions
curl -X PUT "https://backend.example.com/shepard/api/collections/123/permissions" \
  -H "Authorization: Bearer <token>" \
  -H "Content-Type: application/json" \
  -d '{
    "permissionType": "PublicReadable",
    "reader": ["colleague1", "colleague2", "colleague3"],
    "writer": ["labassistant1"],
    "manager": ["projectlead"],
    "readerGroupIds": [10, 11]
  }'

# Create a user group
curl -X POST "https://backend.example.com/shepard/api/userGroups" \
  -H "Authorization: Bearer <token>" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Project Alpha Team",
    "usernames": ["researcher1", "researcher2", "labassistant1"]
  }'
```

--------------------------------

### Local Development Setup for Shepard (Bash)

Source: https://context7.com/dlr-shepard/shepard/llms.txt

This snippet provides the initial commands to set up a local development environment for Shepard using Docker Compose. It involves cloning the repository and navigating to the local infrastructure directory.

```bash
# Clone repository
git clone https://gitlab.com/dlr-shepard/shepard.git
cd shepard/infrastructure-local
```

--------------------------------

### Semantic Annotation Example in XML

Source: https://gitlab.com/dlr-shepard/shepard/-/blob/main/architecture/src/08_concepts/ontologies.adoc

An example of how semantic annotations can be represented using XML. It uses URIs to define properties and their corresponding values.

```xml
<annotation>
    <propertyURI label="is about">http://purl.obolibrary.org/obo/IAO_0000136</propertyURI>
    <valueURI label="grassland biome">http://purl.obolibrary.org/obo/ENVO_01000177</valueURI>
</annotation>
```

--------------------------------

### Python Client Configuration and Usage

Source: https://context7.com/dlr-shepard/shepard/llms.txt

Demonstrates how to configure the Shepard Python client using API keys or JWT tokens, initialize API clients for collections and data objects, and perform operations like retrieving all collections, creating a new collection, and creating a data object.

```python
from shepard_client import ApiClient, Configuration, CollectionApi, DataObjectApi

# Configure the client
config = Configuration(
    host="https://backend.example.com/shepard/api",
    api_key={"apikey": "your-api-key-uuid"}
)

# Or use bearer token
config = Configuration(
    host="https://backend.example.com/shepard/api",
    access_token="your-jwt-token"
)

# Initialize API client
client = ApiClient(config)
collection_api = CollectionApi(client)
data_object_api = DataObjectApi(client)

# Get all collections
collections = collection_api.get_all_collections(page=0, size=20)
for c in collections:
    print(f"Collection: {c.name} (ID: {c.id})")

# Create a new collection
from shepard_client.models import Collection
new_collection = Collection(
    name="Python Created Collection",
    description="Created via Python client"
)
created = collection_api.create_collection(new_collection)
print(f"Created collection ID: {created.id}")

# Create DataObject
from shepard_client.models import DataObject
data_obj = DataObject(
    name="Sample Data",
    attributes={"key": "value"}
)
created_obj = data_object_api.create_data_object(created.id, data_obj)
```

--------------------------------

### Prepare Local Python Client Files

Source: https://gitlab.com/dlr-shepard/shepard/-/blob/main/clients/tests/python/README.md

Copies the generated Python client from the Docker output directory to the test directory and renames the package to avoid naming conflicts. This step is crucial for local testing.

```bash
cp -r ../../python/shepard_client src/shepard_client_local
```

--------------------------------

### POST /examplepath - File Upload

Source: https://gitlab.com/dlr-shepard/shepard/-/blob/main/architecture/src/08_concepts/openapi-specification.adoc

This endpoint allows clients to upload files using a multipart/form-data request. The implementation leverages specific Java classes and annotations to ensure proper schema generation for Swagger UI.

```APIDOC
## POST /examplepath

### Description
This endpoint handles file uploads. It expects a `multipart/form-data` request containing the file to be uploaded. The backend implementation uses a specific schema structure to ensure compatibility with Swagger UI.

### Method
POST

### Endpoint
/examplepath

### Parameters
#### Query Parameters
None

#### Request Body
- **file** (File) - Required - The file to be uploaded.

### Request Example
```json
{
  "file": "<binary data>"
}
```

### Response
#### Success Response (200)
- **message** (string) - A success message indicating the file was uploaded.

#### Response Example
```json
{
  "message": "File uploaded successfully"
}
```

### Error Handling
- **400 Bad Request**: If the request is malformed or the file is missing.
- **500 Internal Server Error**: If there is an issue processing the file on the server.
```

--------------------------------

### Create and Manage File Containers and References

Source: https://context7.com/dlr-shepard/shepard/llms.txt

Handles file storage by creating file containers, uploading files, listing files within a container, and creating references linking DataObjects to uploaded files. Supports downloading specific file payloads. Requires a valid authentication token.

```bash
# Create a file container
curl -X POST "https://backend.example.com/shepard/api/fileContainers" \
  -H "Authorization: Bearer <token>" \
  -H "Content-Type: application/json" \
  -d '{"name": "Test Images Container"}'

# Upload a file to container
curl -X POST "https://backend.example.com/shepard/api/fileContainers/789/payload" \
  -H "Authorization: Bearer <token>" \
  -F "file=@/path/to/test-image.png"

# List files in container
curl -X GET "https://backend.example.com/shepard/api/fileContainers/789/payload" \
  -H "Authorization: Bearer <token>"

# Create file reference linking DataObject to file container
curl -X POST "https://backend.example.com/shepard/api/collections/123/dataObjects/456/fileReferences" \
  -H "Authorization: Bearer <token>" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Microscopy Images",
    "fileContainerId": 789,
    "fileOids": ["abc123def456"]
  }'

# Get files through reference
curl -X GET "https://backend.example.com/shepard/api/collections/123/dataObjects/456/fileReferences/101/payload" \
  -H "Authorization: Bearer <token>"

# Download specific file payload
curl -X GET "https://backend.example.com/shepard/api/collections/123/dataObjects/456/fileReferences/101/payload/abc123def456" \
  -H "Authorization: Bearer <token>" \
  --output downloaded-file.png
```

--------------------------------

### Run Unit Tests with Maven

Source: https://gitlab.com/dlr-shepard/shepard/-/blob/main/CONTRIBUTING.md

Executes all unit tests for the project using the Maven wrapper. This command verifies the correctness of individual components and functions within the application.

```shell
./mvnw test
```

--------------------------------

### Manage Collections using Shepard API

Source: https://context7.com/dlr-shepard/shepard/llms.txt

Provides examples for interacting with the Collections API in Shepard. This includes retrieving all collections with pagination, creating new collections with metadata, fetching a specific collection, updating existing collections, deleting collections, and exporting collections as RO-Crate packages.

```bash
# Get all collections with pagination
curl -X GET "https://backend.example.com/shepard/api/collections?page=0&size=20&orderBy=createdAt&orderDesc=true" \
  -H "Authorization: Bearer <token>"

# Response:
# [
#   {
#     "id": 123,
#     "name": "Wind Tunnel Experiment 2024",
#     "description": "High-speed aerodynamics test data",
#     "createdAt": "2024-08-15T11:18:44.632+00:00",
#     "createdBy": "researcher1",
#     "dataObjectIds": [456, 457, 458],
#     "incomingIds": []
#   }
# ]

# Create a new collection
curl -X POST "https://backend.example.com/shepard/api/collections" \
  -H "Authorization: Bearer <token>" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Material Test Campaign",
    "description": "Composite material fatigue testing",
    "attributes": {
      "project": "CFRP-2024",
      "facility": "DLR Stuttgart"
    }
  }'

# Get a specific collection
curl -X GET "https://backend.example.com/shepard/api/collections/123" \
  -H "Authorization: Bearer <token>"

# Update a collection
curl -X PUT "https://backend.example.com/shepard/api/collections/123" \
  -H "Authorization: Bearer <token>" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Material Test Campaign - Updated",
    "description": "Completed composite material fatigue testing"
  }'

# Delete a collection
curl -X DELETE "https://backend.example.com/shepard/api/collections/123" \
  -H "Authorization: Bearer <token>"

# Export collection as RO-Crate (FAIR data package)
curl -X GET "https://backend.example.com/shepard/api/collections/123/export" \
  -H "Authorization: Bearer <token>" \
  --output collection-export.zip
```

--------------------------------

### Role-Based Access Control

Source: https://gitlab.com/dlr-shepard/shepard/-/blob/main/infrastructure/README.md

Restrict access to Shepard based on user roles defined in OpenID Connect identity providers.

```APIDOC
## Restrict access to users with specific roles

### Description
Configure Shepard to allow access only to users possessing a specific role, which can be included in OpenID Connect access tokens (e.g., from Keycloak).

### Method
Environment Variable Configuration

### Parameters
#### Environment Variables
- **OIDC_ROLE** (string) - Optional - The specific role required for users to access Shepard. If set, users without this role will be denied access after the next restart.
```

--------------------------------

### Timeseries Annotation Structure (JSON)

Source: https://gitlab.com/dlr-shepard/shepard/-/blob/main/architecture/src/09_architecture_decisions/018-semantic-annotations-on-data.adoc

An example of how a semantic annotation for timeseries data could be structured. It includes identifiers for the specific timeseries and metadata about the annotation itself.

```json
{
  "valueIRI": "...",
  "IRI": "",
  "Repository": "",
  "Repository2": "",
  "type": "timeseriesAnnotation",
  "identifier": {
    "containerId": 1,
    "timeseriesId": 123
  }
}
```

--------------------------------

### GET /shepard/api/collections/{collectionId}/dataObjects

Source: https://context7.com/dlr-shepard/shepard/llms.txt

Retrieves a list of DataObjects within a collection, with optional filtering by name and parent ID.

```APIDOC
## GET /shepard/api/collections/{collectionId}/dataObjects

### Description
Retrieves a list of DataObjects within a collection, with optional filtering by name and parent ID.

### Method
GET

### Endpoint
/shepard/api/collections/{collectionId}/dataObjects

### Parameters
#### Path Parameters
- **collectionId** (integer) - Required - The ID of the collection to retrieve DataObjects from.

#### Query Parameters
- **name** (string) - Optional - Filters DataObjects by their name.
- **parentId** (integer) - Optional - Filters DataObjects by their parent ID.

### Request Example
```
GET /shepard/api/collections/123/dataObjects?name=Test%20Sample&parentId=456
```

### Response
#### Success Response (200)
- **dataObjects** (array of objects) - A list of DataObjects matching the criteria.
  - **id** (integer) - The unique identifier of the DataObject.
  - **name** (string) - The name of the DataObject.
  - **description** (string) - The description of the DataObject.
  - **parentId** (integer) - The ID of the parent DataObject.
  - **predecessorIds** (array of integers) - The list of predecessor IDs.

#### Response Example
```json
{
  "dataObjects": [
    {
      "id": 102,
      "name": "Test Sample A-001",
      "description": "First fatigue cycle",
      "parentId": 456,
      "predecessorIds": []
    }
  ]
}
```
```

--------------------------------

### Build Backend Client Package with npm

Source: https://gitlab.com/dlr-shepard/shepard/-/blob/main/backend-client/README.md

Installs dependencies and compiles TypeScript sources to JavaScript for the backend client package. This is a prerequisite for publishing or local consumption.

```bash
npm install
npm run build
```

--------------------------------

### Import Backend Client in Frontend (JavaScript)

Source: https://gitlab.com/dlr-shepard/shepard/-/blob/main/architecture/src/08_concepts/backend-clients.adoc

Demonstrates how to import and use the generated backend client within frontend files. It shows the instantiation of the SemanticRepositoryApi and the usage of its methods. This requires the '@dlr-shepard/backend-client' package.

```javascript
import { SemanticRepositoryApi } from "@dlr-shepard/backend-client";
import { getConfiguration } from "./serviceHelper";

export default class SemanticRepositoryService {
  static createSemanticRepository(params: CreateSemanticRepositoryRequest) {
    const api = new SemanticRepositoryApi(getConfiguration());
    return api.createSemanticRepository(params);
  }
}
```

--------------------------------

### Configure Grafana Admin Credentials (.env)

Source: https://gitlab.com/dlr-shepard/shepard/-/blob/main/infrastructure/README.md

This snippet demonstrates how to set the administrative username and password for Grafana within the .env file. These credentials are required for accessing the Grafana UI.

```text
GRAFANA_ADMIN_USERNAME=grafana
GRAFANA_ADMIN_PASSWORD=secure_password
```

--------------------------------

### Generate Frontend Auth Secret

Source: https://gitlab.com/dlr-shepard/shepard/-/blob/main/infrastructure/README.md

Generates a random secret string suitable for hashing JWT tokens. This is a bash command that utilizes OpenSSL to produce a base64 encoded secret.

```bash
openssl rand -base64 32
```

--------------------------------

### Apply Database Tweaks via SQL Script

Source: https://gitlab.com/dlr-shepard/shepard/-/blob/main/infrastructure/README.md

This command applies a SQL script to a running PostgreSQL database container to adjust system settings. Ensure the script is named `tweak-db-settings.sql` and replace `DB_CONTAINER_NAME` and `ADMIN_USERNAME` with your specific values. The changes require a database or container restart to take effect.

```bash
cat tweak-db-settings.sql | docker exec -i DB_CONTAINER_NAME psql -U ADMIN_USERNAME
```