### Create Installation Directory and Copy Example Config

Source: https://github.com/nebius/nebius-solutions-library/blob/main/soperator/README.md

Create a new directory for your installation and copy example configuration files into it.

```bash
cd soperator
export INSTALLATION_NAME=<your-name> # e.g. company name
mkdir -p installations/$INSTALLATION_NAME
cd installations/$INSTALLATION_NAME
cp -r ../example/* ../example/.* .
```

--------------------------------

### SkyPilot Nebius Setup Script Download and Execution

Source: https://github.com/nebius/nebius-solutions-library/blob/main/skypilot/README.md

Download and make the Nebius setup script executable, then run it to complete the SkyPilot setup for Nebius. You will be prompted for tenant and project ID.

```bash
wget https://raw.githubusercontent.com/nebius/nebius-solution-library/refs/heads/main/skypilot/nebius-setup.sh
chmod +x nebius-setup.sh 
./nebius-setup.sh
```

--------------------------------

### Install and Check Tools

Source: https://github.com/nebius/nebius-solutions-library/blob/main/applications/osmo/deploy/example/README.md

Installs essential tools like Terraform, kubectl, Helm, and Nebius CLI. Use the --check flag to verify existing installations.

```bash
cd 000-prerequisites

# Install required tools (Terraform, kubectl, Helm, Nebius CLI)
./install-tools.sh

# Check if tools are installed
./install-tools.sh --check
```

--------------------------------

### Install KubeRay Prerequisites

Source: https://github.com/nebius/nebius-solutions-library/blob/main/modules/kuberay/README.md

Installs necessary packages for Infiniband support. Ensure these are installed on your system or included in your custom Docker image.

```shell
sudo apt update && sudo apt upgrade -y && sudo apt install -y kmod infiniband-diags ibverbs-utils libibverbs-dev perftest net-tools
```

--------------------------------

### VM Instance Configuration with S3 Bucket Mounting

Source: https://github.com/nebius/nebius-solutions-library/blob/main/vm-instance/README.md

Example of provisioning VM instances with AWS CLI installed and configured, and mounting an S3 bucket to a specified path. The default mount path is `/mnt/s3`.

```hcl
preset = "16vcpu-64gb"
platform = "cpu-e2"

users = [
  {
    user_name = "admin",
    ssh_key_path = "~/.ssh/id_rsa.pub"
  }
]
mount_bucket = "my-bucket-name"
s3_mount_path = "/mnt/s3" # optional
public_ip = true
instance_count = 2
```

--------------------------------

### Install Tailscale Operator with Inline Credentials

Source: https://github.com/nebius/nebius-solutions-library/blob/main/modules/tailscale-operator/README.md

Use this example to install the Tailscale operator using inline Terraform variables for OAuth credentials. Ensure the Tailscale OAuth client is created beforehand.

```hcl
module "tailscale_operator" {
  source = "../../modules/tailscale-operator"

  oauth_client_id     = var.tailscale_oauth_client_id
  oauth_client_secret = var.tailscale_oauth_client_secret
  operator_hostname   = "ts-operator-${var.cluster_name}"

  providers = {
    kubernetes = kubernetes
    helm       = helm
  }
}
```

--------------------------------

### Set up SkyPilot on Kubernetes

Source: https://github.com/nebius/nebius-solutions-library/blob/main/skypilot/multiregion/README.md

Create a virtual environment, install SkyPilot, and register your Terraform-created Kubernetes cluster contexts with SkyPilot. Verify the setup with `sky check kubernetes`.

```bash
cd skypilot/multiregion
mkdir -p .uv-cache
UV_CACHE_DIR=$PWD/.uv-cache uv venv .venv
UV_CACHE_DIR=$PWD/.uv-cache uv pip install --python .venv/bin/python "skypilot-nightly[kubernetes]==1.0.0.dev20260219"
source .venv/bin/activate
sky --version
```

```bash
cd skypilot/multiregion
./setup-skypilot-k8s.sh
sky check kubernetes
```

--------------------------------

### Install Tools and Initialize Environment

Source: https://github.com/nebius/nebius-solutions-library/blob/main/applications/osmo/iam-register/README.md

Installs necessary tools and initializes the Nebius environment variables. Ensure you are in the correct directory before running.

```bash
cd applications/osmo/deploy/example/000-prerequisites
./install-tools.sh --check
source ./nebius-env-init.sh
source ./secrets-init.sh
```

--------------------------------

### Install Deployment Tools

Source: https://github.com/nebius/nebius-solutions-library/blob/main/applications/osmo/README.md

Installs Terraform, kubectl, Helm, Nebius CLI, and OSMO CLI. Use --check to verify without installing. Supports Linux, WSL, and macOS.

```bash
cd deploy/000-prerequisites
./install-tools.sh        # Installs: Terraform, kubectl, Helm, Nebius CLI, OSMO CLI
./install-tools.sh --check  # Verify without installing
```

--------------------------------

### Setup WireGuard Client

Source: https://github.com/nebius/nebius-solutions-library/blob/main/applications/osmo/README.md

After deploying with WireGuard enabled, run this script from the `deploy/000-prerequisites` directory to set up the client.

```bash
cd deploy/000-prerequisites
./wireguard-client-setup.sh
```

--------------------------------

### Install Tailscale Operator with Cilium Compatibility

Source: https://github.com/nebius/nebius-solutions-library/blob/main/modules/tailscale-operator/README.md

This example demonstrates installing the Tailscale operator and enabling the Cilium compatibility workaround for clusters using kube-proxy replacement. It requires the Tailscale OAuth credentials to be provided.

```hcl
module "tailscale_operator" {
  source = "../../modules/tailscale-operator"

  oauth_client_id     = var.tailscale_oauth_client_id
  oauth_client_secret = var.tailscale_oauth_client_secret
  operator_hostname   = "ts-operator-${var.cluster_name}"
  enable_cilium_bpf_lb_sock_hostns_only = true

  providers = {
    kubernetes = kubernetes
    helm       = helm
  }
}
```

--------------------------------

### Setup WireGuard Client

Source: https://github.com/nebius/nebius-solutions-library/blob/main/applications/osmo/deploy/example/001-iac/README.md

Commands to set up the WireGuard client after deploying with private access enabled.

```bash
cd ../000-prerequisites
./wireguard-client-setup.sh
```

--------------------------------

### Install Device Plugin

Source: https://github.com/nebius/nebius-solutions-library/blob/main/gpu_slicing/timeslice/README.md

Upgrades or installs the NVIDIA Device Plugin using Helm. This is a necessary step for GPU management.

```bash
helm upgrade -i nvdp nvdp/nvidia-device-plugin \
  --namespace nvidia-device-plugin \
  --create-namespace \
  --version 0.17.1
```

--------------------------------

### Quick Start Deployment Commands

Source: https://github.com/nebius/nebius-solutions-library/blob/main/applications/osmo/deploy/example/001-iac/README.md

Commands to copy a configuration, edit variables, and deploy the infrastructure using Terraform.

```bash
# Recommended: Cost-optimized with secure private access
cp terraform.tfvars.cost-optimized-secure.example terraform.tfvars

# Edit if needed (tenant_id and parent_id set via environment); optional SSH key
vim terraform.tfvars

# Deploy
terraform init
terraform plan
terraform apply
```

--------------------------------

### Install Required Tools

Source: https://github.com/nebius/nebius-solutions-library/blob/main/applications/osmo/deploy/example/000-prerequisites/README.md

Execute this script to install all necessary tools for OSMO deployment. Use the --check flag to verify existing installations without performing new ones.

```bash
# Install all required tools
./install-tools.sh
```

```bash
# Or check what's already installed
./install-tools.sh --check
```

--------------------------------

### Install Nebius CLI Tools

Source: https://github.com/nebius/nebius-solutions-library/blob/main/applications/osmo/deploy/example/000-prerequisites/README.md

Run this script to install the Nebius CLI tools. Ensure you have the necessary permissions.

```bash
./install-tools.sh
```

--------------------------------

### Install Nebius CLI

Source: https://github.com/nebius/nebius-solutions-library/blob/main/dsvm/README.md

Installs the Nebius CLI tool. Ensure you have curl installed.

```bash
curl -sSL https://storage.ai.nebius.cloud/nebius/install.sh | bash
```

--------------------------------

### Install Required Tools

Source: https://github.com/nebius/nebius-solutions-library/blob/main/applications/osmo/deploy/example/001-iac/README.md

Execute this script to install necessary tools for Terraform deployment.

```bash
cd ../000-prerequisites
./install-tools.sh
```

--------------------------------

### Deploy OSMO Backend

Source: https://github.com/nebius/nebius-solutions-library/blob/main/applications/osmo/deploy/example/README.md

Installs the backend services for the OSMO application.

```bash
# 6. Deploy OSMO backend
./05-deploy-osmo-backend.sh
```

--------------------------------

### Install JQ on MacOS

Source: https://github.com/nebius/nebius-solutions-library/blob/main/k8s-training/README.md

Installs the JQ command-line JSON processor on macOS using Homebrew.

```bash
brew install jq
```

--------------------------------

### Basic VM Instance Configuration

Source: https://github.com/nebius/nebius-solutions-library/blob/main/vm-instance/README.md

Example of a basic VM instance configuration with a public IP, specifying CPU and memory, and adding a single user with SSH access.

```hcl
preset = "16vcpu-64gb"
platform = "cpu-e2"

users = [
  {
    user_name = "admin",
    ssh_key_path = "~/.ssh/id_rsa.pub"
  }
]

public_ip = true
instance_count = 1
```

--------------------------------

### Copy and Edit Deploy Environment File

Source: https://github.com/nebius/nebius-solutions-library/blob/main/applications/osmo/deploy/example/002-setup/README.md

Before running deployment scripts, copy the example environment file and edit it to set necessary configuration parameters like ingress hostname, SSO credentials, and database password.

```bash
cp osmo-deploy.env.example osmo-deploy.env
# Edit osmo-deploy.env: set OSMO_INGRESS_HOSTNAME (e.g. osmo.<LB_IP_DASHED>.nip.io), NEBIUS_SSO_CLIENT_ID, NEBIUS_SSO_CLIENT_SECRET, OSMO_POSTGRESQL_PASSWORD
```

--------------------------------

### Install Nebius CLI

Source: https://github.com/nebius/nebius-solutions-library/blob/main/bastion/README.md

Installs the Nebius CLI using a curl script. Ensure you reload your shell session after installation.

```bash
curl -sSL https://storage.eu-north1.nebius.cloud/cli/install.sh | bash
```

--------------------------------

### Deploy GPU Infrastructure and Observability

Source: https://github.com/nebius/nebius-solutions-library/blob/main/applications/osmo/README.md

Installs essential components for GPU workload management and monitoring, including NVIDIA operators, KAI Scheduler, Prometheus, Grafana, and Loki.

```bash
cd ../002-setup
./01-deploy-gpu-infrastructure.sh
./02-deploy-observability.sh
```

--------------------------------

### Install Nebius CLI via Curl

Source: https://github.com/nebius/nebius-solutions-library/blob/main/applications/osmo/deploy/example/000-prerequisites/README.md

Alternative method to install the Nebius CLI using curl. Appends the binary to your PATH.

```bash
curl -sSL https://storage.eu-north1.nebius.cloud/nebius/install.sh | bash
export PATH="$HOME/.nebius/bin:$PATH"
```

--------------------------------

### Example Rclone S3 Configuration

Source: https://github.com/nebius/nebius-solutions-library/blob/main/data-transfer/Readme.md

An example rclone.conf file for connecting to an S3-compatible storage service. Ensure access keys and endpoint are correctly configured.

```ini
[s3]
type = s3
provider = AWS
env_auth = false
region = eu-north1
no_check_bucket = true
endpoint = https://storage.eu-north1.nebius.cloud:443
acl = private
bucket_acl = private
access_key_id = <your-access-key-id>
secret_access_key = <your-secret-access-key>
```

--------------------------------

### Install SkyPilot with Nebius Support

Source: https://github.com/nebius/nebius-solutions-library/blob/main/skypilot/README.md

Install the SkyPilot package with Nebius support using pip. Ensure you have Python 3.10 or higher.

```bash
pip install "skypilot-nightly[nebius]"
```

--------------------------------

### Run WireGuard Client Setup Script

Source: https://github.com/nebius/nebius-solutions-library/blob/main/applications/osmo/deploy/example/000-prerequisites/README.md

Execute this script if WireGuard VPN was enabled during deployment. It assists in setting up the client configuration.

```bash
./wireguard-client-setup.sh
```

--------------------------------

### Deploy Observability Stack

Source: https://github.com/nebius/nebius-solutions-library/blob/main/applications/osmo/deploy/example/002-setup/README.md

Installs Prometheus, Grafana, Loki, and Promtail for monitoring and logging within the Kubernetes cluster. This script should be run after GPU infrastructure is set up.

```bash
./02-deploy-observability.sh
```

--------------------------------

### Nebius CLI First-Time Authentication

Source: https://github.com/nebius/nebius-solutions-library/blob/main/applications/osmo/deploy/example/000-prerequisites/README.md

Use this command to create a new Nebius CLI profile. It interactively guides you through naming the profile, authenticating via a browser, and selecting your tenant and project.

```bash
# Create a profile (interactive)
nebius profile create
```

--------------------------------

### Configure Terraform Variables

Source: https://github.com/nebius/nebius-solutions-library/blob/main/applications/osmo/deploy/example/README.md

Copies an example Terraform variable file to terraform.tfvars. The cost-optimized secure option is recommended for a balance of cost and security.

```bash
cd 001-iac

# Recommended: Cost-optimized with secure private access
cp terraform.tfvars.cost-optimized-secure.example terraform.tfvars

# Other options:
#   terraform.tfvars.cost-optimized.example  - Cheapest (public endpoints)
#   terraform.tfvars.production.example      - Full production setup
#   terraform.tfvars.secure.example          - H100 with WireGuard

# Edit configuration
vim terraform.tfvars
```

--------------------------------

### SSH Configuration Example

Source: https://github.com/nebius/nebius-solutions-library/blob/main/k8s-training/README.md

Defines SSH configuration for connecting to cluster nodes. Specify the username and either the public key string or its path.

```hcl
# SSH config
ssh_user_name  = "" # Username you want to use to connect to the nodes
ssh_public_key = {
  key  = "Enter your public SSH key here" OR
  path = "Enter the path to your public SSH key here"
}
```

--------------------------------

### Verify Keycloak Setup Job Logs

Source: https://github.com/nebius/nebius-solutions-library/blob/main/applications/osmo/deploy/example/002-setup/AUTHENTICATION.md

Check the logs of the 'keycloak-osmo-setup' job to confirm that the Nebius SSO Identity Provider was successfully created. Look for the success message indicating creation.

```bash
kubectl logs -n osmo job/keycloak-osmo-setup | grep -A2 "Step 4c"
```

--------------------------------

### Osmo Workflow with GPU Resources

Source: https://github.com/nebius/nebius-solutions-library/blob/main/applications/osmo/deploy/example/002-setup/README.md

Example of an Osmo workflow definition specifying GPU resources. Ensure the `platform: gpu` is set to utilize the configured GPU platform.

```yaml
workflow:
  name: my-gpu-job
  resources:
    gpu-resource:
      platform: gpu    # <-- Selects GPU platform with tolerations
      gpu: 1
      memory: 4Gi
  tasks:
  - name: train
    image: nvcr.io/nvidia/cuda:12.6.3-base-ubuntu24.04
    command: ["nvidia-smi"]
    resource: gpu-resource
```

--------------------------------

### Add GPU Resources

Source: https://github.com/nebius/nebius-solutions-library/blob/main/applications/osmo/workflows/README.md

Specify the number of GPUs required for the training job within the 'resources' section of the workflow YAML. This example requests 8 GPUs.

```yaml
resources:
  limits:
    nvidia.com/gpu: 8
```

--------------------------------

### Expose Internal REST API with Tailscale

Source: https://github.com/nebius/nebius-solutions-library/blob/main/modules/tailscale-service/README.md

This example demonstrates exposing an internal REST API. It includes a dependency on the tailscale operator and uses a more specific selector.

```hcl
module "my_rest_api_tailscale" {
  source = "../../modules/tailscale-service"

  namespace        = "platform"
  name             = "my-rest-api-ts"
  tailnet_hostname = "my-rest-api"

  selector = {
    "app.kubernetes.io/name"      = "my-rest-api"
    "app.kubernetes.io/instance"  = "platform"
    "app.kubernetes.io/component" = "api"
  }

  ports = [
    {
      name        = "http"
      port        = 8080
      target_port = 8080
    }
  ]

  depends_on = [module.tailscale_operator]

  providers = {
    kubernetes = kubernetes
  }
}
```

--------------------------------

### VM Instance Configuration for GPU Cluster

Source: https://github.com/nebius/nebius-solutions-library/blob/main/vm-instance/README.md

Configuration for provisioning VM instances within a GPU cluster, enabling high-speed Infiniband connectivity between them. This setup is not modifiable for running instances.

```hcl
preset = "8gpu-128vcpu-1600gb"
platform = "gpu-h100-sxm"

users = [
  {
    user_name = "admin",
    ssh_key_path = "~/.ssh/id_rsa.pub"
  }
]

public_ip = true
instance_count = 2

fabric = "fabric-6"

```

--------------------------------

### Run Ray Job Submission

Source: https://github.com/nebius/nebius-solutions-library/blob/main/modules/kuberay/README.md

Submit a simple Ray job to the cluster using the RAY_ADDRESS environment variable and the 'ray job submit' command. This example prints the cluster's resources.

```shell
RAY_ADDRESS="http://localhost:8265" ray job submit -- python -c "import ray; ray.init(); print(ray.cluster_resources())"
```

--------------------------------

### Get Kubernetes Cluster Credentials

Source: https://github.com/nebius/nebius-solutions-library/blob/main/k8s-training/README.md

Fetches credentials for a Nebius Kubernetes cluster and merges them into the kubectl configuration file. Requires Nebius CLI, kubectl, and jq to be installed.

```bash
nebius mk8s v1 cluster get-credentials --id $(cat terraform.tfstate | jq -r '.resources[] | select(.type == "nebius_mk8s_v1_cluster") | .instances[].attributes.id') --external
```

--------------------------------

### Create OIDC Client Request Example

Source: https://github.com/nebius/nebius-solutions-library/blob/main/applications/osmo/deploy/example/002-setup/AUTHENTICATION.md

This demonstrates the structure of a request to create an OIDC client using the Nebius IAM gRPC API. It includes essential fields like parent ID, client name, authentication methods, redirect URIs, scopes, and grant types.

```protobuf
metadata.parent_id: "Your project NID (required)"
metadata.name: "Unique client name, e.g. `keycloak-osmo`"
spec.client_authentication_methods: ["client_secret_basic"]
spec.redirect_uris: ["Exact callback URL, no trailing slash. Example (nip.io): `https://auth-osmo.89-169-122-246.nip.io/realms/osmo/broker/nebius-sso/endpoint`. Max 5 URIs."]
spec.scopes: ["openid"]
spec.authorization_grant_types: ["authorization_code"]
spec.pkce_enabled: true
```

--------------------------------

### SkyPilot Nebius Configuration Example

Source: https://github.com/nebius/nebius-solutions-library/blob/main/skypilot/README.md

Configure SkyPilot to use Nebius by specifying region, project ID, and filesystem details. Ensure your Nebius API endpoint is correctly set.

```yaml
nebius:
  region_configs:
    <your_region>:
      project_id: project-<your_project_id>
      filesystems:
      - filesystem_id: computefilesystem-<your_filesystem_id>
        mount_path: <your_mount_path>
        attach_mode: READ_WRITE
api_server:
  endpoint: https://USER:PASSWORD@<skypilot_instance_public_ip_endpoint>
```

--------------------------------

### Initialize Terraform

Source: https://github.com/nebius/nebius-solutions-library/blob/main/anyscale/README.md

Initialize Terraform in the 'prepare' directory to set up the NFS server and object storage.

```bash
terraform -chdir=prepare init
```

--------------------------------

### Terraform Initialization and Deployment

Source: https://github.com/nebius/nebius-solutions-library/blob/main/nfs-server/README.md

Run these commands to initialize Terraform, plan the changes, and apply the configuration to provision the NFS server.

```bash
terraform init
terraform plan
terraform apply
```

--------------------------------

### Configure GPU Platform for Workloads

Source: https://github.com/nebius/nebius-solutions-library/blob/main/applications/osmo/README.md

Sets up the default pool with GPU platform configuration, including a pod template with appropriate node selectors and tolerations for GPU nodes.

```bash
./08-configure-gpu-platform.sh
```

--------------------------------

### Install JQuery

Source: https://github.com/nebius/nebius-solutions-library/blob/main/bastion/README.md

Installs the JQuery package on Debian-based distributions. This is a prerequisite for certain operations.

```bash
sudo apt install jq -y
```

--------------------------------

### Submit CPU Workflow

Source: https://github.com/nebius/nebius-solutions-library/blob/main/applications/osmo/workflows/README.md

Submits the 'hello_nebius.yaml' workflow to test CPU functionality. This workflow runs on a GPU node and prints 'Hello Nebius!'.

```bash
osmo workflow submit osmo/hello_nebius.yaml
```

--------------------------------

### Deploy Observability Stack

Source: https://github.com/nebius/nebius-solutions-library/blob/main/applications/osmo/deploy/example/README.md

Sets up the observability stack, which includes monitoring and logging tools for the deployed environment.

```bash
# 2. Deploy observability stack
./02-deploy-observability.sh
```

--------------------------------

### Install GPU Operator

Source: https://github.com/nebius/nebius-solutions-library/blob/main/gpu_slicing/timeslice/README.md

Installs the NVIDIA GPU Operator using Helm. Ensure to use the specified version for compatibility.

```bash
helm install --wait --generate-name \
    -n gpu-operator --create-namespace \
    nvidia/gpu-operator \
    --version=v25.3.2
```

--------------------------------

### Install GPU Operator

Source: https://github.com/nebius/nebius-solutions-library/blob/main/gpu_slicing/mig/README.md

Installs the NVIDIA GPU Operator using Helm. Ensure a driverless cluster is deployed beforehand.

```bash
helm install --wait --generate-name -n gpu-operator --set migStrategy=single --create-namespace nvidia/gpu-operator --version=v25.3.2
```

--------------------------------

### Create MPS Configuration

Source: https://github.com/nebius/nebius-solutions-library/blob/main/gpu_slicing/mps/README.md

Create a YAML configuration file for the device plugin to enable MPS with a specified number of GPU replicas.

```bash
cat << EOF > /tmp/dp-mps-10.yaml
version: v1
sharing:
  mps:
    resources:
    - name: nvidia.com/gpu
      replicas: 10
EOF
```

--------------------------------

### Initialize and Apply Terraform for Infrastructure

Source: https://github.com/nebius/nebius-solutions-library/blob/main/applications/osmo/iam-register/README.md

Initializes Terraform and applies the configuration to set up the necessary infrastructure, including Kubernetes cluster and PostgreSQL.

```bash
cd 001-iac
terraform init
terraform apply
```

--------------------------------

### Install GPU Operator

Source: https://github.com/nebius/nebius-solutions-library/blob/main/gpu_slicing/mps/README.md

Install the NVIDIA GPU Operator using Helm, disabling GFD and the Device Plugin for manual MPS configuration.

```bash
helm install \
      -n gpu-operator \
      --generate-name \
      --create-namespace \
      --set devicePlugin.enabled=false \
      --set gfd.enabled=false \
      nvidia/gpu-operator
```

--------------------------------

### Install External Secrets Operator with Helm

Source: https://github.com/nebius/nebius-solutions-library/blob/main/modules/external-secrets-operator/README.md

Use this Terraform module to install the External Secrets Operator. Ensure the Kubernetes and Helm providers are configured.

```hcl
module "external_secrets_operator" {
  source = "../../modules/external-secrets-operator"

  providers = {
    kubernetes = kubernetes
    helm       = helm
  }
}
```

--------------------------------

### Deploy Infrastructure with Terraform

Source: https://github.com/nebius/nebius-solutions-library/blob/main/applications/osmo/deploy/example/README.md

Initializes Terraform, plans the deployment, and applies the infrastructure changes. This creates core cloud resources like VPC, Kubernetes clusters, and databases.

```bash
# Deploy
terraform init
terraform plan
terraform apply
```

--------------------------------

### Initialize Terraform

Source: https://github.com/nebius/nebius-solutions-library/blob/main/applications/dstack/README.md

Initialize the Terraform working directory to download necessary providers and modules.

```bash
terraform init
```

--------------------------------

### Create GPU Pod Template via API

Source: https://github.com/nebius/nebius-solutions-library/blob/main/applications/osmo/deploy/example/002-setup/README.md

Use this command to create a GPU pod template configuration. Ensure `port-forward` is running and the JSON file exists.

```bash
curl -X PUT 'http://localhost:8080/api/configs/pod_template/gpu_tolerations' \
  -H 'Content-Type: application/json' \
  -d @gpu_pod_template.json
```

```json
{
  "configs": {
    "spec": {
      "tolerations": [
        {
          "key": "nvidia.com/gpu",
          "operator": "Exists",
          "effect": "NoSchedule"
        }
      ],
      "nodeSelector": {
        "nvidia.com/gpu.present": "true"
      }
    }
  }
}
```

--------------------------------

### VM Instance Configuration with Shared Filesystem

Source: https://github.com/nebius/nebius-solutions-library/blob/main/vm-instance/README.md

Configuration for creating VM instances that connect to a pre-existing shared filesystem. The filesystem will be mounted by default to `/mnt/share`.

```hcl
preset = "16vcpu-64gb"
platform = "cpu-e2"

users = [
  {
    user_name = "admin",
    ssh_key_path = "~/.ssh/id_rsa.pub"
  }
]
shared_filesystem_mount = "/mnt/share"  # optional
shared_filesystem_id = "computefilesystem-xxxxx"
public_ip = true
instance_count = 2
```

--------------------------------

### Deploy NGINX Ingress Controller

Source: https://github.com/nebius/nebius-solutions-library/blob/main/applications/osmo/deploy/example/README.md

Installs the NGINX Ingress Controller to manage external access to services within the Kubernetes cluster.

```bash
# 3. Deploy NGINX Ingress Controller
./03-deploy-nginx-ingress.sh
```

--------------------------------

### Configure Nebius Environment

Source: https://github.com/nebius/nebius-solutions-library/blob/main/applications/osmo/deploy/example/README.md

Sets up the Nebius environment variables necessary for deployment. It's recommended to initialize secrets using MysteryBox as well.

```bash
# Configure Nebius environment
source ./nebius-env-init.sh

# (Recommended) Initialize secrets in MysteryBox
source ./secrets-init.sh
```

--------------------------------

### Reload Shell Session

Source: https://github.com/nebius/nebius-solutions-library/blob/main/bastion/README.md

Reloads the current shell session to apply changes, such as newly installed CLIs. Alternatively, use 'source ~/.bashrc'.

```bash
exec -l $SHELL
```

```bash
source ~/.bashrc
```

--------------------------------

### Deploy OSMO Backend

Source: https://github.com/nebius/nebius-solutions-library/blob/main/applications/osmo/deploy/example/002-setup/README.md

Installs the OSMO backend operator, which is necessary for the full functionality of the OSMO platform. This should be run after the control plane deployment.

```bash
./06-deploy-osmo-backend.sh
```

--------------------------------

### Copy Configuration Template

Source: https://github.com/nebius/nebius-solutions-library/blob/main/applications/dstack/README.md

Make a copy of the default configuration template to customize it for your deployment.

```bash
cp default.yaml.tpl default.yaml
```

--------------------------------

### Configure Existing Filesystems for Reuse

Source: https://github.com/nebius/nebius-solutions-library/blob/main/soperator/README.md

Set up retained filesystems as existing resources for future cluster configurations using their recorded IDs.

```hcl
filestore_jail = {
  existing = {
    id = "computefilesystem-<RETAINED-JAIL-ID>"
  }
}

filestore_jail_submounts = [{
  name       = "data"
  mount_path = "/data"
  existing = {
    id = "computefilesystem-<RETAINED-DATA-ID>"
  }
}]
```

--------------------------------

### Display Kubectl Configuration Snippet

Source: https://github.com/nebius/nebius-solutions-library/blob/main/k8s-training/README.md

Shows a sample output of the `kubectl config view` command, illustrating the structure of a Kubernetes configuration file.

```yaml
apiVersion: v1
clusters:
  - cluster:
    certificate-authority-data: DATA+OMITTED
```

--------------------------------

### Configure OSMO Storage

Source: https://github.com/nebius/nebius-solutions-library/blob/main/applications/osmo/deploy/example/002-setup/README.md

Sets up S3-compatible storage for workflow logs and data. This script may require a port-forwarding setup, as detailed in the main README.

```bash
./07-configure-storage.sh
```

--------------------------------

### Troubleshoot InfiniBand Issues

Source: https://github.com/nebius/nebius-solutions-library/blob/main/applications/osmo/deploy/example/002-setup/README.md

Check the Network Operator and verify RDMA devices using `ibstat` within the DCGM exporter pod.

```bash
kubectl get pods -n network-operator
kubectl exec -n gpu-operator <dcgm-pod> -- ibstat
```

--------------------------------

### Configure SSH and WireGuard Access

Source: https://github.com/nebius/nebius-solutions-library/blob/main/bastion/README.md

Example Terraform configuration for setting allowed CIDRs for SSH and WireGuard access, and optionally exposing the WireGuard UI.

```hcl
bastion_allowed_ssh_cidrs = ["203.0.113.10/32"]

# Optional: expose WireGuard UI only to trusted admin CIDRs.
bastion_allowed_wireguard_ui_cidrs = ["203.0.113.10/32"]
```

--------------------------------

### Example: Cluster-Admin Access

Source: https://github.com/nebius/nebius-solutions-library/blob/main/modules/k8s-rbac-bindings/README.md

Use this snippet to grant cluster-admin privileges to a Nebius IAM group. Ensure this is explicitly approved due to its broad permissions.

```hcl
module "k8s_rbac_bindings" {
  source = "../../modules/k8s-rbac-bindings"

  cluster_role_bindings = {
    nebius_viewer_cluster_admin = {
      name      = "nebius-cluster-admin"
      role_name = "cluster-admin"
      subjects = [
        {
          kind      = "Group"
          name      = "nebius:viewer"
          api_group = "rbac.authorization.k8s.io"
        }
      ]
    }
  }

  providers = {
    kubernetes = kubernetes
  }
}
```

--------------------------------

### Ray Job Submission Output

Source: https://github.com/nebius/nebius-solutions-library/blob/main/modules/kuberay/README.md

Example output from a successful Ray job submission, including job ID, logs, and cluster resource information.

```shell
Job submission server address: http://localhost:8265
                                           
-------------------------------------------------------
Job 'raysubmit_C3wurkv53yLxKwSQ' submitted successfully
-------------------------------------------------------
                                           
Next steps
  Query the logs of the job:
    ray job logs raysubmit_C3wurkv53yLxKwSQ
  Query the status of the job:
    ray job status raysubmit_C3wurkv53yLxKwSQ
  Request the job to be stopped:
    ray job stop raysubmit_C3wurkv53yLxKwSQ

Tailing logs until the job exits (disable with --no-wait):
2024-08-02 05:10:54,258 INFO worker.py:1405 -- Using address 172.17.132.18:6379 set in the environment variable RAY_ADDRESS
2024-08-02 05:10:54,259 INFO worker.py:1540 -- Connecting to existing Ray cluster at address: 172.17.132.18:6379...
2024-08-02 05:10:54,266 INFO worker.py:1715 -- Connected to Ray cluster. View the dashboard at http://172.17.132.18:8265 
{'object_store_memory': 49007028633.0, 'GPU': 16.0, 'memory': 164282499072.0, 'node:172.17.131.19': 1.0, 'accelerator_type:H100': 2.0, 'CPU': 22.0, 'node:__internal_head__'
: 1.0, 'node:172.17.132.18': 1.0}

------------------------------------------
Job 'raysubmit_C3wurkv53yLxKwSQ' succeeded
------------------------------------------
```

--------------------------------

### Create GPU Platform via API

Source: https://github.com/nebius/nebius-solutions-library/blob/main/applications/osmo/deploy/example/002-setup/README.md

This command creates or updates a GPU platform configuration. It specifies resource allowances and overrides the pod template.

```bash
curl -X PUT 'http://localhost:8080/api/configs/pool/default/platform/gpu' \
  -H 'Content-Type: application/json' \
  -d @gpu_platform_update.json
```

```json
{
  "configs": {
    "description": "GPU platform for L40S nodes",
    "host_network_allowed": false,
    "privileged_allowed": false,
    "allowed_mounts": [],
    "default_mounts": [],
    "default_variables": {
      "USER_GPU": 1
    },
    "resource_validations": [],
    "override_pod_template": ["gpu_tolerations"]
  }
}
```

--------------------------------

### Use NFS Module in Terraform

Source: https://github.com/nebius/nebius-solutions-library/blob/main/modules/nfs-server/README.md

Example of how to include and configure the NFS server module within a Terraform project. Ensure all required variables are provided.

```terraform
module "nfs-module" {
  providers = {
    nebius = nebius
  }
  source            = "../../modules/nfs-module"
  parent_id         = var.parent_id
  subnet_id         = var.subnet_id
  ssh_user_name     = var.ssh_user_name
  ssh_public_keys   = var.ssh_public_keys
  nfs_ip_range      = var.nfs_ip_range
  nfs_size          = var.nfs_size
}
```

--------------------------------

### Initialize Nebius CLI Configuration

Source: https://github.com/nebius/nebius-solutions-library/blob/main/dsvm/README.md

Initializes the Nebius CLI configuration. It is recommended to use a service account for configuration.

```bash
nebius init
```

--------------------------------

### Deploy OSMO Backend Operator

Source: https://github.com/nebius/nebius-solutions-library/blob/main/applications/osmo/README.md

Installs the backend operator responsible for managing GPU workloads, connecting to the OSMO control plane, and configuring resource pools.

```bash
./05-deploy-osmo-backend.sh
```

--------------------------------

### Configure Strict Egress Rules

Source: https://github.com/nebius/nebius-solutions-library/blob/main/bastion/README.md

Example Terraform configuration to disable the default egress rule, enforcing a stricter egress posture by requiring explicit destinations.

```hcl
bastion_egress_cidrs = []
```

--------------------------------

### Configure Nebius Environment

Source: https://github.com/nebius/nebius-solutions-library/blob/main/applications/osmo/deploy/example/001-iac/README.md

Source this script to initialize your Nebius environment variables.

```bash
source ../000-prerequisites/nebius-env-init.sh
```

--------------------------------

### Example: Read-Only Cluster Visibility

Source: https://github.com/nebius/nebius-solutions-library/blob/main/modules/k8s-rbac-bindings/README.md

Use this configuration to grant read-only access across the entire cluster to a specified group. This is suitable for monitoring or auditing purposes.

```hcl
module "k8s_rbac_bindings" {
  source = "../../modules/k8s-rbac-bindings"

  cluster_role_bindings = {
    nebius_viewer_read_only = {
      name      = "nebius-viewer-read-only"
      role_name = "view"
      subjects = [
        {
          kind      = "Group"
          name      = "nebius:viewer"
          api_group = "rbac.authorization.k8s.io"
        }
      ]
    }
  }

  providers = {
    kubernetes = kubernetes
  }
}
```

--------------------------------

### Verify Osmo Configuration

Source: https://github.com/nebius/nebius-solutions-library/blob/main/applications/osmo/deploy/example/002-setup/README.md

These commands verify the created pod templates and platforms. They also check resource allocation for GPUs.

```bash
# Check pod templates
curl -s http://localhost:8080/api/configs/pod_template | jq 'keys'
# Should include: "gpu_tolerations"

# Check GPU platform
curl -s http://localhost:8080/api/configs/pool/default | jq '.platforms.gpu'

# Check resources (GPU nodes should now be visible)
curl -s http://localhost:8080/api/resources | jq '.resources[] | {name: .name, gpu: .allocatable_fields.gpu}'
```

--------------------------------

### Preview Terraform Plan

Source: https://github.com/nebius/nebius-solutions-library/blob/main/applications/dstack/README.md

Generate a plan to preview the infrastructure changes before applying them.

```bash
terraform plan
```

--------------------------------

### Configure Nebius Environment

Source: https://github.com/nebius/nebius-solutions-library/blob/main/applications/osmo/README.md

Initializes the Nebius environment by checking CLI, authentication, listing tenants, configuring the project, setting the region, and exporting necessary environment variables.

```bash
source ./nebius-env-init.sh
```

--------------------------------

### Uninstall OSMO Components

Source: https://github.com/nebius/nebius-solutions-library/blob/main/applications/osmo/deploy/example/README.md

Removes Kubernetes components in reverse order of deployment, starting with the OSMO backend and control plane, followed by observability and GPU infrastructure.

```bash
# 1. Remove Kubernetes components
cd 002-setup/cleanup
./uninstall-osmo-backend.sh
./uninstall-osmo-control-plane.sh
./uninstall-observability.sh
./uninstall-gpu-infrastructure.sh
```

--------------------------------

### Get LoadBalancer IP for nip.io Hostname

Source: https://github.com/nebius/nebius-solutions-library/blob/main/applications/osmo/deploy/example/002-setup/README.md

Retrieve the external IP address of the ingress-nginx controller service to construct a nip.io hostname for accessing OSMO services.

```bash
kubectl get svc -n ingress-nginx ingress-nginx-controller -o jsonpath='{.status.loadBalancer.ingress[0].ip}'
```

--------------------------------

### Configure GPU Platform

Source: https://github.com/nebius/nebius-solutions-library/blob/main/applications/osmo/deploy/example/002-setup/README.md

This script configures the GPU platform, including setting up tolerations and node selectors, which is required for GPU-accelerated workflows. It should be run after core OSMO components are deployed.

```bash
./09-configure-gpu-platform.sh
```

--------------------------------

### Verify GPU Pod Template Configuration

Source: https://github.com/nebius/nebius-solutions-library/blob/main/applications/osmo/README.md

Verifies that the pod template for GPU tolerations is correctly configured with the necessary node selectors and tolerations.

```bash
osmo config show POD_TEMPLATE gpu_tolerations
```

--------------------------------

### Deploy OSMO Control Plane

Source: https://github.com/nebius/nebius-solutions-library/blob/main/applications/osmo/README.md

Installs the core OSMO services, including the namespace, databases, Redis, API, agent, worker, logger, and Kubernetes Ingress resources.

```bash
./04-deploy-osmo-control-plane.sh
```

--------------------------------

### Configure KubeRay Module in Terraform

Source: https://github.com/nebius/nebius-solutions-library/blob/main/modules/kuberay/README.md

Add this module call to your root main.tf to integrate KubeRay. Configure CPU and GPU worker setups with appropriate images and resources.

```terraform
module "kuberay" {
  providers = {
    nebius = nebius
    helm   = helm
  }

  parent_id  = var.parent_id
  cluster_id = nebius_mk8s_v1_cluster.k8s-cluster.id
  #cpu worker setup
  cpu_platform     = local.cpu_nodes_platform
  cpu_worker_image = var.kuberay_cpu_worker_image
  min_cpu_replicas = var.kuberay_min_cpu_replicas
  max_cpu_replicas = var.kuberay_max_cpu_replicas
  cpu_resources    = var.kuberay_cpu_resources
  #gpu worker setup
  gpu_platform     = local.gpu_nodes_platform
  gpu_worker_image = var.kuberay_gpu_worker_image
  min_gpu_replicas = var.kuberay_min_gpu_replicas
  max_gpu_replicas = var.kuberay_max_gpu_replicas
  gpu_resources    = var.kuberay_gpu_resources

}
```

--------------------------------

### Run hello.sh Script

Source: https://github.com/nebius/nebius-solutions-library/blob/main/soperator/test/quickcheck/README.md

Execute the hello.sh script to perform basic checks of the Slurm cluster, ensuring jobs can be executed and resources allocated. The output is monitored using tail -f.

```shell
sbatch hello.sh && \
tail -f results/hello.out
```

--------------------------------

### Display deliver.sh Usage

Source: https://github.com/nebius/nebius-solutions-library/blob/main/soperator/test/README.md

Use this command to display the help message for the `deliver.sh` script, outlining its required and optional flags for uploading test data and scripts.

```console
$ ./deliver.sh -h
Usage: ./deliver.sh <REQUIRED_FLAGS> [FLAGS] [-h]
Required flags:
  -t  [str ]  Test type. One of:
                quickcheck
  -u  [str ]  SSH username
  -k  [path]  Path to private SSH key
  -a  [str ]  Address of login node (IP or domain name)

Flags:
  -p  [int ]  SSH port of login node.
              By default, 22

  -h  Print help and exit
```

--------------------------------

### Deploy OSMO Control Plane Components

Source: https://github.com/nebius/nebius-solutions-library/blob/main/applications/osmo/iam-register/README.md

Deploys GPU infrastructure, observability tools, Nginx ingress, and the OSMO control plane. TLS for Keycloak is recommended for production environments.

```bash
cd applications/osmo/deploy/example/002-setup

./01-deploy-gpu-infrastructure.sh
./02-deploy-observability.sh
./03-deploy-nginx-ingress.sh

./04-deploy-osmo-control-plane.sh
```

--------------------------------

### Initialize Secrets with MysteryBox

Source: https://github.com/nebius/nebius-solutions-library/blob/main/applications/osmo/deploy/example/001-iac/README.md

Commands to initialize secrets in MysteryBox before running `terraform apply`.

```bash
# Before terraform apply:
cd ../000-prerequisites
source ./secrets-init.sh  # Creates secrets in MysteryBox
cd ../001-iac
terraform apply           # Uses TF_VAR_* env vars set by script
```

--------------------------------

### Temporary Unrestricted Ingress for Testing

Source: https://github.com/nebius/nebius-solutions-library/blob/main/bastion/README.md

Example Terraform configuration for temporary unrestricted SSH ingress, useful for testing from dynamic IP addresses. Avoid using this in production environments.

```hcl
bastion_allowed_ssh_cidrs          = ["0.0.0.0/0"]
bastion_allow_unrestricted_ingress_rules = true
```