SUSE Private AI Stack

https://github.com/suse/suse-ai-stack
Admin
A lightweight virtualized private AI stack that combines SUSE Linux, Kubernetes, Rancher, Ollama LLM...
Tokens:33,453
Snippets:459
Trust Score:9.1
License:Apache-2.0
Update:2 months ago
Show doc for...
Tokens:
Raw
### Install KServe Quickstart Environment

Source: https://github.com/suse/suse-ai-stack/blob/main/docs/cncf-ai-conformance/v1.34/SUSE-AI/specs/robust_controller.md

Executes a script to set up the KServe quickstart environment. This script is a modified version of the official KServe quickstart script.

```bash
./kserve/quickstart.sh
```

--------------------------------

### Example /etc/hosts Configuration Instructions

Source: https://github.com/suse/suse-ai-stack/blob/main/docs/AWS_Setup.md

Sample output provided by the setup script detailing the required entries for the local /etc/hosts file.

```text
"Make sure to update the /etc/hosts file: 35.87.100.210 mgmt-rancher suse-rancher.demo",
"Make sure to update the /etc/hosts file: 35.87.100.210 suse-ai suse-ollama-webui",
"Make sure to update the /etc/hosts file: 35.87.100.210 suse-observability",
"To access rancher UI, point your browser to https://suse-rancher.demo and login with user=admin and password=rancher",
"To access open-webui, point your browser to https://suse-ollama-webui and login with user=admin@suse-private-ai.org and password=WelcomeToAI",
"To access suse-observability, point your browser to https://suse-observability and login with user=admin and password=MldDMgj2eRHPUXMT"
```

--------------------------------

### Full Deployment Sequence Tree

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/playbooks-reference.md

Visual representation of the deployment hierarchy starting from the main setup script.

```text
setup_private_ai_stack.sh
  ├─ setup_nodes.yml
  │  └─ roles/vm (create infrastructure)
  ├─ deploy_rke2_rancher.yml (mgmt cluster)
  ├─ deploy_rke2_rancher.yml (suse-ai cluster)
  ├─ deploy_rke2_rancher.yml (suse-observability cluster)
  ├─ deploy_nvidia_driver_installer.yml
  ├─ setup_dns_storage.yml (all clusters)
  ├─ deploy_suse_observability.yml
  ├─ deploy_suse_ai.yml
  └─ display.yml
```

--------------------------------

### Deploy NVIDIA Driver Installer Playbook

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/playbooks-reference.md

Command to execute the NVIDIA driver installation playbook with specified inventory and extra variables.

```bash
ansible-playbook playbooks/deploy_nvidia_driver_installer.yml \
  -i inventories/localhost_inventory.yml \
  -i inventories/suse-ai_inventory.ini \
  -e @extra_vars.yml
```

--------------------------------

### Execute Setup Script

Source: https://github.com/suse/suse-ai-stack/blob/main/docs/Local_Setup.md

Run the provided shell script to initialize the private AI stack environment.

```console
./setup_private_ai_stack.sh
```

--------------------------------

### Setup DNS and Storage Playbook

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/playbooks-reference.md

Command to execute the DNS and storage configuration playbook for a specific cluster.

```bash
ansible-playbook playbooks/setup_dns_storage.yml \
  -i inventories/localhost_inventory.yml \
  -i inventories/suse-ai_inventory.ini \
  -e @extra_vars.yml \
  -e cluster=suse-ai
```

--------------------------------

### Execute Infrastructure Setup Playbook

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/entry-points-and-scripts.md

Runs the setup_nodes playbook to create VM instances and define cluster nodes.

```bash
playbook="setup_nodes"
ansible-playbook "${PROJECT_DIR}/playbooks/${playbook}.yml" \
  -e current_project_dir="${PROJECT_DIR}" \
  -e @extra_vars.yml
```

--------------------------------

### Deploy NVIDIA GPU Support

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/entry-points-and-scripts.md

Installs GPU drivers if hardware is detected and enabled in configuration.

```bash
playbook="deploy_nvidia_driver_installer"
ansible-playbook "${PROJECT_DIR}/playbooks/${playbook}.yml" \
  -i inventories/localhost_inventory.yml \
  -i "${PROJECT_DIR}/inventories/${cluster_inv_name}" \
  -e @extra_vars.yml
```

--------------------------------

### Install mkisofs dependency

Source: https://github.com/suse/suse-ai-stack/blob/main/docs/Local_Setup.md

Install the required mkisofs utility using the zypper package manager.

```console
sudo zypper install mkisofs
```

--------------------------------

### Install Volcano

Source: https://github.com/suse/suse-ai-stack/blob/main/docs/cncf-ai-conformance/v1.34/SUSE-AI/specs/gang_scheduling.md

Installs the Volcano gang scheduling system into the 'volcano-system' namespace. Ensure the namespace exists or is created.

```bash
helm install volcano volcano-sh/volcano -n volcano-system --create-namespace
```

--------------------------------

### Configure Network Settings

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/deployment-architecture.md

Configuration parameters for default and custom libvirt network setups.

```yaml
Network: default
CIDR: 192.168.122.0/24
DHCP Range: 192.168.122.1-254
Default Route: 192.168.122.1
DNS: Host system resolver
```

```yaml
Network: {{ private_ai_vm_network }}
Static IP: {{ private_ai_vm_ip }}
MAC Address: {{ private_ai_vm_mac_address }}
```

--------------------------------

### Install Ansible Dependencies

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/entry-points-and-scripts.md

Command to install collections and roles defined in the requirements file.

```bash
ansible-galaxy install -r requirements.yml
```

--------------------------------

### Define Cloud-Init Configuration

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/deployment-architecture.md

YAML configuration for VM provisioning, including user setup and required system packages.

```yaml
#cloud-config
hostname: ai-vm
fqdn: ai-vm.local
manage_etc_hosts: true
users:
  - name: ai
    sudo: ALL=(ALL) NOPASSWD:ALL
    ssh_authorized_keys:
      - {{ vm_authorized_ssh_keys }}
ssh_pwauth: true
packages:
  - curl
  - git
  - jq
  - python3-yaml
  - qemu-guest-agent
```

--------------------------------

### Install APISIX Gateway with Gateway API Enabled

Source: https://github.com/suse/suse-ai-stack/blob/main/docs/cncf-ai-conformance/v1.34/SUSE-AI/specs/ai_inference.md

Deploys the APISIX Ingress Controller using Helm, enabling Gateway API support and configuring necessary parameters for integration.

```bash
helm upgrade -i apisix oci://dp.apps.rancher.io/charts/apache-apisix --version 2.11.3 --create-namespace  --namespace apisix \
    --set global.imagePullSecrets={application-collection} --set apisix.admin.enabled=true --set apisix.admin.type=NodePort \
    --set apisix.admin.allow.ipList={0.0.0.0/0} --set ingress-controller.enabled=true --set ingress-controller.config.apisix.serviceNamespace=apisix \
    --set ingress-controller.config.apisix.adminAPIVersion="v3" --set ingress-controller.config.kubernetes.enableGatewayAPI=true
```

--------------------------------

### Deploy RKE2 and Rancher

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/entry-points-and-scripts.md

Iterates through clusters to install RKE2 and Rancher Prime using the deploy_rke2_rancher playbook.

```bash
playbook="deploy_rke2_rancher"
for cluster in "${clusters[@]}"; do
  if [ -e "${PROJECT_DIR}/inventories/${cluster}_inventory.ini" ]; then
    ansible-playbook "${PROJECT_DIR}/playbooks/${playbook}.yml" \
      -i "${PROJECT_DIR}/inventories/${cluster}_inventory.ini" \
      -e "@${DEST_DIR}/${cluster}_extra_vars.yml" \
      -e cluster="${cluster}"
  fi
done
```

--------------------------------

### Example virtual network definition

Source: https://github.com/suse/suse-ai-stack/blob/main/docs/Local_Setup.md

A sample XML configuration for a libvirt network including a static DHCP host entry.

```xml
<network>
  <name>default</name>
  <uuid>314e5390-370b-4d2c-a0e7-57b220b43754</uuid>
  <forward mode='nat'/>
  <bridge name='virbr0' stp='on' delay='0'/>
  <mac address='52:54:00:49:f7:9e'/>
  <ip address='192.168.122.1' netmask='255.255.255.0'>
    <dhcp>
      <range start='192.168.122.2' end='192.168.122.254'/>
      <host mac='52:54:00:6C:3C:88' ip='192.168.122.100'/>
    </dhcp>
  </ip>
</network>
```

--------------------------------

### Install Ansible collections

Source: https://github.com/suse/suse-ai-stack/blob/main/docs/Local_Setup.md

Install the required Ansible modules defined in the requirements.yml file.

```console
ansible-galaxy collection install -r requirements.yml
```

--------------------------------

### Execute Playbook for Specific Clusters

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/playbooks-reference.md

Examples of running the playbook for management, AI, and observability clusters using cluster-specific inventory and variable files.

```bash
# Management cluster
ansible-playbook playbooks/deploy_rke2_rancher.yml \
  -i inventories/mgmt_inventory.ini \
  -e @mgmt_extra_vars.yml

# AI cluster
ansible-playbook playbooks/deploy_rke2_rancher.yml \
  -i inventories/suse-ai_inventory.ini \
  -e @suse-ai_extra_vars.yml

# Observability cluster
ansible-playbook playbooks/deploy_rke2_rancher.yml \
  -i inventories/suse-observability_inventory.ini \
  -e @suse-observability_extra_vars.yml
```

--------------------------------

### Run Verbose Debugging

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/entry-points-and-scripts.md

Enable verbose output for setup scripts and Ansible playbooks to troubleshoot deployment issues.

```bash
# Run setup with verbose Ansible output
DEBUG=1 ./setup_private_ai_stack.sh -vvv

# Run specific playbook with debug
ansible-playbook playbooks/deploy_suse_ai.yml \
  -i inventories/suse-ai_inventory.ini \
  -e @extra_vars.yml \
  -vvv
```

--------------------------------

### Add Volcano Helm Repository and Update

Source: https://github.com/suse/suse-ai-stack/blob/main/docs/cncf-ai-conformance/v1.34/SUSE-AI/specs/gang_scheduling.md

Installs the Volcano Helm chart repository and updates the local Helm chart repository cache. This is a prerequisite for installing Volcano.

```bash
helm repo add volcano-sh https://volcano-sh.github.io/helm-charts
helm repo update
```

--------------------------------

### Install Ollama Operator

Source: https://github.com/suse/suse-ai-stack/blob/main/docs/cncf-ai-conformance/v1.34/SUSE-AI/specs/robust_controller.md

Installs the Ollama community operator using kubectl. Ensure server-side apply is enabled.

```bash
kubectl apply \
  --server-side=true \
  -f https://raw.githubusercontent.com/nekomeowww/ollama-operator/v0.10.1/dist/install.yaml
```

--------------------------------

### Deploy NVIDIA Driver Installer

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/README.md

Executes the Ansible playbook to install NVIDIA drivers on an existing cluster with GPU operator enabled.

```bash
ansible-playbook playbooks/deploy_nvidia_driver_installer.yml \
  -i inventories/suse-ai_inventory.ini \
  -e extra_vars.yml \
  -e enable_gpu_operator=true
```

--------------------------------

### Observability Cluster Inventory Format

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/entry-points-and-scripts.md

Example of the generated INI inventory file for the observability cluster.

```ini
[rke2_servers]
obs-rancher ansible_host=192.168.122.104

[rke2_agents]
```

--------------------------------

### NVIDIA Driver Installer Helm Release Configuration

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/ansible-roles-reference.md

Displays the specific Helm release details for the nvidia-driver-installer chart.

```yaml
Release: nvidia-driver-installer
Chart: oci://ghcr.io/suse/chart/nvidia-driver-installer:0.1.2
Namespace: kube-system
```

--------------------------------

### Display Cluster Node Information

Source: https://github.com/suse/suse-ai-stack/blob/main/docs/AWS_Setup.md

Example output from the Ansible task showing the hostname and IP address of provisioned cluster nodes.

```console
TASK [vm : Display all cluster nodes mgmt cluster] *************************************************************************************************************************************
Thursday 05 June 2025  12:02:45 -0700 (0:00:00.023)       0:03:24.333 *********
ok: [localhost] => {
   "msg": [
      {
            "hostname": "mgmt-rancher",
            "ip": "52.25.204.43",
            "name": "mgmt-rancher"
      }
   ]
}
.
.
TASK [vm : Display all cluster nodes suse ai cluster] **********************************************************************************************************************************
Thursday 05 June 2025  14:17:24 -0700 (0:00:00.021)       0:00:22.312 ********* 
ok: [localhost] => {
   "msg": [
      {
            "hostname": "suse-ai",
            "ip": "54.245.160.93",
            "name": "suse-ai"
      }
   ]
}
```

--------------------------------

### Configure AWS SSH Public Key

Source: https://github.com/suse/suse-ai-stack/blob/main/docs/AWS_Setup.md

Example format for the aws_ssh_public_key variable in extra_vars.yml.

```console
aws_ssh_public_key: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/PFrEQRjraJTx5WulyLfPHiDf6OO0rLU3atox2xu18suohUjCrLTIuRaSMX6mHAX8wb/wPFd2hlk8oXKwBxUMFOn1sOlXFti0tYbtR+TidlKMB22hehCa2K6ckQg07l9IQOQhcccSprT4jXxKW3H4PzC5tA+LfrbaUE8eHEv1/5vBK51AsYRf2T2vbSjnUHIP3bWoYbVx1fdLPvCQsYVRwnP7bLcoaIkciWVqjDW6/xEfw9GrCZCl5QfCUs5lRT2TqrgalODJmBg3tWLO2Bfgmvr9+V4j1DGHX7TqSiVGjgqhruXjGZC675/jML2TXnAxvXDQIMaSz0KSsQSKpC/p foo@somedomain
```

--------------------------------

### Management Cluster Inventory Format

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/entry-points-and-scripts.md

Example of the generated INI inventory file for the management cluster.

```ini
[rke2_servers]
mgmt-rancher ansible_host=192.168.122.100

[rke2_servers:vars]
ansible_user=ai
ansible_python_interpreter=/usr/bin/python3

[rke2_agents]

[rke2_agents:vars]
ansible_user=ai
ansible_python_interpreter=/usr/bin/python3
```

--------------------------------

### AI Cluster Inventory Format

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/entry-points-and-scripts.md

Example of the generated INI inventory file for the AI cluster, including GPU and non-GPU agent groups.

```ini
[rke2_servers]
ai-rancher ansible_host=192.168.122.101

[rke2_agents_gpu]
ai-worker-gpu ansible_host=192.168.122.102

[rke2_agents_nongpu]
ai-worker-nongpu ansible_host=192.168.122.103

[rke2_agents]
ai-worker-gpu
ai-worker-nongpu
```

--------------------------------

### Define Cluster Nodes in INI Inventory

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/playbooks-reference.md

Example of the INI format used for defining RKE2 server and agent nodes in the dynamic inventory.

```ini
[rke2_servers]
mgmt-rancher ansible_host=192.168.122.100

[rke2_agents_gpu]
ai-worker-gpu-1 ansible_host=192.168.122.101

[rke2_agents_nongpu]
ai-worker-1 ansible_host=192.168.122.102
```

--------------------------------

### Cleanup Kubernetes Resources

Source: https://github.com/suse/suse-ai-stack/blob/main/docs/cncf-ai-conformance/v1.34/SUSE-AI/specs/ai_inference.md

Uninstalls the APISIX Helm release, deletes the APISIX namespace, removes the Gateway API experimental installation, and provides instructions for restarting the rke2-server service after configuration changes.

```bash
helm uninstall apisix -n apisix
kubectl delete namespace apisix
kubectl delete -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.3.0/experimental-install.yaml
# Remove the `ingress-controller: none` setting on the `/etc/rancher/rke2/config.yaml` file and restart the rke2-server service
```

--------------------------------

### Prometheus Adapter Values for GPU Metrics

Source: https://github.com/suse/suse-ai-stack/blob/main/docs/cncf-ai-conformance/v1.34/SUSE-AI/specs/pod_autoscaling.md

Configure the Prometheus Adapter to scrape GPU utilization metrics from VictoriaMetrics. This setup defines the metrics endpoint and the query for GPU utilization.

```yaml
prometheus:
  url: http://suse-observability-victoria-metrics-0
  port: 8428

rules:
  custom:
    - seriesQuery: 'DCGM_FI_DEV_GPU_UTIL{namespace!=""}'
      resources:
        overrides:
          namespace:
            resource: namespace
      name:
        matches: "DCGM_FI_DEV_GPU_UTIL"
        as: "gpu_utilization"
      metricsQuery: 'avg(DCGM_FI_DEV_GPU_UTIL{<<.LabelMatchers>>}) by (<<.GroupBy>>)'
```

--------------------------------

### Ollama Helm Chart Values for Multi-Node Scaling

Source: https://github.com/suse/suse-ai-stack/blob/main/docs/cncf-ai-conformance/v1.34/SUSE-AI/specs/pod_autoscaling.md

Configure Ollama deployment for multi-node scaling with a shared Persistent Volume (PV) to avoid re-downloading models. This setup enables GPU usage and ReadWriteMany access mode.

```yaml
global:
  imagePullSecrets:
  - application-collection
ingress:
  enabled: false
defaultModel: "gemma:2b"
ollama:
  models:
    pull:
      - "gemma:2b"
    run:
      - "gemma:2b"
  gpu:
    enabled: true
    type: 'nvidia'
    number: 1
persistentVolume:
  enabled: true
  storageClass: longhorn
  size: 20Gi
  accessModes:
    - ReadWriteMany
extraEnv:
  - name: OLLAMA_DEBUG
    value: "1"
runtimeClassName: "nvidia"
affinity:
  podAntiAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 100
        podAffinityTerm:
          topologyKey: kubernetes.io/hostname
          labelSelector:
            matchExpressions:
              - key: app.kubernetes.io/name
                operator: In
                values:
                  - ollama
```

--------------------------------

### Invoke setup_nodes.yml Playbook

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/playbooks-reference.md

Executes the setup_nodes.yml playbook using a local inventory file and external variables.

```bash
ansible-playbook playbooks/setup_nodes.yml \
  -i inventories/localhost_inventory.yml \
  -e @extra_vars.yml
```

--------------------------------

### Initialize Path and Environment Variables

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/entry-points-and-scripts.md

Sets up the project directory, updates the PATH, and defines the Ansible roles path.

```bash
PROJECT_DIR=$(dirname -- $(readlink -e -- ${BASH_SOURCE[0]}))
PATH=${PROJECT_DIR}/bin:${PATH}
EXTRA_VARS_FILE=$PROJECT_DIR/extra_vars.yml
export ANSIBLE_ROLES_PATH=${PROJECT_DIR}/roles:${PROJECT_DIR}/external_playbooks/roles
```

--------------------------------

### Install Gateway API CRDs

Source: https://github.com/suse/suse-ai-stack/blob/main/docs/cncf-ai-conformance/v1.34/SUSE-AI/specs/ai_inference.md

Installs the necessary Custom Resource Definitions for the Kubernetes Gateway API. Ensure the version matches your APISIX Ingress Controller requirements.

```bash
kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.3.0/experimental-install.yaml
```

--------------------------------

### Display stack configuration

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/playbooks-reference.md

Executes the display playbook to output Kubernetes, Rancher, and service access information.

```bash
ansible-playbook playbooks/display.yml \
  -i inventories/localhost_inventory.yml \
  -e @extra_vars.yml
```

--------------------------------

### Deploy Observability Platform

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/entry-points-and-scripts.md

Deploys the SUSE Observability platform if enabled.

```bash
playbook="deploy_suse_observability"
ansible-playbook "${PROJECT_DIR}/playbooks/${playbook}.yml" \
  -i inventories/localhost_inventory.yml \
  -i "${PROJECT_DIR}/inventories/${cluster_inv_name}" \
  -e @extra_vars.yml
```

--------------------------------

### Deploy KServe InferenceService

Source: https://github.com/suse/suse-ai-stack/blob/main/docs/cncf-ai-conformance/v1.34/SUSE-AI/specs/robust_controller.md

Applies a YAML configuration file to deploy an InferenceService for a Qwen LLM using KServe.

```bash
kubectl apply -f ./kserve/qwen-kserve.yaml
```

--------------------------------

### Visualize VM Infrastructure Architecture

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/deployment-architecture.md

A diagram representing the local host machine, libvirt network, and the virtual machine structure.

```text
┌──────────────────────────────────────────────────┐
│  Local Host Machine (Developer Workstation)      │
│                                                   │
│  libvirt Network: 192.168.122.0/24                │
│  ┌────────────────────────────────────────────┐  │
│  │ Virtual Machine                            │  │
│  │ IP: 192.168.122.100                        │  │
│  │ CPUs: 8 (configurable)                     │  │
│  │ Memory: 32 GB (configurable)               │  │
│  │ Disk: 50+ GB (configurable)                │  │
│  │                                            │  │
│  │ ┌──────────────────────────────────────┐  │  │
│  │ │ SLE Micro or SLES 15SP6              │  │  │
│  │ ├──────────────────────────────────────┤  │  │
│  │ │ RKE2 Kubernetes Cluster              │  │  │
│  │ │ ├─ Control Plane Node                │  │  │
│  │ │ ├─ Worker Node (GPU optional)        │  │  │
│  │ │ └─ Worker Node (Non-GPU)             │  │  │
│  │ └──────────────────────────────────────┘  │  │
│  └────────────────────────────────────────────┘  │
│                                                   │
│  libvirt Directories:                            │
│  ├─ Cloud Images: libvirt_images/                │
│  ├─ VM Disks: libvirt_images/*-instance.qcow2   │
│  └─ ISO Files: libvirt_images/*-cidata.iso      │
└──────────────────────────────────────────────────┘
```

--------------------------------

### Configure DNS and Storage

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/entry-points-and-scripts.md

Configures external DNS and storage backends for each cluster.

```bash
playbook="setup_dns_storage"
for cluster in "${clusters[@]}"; do
  ansible-playbook "${PROJECT_DIR}/playbooks/${playbook}.yml" \
    -i inventories/localhost_inventory.yml \
    -i "${PROJECT_DIR}/inventories/${cluster}_inventory.ini" \
    -e @extra_vars.yml \
    -e cluster="${cluster}"
done
```

--------------------------------

### Main Configuration File Structure

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/entry-points-and-scripts.md

Template for the user-created extra_vars.yml file containing registration, cloud provider, and cluster settings.

```yaml
# Required variables
registration_email: user@company.com
github_username: octocat
application_collection_user_email: user@suse.okta.com
application_collection_user_token: token123

# Optional variables (with defaults)
open_webui_hostname: suse-ollama-webui
enable_gpu_operator: false
enable_suse_observability: false

# Cloud provider specific
cloud_provider: local  # or aws
aws_region: us-west-2
aws_access_key: AKIA...
aws_secret_key: ...

# Kubernetes cluster configuration
cluster:
  user: ai
  user_home: /home/ai
  root_volume_size: 350
  instance_type_cp: g4dn.2xlarge
  num_cp_nodes: 1
```

--------------------------------

### Deploy AI Stack Components

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/entry-points-and-scripts.md

Deploys AI components including Ollama, Open WebUI, Milvus, OpenSearch, and vLLM.

```bash
playbook="deploy_suse_ai"
ansible-playbook "${PROJECT_DIR}/playbooks/${playbook}.yml" \
  -i inventories/localhost_inventory.yml \
  -i "${PROJECT_DIR}/inventories/${cluster_inv_name}" \
  -e @extra_vars.yml
```

--------------------------------

### Verify KServe CRDs

Source: https://github.com/suse/suse-ai-stack/blob/main/docs/cncf-ai-conformance/v1.34/SUSE-AI/specs/robust_controller.md

Lists all Custom Resource Definitions (CRDs) and filters for those related to KServe, confirming their creation after installation.

```bash
> kubectl get crd | grep -i kserve
clusterservingruntimes.serving.kserve.io                          2026-02-13T14:48:14Z
clusterstoragecontainers.serving.kserve.io                        2026-02-13T14:48:14Z
inferencegraphs.serving.kserve.io                                 2026-02-13T14:48:14Z
inferenceservices.serving.kserve.io                                 2026-02-13T14:48:14Z
localmodelcaches.serving.kserve.io                                2026-02-13T14:48:14Z
localmodelnodegroups.serving.kserve.io                            2026-02-13T14:48:14Z
localmodelnodes.serving.kserve.io                                 2026-02-13T14:48:14Z
servingruntimes.serving.kserve.io                                 2026-02-13T14:48:14Z
trainedmodels.serving.kserve.io                                   2026-02-13T14:48:14Z


```

--------------------------------

### Deploy SUSE Observability via Ansible

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/playbooks-reference.md

Execute the deployment playbook using the specified inventory files and extra variables.

```bash
ansible-playbook playbooks/deploy_suse_observability.yml \
  -i inventories/localhost_inventory.yml \
  -i inventories/suse-observability_inventory.ini \
  -e @extra_vars.yml
```

--------------------------------

### Deploy Component via Helm

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/ansible-roles-reference.md

Pattern for authenticating with a Helm registry and performing an upgrade/install with retry logic.

```yaml
- name: Deploy component via Helm
  shell: |
    if [[ {{ helm_repo }} == *"dp.apps.rancher.io"* ]]; then
        helm registry login dp.apps.rancher.io/charts \
          -u {{ application_collection_user_email }} \
          -p {{ application_collection_user_token }}
    fi
    helm upgrade --install {{ release_name }} {{ helm_repo }} \
      -n {{ namespace }} --create-namespace \
      --version {{ chart_version }} -f values.yaml
  register: helm_deploy_result
  until: helm_deploy_result.rc == 0
  retries: 3
  delay: 20
```

--------------------------------

### Get Ollama CRDs

Source: https://github.com/suse/suse-ai-stack/blob/main/docs/cncf-ai-conformance/v1.34/SUSE-AI/specs/robust_controller.md

Lists all Custom Resource Definitions (CRDs) and filters for those related to Ollama, verifying that the 'models.ollama.ayaka.io' CRD has been created.

```bash
$ kubectl get crd | grep -i ollama
models.ollama.ayaka.io                      2025-10-06T21:57:57Z
```

--------------------------------

### Configure Open WebUI and Pipelines

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/kubernetes-resources.md

ConfigMaps for Open WebUI settings and Python pipeline scripts.

```yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: open-webui-config
  namespace: suse-private-ai
data:
  config.json: |
    {
      "default_model": "neural-chat",
      "model_list": [...]
    }
```

```yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: open-webui-pipelines
  namespace: suse-private-ai
data:
  observability_filter.py: |
    # Python code for observability pipeline
  conversation_limit_filter.py: |
    # Python code for conversation length limiting
```

--------------------------------

### Viewing Deployment Logs

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/entry-points-and-scripts.md

Commands to monitor and search through deployment log files.

```bash
# Live stream
tail -f deployments/deployment_*.log

# Search for errors
grep ERROR deployments/*.log

# Check specific component
grep -i "ollama" deployments/deployment_*.log
```

--------------------------------

### Deploy SUSE AI Stack via Ansible

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/playbooks-reference.md

Execute the deployment playbook using the specified inventory files and extra variables.

```bash
ansible-playbook playbooks/deploy_suse_ai.yml \
  -i inventories/localhost_inventory.yml \
  -i inventories/suse-ai_inventory.ini \
  -e @extra_vars.yml
```

--------------------------------

### Perform Manual Etcd Backup

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/deployment-architecture.md

Use these commands to export cluster resources and create a snapshot of the etcd database.

```bash
kubectl get -A -o yaml all > backup.yaml
kubectl exec -n kube-system -it <etcd-pod> -- \
  etcdctl snapshot save /tmp/etcd-backup.db
```

--------------------------------

### Get Ollama Pods

Source: https://github.com/suse/suse-ai-stack/blob/main/docs/cncf-ai-conformance/v1.34/SUSE-AI/specs/robust_controller.md

Retrieves the status of Ollama-related pods, including the model and model store pods. Useful for verifying deployment status.

```bash
$ kubectl get pods
NAME                               READY   STATUS    RESTARTS   AGE
ollama-model-phi-7cd69569f7-pq9gl   1/1     Running   0          153m
ollama-models-store-0               1/1     Running   0          155m
```

--------------------------------

### Storage Backend Selection Logic

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/playbooks-reference.md

Jinja2 template logic used to determine the storage class based on the cloud provider and Longhorn enablement.

```yaml
storage_class: "{% if cloud_provider == 'aws' and enable_longhorn %}longhorn{% else %}local-path{% endif %}"
```

--------------------------------

### Deployment Log Directory Structure

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/entry-points-and-scripts.md

The file structure for logs generated during deployment.

```bash
deployments/
├── deployment_<timestamp>.log
├── rollout_status.log
├── ingress_output.log
└── cluster_summary.log
```

--------------------------------

### Configure Libvirt Images Directory

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/entry-points-and-scripts.md

Sets the libvirt images directory if it is not already defined.

```bash
if [ ! -d ${LIBVIRT_IMAGES_DIR} ]; then
  LIBVIRT_IMAGES_DIR=${PROJECT_DIR}/libvirt_images
fi
```

--------------------------------

### Get API Gateway Port and Test Inference

Source: https://github.com/suse/suse-ai-stack/blob/main/docs/cncf-ai-conformance/v1.34/SUSE-AI/specs/ai_inference.md

Retrieves the NodePort for the APISIX gateway service and then sends a POST request to the /v1/chat/completions endpoint to test AI inference with a specified model and message.

```bash
PORT=$(kubectl get svc -n apisix --selector app.kubernetes.io/service=apisix-gateway -o jsonpath='{.items[0].spec.ports[0].nodePort}')
curl --location -X POST "http://localhost:${PORT}/v1/chat/completions" -H "Content-Type: application/json" -H "OpenAI-Model: gemma:2b" -d '{ "model": "gemma:2b", "messages": [{ "role": "system", "content": "You are a helpful assistant."}, {"role": "user","content": "Hello!"}]}'
```

--------------------------------

### Deployment Sequence Workflow

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/README.md

The order of execution for Ansible playbooks required to deploy the SUSE AI Stack.

```text
1. setup_nodes.yml
   └─ Create infrastructure (VMs or EC2)
   └─ Define cluster nodes in inventory

2. deploy_rke2_rancher.yml (per cluster)
   └─ Install RKE2 Kubernetes
   └─ Deploy Rancher Prime (mgmt cluster)

3. deploy_nvidia_driver_installer.yml
   └─ Install GPU drivers (if GPU available)

4. setup_dns_storage.yml (per cluster)
   └─ Configure DNS (optional)
   └─ Deploy storage backend

5. deploy_suse_observability.yml
   └─ Deploy observability platform (if enabled)

6. deploy_suse_ai.yml
   └─ Deploy all AI components

7. display.yml
   └─ Output access information
```

--------------------------------

### Deploy Ollama using Helm

Source: https://github.com/suse/suse-ai-stack/blob/main/docs/cncf-ai-conformance/v1.34/SUSE-AI/specs/pod_autoscaling.md

Deploy or upgrade the Ollama Helm chart to your SUSE AI environment using the specified version and custom values file.

```bash
helm upgrade -i ollama oci://dp.apps.rancher.io/charts/ollama -n suse-private-ai --version 1.26.0 -f ollama-values.yaml
```

--------------------------------

### Display Access Information

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/entry-points-and-scripts.md

Runs the display playbook to output access details for deployed components.

```bash
playbook="display"
ansible-playbook "${PROJECT_DIR}/playbooks/${playbook}.yml" \
  -i inventories/localhost_inventory.yml \
  -e @extra_vars.yml
```

--------------------------------

### Create Kubernetes Secrets for OpenTelemetry

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/ansible-roles-reference.md

Commands to initialize the observability namespace and create the necessary API key and registry credentials secrets.

```bash
kubectl create namespace observability
kubectl create secret generic open-telemetry-collector \
  --from-literal=API_KEY='<suse_observability_api_key>' \
  -n observability
kubectl create secret docker-registry suse-ai-registry \
  --docker-server=registry.suse.com \
  --docker-username=regcode \
  --docker-password='<suse_ai_registration_code>' \
  -n observability
```

--------------------------------

### Execute Individual Playbooks

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/playbooks-reference.md

Commands for running specific deployment playbooks with custom inventory and extra variables.

```bash
# Just deploy AI without observability
ansible-playbook playbooks/deploy_suse_ai.yml \
  -i inventories/suse-ai_inventory.ini \
  -e @suse-ai_extra_vars.yml

# Redeploy GPU drivers
ansible-playbook playbooks/deploy_nvidia_driver_installer.yml \
  -i inventories/suse-ai_inventory.ini \
  -e @extra_vars.yml
```

--------------------------------

### Deploy Open WebUI

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/kubernetes-resources.md

Defines the deployment configuration for the Open WebUI interface.

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: open-webui
  namespace: suse-private-ai
spec:
  replicas: 1
  strategy:
    type: Recreate
```

--------------------------------

### Set up Ingress Gateway Port Forwarding

Source: https://github.com/suse/suse-ai-stack/blob/main/docs/cncf-ai-conformance/v1.34/SUSE-AI/specs/robust_controller.md

Establishes a port-forwarding connection to the Istio ingress gateway service. This is necessary to access services deployed within the cluster for testing purposes.

```bash
> INGRESS_GATEWAY_SERVICE=$(kubectl get svc --namespace istio-system --selector="app=istio-ingressgateway" --output jsonpath='{.items[0].metadata.name}')
> kubectl port-forward --namespace istio-system svc/${INGRESS_GATEWAY_SERVICE} 8080:80
```

--------------------------------

### Wait for Rollout

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/ansible-roles-reference.md

Pattern for monitoring deployment rollout status with retries until success.

```yaml
- name: Wait for rollout
  shell: >
    kubectl -n {{ namespace }} rollout status deployment/{{ name }} \
      --timeout=120s
  register: rollout_result
  retries: 6
  delay: 30
  until: "'successfully rolled out' in rollout_result.stdout"
```

--------------------------------

### Invoke the main deployment script

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/entry-points-and-scripts.md

Execute the orchestration script with optional Ansible arguments.

```bash
./setup_private_ai_stack.sh [ANSIBLE_OPTIONS]
```

--------------------------------

### Accessing Libvirt VM via SSH

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/entry-points-and-scripts.md

Command to access a local libvirt-based VM.

```bash
ssh -i ~/.ssh/id_rsa ai@192.168.122.100
```

--------------------------------

### Perform Inference Request

Source: https://github.com/suse/suse-ai-stack/blob/main/docs/cncf-ai-conformance/v1.34/SUSE-AI/specs/robust_controller.md

Sends a sample inference request to a deployed AI model (e.g., qwen-llm) via the ingress gateway. This verifies the model's availability and the correctness of the inference pipeline.

```bash
> export INGRESS_HOST=localhost
> export INGRESS_PORT=8080
> export SERVICE_HOSTNAME=$(kubectl get inferenceservice qwen-llm -n default -o jsonpath='{.status.url}' | cut -d "/" -f 3)
> export MODEL_NAME=qwen

> curl -v http://${INGRESS_HOST}:${INGRESS_PORT}/openai/v1/completions -H "Host: ${SERVICE_HOSTNAME}" -H "Content-Type: application/json" -d '{
  "model": "'"${MODEL_NAME}"'",
  "prompt": "Write a poem about colors",
  "max_tokens": 100,
  "stream": false
}'
```

--------------------------------

### Debug and Dry-Run Playbook Execution

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/playbooks-reference.md

Commands for troubleshooting deployments using verbose output, fact caching, or check mode.

```bash
ansible-playbook playbooks/deploy_suse_ai.yml \
  -i inventories/suse-ai_inventory.ini \
  -e @extra_vars.yml \
  -vvv

# Enable fact caching
ANSIBLE_CACHE_PLUGIN=jsonfile \
ANSIBLE_CACHE_PLUGIN_CONNECTION=/tmp/ansible_facts \
ansible-playbook playbooks/deploy_suse_ai.yml ...

# Dry-run (check mode)
ansible-playbook playbooks/deploy_suse_ai.yml \
  --check
```

--------------------------------

### Verify Weighted Load Balancing

Source: https://github.com/suse/suse-ai-stack/blob/main/docs/cncf-ai-conformance/v1.34/SUSE-AI/specs/ai_inference.md

Tests the weighted traffic splitting by sending a POST request to the inference service endpoint and observing the distribution of traffic.

```bash
# Get the proper port
PORT=$(kubectl get svc -n apisix --selector app.kubernetes.io/service=apisix-gateway -o jsonpath='{.items[0].spec.ports[0].nodePort}')
curl --location -X POST "http://localhost:${PORT}/v1/chat/completions" -H "Host: inference.service.com" -d '{ "model": "gemma:2b", "messages": [{ "role": "system", "content": "You are a helpful assistant."}, {"role": "user","content": "Hello!"}]}'
```

--------------------------------

### Deploy SUSE Observability

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/README.md

Executes the Ansible playbook to deploy observability components to the cluster.

```bash
ansible-playbook playbooks/deploy_suse_observability.yml \
  -i inventories/suse-ai_inventory.ini \
  -e extra_vars.yml \
  -e enable_suse_observability=true
```

--------------------------------

### Apply GPU Blocker Deployment

Source: https://github.com/suse/suse-ai-stack/blob/main/docs/cncf-ai-conformance/v1.34/SUSE-AI/specs/gang_scheduling.md

Deploys a set of pods designed to consume available GPU resources, simulating a scenario where insufficient GPUs are available for other workloads. The number of replicas should be adjusted based on the cluster's GPU count.

```bash
kubectl apply -f ./volcano/gpu-blocker.yaml
```

--------------------------------

### Create Namespace and Registry Secret

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/ansible-roles-reference.md

Pattern for initializing a Kubernetes namespace and creating a docker-registry secret.

```yaml
- name: Create namespace
  shell: kubectl create namespace {{ namespace }}

- name: Create registry secret
  shell: |
    kubectl create secret docker-registry {{ secret_name }} \
      --docker-server={{ registry }} \
      --docker-username={{ username }} \
      --docker-password='{{ password }}' \
      -n {{ namespace }}
```

--------------------------------

### Define Global Host Variables

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/playbooks-reference.md

Configuration for common host variables applied to the inventory.

```ini
[hosts:vars]
ansible_user=ai
ansible_python_interpreter=/usr/bin/python3
```

--------------------------------

### Configure Ingress for API and UI

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/kubernetes-resources.md

Sets up NGINX ingress rules to route traffic to the observability router service.

```yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: suse-observability
  namespace: suse-observability
spec:
  ingressClassName: nginx
  tls:
  - hosts:
    - suse-observability
      secretName: suse-observability-tls
  rules:
  - host: suse-observability
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: suse-observability-router
            port:
              number: 8080
```

--------------------------------

### Deployment Task Execution Flow

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/ansible-roles-reference.md

Represents the sequential order of task execution for the suse-private-ai role.

```text
selinux → namespace → appcosecret → {milvus, opensearch, ollama} → {vllm if GPU} → {open-webui-mcpo} → open-webui
```

--------------------------------

### Configure Root Volume Settings

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/deployment-architecture.md

Storage configuration for the root volume of the nodes.

```yaml
root_volume_size: 350  # GB per node
volume_type: gp3
encrypted: true
delete_on_termination: true
```

--------------------------------

### Deploy Prometheus Adapter using Helm

Source: https://github.com/suse/suse-ai-stack/blob/main/docs/cncf-ai-conformance/v1.34/SUSE-AI/specs/pod_autoscaling.md

Deploy or upgrade the Prometheus Adapter Helm chart to expose GPU metrics to the Horizontal Pod Autoscaler (HPA) from SUSE Observability.

```bash
helm upgrade -i prom-adapter oci://ghcr.io/prometheus-community/charts/prometheus-adapter -n suse-observability -f prom-adapter-values.yaml
```

--------------------------------

### Deploy AI Observability Extension

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/kubernetes-resources.md

Deployment manifest for the AI-specific dashboard and monitoring extension.

```yaml
apiVersion: v1
kind: Deployment
metadata:
  name: suse-ai-observability-extension
  namespace: suse-observability
```

--------------------------------

### Local /etc/hosts Configuration

Source: https://github.com/suse/suse-ai-stack/blob/main/docs/AWS_Setup.md

The required format for the local /etc/hosts file to enable DNS resolution for the private stack services.

```console
  35.87.100.210 mgmt-rancher suse-rancher.demo
  35.87.100.210 suse-ai suse-ollama-webui
  35.87.100.210 suse-observability
```

--------------------------------

### Deploy Milvus Vector Database

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/kubernetes-resources.md

Configuration for Milvus standalone and cluster deployments.

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: milvus-standalone
  namespace: suse-private-ai
```

```yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: milvus-datanode
  namespace: suse-private-ai
---
kind: StatefulSet
metadata:
  name: milvus-indexnode
  namespace: suse-private-ai
---
kind: StatefulSet
metadata:
  name: milvus-querynode
  namespace: suse-private-ai
```

--------------------------------

### Verify AWS Instance Type Offerings

Source: https://github.com/suse/suse-ai-stack/blob/main/docs/AWS_Setup.md

Command to check if specific instance types are available in a given AWS region.

```bash
aws ec2 describe-instance-type-offerings --region <REGION>
```

--------------------------------

### Observe Pending Distributed Job Pods with Insufficient Resources

Source: https://github.com/suse/suse-ai-stack/blob/main/docs/cncf-ai-conformance/v1.34/SUSE-AI/specs/gang_scheduling.md

After applying the GPU blocker, this command monitors the distributed job's pods. It demonstrates the 'all-or-nothing' gang scheduling behavior, where pods remain in 'Pending' if all required resources are not available.

```bash
> kubectl get pods -l volcano.sh/job-name=distributed-job -w
NAME                            READY   STATUS    RESTARTS   AGE
distributed-job-coordinator-0   0/1     Pending   0          9s
distributed-job-worker-0        0/1     Pending   0          9s
distributed-job-worker-1        0/1     Pending   0          9s

```

--------------------------------

### Ansible Configuration Settings

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/entry-points-and-scripts.md

Defines default Ansible behavior including inventory paths, roles, and privilege escalation settings.

```ini
[defaults]
inventory = inventories/localhost_inventory.yml
roles_path = roles
host_key_checking = False
remote_user = root
timeout = 120

[privilege_escalation]
become = False
become_method = sudo
```

--------------------------------

### Terraform State and Variable Files

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/entry-points-and-scripts.md

Lists the standard files generated and used by Terraform for infrastructure state management.

```bash
terraform.tfstate      # Current state
terraform.tfstate.backup  # Previous state backup
terraform.tfvars       # Variable definitions
```

--------------------------------

### Define Liveness and Readiness Probes

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/deployment-architecture.md

Configures HTTP health checks to monitor service availability and readiness.

```yaml
livenessProbe:
  httpGet:
    path: /health
    port: 8080
  initialDelaySeconds: 30
  periodSeconds: 10

readinessProbe:
  httpGet:
    path: /ready
    port: 8080
  initialDelaySeconds: 5
  periodSeconds: 5
```

--------------------------------

### Pass variables via inventory files

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/playbooks-reference.md

Use the -i flag to specify inventory files in YAML or INI format.

```bash
-i inventories/localhost_inventory.yml
-i inventories/suse-ai_inventory.ini
```

--------------------------------

### Manage Kubeconfig

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/entry-points-and-scripts.md

Commands for exporting, merging, and specifying custom kubeconfig files for cluster access.

```bash
# Export kubeconfig for sharing
kubectl config view --raw > kubeconfig_suse_ai.yaml

# Set KUBECONFIG environment variable
export KUBECONFIG=$HOME/.kube/config:/path/to/other/config

# Use specific kubeconfig
kubectl --kubeconfig=/path/to/config get nodes
```

--------------------------------

### Deploy Open WebUI MCPO

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/kubernetes-resources.md

Defines the deployment configuration for the Multi-Chat Progressive OpenWebUI extension.

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: open-webui-mcpo
  namespace: suse-private-ai
spec:
  replicas: 1
```

--------------------------------

### Configure SSH Authorized Keys

Source: https://github.com/suse/suse-ai-stack/blob/main/docs/Local_Setup.md

Add your public SSH key to the extra_vars.yml file to enable passwordless access to the VM.

```console
ssh_authorized_keys:
  - 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/PFrEQRjraJTx5WulyLfPHiDf6OO0rLU3atox2xu18suohUjCrLTIuRaSMX6mHAX8wb/wPFd2hlk8oXKwBxUMFOn1sOlXFti0tYbtR+TidlKMB22hehCa2K6ckQg07l9IQOQhcccSprT4jXxKW3H4PzC5tA+LfrbaUE8eHEv1/5vBK51AsYRf2T2vbSjnUHIP3bWoYbVx1fdLPvCQsYVRwnP7bLcoaIkciWVqjDW6/xEfw9GrCZCl5QfCUs5lRT2TqrgalODJmBg3tWLO2Bfgmvr9+V4j1DGHX7TqSiVGjgqhruXjGZC675/jML2TXnAxvXDQIMaSz0KSsQSKpC/p foo@somedomain'
```

--------------------------------

### Configure static IP in DHCP

Source: https://github.com/suse/suse-ai-stack/blob/main/docs/Local_Setup.md

Add a host entry to the libvirt network configuration to assign a static IP to a specific MAC address.

```xml
<host mac='52:54:00:6C:3C:88' ip='192.168.122.100'/>
```

--------------------------------

### Deploy OpenSearch Cluster

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/kubernetes-resources.md

Configuration for OpenSearch standalone and cluster deployments.

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: opensearch-cluster-master
  namespace: suse-private-ai
```

```yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: opensearch-cluster-master
  namespace: suse-private-ai
---
kind: StatefulSet
metadata:
  name: opensearch-cluster-data
  namespace: suse-private-ai
---
kind: StatefulSet
metadata:
  name: opensearch-cluster-coordinating
  namespace: suse-private-ai
```

--------------------------------

### Managing AWS SSH Key Pairs

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/entry-points-and-scripts.md

Commands to create and configure SSH key pairs for AWS deployments.

```bash
# Create new key pair
aws ec2 create-key-pair --key-name suse-ai-stack-key \
  --output text > ~/.ssh/suse-ai-stack-key.pem
chmod 400 ~/.ssh/suse-ai-stack-key.pem

# Use in configuration
aws_ssh_key_name: suse-ai-stack-key
aws_ssh_public_key: $(cat ~/.ssh/suse-ai-stack-key.pub)
```

--------------------------------

### Invoke RKE2 Rancher Playbook

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/playbooks-reference.md

Standard command to execute the deployment playbook with specific inventory and extra variables.

```bash
ansible-playbook playbooks/deploy_rke2_rancher.yml \
  -i inventories/mgmt_inventory.ini \
  -e @extra_vars.yml \
  -e cluster=mgmt
```

--------------------------------

### Ansible Configuration Variable Hierarchy

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/README.md

Visual representation of the precedence order for configuration variables in the SUSE AI Stack.

```text
extra_vars.yml (User-supplied)
    ↓ (overrides)
Role defaults (roles/*/defaults/main.yml)
    ↓ (overrides)
Playbook variables
    ↓ (overrides)
Ansible command-line (-e flags)
```

--------------------------------

### Accessing Kubeconfig Contexts

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/entry-points-and-scripts.md

Commands to interact with cluster contexts using kubectl.

```bash
# All clusters merged into single file
kubectl get nodes

# Specific cluster context
kubectl config use-context suse-ai
kubectl get nodes

# View all contexts
kubectl config get-contexts
```

--------------------------------

### Default Kubeconfig Location

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/entry-points-and-scripts.md

The standard file path for the kubeconfig file.

```bash
~/.kube/config
```
Tokens:
Raw