### Install KServe Quickstart Environment

Source: https://github.com/suse/suse-ai-stack/blob/main/docs/cncf-ai-conformance/v1.34/SUSE-AI/specs/robust_controller.md

Executes a script to set up the KServe quickstart environment. This script is a modified version of the official KServe quickstart script.

```bash
./kserve/quickstart.sh
```

--------------------------------

### Copy Example Configuration Files

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/configuration.md

Copy the appropriate example configuration file based on your deployment environment (local or AWS).

```bash
cp extra_vars.yml.local.example extra_vars.yml   # For local deployment
cp extra_vars.yml.aws.example extra_vars.yml     # For AWS deployment
```

--------------------------------

### Local Single-Node Deployment

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/setup.md

Use this snippet for a basic local deployment on a single node. Ensure to copy and edit the example variables file before running the setup script.

```bash
cp extra_vars.yml.local.example extra_vars.yml
# Edit extra_vars.yml with required settings
./setup_private_ai_stack.sh
```

--------------------------------

### RKE2 Server Node Initialization

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/cluster-architecture.md

Steps for initializing the first RKE2 server node, including downloading the installer, running the installation, and generating a bootstrap token.

```text
Target: rke2_servers[0] (first control plane node)

1. Download RKE2 installer
2. Run RKE2 server installation
   - Creates kubeconfig
   - Initializes etcd
   - Starts API server, controller, scheduler
3. Wait for node ready
4. Extract kubeconfig
5. Generate bootstrap token for subsequent nodes
```

--------------------------------

### Troubleshoot Pod Startup Issues

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/cluster-architecture.md

When a pod is not starting, use `kubectl describe pod` to get detailed information about the pod's status, events, and potential reasons for failure. Use `kubectl logs` to view the logs of the pod for further debugging.

```bash
kubectl describe pod <pod> -n <namespace>
kubectl logs <pod> -n <namespace>
```

--------------------------------

### Create Local Configuration File

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/setup.md

Copy the example local configuration file to create your `extra_vars.yml` for a local setup. This file is used to override default Ansible variables.

```bash
cp extra_vars.yml.local.example extra_vars.yml
```

--------------------------------

### Run Private AI Stack Setup Script

Source: https://github.com/suse/suse-ai-stack/blob/main/docs/Local_Setup.md

Execute the main setup script to deploy the virtualized private AI stack. Ensure `extra_vars.yml` is correctly configured beforehand.

```bash
./setup_private_ai_stack.sh
```

--------------------------------

### Install mkisofs Package

Source: https://github.com/suse/suse-ai-stack/blob/main/docs/Local_Setup.md

Installs the 'mkisofs' utility, which may be part of the 'genisoimage' package on some distributions. This is a prerequisite for setting up the AI stack.

```console
sudo zypper install mkisofs
```

--------------------------------

### Create AWS Configuration File

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/setup.md

Copy the example AWS configuration file to create your `extra_vars.yml` for an AWS setup. This file is used to override default Ansible variables.

```bash
cp extra_vars.yml.aws.example extra_vars.yml
```

--------------------------------

### Dynamic Inventory Generation Examples

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/configuration.md

Examples of dynamically generated INI inventory files for RKE2 servers and agents, used for management, SUSE AI, and SUSE Observability clusters.

```ini
# inventories/mgmt_inventory.ini
[rke2_servers]
mgmt-server1 ansible_host=<ip>

[rke2_agents]
mgmt-agent1 ansible_host=<ip>

# inventories/suse-ai_inventory.ini
[rke2_servers]
suse-ai-server1 ansible_host=<ip>

# inventories/suse-observability_inventory.ini
[rke2_servers]
suse-obs-server1 ansible_host=<ip>

```

--------------------------------

### Install Volcano

Source: https://github.com/suse/suse-ai-stack/blob/main/docs/cncf-ai-conformance/v1.34/SUSE-AI/specs/gang_scheduling.md

Installs the Volcano gang scheduling system into the 'volcano-system' namespace. Ensure the namespace exists or is created.

```bash
helm install volcano volcano-sh/volcano -n volcano-system --create-namespace
```

--------------------------------

### Example libvirt Network Definition with Static IP

Source: https://github.com/suse/suse-ai-stack/blob/main/docs/Local_Setup.md

An example of a libvirt network definition file after adding a static IP configuration for a specific MAC address. This ensures consistent IP assignment for virtual machines.

```xml
<network>
  <name>default</name>
  <uuid>314e5390-370b-4d2c-a0e7-57b220b43754</uuid>
  <forward mode='nat'/>
  <bridge name='virbr0' stp='on' delay='0'/>
  <mac address='52:54:00:49:f7:9e'/>
  <ip address='192.168.122.1' netmask='255.255.255.0'>
    <dhcp>
      <range start='192.168.122.2' end='192.168.122.254'/>
      <host mac='52:54:00:6C:3C:88' ip='192.168.122.100'/>
    </dhcp>
  </ip>
</network>
```

--------------------------------

### Role Dependencies Example

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/roles.md

Illustrates the dependency chain for the `suse-private-ai` role and the `deploy_suse_ai.yml` playbook. Understanding these dependencies is crucial for correct deployment order.

```text
suse-private-ai
  ├─ requires → namespace creation
  ├─ requires → application-collection secret
  ├─ can use → milvus (if enabled)
  └─ can use → opensearch (if enabled)

deploy_suse_ai.yml
  ├─ requires → setup_nodes.yml (cluster exists)
  ├─ requires → deploy_rke2_rancher.yml (K8s running)
  ├─ requires → setup_dns_storage.yml (storage available)
  └─ optionally uses → suse-observability-agent
```

--------------------------------

### Create Admin User for Open WebUI

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/api-reference.md

Sign up for a new user account in Open WebUI. This example demonstrates creating an admin user with specified credentials.

```bash
curl -X POST https://open-webui-host/api/v1/auths/signup \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Admin",
    "email": "admin@example.com",
    "password": "password123"
  }'
```

--------------------------------

### Enable Debug Mode for Setup Script

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/setup.md

Activates debug mode for the setup script, which echoes all commands without executing them. Useful for verifying the script's intended actions.

```bash
DEBUG=1 ./setup_private_ai_stack.sh
```

--------------------------------

### List All Deployments

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/operations.md

Get a list of all deployments across all namespaces in the cluster.

```bash
# List all deployments
kubectl get deployments -A
```

--------------------------------

### AI Stack Only Deployment (Skip Observability)

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/setup.md

Run the setup script to deploy only the AI Stack components, skipping the observability features. This is useful if observability is not required.

```bash
./setup_private_ai_stack.sh --skip-tags observability
```

--------------------------------

### Internal Service Discovery Example

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/cluster-architecture.md

Illustrates how to access internal services within the Kubernetes cluster using their fully qualified domain names.

```text
Ollama: http://ollama.suse-private-ai.svc.cluster.local:80
Milvus: milvus.suse-private-ai.svc.cluster.local:19530
OpenSearch: opensearch.suse-private-ai.svc.cluster.local:9200
```

--------------------------------

### Cluster Import Role Example

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/cluster-architecture.md

Ansible role example for importing a cluster into Rancher, demonstrating the use of 'rancher-token' and 'import-cluster' roles.

```yaml
# In deploy_suse_ai.yml

- name: Get rancher token
  import_role:
    name: rancher-token
  vars:
    rancher_host: "{{ rancher_host }}"
    rancher_bootstrap_password: "rancher"

- name: Import suse_ai_cluster
  import_role:
    name: import-cluster
  vars:
    rancher_url: "https://{{ rancher_host }}"
    rancher_token: "{{ hostvars['mgmt-rancher']['rancher_token'] }}"
    cluster_name: suse-ai
```

--------------------------------

### Kubeflow Application with Kustomization Structure

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/fleet-applications.md

Shows an example of an application directory structure using Kustomization for Kubeflow, including base and overlay directories.

```text
fleet/applications/kubeflow/
├── fleet.yaml
├── kustomization.yaml
├── base/
│   ├── kustomization.yaml
│   └── manifests/
└── overlays/
    ├── prod/
    └── dev/
```

--------------------------------

### Example /etc/hosts File Update

Source: https://github.com/suse/suse-ai-stack/blob/main/docs/AWS_Setup.md

Illustrative content for updating the local '/etc/hosts' file with public IPs and hostnames for accessing various SUSE AI Stack services.

```text
"Make sure to update the /etc/hosts file: 35.87.100.210 mgmt-rancher suse-rancher.demo",
"Make sure to update the /etc/hosts file: 35.87.100.210 suse-ai suse-ollama-webui",
"Make sure to update the /etc/hosts file: 35.87.100.210 suse-observability",
"To access rancher UI, point your browser to https://suse-rancher.demo and login with user=admin and password=rancher",
"To access open-webui, point your browser to https://suse-ollama-webui and login with user=admin@suse-private-ai.org and password=WelcomeToAI",
"To access suse-observability, point your browser to https://suse-observability and login with user=admin and password=MldDMgj2eRHPUXMT"
```

--------------------------------

### Install Ollama Operator

Source: https://github.com/suse/suse-ai-stack/blob/main/docs/cncf-ai-conformance/v1.34/SUSE-AI/specs/robust_controller.md

Installs the Ollama community operator using kubectl. Ensure server-side apply is enabled.

```bash
kubectl apply \
  --server-side=true \
  -f https://raw.githubusercontent.com/nekomeowww/ollama-operator/v0.10.1/dist/install.yaml
```

--------------------------------

### Add Volcano Helm Repository and Update

Source: https://github.com/suse/suse-ai-stack/blob/main/docs/cncf-ai-conformance/v1.34/SUSE-AI/specs/gang_scheduling.md

Installs the Volcano Helm chart repository and updates the local Helm chart repository cache. This is a prerequisite for installing Volcano.

```bash
helm repo add volcano-sh https://volcano-sh.github.io/helm-charts
helm repo update
```

--------------------------------

### Kubernetes Readiness Probe Configuration

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/api-reference.md

Example configuration for a Kubernetes Readiness Probe, used to determine when a container is ready to serve traffic.

```yaml
readinessProbe:
  httpGet:
    path: /ready
    port: 80
  initialDelaySeconds: 10
  periodSeconds: 5
  timeoutSeconds: 3
  failureThreshold: 2
```

--------------------------------

### Install APISIX Gateway with Gateway API Enabled

Source: https://github.com/suse/suse-ai-stack/blob/main/docs/cncf-ai-conformance/v1.34/SUSE-AI/specs/ai_inference.md

Deploys the APISIX Ingress Controller using Helm, enabling Gateway API support and configuring necessary parameters for integration.

```bash
helm upgrade -i apisix oci://dp.apps.rancher.io/charts/apache-apisix --version 2.11.3 --create-namespace  --namespace apisix \
    --set global.imagePullSecrets={application-collection} --set apisix.admin.enabled=true --set apisix.admin.type=NodePort \
    --set apisix.admin.allow.ipList={0.0.0.0/0} --set ingress-controller.enabled=true --set ingress-controller.config.apisix.serviceNamespace=apisix \
    --set ingress-controller.config.apisix.adminAPIVersion="v3" --set ingress-controller.config.kubernetes.enableGatewayAPI=true
```

--------------------------------

### MinIO S3-compatible API Client

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/api-reference.md

Python example using the MinIO SDK to interact with an S3-compatible object storage. Demonstrates client initialization, listing buckets, uploading, and downloading objects.

```python
from minio import Minio

client = Minio(
    "milvus-minio:9000",
    access_key="minioadmin",
    secret_key="minioadmin",
    secure=False
)

# List buckets
buckets = client.list_buckets()

# Upload object
client.fput_object(
    "mybucket",
    "myobject.txt",
    "/path/to/file.txt"
)

# Download object
client.fget_object(
    "mybucket",
    "myobject.txt",
    "/local/path/file.txt"
)
```

--------------------------------

### Local Single-Node Setup

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/configuration.md

Configuration for a single-node local deployment of SUSE AI Stack. Includes registration details, user credentials, SSH keys, and component enablement.

```yaml
cloud_provider: local

registration_email: admin@company.com
sle_micro_registration_code: <code>
sles_registration_code: <code>
suse_ai_registration_code: <code>

github_username: myusername
application_collection_user_email: user@suse.com
application_collection_user_token: <token>

vm_authorized_ssh_keys:
  - "ssh-rsa AAAA... user@workstation"

# Optional GPU
enable_gpu_operator: false

# Default components enabled
enable_milvus: true
enable_opensearch: true
enable_vllm_deployment: false
enable_suse_observability: false

# Simple TLS
tls_source: suse-private-ai

pipelines_enabled: true

```

--------------------------------

### Deploy Milvus Helm Chart with Dependencies

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/fleet-applications.md

Example of deploying the Milvus Helm chart using Fleet. Fleet automatically manages its dependencies like MinIO and etcd.

```yaml
helm:
  chart: oci://dp.apps.rancher.io/charts/milvus
  releaseName: milvus
  # Dependencies automatically managed
```

--------------------------------

### Local Multi-Node Deployment with GPU

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/setup.md

Deploy the AI Stack on a local multi-node setup with GPU support. Requires setting GPU-specific variables in the extra_vars.yml file.

```bash
cp extra_vars.yml.local.example extra_vars.yml
# Edit extra_vars.yml:
# - Set enable_gpu_operator: true
# - Set host_device: <gpu-device-id>
./setup_private_ai_stack.sh
```

--------------------------------

### Run Specific Ansible Tags

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/playbooks.md

Execute tasks tagged with 'observability' using the setup script. This allows for targeted playbook runs.

```bash
./setup_private_ai_stack.sh --tags observability
```

--------------------------------

### Minimal Required Configuration

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/README.md

This YAML snippet outlines the essential configuration parameters required for a minimal setup of the SUSE AI Stack. Ensure all listed fields are populated with valid values.

```yaml
registration_email: user@example.com
sle_micro_registration_code: XXXX-XXXX-XXXX
sles_registration_code: XXXX-XXXX-XXXX
suse_ai_registration_code: XXXX-XXXX-XXXX
github_username: myusername
application_collection_user_email: user@suse.com
application_collection_user_token: token_value
vm_authorized_ssh_keys:
  - "ssh-rsa AAAA... user@host"
```

--------------------------------

### Rancher Installation on Management Cluster

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/cluster-architecture.md

Procedure for deploying Rancher on the management cluster's first server node using Helm.

```text
Target: rke2_servers[0] in mgmt cluster

1. Deploy Rancher via Helm
2. Wait for Rancher deployment ready
3. Configure Rancher with:
   - Bootstrap password
   - Feature flags
   - License (if using Prime)
4. Create API tokens for cluster operations
```

--------------------------------

### Rancher Fleet GitOps Configuration

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/api-reference.md

Example configuration for Rancher Fleet to trigger GitOps synchronization based on repository changes.

```yaml
spec:
  repo: https://github.com/SUSE/suse-ai-stack
  branch: main
  forceSyncGeneration: 1  # Increment to force sync
```

--------------------------------

### SUSE Observability API Token Usage

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/api-reference.md

Example of how to include the SUSE Observability API token in request headers for authentication.

```bash
# Include token in request headers
curl -H "Authorization: Bearer <token>" \
  http://suse-observability-host/api/endpoint
```

--------------------------------

### Get Pod Resource Metrics

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/operations.md

Monitor CPU and memory consumption for pods across different namespaces.

```bash
# CPU and memory per pod
kubectl top pods -n suse-private-ai
kubectl top pods -A
```

--------------------------------

### Environment Paths for Project Setup

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/setup.md

Defines essential environment variables for locating project directories and Ansible roles. Ensure these paths are correctly set before execution.

```bash
PROJECT_DIR=$(dirname -- $(readlink -e -- ${BASH_SOURCE[0]}))
EXTRA_VARS_FILE=$PROJECT_DIR/extra_vars.yml
ANSIBLE_ROLES_PATH=${PROJECT_DIR}/roles:${PROJECT_DIR}/external_playbooks/roles
```

--------------------------------

### Describe Kubernetes Resource

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/api-reference.md

Get detailed information about a specific Kubernetes resource, such as a node. Replace <node> with the resource name.

```bash
# Describe resource
kubectl describe node <node>
```

--------------------------------

### GPU-Enabled Setup

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/configuration.md

Configuration for a local deployment with GPU support enabled. Specifies GPU operator, time slicing, host device, and vLLM deployment.

```yaml
cloud_provider: local

enable_gpu_operator: true
enable_time_slicing: false
host_device: "04:00.0"

enable_vllm_deployment: true
enable_milvus: true

# Recommend larger instance for GPU workloads
vm_memory: 65536
vm_vcpus: 16

pipelines_enabled: true

```

--------------------------------

### Get Kubernetes Events for Troubleshooting

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/operations.md

Retrieve events within a namespace to help diagnose pod or cluster issues.

```bash
# Get events
kubectl get events -n <namespace>
```

--------------------------------

### Kubernetes Pod Environment Variables Example

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/api-reference.md

YAML snippet demonstrating how to set environment variables for a Kubernetes pod, often configured within Helm chart values.

```yaml
extraEnv:
  - name: OLLAMA_DEBUG
    value: "1"
  - name: OLLAMA_HOST
    value: "0.0.0.0:80"
  - name: OLLAMA_NUM_PARALLEL
    value: "4"
```

--------------------------------

### Run SUSE Private AI Stack Setup

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/setup.md

Execute the main orchestration script to deploy the entire SUSE Private AI Stack. This script runs several Ansible playbooks in sequence. It accepts standard Ansible options.

```bash
./setup_private_ai_stack.sh [Ansible options]
```

--------------------------------

### AWS Deployment

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/setup.md

Deploy the SUSE AI Stack on AWS. This involves copying AWS-specific example variables and configuring them with your AWS settings and credentials.

```bash
cp extra_vars.yml.aws.example extra_vars.yml
# Edit extra_vars.yml with AWS settings and credentials
./setup_private_ai_stack.sh
```

--------------------------------

### AWS Multi-Cluster Setup

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/configuration.md

Configuration for a multi-cluster AWS deployment. Enables SUSE Observability, external DNS via CloudFlare, and Longhorn for storage.

```yaml
cloud_provider: aws

registration_email: admin@company.com
# ... registration codes ...

enable_suse_observability: true
suse_observability_license: <license>

# DNS via CloudFlare
enable_external_dns: true
cloudflare_api_token: <token>

open_webui_hostname: ai.example.com
tls_source: letsEncrypt
letsencrypt_environment: production

# AWS-specific
enable_longhorn: true

pipelines_enabled: true

```

--------------------------------

### Get Kubernetes Services in a Namespace

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/api-reference.md

List all services within a specified namespace in your Kubernetes cluster. This is useful for understanding network configurations.

```bash
# Get services
kubectl get svc -n <namespace>
```

--------------------------------

### Rancher API: Login and Get Token

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/api-reference.md

Example of logging into the Rancher API using curl to obtain an authentication token. This can be done via the UI or programmatically.

```bash
# Login via Rancher UI
# Settings → API & Keys → Create API Key

# Or via API
curl -X POST https://rancher-host/v3-public/localProviders/local?action=login \
  -H "Content-Type: application/json" \
  -d '{
    "username": "admin",
    "password": "password"
  }'
```

--------------------------------

### Search Documents in OpenSearch

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/api-reference.md

Perform a search query against an OpenSearch index using curl. The example searches for documents containing a specific keyword.

```bash
# Search
curl -X GET -u admin:password http://opensearch:9200/documents/_search \
  -H "Content-Type: application/json" \
  -d '{
    "query": {
      "match": {
        "content": "keyword"
      }
    }
  }'
```

--------------------------------

### Configure and Create OpenSearch Snapshot Repository

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/operations.md

Sets up a filesystem repository for OpenSearch snapshots and then creates an initial snapshot named 'snapshot1'. Ensure the path specified in 'location' is accessible by OpenSearch.

```bash
curl -X PUT "http://opensearch:9200/_snapshot/backup" \
  -H "Content-Type: application/json" \
  -d '{"type":"fs","settings":{"location":"/usr/share/opensearch/backup"}}'

curl -X PUT "http://opensearch:9200/_snapshot/backup/snapshot1"
```

--------------------------------

### Setup Python Interpreter

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/playbooks.md

Ensures Python is available on target nodes for Ansible tasks. It detects existing Python versions, sets the Ansible interpreter, and installs Python if it's missing.

```yaml
--- 
- name: Setup Python Interpreter
  hosts: all
  become: yes

  tasks:
    - name: Detect available Python versions
      ansible.builtin.shell: "ls /usr/bin/python*"
      register: python_versions
      changed_when: false

    - name: Set Ansible Python interpreter
      ansible.builtin.set_fact:
        ansible_python_interpreter: "{{ python_versions.stdout_lines[0] if python_versions.stdout_lines else '/usr/bin/python3' }}"

    - name: Install Python if missing (Debian/Ubuntu)
      ansible.builtin.apt: 
        name: "python3"
        state: "present"
      when: ansible_os_family == "Debian"

    - name: Install Python if missing (RHEL/CentOS)
      ansible.builtin.yum: 
        name: "python3"
        state: "present"
      when: ansible_os_family == "RedHat"

```

--------------------------------

### Install Required Ansible Modules

Source: https://github.com/suse/suse-ai-stack/blob/main/docs/Local_Setup.md

Installs the necessary Ansible collections specified in 'requirements.yml' after Ansible has been installed. This ensures all required modules are available for automation.

```console
ansible-galaxy collection install -r requirements.yml
```

--------------------------------

### Restart libvirt Network

Source: https://github.com/suse/suse-ai-stack/blob/main/docs/Local_Setup.md

Commands to destroy and then start a libvirt virtual network. This is necessary to apply changes made to the network configuration, such as adding static IP assignments.

```console
virsh net-destroy default
virsh net-start default
```

--------------------------------

### Local Cluster Provisioning Flow

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/cluster-architecture.md

Outlines the steps for setting up a local cluster using libvirt, including VM creation, cloud-init, and inventory file generation.

```yaml
setup_nodes.yml
    ↓
Call vm role → create_local_vm.yml
    ├─ Download cloud image (SL-Micro)
    ├─ Create QCOW2 disk image
    ├─ Generate cloud-init configuration
    ├─ Create VM via virt-install
    └─ Wait for VM boot
    ↓
Generate inventory files:
    ├─ inventories/mgmt_inventory.ini
    ├─ inventories/suse-ai_inventory.ini
    └─ inventories/suse-observability_inventory.ini
    ↓
Define node groups:
    ├─ [rke2_servers] ← Master nodes
    └─ [rke2_agents] ← Worker nodes
```

--------------------------------

### AWS SSH Public Key Configuration Example

Source: https://github.com/suse/suse-ai-stack/blob/main/docs/AWS_Setup.md

Example of how to configure the 'aws_ssh_public_key' variable in extra_vars.yml. This key is used for passwordless SSH access to the EC2 node.

```yaml
aws_ssh_public_key: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/PFrEQRjraJTx5WulyLfPHiDf6OO0rLU3atox2xu18suohUjCrLTIuRaSMX6mHAX8wb/wPFd2hlk8oXKwBxUMFOn1sOlXFti0tYbtR+TidlKMB22hehCa2K6ckQg07l9IQOQhcccSprT4jXxKW3H4PzC5tA+LfrbaUE8eHEv1/5vBK51AsYRf2T2vbSjnUHIP3bWoYbVx1fdLPvCQsYVRwnP7bLcoaIkciWVqjDW6/xEfw9GrCZCl5QfCUs5lRT2TqrgalODJmBg3tWLO2Bfgmvr9+V4j1DGHX7TqSiVGjgqhruXjGZC675/jML2TXnAxvXDQIMaSz0KSsQSKpC/p foo@somedomain
```

--------------------------------

### Milvus Backup via SDK

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/operations.md

Demonstrates how to initiate a backup for Milvus using its Python SDK. Note that the specific backup API call might vary or be commented out if not directly available.

```python
from pymilvus import MilvusClient

client = MilvusClient(uri="http://milvus:19530")
# Use Milvus backup API
# client.backup()  # If available
```

--------------------------------

### Install Gateway API CRDs

Source: https://github.com/suse/suse-ai-stack/blob/main/docs/cncf-ai-conformance/v1.34/SUSE-AI/specs/ai_inference.md

Installs the necessary Custom Resource Definitions for the Kubernetes Gateway API. Ensure the version matches your APISIX Ingress Controller requirements.

```bash
kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.3.0/experimental-install.yaml
```

--------------------------------

### Kubernetes Get Nodes

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/api-reference.md

Retrieves a list of nodes in the Kubernetes cluster.

```APIDOC
## GET /api/v1/nodes

### Description
Retrieves a list of nodes in the Kubernetes cluster.

### Method
GET

### Endpoint
/api/v1/nodes

### Parameters
None

### Request Example
```bash
kubectl get nodes
```

### Response
#### Success Response (200)
A list of Kubernetes nodes.
```

--------------------------------

### Stream Open WebUI Component Logs

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/operations.md

Follow the logs for the Open WebUI component in real-time.

```bash
# Open WebUI logs
kubectl logs -n suse-private-ai -l app=open-webui -f
```

--------------------------------

### Diagnose Persistent Volume Problems

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/cluster-architecture.md

To diagnose persistent volume issues, first check the status of Persistent Volume Claims (PVCs) within a specific namespace using `kubectl get pvc`. Then, review all available Persistent Volumes (PVs) in the cluster using `kubectl get pv`.

```bash
kubectl get pvc -n <namespace>
kubectl get pv
```

--------------------------------

### Check Kubernetes Cluster Version

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/operations.md

Verify the installed Kubernetes version for the cluster.

```bash
# Check cluster version
kubectl version
```

--------------------------------

### Rancher Get Kubeconfig

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/api-reference.md

Retrieves the kubeconfig for a specific cluster managed by Rancher.

```APIDOC
## POST /v3/clusters/{id}/kubeconfig

### Description
Retrieves the kubeconfig for a specific cluster managed by Rancher.

### Method
POST

### Endpoint
/v3/clusters/{id}/kubeconfig

### Parameters
#### Path Parameters
- **id** (string) - Required - The ID of the cluster.

### Request Example
```bash
curl -H "Authorization: Bearer $TOKEN" \
  https://rancher-host/v3/clusters/local/kubeconfig
```

### Response
#### Success Response (200)
The kubeconfig for the specified cluster.
```

--------------------------------

### Kubernetes Get Services

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/api-reference.md

Retrieves a list of services in a specified Kubernetes namespace.

```APIDOC
## GET /api/v1/namespaces/{namespace}/services

### Description
Retrieves a list of services in a specified Kubernetes namespace.

### Method
GET

### Endpoint
/api/v1/namespaces/{namespace}/services

### Parameters
#### Path Parameters
- **namespace** (string) - Required - The namespace to list services from.

### Request Example
```bash
kubectl get svc -n <namespace>
```

### Response
#### Success Response (200)
A list of services in the specified namespace.
```

--------------------------------

### Deploy KServe InferenceService

Source: https://github.com/suse/suse-ai-stack/blob/main/docs/cncf-ai-conformance/v1.34/SUSE-AI/specs/robust_controller.md

Applies a YAML configuration file to deploy an InferenceService for a Qwen LLM using KServe.

```bash
kubectl apply -f ./kserve/qwen-kserve.yaml
```

--------------------------------

### Kubernetes Get Pods

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/api-reference.md

Retrieves a list of pods in a specified Kubernetes namespace.

```APIDOC
## GET /api/v1/namespaces/{namespace}/pods

### Description
Retrieves a list of pods in a specified Kubernetes namespace.

### Method
GET

### Endpoint
/api/v1/namespaces/{namespace}/pods

### Parameters
#### Path Parameters
- **namespace** (string) - Required - The namespace to list pods from.

### Request Example
```bash
kubectl get pods -n <namespace>
```

### Response
#### Success Response (200)
A list of pods in the specified namespace.
```

--------------------------------

### Open WebUI Configuration

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/configuration.md

Configure the hostname, admin credentials, and pipeline enablement for the Open WebUI.

```yaml
# Hostname for Open WebUI access
# For Let's Encrypt, must be a public FQDN
# open_webui_hostname: ai.example.com

# Admin user configuration
open_webui_admin_name: admin
open_webui_admin_email: admin@suse-private-ai.org
open_webui_admin_password: WelcomeToAI

# Enable Open WebUI pipelines for function calling
pipelines_enabled: true
```

--------------------------------

### Get Node Resource Metrics

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/operations.md

Retrieve CPU and memory usage for nodes in the cluster.

```bash
# CPU and memory usage per node
kubectl top nodes
```

--------------------------------

### Check Namespace Resources

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/operations.md

Verify all resources within specific namespaces like 'suse-private-ai' and 'suse-observability'.

```bash
# Check namespace resources
kubectl get all -n suse-private-ai
kubectl get all -n suse-observability
```

--------------------------------

### Ollama API Endpoints

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/README.md

Details on Ollama API endpoints, including authentication and usage examples.

```APIDOC
## Ollama API Endpoints

### Description
Provides access to Ollama's functionalities for managing and interacting with language models.

### Method
GET, POST, etc. (Specific methods depend on the operation)

### Endpoint
`/api/...` (Specific endpoints vary)

### Parameters
(Details not provided in source)

### Request Example
(Details not provided in source)

### Response
(Details not provided in source)
```

--------------------------------

### Rancher Get Cluster Details

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/api-reference.md

Retrieves detailed information about a specific cluster managed by Rancher.

```APIDOC
## GET /v3/clusters/{id}

### Description
Retrieves detailed information about a specific cluster managed by Rancher.

### Method
GET

### Endpoint
/v3/clusters/{id}

### Parameters
#### Path Parameters
- **id** (string) - Required - The ID of the cluster.

### Request Example
```bash
curl -H "Authorization: Bearer $TOKEN" \
  https://rancher-host/v3/clusters/local
```

### Response
#### Success Response (200)
Detailed information about the specified cluster.
```

--------------------------------

### Kubernetes Storage Class Operations

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/api-reference.md

Commands to list, get details of, and view provisioned PersistentVolumes and PersistentVolumeClaims in Kubernetes.

```bash
# List storage classes
kubectl get storageclass

# Get storage class details
kubectl get storageclass local-path -o yaml

# Provisioned PVs
kubectl get pv
kubectl get pvc -n suse-private-ai
```

--------------------------------

### Create Manual Etcd Snapshot

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/operations.md

Creates a manual backup of the etcd database. This is crucial for disaster recovery of the RKE2 cluster state.

```bash
sudo /opt/rke2/bin/etcdctl --endpoints=127.0.0.1:2379 \
  snapshot save /tmp/etcd-backup-$(date +%s).db
```

--------------------------------

### Describe Persistent Volume Claim

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/operations.md

Get detailed information about a Persistent Volume Claim, including its status and capacity.

```bash
# Check persistent volume usage
kubectl get pvc -n suse-private-ai
kubectl describe pvc <pvc-name> -n <namespace>
```

--------------------------------

### OpenWebUI User Signup API

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/api-reference.md

API call to create a new user in OpenWebUI. Requires email, password, and name.

```bash
# Create user via API
curl -X POST https://open-webui-host/api/v1/auths/signup \
  -H "Content-Type: application/json" \
  -d '{
    "email": "user@example.com",
    "password": "password",
    "name": "User Name"
  }'
```

--------------------------------

### Product Registration Variables

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/configuration.md

Configure these variables for SUSE product registration and AI registry access.

```yaml
# SUSE SCC account email
registration_email: user@example.com

# SUSE Linux Enterprise Micro registration code
sle_micro_registration_code: XXXX-XXXX-XXXX-XXXX

# SUSE Linux Enterprise Server registration code
sles_registration_code: XXXX-XXXX-XXXX-XXXX

# SUSE AI registry registration code
suse_ai_registration_code: XXXX-XXXX-XXXX-XXXX
```

--------------------------------

### Verify KServe CRDs

Source: https://github.com/suse/suse-ai-stack/blob/main/docs/cncf-ai-conformance/v1.34/SUSE-AI/specs/robust_controller.md

Lists all Custom Resource Definitions (CRDs) and filters for those related to KServe, confirming their creation after installation.

```bash
> kubectl get crd | grep -i kserve
clusterservingruntimes.serving.kserve.io                          2026-02-13T14:48:14Z
clusterstoragecontainers.serving.kserve.io                        2026-02-13T14:48:14Z
inferencegraphs.serving.kserve.io                                 2026-02-13T14:48:14Z
inferenceservices.serving.kserve.io                                 2026-02-13T14:48:14Z
localmodelcaches.serving.kserve.io                                2026-02-13T14:48:14Z
localmodelnodegroups.serving.kserve.io                            2026-02-13T14:48:14Z
localmodelnodes.serving.kserve.io                                 2026-02-13T14:48:14Z
servingruntimes.serving.kserve.io                                 2026-02-13T14:48:14Z
trainedmodels.serving.kserve.io                                   2026-02-13T14:48:14Z


```

--------------------------------

### Kubernetes Liveness Probe Configuration

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/api-reference.md

Example configuration for a Kubernetes Liveness Probe, used to detect and restart unhealthy containers.

```yaml
livenessProbe:
  httpGet:
    path: /health
    port: 80
  initialDelaySeconds: 30
  periodSeconds: 10
  timeoutSeconds: 5
  failureThreshold: 3
```

--------------------------------

### Display Access Information

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/playbooks.md

This playbook displays crucial access information after a successful deployment. It runs on localhost and reads facts gathered during the deployment process.

```yaml
--- 
- name: Display Access Information
  hosts: localhost
  connection: local
  become: no

  tasks:
    - name: Display Rancher management interface URL
      ansible.builtin.debug:
        msg: "Rancher Management URL: {{ rancher_url }}"
      when: rancher_url is defined

    - name: Display Open WebUI access URL
      ansible.builtin.debug:
        msg: "Open WebUI URL: http://{{ open_webui_host }}:9600"
      when: open_webui_host is defined

    - name: Display SSH access commands for VMs
      ansible.builtin.debug:
        msg: "SSH Access for {{ inventory_hostname }}: ssh user@{{ ansible_host }}"
      loop: "{{ groups['all'] }}"
      loop_control:
        loop_var: "item"
      when: ansible_host is defined

    - name: Display default credentials for services
      ansible.builtin.debug:
        msg: "Default Credentials (Example): user=admin, password=password123"
      # Actual credentials should be securely retrieved or defined in vars

    - name: Display Kubernetes API endpoints
      ansible.builtin.debug:
        msg: "Kubernetes API Endpoint: {{ management_cluster_api_endpoint }}"
      when: management_cluster_api_endpoint is defined

```

--------------------------------

### vLLM Service Health Check

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/api-reference.md

Check the health status of the vLLM service. This is a simple GET request to the health endpoint.

```bash
curl http://vllm-service:8000/health
```

--------------------------------

### Describe Pod Resource Requests and Limits

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/operations.md

View the resource requests and limits configured for a specific pod.

```bash
# Get resource requests/limits
kubectl describe pod <pod-name> -n <namespace>
```

--------------------------------

### Manually Snapshot Etcd (RKE2)

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/operations.md

Perform a manual snapshot of the RKE2 etcd database.

```bash
bash

```

--------------------------------

### Manually Scale Deployments and StatefulSets

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/operations.md

Adjust the number of replicas for deployments and stateful sets.

```bash
# Scale deployment
kubectl scale deployment open-webui -n suse-private-ai --replicas=3

# Scale stateful set
kubectl scale statefulset milvus -n suse-private-ai --replicas=3
```

--------------------------------

### Get OpenSearch Cluster Health

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/api-reference.md

Use curl to check the health status of your OpenSearch cluster. Ensure you have the correct credentials and endpoint.

```bash
# Cluster health
curl -u admin:password http://opensearch:9200/_cluster/health
```

--------------------------------

### RKE2 Additional Server Node Join

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/cluster-architecture.md

Instructions for joining additional RKE2 server nodes to an existing cluster using a bootstrap token.

```text
Target: rke2_servers[1:] (additional control planes)

1. Get bootstrap token from first node
2. Run RKE2 server join
3. Join to existing cluster via etcd
```

--------------------------------

### Get Ollama CRDs

Source: https://github.com/suse/suse-ai-stack/blob/main/docs/cncf-ai-conformance/v1.34/SUSE-AI/specs/robust_controller.md

Lists all Custom Resource Definitions (CRDs) and filters for those related to Ollama, verifying that the 'models.ollama.ayaka.io' CRD has been created.

```bash
$ kubectl get crd | grep -i ollama
models.ollama.ayaka.io                      2025-10-06T21:57:57Z
```

--------------------------------

### View Kubernetes Roles and Role Bindings

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/operations.md

List all Roles, ClusterRoles, RoleBindings, and ClusterRoleBindings in the cluster. This helps in understanding and auditing RBAC configurations.

```bash
kubectl get roles -A
kubectl get clusterroles

# Get role bindings
kubectl get rolebindings -A
kubectl get clusterrolebindings
```

--------------------------------

### Deploy Ollama using Helm

Source: https://github.com/suse/suse-ai-stack/blob/main/docs/cncf-ai-conformance/v1.34/SUSE-AI/specs/pod_autoscaling.md

Deploy or upgrade the Ollama Helm chart to your SUSE AI environment using the specified version and custom values file.

```bash
helm upgrade -i ollama oci://dp.apps.rancher.io/charts/ollama -n suse-private-ai --version 1.26.0 -f ollama-values.yaml
```

--------------------------------

### Check Kubelet Logs

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/operations.md

Connect to a node via SSH and use journalctl to view Kubelet logs. This is useful for diagnosing node-level issues.

```bash
ssh <node>
journalctl -u rke2-server -f  # or rke2-agent
```

--------------------------------

### List Ollama Models

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/api-reference.md

Use this command to list all available models in the Ollama service.

```bash
curl http://ollama:80/api/tags
```

--------------------------------

### List Models via Open WebUI API

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/api-reference.md

Retrieve a list of available models from the Open WebUI API. Requires an Authorization header with a Bearer token.

```bash
curl -X GET https://open-webui-host/api/v1/models \
  -H "Authorization: Bearer $TOKEN"
```

--------------------------------

### Get Kubernetes Nodes

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/api-reference.md

List all nodes in your Kubernetes cluster using kubectl. This command helps in monitoring the health and status of your cluster nodes.

```bash
# Get nodes
kubectl get nodes
```

--------------------------------

### Minimal Local Configuration File

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/configuration.md

A minimal configuration file for local SUSE AI Stack deployments. It includes essential registration details, GitHub credentials, and authorized SSH keys for VM access.

```yaml
registration_email: user@example.com
sle_micro_registration_code: CODE1
sles_registration_code: CODE2
suse_ai_registration_code: CODE3
github_username: myuser
application_collection_user_email: user@suse.com
application_collection_user_token: token
vm_authorized_ssh_keys:
  - "ssh-rsa AAAA... user@host"
```

--------------------------------

### Persistence Overview Table

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/components.md

An overview of persistence requirements for various components, detailing the storage type (PVC), size, and purpose. All PVCs utilize the configured storage class, defaulting to `local-path`.

```text
| Component | Storage | Size | Purpose |
|-----------|---------|------|---------|
| Ollama | PVC | 20Gi | Model files cache |
| Open WebUI | PVC | Auto | User data, history |
| Milvus | PVC | Auto | Vector indices |
| MinIO | PVC | Auto | Document storage |
| OpenSearch | PVC | Auto | Search indices |
| SUSE Observability | PVC | Auto | Metrics/logs data |

All PVCs use configured storage class (default: `local-path`)
```

--------------------------------

### Describe Fleet Application Status

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/fleet-applications.md

Get detailed status and events for a specific Fleet GitRepo resource. Use this to troubleshoot deployment issues.

```bash
kubectl describe gitrepo <name> -n fleet-local
```

--------------------------------

### Stream OpenSearch Component Logs

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/operations.md

Follow the logs for the OpenSearch component in real-time.

```bash
# OpenSearch logs
kubectl logs -n suse-private-ai -l app=opensearch -f
```

--------------------------------

### Get Ollama Pods

Source: https://github.com/suse/suse-ai-stack/blob/main/docs/cncf-ai-conformance/v1.34/SUSE-AI/specs/robust_controller.md

Retrieves the status of Ollama-related pods, including the model and model store pods. Useful for verifying deployment status.

```bash
$ kubectl get pods
NAME                               READY   STATUS    RESTARTS   AGE
ollama-model-phi-7cd69569f7-pq9gl   1/1     Running   0          153m
ollama-models-store-0               1/1     Running   0          155m
```

--------------------------------

### Rancher API: Get Cluster List

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/api-reference.md

Retrieve a list of all clusters managed by Rancher using a curl command. Requires an Authorization Bearer token.

```bash
# Get cluster list
curl -H "Authorization: Bearer $TOKEN" \
  https://rancher-host/v3/clusters
```

--------------------------------

### Troubleshoot Fleet Deployment Issues

Source: https://github.com/suse/suse-ai-stack/blob/main/_autodocs/fleet-applications.md

A set of commands to diagnose application deployment problems. Includes checking GitRepo status, describing resources, and viewing agent logs.

```bash
kubectl get gitrepo -n fleet-local
kubectl describe gitrepo <name> -n fleet-local
kubectl logs -n fleet-system -l app=fleet-agent
```