### Download Kubernetes Installation Script

Source: https://github.com/run-ai/docs/blob/master/graveyard/install-k8s.md

This command downloads the Kubernetes installation script from a GitHub repository. Ensure you have wget installed and internet connectivity. The script is used for installing, resetting, and managing Kubernetes nodes.

```bash
wget https://raw.githubusercontent.com/run-ai/docs/v2.17/install/kube-install.sh
```

--------------------------------

### Run:ai CLI V2 Output for Port Forwarding

Source: https://github.com/run-ai/docs/blob/master/docs/Researcher/Walkthroughs/walkthrough-build-ports.md

Example output from the Run:ai CLI V2 after successfully submitting a workload and starting port forwarding. It confirms the workspace creation and port forwarding status.

```shell
Creating workspace nginx-test...
To track the workload's status, run 'runai workspace list'

port-forward stared, opening ports [8080:80]
```

--------------------------------

### Install Kubernetes Master

Source: https://github.com/run-ai/docs/blob/master/graveyard/install-k8s.md

Installs a Kubernetes master node. This involves running the downloaded script with root privileges and selecting the 'install Kubernetes master' option. You will be prompted to choose Kubernetes and CNI versions. A join command for workers will be generated upon completion.

```bash
sudo ./kube-install.sh
# Follow prompts to install master, select versions, and save join command.
```

--------------------------------

### Example Startup Script

Source: https://github.com/run-ai/docs/blob/master/docs/Researcher/best-practices/convert-to-unattended.md

A basic shell script to install Python dependencies and run a training script. It's designed to be executed at the start of an unattended job, ensuring all necessary libraries are present and then launching the main training process. The `$@` allows passing arguments to the training script.

```shell
pip install -r requirements.txt
...
python training.py $@

```

--------------------------------

### Install Kubernetes Worker

Source: https://github.com/run-ai/docs/blob/master/graveyard/install-k8s.md

Installs a Kubernetes worker node. Similar to the master installation, download and run the script with root privileges. Select 'install Kubernetes worker' and ensure the Kubernetes version matches the master. Finally, execute the saved join command on the worker.

```bash
sudo ./kube-install.sh
# Follow prompts to install worker, select same version as master.
# Then run the join command on the worker node.
```

--------------------------------

### Execute Run:ai air-gapped setup script

Source: https://github.com/run-ai/docs/blob/master/docs/admin/runai-setup/self-hosted/ocp/preparations.md

Executes the `setup.sh` script for air-gapped installations to upload images to a local Docker Registry. Requires at least 20GB of free disk space. `sudo` may be omitted if Docker is configured for non-root execution.

```bash
./setup.sh
```

--------------------------------

### Run Run.ai Setup Script (Bash)

Source: https://github.com/run-ai/docs/blob/master/docs/admin/runai-setup/self-hosted/k8s/preparations.md

Executes the setup script for Run.ai in an air-gapped environment. Requires Docker installation and sufficient disk space. Optionally, sudo may not be needed if Docker is configured for non-root users.

```bash
sudo -E ./setup.sh
```

--------------------------------

### Execute Postgres Setup Script

Source: https://github.com/run-ai/docs/blob/master/docs/admin/runai-setup/self-hosted/k8s/preparations.md

This bash command executes a locally saved SQL script against a remote PostgreSQL database using the psql client. Ensure the PostgreSQL client is installed and all placeholders in the command are replaced with your specific database connection details and the script path.

```bash
psql --host <POSTGRESQL_HOST> \
--user <POSTGRESQL_USER> \
--port <POSTGRESQL_PORT> \
--dbname <POSTGRESQL_DB> \
-a -f <SQL_FILE> \

```

--------------------------------

### Download Run:ai Installation Log Script

Source: https://github.com/run-ai/docs/blob/master/docs/admin/runai-setup/cluster-setup/bcm-install.md

This command downloads a shell script used to gather Run:ai cluster installation logs. This script is useful for diagnosing failures during the cluster setup process.

```bash
curl -fsSL https://raw.githubusercontent.com/run-ai/public/main/installation/get-installation-logs.sh
```

--------------------------------

### Run:ai CLI V1 Output for Port Forwarding

Source: https://github.com/run-ai/docs/blob/master/docs/Researcher/Walkthroughs/walkthrough-build-ports.md

Example output from the deprecated Run:ai CLI V1 after submitting a job and initiating port forwarding. It indicates successful job submission and the forwarding configuration.

```shell
The job 'nginx-test-0' has been submitted successfully
You can run `runai describe job nginx-test-0 -p team-a` to check the job status

Forwarding from 127.0.0.1:8080 -> 80
Forwarding from [::1]:8080 -> 80
```

--------------------------------

### Testing CLI Installation

Source: https://github.com/run-ai/docs/blob/master/docs/admin/researcher-setup/new-cli-install.md

Verify that the CLI client has been installed correctly by running the version command in the terminal.

```bash
runai version
```

--------------------------------

### Python Virtual Environment Setup

Source: https://github.com/run-ai/docs/blob/master/examples/api/python_client/README.md

Sets up and activates a Python virtual environment, a recommended practice for managing project dependencies. This ensures that project-specific packages do not conflict with globally installed ones.

```bash
python3 -m venv venv
source venv/bin/activate
```

```bash
python -m venv venv
venv\Scripts\activate
```

--------------------------------

### Install Dependencies and Build Go Client

Source: https://github.com/run-ai/docs/blob/master/examples/api/go_client/README.md

Commands to install Go project dependencies using 'go mod tidy' and build the client binary.

```bash
go mod tidy
go build -o runai-go-client cmd/main.go
```

--------------------------------

### Reset and Remove Kubernetes Installation

Source: https://github.com/run-ai/docs/blob/master/graveyard/install-k8s.md

Completely removes Kubernetes from a node (master or worker). Download and execute the installation script with root privileges, then select the 'reset/delete kubernetes' option and confirm the cluster reset and package removal.

```bash
sudo ./kube-install.sh
# Follow prompts to reset/delete kubernetes and confirm removal.
```

--------------------------------

### Get Shell to Workload Container using CLI V2

Source: https://github.com/run-ai/docs/blob/master/docs/Researcher/Walkthroughs/walkthrough-build.md

Opens a direct shell into the container of a specified workload using the Run:ai CLI V2. Replace 'build1' with the target workload name.

```bash
runai workspace bash build1
```

--------------------------------

### Submit Workload with Bash Command and Interact

Source: https://github.com/run-ai/docs/blob/master/docs/Researcher/cli-reference/new-cli/cli-examples.md

Demonstrates submitting a workload using a bash command and interacting with it. Includes examples for submitting with bash and then attaching to it.

```shell
runai training pytorch submit  -p "project-name" -i nvidia/cuda:11.8.0-cudnn8-runtime-ubuntu20.04 -g 1 --workers 3 --command -- bash -c 'trap : TERM INT; sleep infinity & wait'
```

```shell
runai training pytorch bash pytorch-06027b585626 -p "project-name"
```

--------------------------------

### Create OpenShift Projects

Source: https://github.com/run-ai/docs/blob/master/docs/admin/runai-setup/self-hosted/ocp/cluster.md

Creates necessary OpenShift projects for Run:ai cluster installation. The 'runai-scale-adjust' project is optional and used for auto-scaling on cloud clusters.

```bash
oc new-project runai
oc new-project runai-reservation
oc new-project runai-scale-adjust
```

--------------------------------

### Submit Workload with Resource Requests and Limits

Source: https://github.com/run-ai/docs/blob/master/docs/Researcher/cli-reference/new-cli/cli-examples.md

Submits a workload specifying CPU, memory, and GPU resource requests and limits. This ensures the workload gets the necessary resources.

```shell
runai workspace submit  -p "project-name" -i runai.jfrog.io/demo/quickstart-demo   --cpu-core-request 0.3 --cpu-core-limit 1 --cpu-memory-request 50M --cpu-memory-limit 1G  --gpu-devices-request 1 --gpu-memory-request 1G
```

--------------------------------

### Examples for Describing XGBoost Workloads

Source: https://github.com/run-ai/docs/blob/master/docs/Researcher/cli-reference/new-cli/runai_training_xgboost_describe.md

Provides examples of how to use the 'runai training xgboost describe' command with different parameters, such as specifying a project, using a UUID, setting the output format, selecting specific sections, and customizing limits for pods and events.

```bash
# Describe a xgboost training workload with a default project
runai training xgboost describe <xgboost-name>
```

```bash
# Describe a xgboost training workload in a specific project
runai training xgboost describe <xgboost-name> -p <project_name>
```

```bash
# Describe a xgboost training workload by UUID
runai training xgboost describe --uuid=<xgboost_uuid>
```

```bash
# Describe a xgboost training workload with specific output format
runai training xgboost describe <xgboost-name> -o json
```

```bash
# Describe a xgboost training workload with specific sections
runai training xgboost describe <xgboost-name> --general --compute --pods --events --networks
```

```bash
# Describe a xgboost training workload with container details and custom limits
runai training xgboost describe <xgboost-name> --containers --pod-limit 20 --event-limit 100
```

--------------------------------

### Configure Pod Resources

Source: https://github.com/run-ai/docs/blob/master/docs/admin/runai-setup/self-hosted/k8s/backend.md

Example of setting resource requests and limits for Run:ai components and third-party services. This configuration is applied using the `--set` flag during `helm upgrade`.

```yaml
<component>
    resources:
        limits:
            cpu: 500m
            memory: 512Mi
        requests:
            cpu: 250m
            memory: 256Mi
```

--------------------------------

### Run Docker Container Interactively

Source: https://github.com/run-ai/docs/blob/master/docs/Researcher/best-practices/bare-metal-to-docker-images.md

Starts a Docker container interactively, allowing for a shell prompt within the container. It mounts a local directory into the container, enabling direct access to code and the installation of prerequisites. This is useful for debugging and setting up development environments.

```bash
docker run -it .... "the well known image" -v /where/my/code/resides bash
```

--------------------------------

### Create OpenShift project for Run:ai

Source: https://github.com/run-ai/docs/blob/master/docs/admin/runai-setup/self-hosted/ocp/preparations.md

Creates a new OpenShift project (namespace) named `runai-backend` which is required for the Run:ai control plane installation. Uses the `oc` command-line interface.

```bash
oc new-project runai-backend
```

--------------------------------

### runai mpi describe Examples

Source: https://github.com/run-ai/docs/blob/master/docs/Researcher/cli-reference/new-cli/runai_mpi_describe.md

Provides practical examples of using the 'runai mpi describe' command, demonstrating how to describe workloads in the default or a specific project, by UUID, with different output formats, and with specific sections or limits.

```bash
# Describe a mpi training workload with a default project
runai training mpi describe <mpi-name>

# Describe a mpi training workload in a specific project
runai training mpi describe <mpi-name> -p <project_name>

# Describe a mpi training workload by UUID
runai training mpi describe --uuid=<mpi_uuid>

# Describe a mpi training workload with specific output format
runai training mpi describe <mpi-name> -o json

# Describe a mpi training workload with specific sections
runai training mpi describe <mpi-name> --general --compute --pods --events --networks

# Describe a mpi training workload with container details and custom limits
runai training mpi describe <mpi-name> --containers --pod-limit 20 --event-limit 100
```

--------------------------------

### List Run:ai Objects

Source: https://github.com/run-ai/docs/blob/master/docs/Researcher/cli-reference/new-cli/cli-examples.md

Provides examples for listing various objects within the Run:ai environment, including all workloads, projects in YAML format, and nodes in JSON format.

```shell
runai workload list -A
```

```shell
runai project list --yaml
```

```shell
runai node list --json
```

--------------------------------

### Get Shell to Workload Container using CLI V1 (Deprecated)

Source: https://github.com/run-ai/docs/blob/master/docs/Researcher/Walkthroughs/walkthrough-build.md

Opens a direct shell into the container of a specified job using the deprecated Run:ai CLI V1. Replace 'build1' with the target job name.

```bash
runai bash build1
```

--------------------------------

### Generate New Kubernetes Join Command

Source: https://github.com/run-ai/docs/blob/master/graveyard/install-k8s.md

If the original Kubernetes join token expires (default 24 hours), this command can be run on the master node to generate a new, up-to-date join command for worker nodes.

```bash
sudo kubeadm token create --print-join-command
```

--------------------------------

### Configure Run:ai Project and Submit Inference Jobs

Source: https://github.com/run-ai/docs/blob/master/graveyard/quickstart-mig.md

Configures the active Run:ai project and submits inference workloads. This example shows submitting jobs with specified GPU memory requirements and using MIG profiles for fine-grained GPU partitioning. It requires the Run:ai CLI to be installed and configured.

```bash
runai config project team-a
runai submit mig1 -i runai.jfrog.io/demo/quickstart-cuda  --gpu-memory 10GB
runai submit mig2 -i runai.jfrog.io/demo/quickstart-cuda  --mig-profile 2g.10gb 
runai submit mig3 -i runai.jfrog.io/demo/quickstart-cuda  --mig-profile 2g.10gb 
```

--------------------------------

### Runai CLI: Examples for Standard Training Logs

Source: https://github.com/run-ai/docs/blob/master/docs/Researcher/cli-reference/new-cli/runai_training_standard_logs.md

A collection of examples showcasing different ways to retrieve standard training logs using the runai CLI. These examples cover filtering by specific pods, containers, limiting lines, adding timestamps, following logs, viewing previous logs, limiting bytes, and specifying time ranges.

```bash
# Get logs for a standard training
runai training standard logs standard-01

# Get logs for a specific pod in a standard training
runai training standard logs standard-01 --pod=standard-01-worker-0

# Get logs for a specific container in a standard training
runai training standard logs standard-01 --container=standard-worker

# Get the last 100 lines of logs
runai training standard logs standard-01 --tail=100

# Get logs with timestamps
runai training standard logs standard-01 --timestamps

# Follow the logs
runai training standard logs standard-01 --follow

# Get logs for the previous instance of the standard training
runai training standard logs standard-01 --previous

# Limit the logs to 1024 bytes
runai training standard logs standard-01 --limit-bytes=1024

# Get logs since the last 5 minutes
runai training standard logs standard-01 --since=300s

# Get logs since a specific timestamp
runai training standard logs standard-01 --since-time=2023-05-30T10:00:00Z

# Wait up to 30 seconds for standard training to be ready for logs
runai training standard logs standard-01 --wait-timeout=30s
```

--------------------------------

### Installing Bash Auto-completion

Source: https://github.com/run-ai/docs/blob/master/docs/admin/researcher-setup/new-cli-install.md

Enable Bash command auto-completion by installing the bash-completion package and sourcing the completion script.

```bash
# Mac:
brew install bash-completion
# Ubuntu/Debian:
sudo apt-get install bash-completion
# Fedora/Centos:
sudo yum install bash-completion

[[ $PS1 && -f /usr/share/bash-completion/bash_completion ]] && . /usr/share/bash-completion/bash_completion
source <(runai completion bash)
```

--------------------------------

### Installing Windows PowerShell Auto-completion

Source: https://github.com/run-ai/docs/blob/master/docs/admin/researcher-setup/new-cli-install.md

Set up command auto-completion for Windows PowerShell by invoking the completion script and configuring key handlers.

```powershell
runai.exe completion powershell | Out-String | Invoke-Expression
Set-PSReadLineKeyHandler -Key Tab -Function MenuComplete
```

--------------------------------

### Retrieve Run:ai Installation Logs

Source: https://github.com/run-ai/docs/blob/master/docs/admin/troubleshooting/troubleshooting.md

This bash script fetches and executes a remote script to gather installation logs, which is useful for diagnosing installation failures.

```bash
curl -fsSL https://raw.githubusercontent.com/run-ai/public/main/installation/get-installation-logs.sh | bash
```

--------------------------------

### Run AI XGBoost Attach Examples

Source: https://github.com/run-ai/docs/blob/master/docs/Researcher/cli-reference/new-cli/runai_training_xgboost_attach.md

Examples demonstrating how to attach to an XGBoost training job. The first example shows attaching to the main worker with interactive TTY and stdin enabled. The second example illustrates attaching to a specific pod within the job.

```bash
# Attaching to the main worker of a xgboost training
runai training xgboost attach xgboost-01 --tty --stdin
```

```bash
# Attaching to a specific pod of a xgboost training
runai training xgboost attach xgboost-01 --pod xgboost-01-worker-1 --tty --stdin
```

--------------------------------

### Retrieve Run:ai Cluster Installer Logs

Source: https://github.com/run-ai/docs/blob/master/docs/admin/runai-setup/cluster-setup/bcm-install.md

This command helps in troubleshooting the Run:ai Installer pod by retrieving its logs. It first finds the pod's name and then prints its logs for analysis.

```bash
kubectl get pods -n runai | grep 'cluster-installer' # Find the cluster installer pod's name
kubectl logs <POD-NAME> -n runai # Print the cluster installer pod logs
```

--------------------------------

### Deploy Run:ai Event Router Chart with Helm

Source: https://github.com/run-ai/docs/blob/master/graveyard/integration/messaging.md

These shell commands demonstrate how to make the post-process.sh script executable and then install the runai-event-router Helm chart. The installation targets a specific namespace and utilizes the post-renderer script.

```shell
chmod +x post-process.sh
helm install runai-event-router . -n runai-monitoring --create-namespace --post-renderer ./post-process.sh
```

--------------------------------

### Apply Run:ai registry credentials on OpenShift

Source: https://github.com/run-ai/docs/blob/master/docs/admin/runai-setup/self-hosted/ocp/preparations.md

Applies the `runai-reg-creds.yaml` file to the `runai-backend` namespace to enable image downloads from the Run:ai Container Registry. Requires `oc` CLI and Docker installed on the node.

```bash
oc apply -f runai-reg-creds.yaml -n runai-backend
```

--------------------------------

### Install Run:ai Control Plane (Connected)

Source: https://github.com/run-ai/docs/blob/master/docs/admin/runai-setup/self-hosted/ocp/backend.md

Installs the Run:ai Control Plane using Helm for a connected environment. It adds the Helm repository, updates it, and then upgrades the release, specifying the domain and Kubernetes distribution.

```bash
helm repo add runai-backend https://runai.jfrog.io/artifactory/cp-charts-prod
helm repo update
helm upgrade -i runai-backend -n runai-backend runai-backend/control-plane --version "~2.21.0" \
    --set global.domain=runai.apps.<OPENSHIFT-CLUSTER-DOMAIN> \
    --set global.config.kubernetesDistribution=openshift
```

--------------------------------

### Kubernetes and OpenShift Support

Source: https://github.com/run-ai/docs/blob/master/docs/home/whats-new-2-16.md

Information on updated installation prerequisites for Kubernetes and OpenShift releases.

```APIDOC
## Kubernetes and OpenShift Support

### Description
Updated installation prerequisites for supporting new Kubernetes and OpenShift releases.

### Endpoint
Installation Prerequisites

### Notes
See [Kubernetes support matrix](../admin/runai-setup/cluster-setup/cluster-prerequisites.md#kubernetes-distribution) for more details.
```

--------------------------------

### Run Pre-install Diagnostics Script

Source: https://github.com/run-ai/docs/blob/master/docs/admin/runai-setup/self-hosted/k8s/preparations.md

This command makes a downloaded diagnostic script executable and then runs it to check system prerequisites for Run:ai installation. It requires specifying a DNS entry relevant to your deployment. The script performs comprehensive checks for Kubernetes, NVIDIA, storage, and networking components.

```bash
chmod +x preinstall-diagnostics-<platform>
./preinstall-diagnostics-<platform> --domain <dns-entry>
```

--------------------------------

### Create Workspace via API

Source: https://github.com/run-ai/docs/blob/master/docs/Researcher/Walkthroughs/walkthrough-build.md

Submit a new workspace workload to the Run:ai platform using the API.

```APIDOC
## POST /api/v1/workloads/workspaces

### Description
Submits a new workspace workload to the Run:ai platform. This allows researchers to launch interactive environments for development tasks.

### Method
POST

### Endpoint
`/api/v1/workloads/workspaces`

### Parameters

#### Request Body
- **name** (string) - Required - The name of the workspace (e.g., "build1").
- **projectId** (string) - Required - The ID of the project the workspace belongs to (e.g., "<PROJECT-ID>").
- **clusterId** (string) - Required - The UUID of the cluster where the workspace will be launched (e.g., "<CLUSTER-UUID>").
- **spec** (object) - Required - Specifies the details of the workspace environment.
    - **command** (string) - Required - The command to run inside the workspace container (e.g., "sleep").
    - **args** (string) - Required - Arguments for the command (e.g., "infinity").
    - **image** (string) - Required - The container image to use (e.g., "ubuntu").
    - **compute** (object) - Required - Defines the compute resources for the workspace.
        - **gpuDevicesRequest** (integer) - Required - The number of GPUs to request (e.g., 1).

### Request Example
```json
{
    "name": "build1", 
    "projectId": "<PROJECT-ID>", 
    "clusterId": "<CLUSTER-UUID>", 
    "spec": {
        "command" : "sleep",
        "args" : "infinity"
        "image": "ubuntu",
        "compute": {
        "gpuDevicesRequest": 1
        }
    }
}
```

### Response
#### Success Response (200)
- **status** (string) - Indicates the status of the workspace creation.
- **id** (string) - The unique identifier of the created workspace.

#### Response Example
```json
{
  "status": "created",
  "id": "workspace-12345"
}
```
```

--------------------------------

### Submit Workspace via CLI V1

Source: https://github.com/run-ai/docs/blob/master/docs/Researcher/Walkthroughs/walkthrough-build.md

Submits a new workspace named 'build1' using an Ubuntu image with 1 GPU, running 'sleep infinity'. Requires configuring the project first.

```bash
runai config project team-a   
runai submit build1 -i ubuntu -g 1 --interactive -- sleep infinity
```

--------------------------------

### Install Run:ai Control Plane (Airgapped)

Source: https://github.com/run-ai/docs/blob/master/docs/admin/runai-setup/self-hosted/ocp/backend.md

Installs the Run:ai Control Plane using Helm for an airgapped environment. This method requires specifying a local chart file, custom CA if needed, and a custom environment configuration file.

```bash
helm upgrade -i runai-backend  ./control-plane-<version>.tgz -n runai-backend \
    --set global.domain=runai.apps.<OPENSHIFT-CLUSTER-DOMAIN> \
    --set global.config.kubernetesDistribution=openshift \
    --set global.customCA.enabled=true \
    -f custom-env.yaml
```

--------------------------------

### Login via CLI V2

Source: https://github.com/run-ai/docs/blob/master/docs/Researcher/Walkthroughs/walkthrough-build.md

Logs into the Run:ai platform using the V2 CLI. Requires user credentials.

```bash
runai login
```

--------------------------------

### MLflow Kubernetes Configuration Example

Source: https://github.com/run-ai/docs/blob/master/graveyard/integration/mlflow.md

An example MLflow Kubernetes configuration file that specifies context, repository URI, and a job template path. This is a foundational configuration for running MLflow projects on Kubernetes.

```json
{
  "kube-context": ...,
  "repository-uri": ...,
  "kube-job-template-path": "/username/path/to/kubernetes_job_template.yaml"
}
```

--------------------------------

### Submit Workspace via CLI V2

Source: https://github.com/run-ai/docs/blob/master/docs/Researcher/Walkthroughs/walkthrough-build.md

Submits a new workspace named 'build1' using an Ubuntu image with 1 GPU, running 'sleep infinity'. Requires the project to be set first.

```bash
runai project set team-a
runai workspace submit build1 -i ubuntu -g 1 --command -- sleep infinity
```

--------------------------------

### Extract Air-gapped Run.ai Package (Bash)

Source: https://github.com/run-ai/docs/blob/master/docs/admin/runai-setup/self-hosted/k8s/preparations.md

Extracts the Run.ai air-gapped package. Requires the package file and a version placeholder. Assumes Docker is installed and a Docker registry is available.

```bash
tar xvf runai-airgapped-package-<VERSION>.tar.gz

kubectl create namespace runai-backend
```

--------------------------------

### Airgapped Helm Installation for OpenShift

Source: https://github.com/run-ai/docs/blob/master/docs/admin/runai-setup/self-hosted/ocp/cluster.md

Installs Run:ai in an airgapped OpenShift environment using a local Helm chart. It requires specifying the image registry and custom CA settings.

```bash
helm upgrade -i runai-cluster runai-cluster-<version>.tgz \
    --set controlPlane.url=... \
    --set controlPlane.clientSecret=... \
    --set cluster.uid=... \
    --set cluster.url=... --create-namespace \
    --set global.image.registry=registry.mycompany.local \
    --set global.customCA.enabled=true
```

--------------------------------

### Describe Workload using CLI V2

Source: https://github.com/run-ai/docs/blob/master/docs/Researcher/Walkthroughs/walkthrough-build.md

Retrieves detailed status and information about a specific workload using the Run:ai CLI V2. Replace 'build1' with the actual workload name.

```bash
runai workspace describe build1
```

--------------------------------

### Get OpenShift Base Domain

Source: https://github.com/run-ai/docs/blob/master/docs/admin/runai-setup/self-hosted/ocp/additional-clusters.md

Retrieves the base domain for your OpenShift cluster, which is required for configuring remote cluster installations.

```bash
oc get dns cluster -oyaml | grep baseDomain
```

--------------------------------

### Submit Workspace via API

Source: https://github.com/run-ai/docs/blob/master/docs/Researcher/Walkthroughs/walkthrough-build.md

Submits a new workspace named 'build1' via the Run:ai API. It specifies an Ubuntu image, requests 1 GPU, and sets the command to 'sleep infinity'. Requires authentication token and project/cluster details.

```bash
curl -L 'https://<COMPANY-URL>/api/v1/workloads/workspaces' \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer <TOKEN>' \
-d '{ 
    "name": "build1", 
    "projectId": "<PROJECT-ID>", \
    "clusterId": "<CLUSTER-UUID>", \
    "spec": {
        "command" : "sleep",
        "args" : "infinity"
        "image": "ubuntu",
        "compute": {
        "gpuDevicesRequest": 1
        }
    }
}'
```

--------------------------------

### Airgapped Helm Installation Command

Source: https://github.com/run-ai/docs/blob/master/docs/admin/runai-setup/self-hosted/k8s/cluster.md

This command is used for installing a Run:ai cluster in an airgapped Kubernetes environment. It requires specifying a local Helm chart file and setting global image registry details.

```bash
helm upgrade -i runai-cluster runai-cluster-<version>.tgz \
    --set controlPlane.url=... \
    --set controlPlane.clientSecret=... \
    --set cluster.uid=... \
    --set cluster.url=... --create-namespace \
    --set global.image.registry=registry.mycompany.local \

```

--------------------------------

### Validating Configuration

Source: https://github.com/run-ai/docs/blob/master/docs/admin/researcher-setup/new-cli-install.md

View the current CLI configuration in JSON format to validate settings.

```bash
runai config generate --json
```

--------------------------------

### List Workloads using CLI V1 (Deprecated)

Source: https://github.com/run-ai/docs/blob/master/docs/Researcher/Walkthroughs/walkthrough-build.md

Lists all jobs (workloads) using the deprecated Run:ai CLI V1. This command is for older versions and is not recommended for new projects.

```bash
runai list jobs
```

--------------------------------

### runai tensorflow logs examples

Source: https://github.com/run-ai/docs/blob/master/docs/Researcher/cli-reference/new-cli/runai_tensorflow_logs.md

Provides practical examples of using the 'runai tensorflow logs' command. Demonstrates fetching logs for a training job, specific pods, containers, and applying filters like tail, timestamps, follow, previous logs, byte limits, and time-based filtering.

```bash
# Get logs for a tf training
runai training tf logs tf-01

# Get logs for a specific pod in a tf training
runai training tf logs tf-01 --pod=tf-01-worker-0

# Get logs for a specific container in a tf training
runai training tf logs tf-01 --container=tf-worker

# Get the last 100 lines of logs
runai training tf logs tf-01 --tail=100

# Get logs with timestamps
runai training tf logs tf-01 --timestamps

# Follow the logs
runai training tf logs tf-01 --follow

# Get logs for the previous instance of the tf training
runai training tf logs tf-01 --previous

# Limit the logs to 1024 bytes
runai training tf logs tf-01 --limit-bytes=1024

# Get logs since the last 5 minutes
runai training tf logs tf-01 --since=300s

# Get logs since a specific timestamp
runai training tf logs tf-01 --since-time=2023-05-30T10:00:00Z

# Wait up to 30 seconds for tf training to be ready for logs
runai training tf logs tf-01 --wait-timeout=30s
```

--------------------------------

### Describe Workload using CLI V1 (Deprecated)

Source: https://github.com/run-ai/docs/blob/master/docs/Researcher/Walkthroughs/walkthrough-build.md

Retrieves detailed status and information about a specific job using the deprecated Run:ai CLI V1. Replace 'build1' with the actual job name.

```bash
runai describe job build1
```

--------------------------------

### Ray Start Parameters Configuration

Source: https://github.com/run-ai/docs/blob/master/graveyard/integration/ray.md

Specifies parameters for starting a Ray instance, particularly the 'block' parameter, which is set to 'true'. This configuration is part of a larger Ray cluster setup.

```yaml
rayStartParams:
  block: 'true'
```

--------------------------------

### Get logs for XGBoost training

Source: https://github.com/run-ai/docs/blob/master/docs/Researcher/cli-reference/new-cli/runai_xgboost_logs.md

Example of retrieving logs for a specific XGBoost training job identified by its workload name.

```bash
runai training xgboost logs xgboost-01
```

--------------------------------

### runai CLI: XGBoost Log Retrieval Examples

Source: https://github.com/run-ai/docs/blob/master/docs/Researcher/cli-reference/new-cli/runai_training_xgboost_logs.md

Provides examples of how to use the 'runai training xgboost logs' command for various log retrieval scenarios, such as getting logs from a specific pod, container, limiting lines, showing timestamps, following logs, and filtering by time.

```bash
# Get logs for a xgboost training
runai training xgboost logs xgboost-01

# Get logs for a specific pod in a xgboost training
runai training xgboost logs xgboost-01 --pod=xgboost-01-worker-0

# Get logs for a specific container in a xgboost training
runai training xgboost logs xgboost-01 --container=xgboost-worker

# Get the last 100 lines of logs
runai training xgboost logs xgboost-01 --tail=100

# Get logs with timestamps
runai training xgboost logs xgboost-01 --timestamps

# Follow the logs
runai training xgboost logs xgboost-01 --follow

# Get logs for the previous instance of the xgboost training
runai training xgboost logs xgboost-01 --previous

# Limit the logs to 1024 bytes
runai training xgboost logs xgboost-01 --limit-bytes=1024

# Get logs since the last 5 minutes
runai training xgboost logs xgboost-01 --since=300s

# Get logs since a specific timestamp
runai training xgboost logs xgboost-01 --since-time=2023-05-30T10:00:00Z

# Wait up to 30 seconds for xgboost training to be ready for logs
runai training xgboost logs xgboost-01 --wait-timeout=30s
```

--------------------------------

### runai pytorch attach Examples

Source: https://github.com/run-ai/docs/blob/master/docs/Researcher/cli-reference/new-cli/runai_pytorch_attach.md

Illustrates how to use the `runai pytorch attach` command to connect to PyTorch training jobs. Examples show attaching to the primary worker and to a specific pod within a distributed training setup, utilizing flags like `--tty` and `--stdin` for interactive sessions.

```bash
# Attaching to the main worker of a pytorch training
runai training pytorch attach pytorch-01 --tty --stdin
```

```bash
# Attaching to a specific pod of a pytorch training
runai training pytorch attach pytorch-01 --pod pytorch-01-worker-1 --tty --stdin
```

--------------------------------

### Example Namespace YAML with Run:ai Scheduler Annotation (YAML)

Source: https://github.com/run-ai/docs/blob/master/docs/admin/config/default-scheduler.md

An example output of `kubectl get ns <namespace> -o yaml` showing a Kubernetes namespace with the `runai/enforce-scheduler-name: "true"` annotation applied, indicating that Run:ai scheduler is enforced.

```yaml
apiVersion: v1
kind: Namespace
metadata:
  annotations:
    runai/enforce-scheduler-name: "true"
  creationTimestamp: "2024-04-09T08:15:50Z"
  labels:
    kubernetes.io/metadata.name: runai-proj-a
    runai/namespace-version: v2
    runai/queue: proj-a
  name: runai-proj-a
  resourceVersion: "388336"
  uid: c53af666-7989-43df-9804-42bf8965ce83
spec:
  finalizers:
  - kubernetes
status:
  phase: Active
```

--------------------------------

### Readiness Probe Configuration Example (YAML)

Source: https://github.com/run-ai/docs/blob/master/docs/platform-admin/workloads/policies/policy-reference.md

This YAML snippet demonstrates how to configure the readiness probe for a container, specifically setting the initial delay in seconds before probes are initiated. This is crucial for ensuring a container is ready to serve traffic.

```yaml
defaults:
  probes:
    readiness:
        initialDelaySeconds: 2
```

--------------------------------

### Creating a Project

Source: https://github.com/run-ai/docs/blob/master/docs/platform-admin/aiinitiatives/org/projects.md

Describes the process of creating a new project.

```APIDOC
## Creating a Project

To create a new project:

1. Navigate to the projects section.
2. Click __CREATE PROJECT__.
```

--------------------------------

### Prevent Accidental Kubernetes Upgrades

Source: https://github.com/run-ai/docs/blob/master/graveyard/install-k8s.md

Prevents automatic upgrades of Kubernetes components (kubeadm, kubelet, kubectl) during system package updates on Ubuntu-based systems. This is done by holding the current versions of these packages.

```bash
sudo apt-mark hold kubeadm kubelet kubectl
```

--------------------------------

### List Workloads using CLI V2

Source: https://github.com/run-ai/docs/blob/master/docs/Researcher/Walkthroughs/walkthrough-build.md

Lists all workloads in the current workspace using the Run:ai CLI V2. It displays details such as Workload name, Type, Status, Project, Preemptible status, Pods, and GPU Allocation.

```bash
runai workspace list
```

--------------------------------

### Log in using runai CLI v2

Source: https://github.com/run-ai/docs/blob/master/docs/Researcher/scheduling/launching-workloads-with-dynamic-gpu-fractions.md

Obtain login options and log in according to your setup using the runai CLI v2.

```bash
runai login --help
```

--------------------------------

### Run Run:ai Pre-install Diagnostics Script

Source: https://github.com/run-ai/docs/blob/master/docs/admin/runai-setup/self-hosted/ocp/preparations.md

This command makes a downloaded diagnostics script executable and then runs it. This script checks Run:ai prerequisites and identifies potential issues related to Kubernetes, NVIDIA, storage, and networking.

```bash
chmod +x preinstall-diagnostics-<platform>
./preinstall-diagnostics-<platform>
```

--------------------------------

### Enable KubeVirt in Run:ai cluster setup

Source: https://github.com/run-ai/docs/blob/master/graveyard/integration/kubevirt.md

This YAML snippet shows how to enable the KubeVirt feature within a Run:ai cluster installation or upgrade. By setting `global.kubevirtCluster.enabled: true` in the `values.yaml` file, KubeVirt components are deployed and configured for VM scheduling.

```yaml
global:
  kubevirtCluster:
    enabled: true
```

--------------------------------

### Project Structure Overview

Source: https://github.com/run-ai/docs/blob/master/examples/api/go_client/README.md

Directory structure of the runai-go-client project, highlighting key Go packages and files.

```plaintext
runai_go_client/
├── cmd/
│   └── main.go              # CLI entry point
├── pkg/
│   ├── client/
│   │   └── runai_client.go  # Main client logic
│   └── kubeconfig/
│       └── kubeconfig.go    # Kubeconfig manipulation
├── go.mod                   # Go module definition
└── README.md               # This file
```

--------------------------------

### Using kubectl explain for Policy Syntax

Source: https://github.com/run-ai/docs/blob/master/docs/platform-admin/workloads/policies/old-policies.md

Instructions on how to use the `kubectl explain` command to retrieve detailed syntax and field descriptions for Run AI policies.

```APIDOC
## Using kubectl explain for Policy Syntax

### Description
This section provides guidance on how to use the `kubectl explain` command to inspect the structure and available fields for various Run AI policy resources.

### Method
Bash Command

### Endpoint
N/A

### Parameters
None

### Request Example
```bash
kubectl explain trainingpolicy.spec
```

### Response
#### Success Response (200)
```yaml
KIND:     TrainingPolicy
VERSION:  run.ai/v2alpha1

RESOURCE: spec <Object>

DESCRIPTION:
The specifications of this TrainingPolicy

FIELDS:
annotations	<Object>
Specifies annotations to be set in the container running the created
workload.

arguments	<Object>
If set, the arguments are sent along with the command which overrides the
image's entry point of the created workload.

command	<Object>
If set, overrides the image's entry point with the supplied command.
...
```

#### Response Example
```bash
kubectl explain trainingpolicy.spec.ports
```

```yaml
KIND:     TrainingPolicy
VERSION:  run.ai/v2alpha1

RESOURCE: ports <Object>

DESCRIPTION:
     Specify the set of ports exposed from the container running the created
     workload. Used together with --service-type.

FIELDS:
   itemRules	<Object>

   items	<map[string]Object>

   rules	<Object>
     these rules apply to a value of type map (=non primitive) as a whole
     additionally there are rules which apply for specific items of the map
```

```bash
kubectl explain trainingpolicy.spec.ports.rules
```

```yaml
KIND:     TrainingPolicy
VERSION:  run.ai/

RESOURCE: rules <Object>

DESCRIPTION:
     these rules apply to a value of type map (=non primitive) as a whole
     additionally there are rules which apply for specific items of the map

FIELDS:
   canAdd	<boolean>
     is it allowed for a workload to add items to this map

   required	<boolean>
     if the map as a whole is required
```
```

--------------------------------

### Install KubeRay Operator

Source: https://github.com/run-ai/docs/blob/master/graveyard/integration/ray.md

Installs the KubeRay operator using Helm. Ensure you have Helm v0.5.0 or greater installed. This command adds the KubeRay Helm repository and installs the operator in a dedicated namespace.

```bash
helm repo add kuberay https://ray-project.github.io/kuberay-helm/

helm install kuberay-operator kuberay/kuberay-operator -n kuberay-operator --version 0.5.0 --create-namespace
```

--------------------------------

### Submit Jupyter Notebook Workload

Source: https://github.com/run-ai/docs/blob/master/docs/Researcher/cli-reference/new-cli/cli-examples.md

Submits a Jupyter Notebook workload with specific configurations, including image, project, GPU request, external URL, and base URL for the notebook.

```shell
runai workspace submit --image jupyter/scipy-notebook -p "project-name" --gpu-devices-request 1 --external-url container=8888 --name-prefix jupyter --command -- start-notebook.sh --NotebookApp.base_url='/${RUNAI_PROJECT}/${RUNAI_JOB_NAME}' --NotebookApp.token=''
```

--------------------------------

### Submit Training Jobs (CLI V2)

Source: https://github.com/run-ai/docs/blob/master/docs/Researcher/Walkthroughs/walkthrough-overquota.md

Submits training jobs with specified image, GPU count, and project for CLI V2. Used to demonstrate over-quota scenarios.

```bash
runai training submit a2 -i runai.jfrog.io/demo/quickstart -g 2 -p team-a
runai training submit a1 -i runai.jfrog.io/demo/quickstart -g 1 -p team-a
runai training submit b1 -i runai.jfrog.io/demo/quickstart -g 1 -p team-b
```

--------------------------------

### CLI v2: Install script environment variable usage

Source: https://github.com/run-ai/docs/blob/master/docs/home/saas-updates.md

Fixes an issue in the CLI v2 install script where setting the install path environment variable did not correctly install all files. Ensures proper file placement according to the specified path.

```bash
export RUNAI_INSTALL_PATH=/path/to/install
./install.sh
```

--------------------------------

### Install Bash-Completion Package (Mac/Ubuntu/Fedora)

Source: https://github.com/run-ai/docs/blob/master/docs/admin/researcher-setup/cli-install.md

These commands install the bash-completion package on different Linux distributions and macOS. This package is a prerequisite for enabling Bash command auto-completion.

```bash
# Mac:
brew install bash-completion

# Ubuntu/Debian:
sudo apt-get install bash-completion

# Fedora/Centos:
sudo yum install bash-completion
```

--------------------------------

### Initialize and Configure Run:ai HPO Library (Python)

Source: https://github.com/run-ai/docs/blob/master/graveyard/walkthrough-hpo.md

This Python code demonstrates how to import and initialize the Run:ai HPO library. It covers selecting an optimization strategy (Grid Search or Random Search) and initializing the library with a specified NFS directory for synchronization. The example also shows how to define hyperparameter configurations for experiments using the `pick` function.

```python
# import Run:ai HPO library
import runai.hpo

# select Random search or grid search
strategy = runai.hpo.Strategy.GridSearch

# initialize the Run:ai HPO library. Send the NFS directory used for sync
runai.hpo.init("/nfs")

# pick a configuration for this HPO experiment
# we pass the options of all hyperparameters we want to test
# `config` will hold a single value for each parameter
config = runai.hpo.pick(
grid=dict(
        batch_size=[32, 64, 128],
        lr=[1, 0.1, 0.01, 0.001]),
strategy=strategy)

....

# Use the selected configuration within your code
# Example using the picked configuration with a hypothetical optimizer
# optimizer = keras.optimizers.SGD(lr=config['lr'])
```

--------------------------------

### Install Run:ai Control Plane (Connected)

Source: https://github.com/run-ai/docs/blob/master/docs/admin/runai-setup/self-hosted/k8s/backend.md

Installs the Run:ai control plane in a connected environment using Helm. Requires adding the Helm repository, updating it, and then upgrading the release with the specified domain. It's possible to install a specific version by adding the --version flag.

```bash
helm repo add runai-backend https://runai.jfrog.io/artifactory/cp-charts-prod
helm repo update
helm upgrade -i runai-backend -n runai-backend runai-backend/control-plane --version "~2.21.0" \
    --set global.domain=<DOMAIN>  # (1)
```

--------------------------------

### Install PyYAML

Source: https://github.com/run-ai/docs/blob/master/graveyard/hpo.md

Installs the PyYAML package, a dependency for the Run:ai HPO library, using pip. Ensure you are using the correct pip installer for your Python environment.

```bash
pip install pyyaml
```

--------------------------------

### Runai CLI Resume Options for PyTorch

Source: https://github.com/run-ai/docs/blob/master/docs/Researcher/cli-reference/new-cli/runai_training_pytorch_resume.md

Describes the available options for the `runai training pytorch resume` command, including help, project specification, and workload UUID.

```bash
  -h, --help             help for resume
  -p, --project string   Specify the project to which the command applies. By default, commands apply to the default project. To change the default project use ‘runai config project <project name>’
  -u, --uuid string      The UUID of the workload
```

--------------------------------

### Fix CLI v2 Install Script Path Issues

Source: https://github.com/run-ai/docs/blob/master/docs/home/changelog/hotfixes-2-18.md

Corrects an issue in the CLI v2 install script where setting the install path environment variable did not correctly place all files.

```Shell
# This fix addresses the installation process for CLI v2, ensuring files are placed in the correct directories.
# It likely involves modifications to the install script itself.
```

--------------------------------

### runai training mpi help options

Source: https://github.com/run-ai/docs/blob/master/docs/Researcher/cli-reference/new-cli/runai_training_mpi.md

Displays help information for the 'mpi' command within the runai training context. This includes general help and inherited options.

```bash
runai training mpi --help
```

--------------------------------

### Install JupyterHub using Helm

Source: https://github.com/run-ai/docs/blob/master/graveyard/integration/jupyterhub.md

Installs JupyterHub using the Helm package manager. It adds the JupyterHub Helm repository, updates it, and then installs the JupyterHub chart with specified values and namespace.

```bash
helm repo add jupyterhub https://jupyterhub.github.io/helm-chart/
helm repo update
helm install jhub jupyterhub/jupyterhub -n jhub  --version=0.11.1 --values config.yaml
```

--------------------------------

### Disable NGINX Installation in Run:ai Cluster YAML

Source: https://github.com/run-ai/docs/blob/master/docs/admin/troubleshooting/troubleshooting.md

This snippet shows how to disable the automatic installation of NGINX in the Run:ai cluster configuration YAML file to resolve ingress conflicts.

```yaml
ingress-nginx:
    enabled: false
```

--------------------------------

### Log in to Run:ai CLI

Source: https://github.com/run-ai/docs/blob/master/docs/Researcher/cli-reference/new-cli/cli-examples.md

Demonstrates how to log in to the Run:ai CLI using either the web sign-in page or terminal credentials. Requires valid user credentials for terminal login.

```shell
runai login
```

```shell
runai login user -u john@acme.com -p "password"
```

--------------------------------

### Install MPI Operator for MPI v2

Source: https://github.com/run-ai/docs/blob/master/docs/admin/runai-setup/cluster-setup/cluster-prerequisites.md

Installs the MPI Operator for MPI v2, required for distributed training with MPI workloads. This installation ensures compatibility with advanced workload features when using MPI v2.

```bash
kubectl apply --server-side -f https://raw.githubusercontent.com/kubeflow/mpi-operator/v0.6.0/deploy/v2beta1/mpi-operator.yaml
```

--------------------------------

### Install Run:ai Control Plane (Airgapped)

Source: https://github.com/run-ai/docs/blob/master/docs/admin/runai-setup/self-hosted/k8s/backend.md

Installs the Run:ai control plane in an airgapped environment using Helm. This method requires a local tarball of the control plane, specifying the domain, enabling custom CA if necessary, and referencing a custom environment file. It's also possible to install a specific version by replacing the placeholder.

```bash
helm upgrade -i runai-backend control-plane-<VERSION>.tgz  # (1) \
    --set global.domain=<DOMAIN>  # (2) \
    --set global.customCA.enabled=true \
    -n runai-backend -f custom-env.yaml  # (4)
```

--------------------------------

### Submit Jobs with CLI V2 and V1

Source: https://github.com/run-ai/docs/blob/master/docs/Researcher/Walkthroughs/walkthrough-queue-fairness.md

Submits multiple training jobs for 'team-a' using the runai CLI. This demonstrates how to request GPU resources for a specific project. It supports both V1 (deprecated) and V2 CLI versions.

```bash
runai training submit a1 -i runai.jfrog.io/demo/quickstart -g 1 -p team-a
runai training submit a2 -i runai.jfrog.io/demo/quickstart -g 1 -p team-a
runai training submit a3 -i runai.jfrog.io/demo/quickstart -g 1 -p team-a
runai training submit a4 -i runai.jfrog.io/demo/quickstart -g 1 -p team-a
```

```bash
runai submit a1 -i runai.jfrog.io/demo/quickstart -g 1 -p team-a
runai submit a2 -i runai.jfrog.io/demo/quickstart -g 1 -p team-a
runai submit a3 -i runai.jfrog.io/demo/quickstart -g 1 -p team-a
runai submit a4 -i runai.jfrog.io/demo/quickstart -g 1 -p team-a
```