### Pre-GitOps Setup

Source: https://github.com/rh-aiservices-bu/rhoai-nightly/blob/main/README.md

Runs the initial setup steps required before installing GitOps, including infrastructure provisioning and pull-secret configuration.

```bash
make setup
```

--------------------------------

### Bootstrap GitOps and Deploy Apps

Source: https://github.com/rh-aiservices-bu/rhoai-nightly/blob/main/README.md

Combines the installation of the GitOps operator/ArgoCD and the deployment of ArgoCD applications. This is a convenient shortcut for initial setup.

```bash
make bootstrap
```

--------------------------------

### Common Commands: Pre-GitOps Setup

Source: https://github.com/rh-aiservices-bu/rhoai-nightly/blob/main/CLAUDE.md

Commands for adding credentials, creating ImageContentPolicy, and setting up GPU/CPU machine sets. Includes a combined setup command.

```bash
make secrets         # Add quay.io/rhoai credentials to global pull secret
make icsp            # Create ImageContentSourcePolicy (triggers node restart)
make gpu             # Create GPU MachineSet (waits for node Ready)
make cpu             # Create CPU worker MachineSet (waits for node Ready)
make setup           # Run all pre-GitOps setup (secrets, icsp, gpu, cpu)
```

--------------------------------

### Run Infrastructure Setup

Source: https://github.com/rh-aiservices-bu/rhoai-nightly/blob/main/README.md

Executes the complete infrastructure setup, including registry mirroring (icsp) and creating both CPU and GPU worker nodes.

```bash
make infra
```

--------------------------------

### Install MaaS Platform

Source: https://github.com/rh-aiservices-bu/rhoai-nightly/blob/main/README.md

Installs the Models as a Service (MaaS) platform, including secrets, Gateway, and Authorino SSL configuration. This does not install observability components.

```bash
make maas
```

--------------------------------

### Install Observability Script Flow

Source: https://github.com/rh-aiservices-bu/rhoai-nightly/blob/main/CLAUDE.md

The installation script performs a series of preflight checks, resource creations, and configuration updates to enable MaaS observability.

```bash
install-observability.sh:
  1. Preflight: cluster connection, Kuadrant + Istio Telemetry CRDs
  2. Phase A:  VERIFY UWM is enabled (fails with pointer to `make uwm` if not)
  3. Phase A2: Scrub openshift.io/cluster-monitoring label from MaaS namespaces (UWM owns scraping)
  4. Phase S:  Settle-gate (CSVs Succeeded, DSC/DSCI Ready, no bad pods, etcd healthy, masters <80% mem)
  5. Phase G:  Create instance-maas-observability ArgoCD Application (TelemetryPolicy + Istio Telemetry sync via ArgoCD)
  6. Phase O:  Flip instance-rhoai source -> overlays/maas-observability (RHOAI operator's monitoring cascade fires)
  7. Phase B:  Verify Kuadrant CR has spec.observability.enable=true (GitOps-managed)
  8. Phase C:  Apply limitador-servicemonitor IF no existing monitor covers it
  9. Phase D:  Apply authorino-server-metrics-servicemonitor IF no existing monitor covers /server-metrics
  10. Phase E: Apply istio-gateway-service + servicemonitor IF maas-default-gateway deployment present
  11. Phase F: Apply kserve-llm-models-servicemonitor IF llm namespace exists
```

--------------------------------

### Autonomous Workflow: Run All

Source: https://github.com/rh-aiservices-bu/rhoai-nightly/blob/main/CLAUDE.md

A single command to execute all setup, bootstrap, sync, and MAAS steps. Each script within waits for readiness.

```bash
make all             # Runs: setup + bootstrap + sync + maas
                     # Each script waits for readiness before proceeding
```

--------------------------------

### Install Observability Stack

Source: https://github.com/rh-aiservices-bu/rhoai-nightly/blob/main/README.md

Installs the observability stack for MaaS, which includes components like Perses, Tempo, and OTel. This step is performed separately after the main MaaS installation and includes a settle-gate to ensure cluster health before proceeding.

```bash
make observability            # Settle-gate → flip instance-rhoai overlay → wait for Perses/Tempo/OTel
```

--------------------------------

### Incremental Deployment: Pre-GitOps Setup

Source: https://github.com/rh-aiservices-bu/rhoai-nightly/blob/main/CLAUDE.md

Steps for setting up necessary credentials, policies, and machine sets for GPU and CPU workers. Verification steps are included.

```bash
make secrets         # Add quay.io/rhoai credentials (pull-secret)
                     # VERIFY: oc get secret/pull-secret -n openshift-config

make icsp            # Create ImageContentSourcePolicy
                     # WAITS: ~10-15 min for all nodes to restart
                     # VERIFY: oc get nodes (all Ready)

make gpu             # Create GPU MachineSet (g6e.2xlarge, autoscale 1-3)
                     # WAITS: Until GPU node is Ready
                     # VERIFY: oc get nodes -l node-role.kubernetes.io/gpu

make cpu             # Create CPU worker MachineSet (m6a.4xlarge, autoscale 1-3)
                     # WAITS: Until CPU worker node is Ready
                     # VERIFY: oc get nodes -l node-role.kubernetes.io/worker
```

--------------------------------

### Configure Manual Credentials

Source: https://github.com/rh-aiservices-bu/rhoai-nightly/blob/main/README.md

Copies the example environment file and prompts the user to edit it with their quay.io credentials. This is for manual credential configuration.

```bash
cp .env.example .env
# Edit .env with your quay.io credentials for quay.io/rhoai access
```

--------------------------------

### Install Observability Stack

Source: https://github.com/rh-aiservices-bu/rhoai-nightly/blob/main/README.md

Installs the observability stack (Perses, Tempo, OTel) after a settle-gate period. This command will refuse to run if the cluster is not healthy.

```bash
make observability
```

--------------------------------

### Install GitOps Operator and ArgoCD

Source: https://github.com/rh-aiservices-bu/rhoai-nightly/blob/main/README.md

Installs the GitOps operator and an ArgoCD instance on the cluster. This is a prerequisite for deploying applications via GitOps.

```bash
make gitops
```

--------------------------------

### Incremental Deployment: Bootstrap GitOps

Source: https://github.com/rh-aiservices-bu/rhoai-nightly/blob/main/CLAUDE.md

Commands to install the GitOps operator and ArgoCD, followed by deploying the root application.

```bash
make gitops          # Install GitOps operator + ArgoCD
                     # WAITS: Until ArgoCD is ready
                     # VERIFY: oc get pods -n openshift-gitops

make deploy          # Deploy root app (triggers all ApplicationSets)
                     # VERIFY: oc get applications.argoproj.io -n openshift-gitops
```

--------------------------------

### Common Commands: GitOps Bootstrap

Source: https://github.com/rh-aiservices-bu/rhoai-nightly/blob/main/CLAUDE.md

Commands to install the GitOps operator and ArgoCD, deploy the root application, or run both together.

```bash
make gitops          # Install GitOps operator + ArgoCD instance
make deploy          # Deploy root app (triggers ApplicationSets)
make bootstrap       # Run gitops + deploy together
```

--------------------------------

### Configure .env File for RHOAI

Source: https://github.com/rh-aiservices-bu/rhoai-nightly/blob/main/CLAUDE.md

Example configuration for the .env file, including Quay.io credentials and optional GPU/CPU MachineSet settings, as well as GitOps configuration.

```bash
# Required: Quay.io credentials for RHOAI nightly images
QUAY_USER=your-username
QUAY_TOKEN=your-token

# Optional: GPU MachineSet configuration
GPU_INSTANCE_TYPE=g6e.2xlarge    # GPU instance type
GPU_REPLICAS=1                   # Initial replicas
GPU_ACCESS_TYPE=SHARED           # SHARED or PRIVATE
GPU_MIN=1                        # Minimum replicas (autoscaling)
GPU_MAX=3                        # Maximum replicas (autoscaling)
GPU_AZ=                          # Auto-detected if empty

# Optional: CPU Worker MachineSet configuration
CPU_INSTANCE_TYPE=m6a.4xlarge   # CPU instance type
CPU_REPLICAS=1                   # Initial replicas
CPU_VOLUME_SIZE=120              # Root volume size (GB)
CPU_MIN=1                        # Minimum replicas (autoscaling)
CPU_MAX=3                        # Maximum replicas (autoscaling)
CPU_AZ=                          # Auto-detected if empty

# Optional: GitOps configuration (for forks)
GITOPS_REPO_URL=https://github.com/your-username/rhoai-nightly
GITOPS_BRANCH=main

# RHOAI version: edit components/operators/rhoai-operator/base/catalogsource.yaml
# See "RHOAI Version Configuration" section above
```

--------------------------------

### MaaS Install Flow Script

Source: https://github.com/rh-aiservices-bu/rhoai-nightly/blob/main/CLAUDE.md

The install-maas.sh script orchestrates the installation of MaaS components. It performs preflight checks, creates PostgreSQL secrets, deploys the ArgoCD Application for MaaS using Helm, configures Authorino SSL, and validates the deployment.

```bash
install-maas.sh:
  1. Preflight: check cluster, detect domain/cert/repo/branch
  2. Create PostgreSQL secrets (idempotent)
  3. Create instance-maas ArgoCD Application (Helm source with values)
     → ArgoCD syncs: GatewayClass, Gateway, PostgreSQL (Deployment + PVC + Service)
  4. Configure Authorino SSL env vars
  5. Validate: wait for maas-api, check health endpoint
```

--------------------------------

### Permanent Fork Setup

Source: https://github.com/rh-aiservices-bu/rhoai-nightly/blob/main/README.md

Configure a permanent fork by setting GITOPS_REPO_URL and GITOPS_BRANCH, then run 'make configure-repo'. This mutates checked-in YAML files and is intended for hard-forking scenarios, not transient branches.

```bash
GITOPS_REPO_URL=https://github.com/my-fork/rhoai-nightly \
GITOPS_BRANCH=my-default \
make configure-repo
git commit -am "chore: point GitOps at my fork"
```

--------------------------------

### Apply ICSP (Registry Mirror)

Source: https://github.com/rh-aiservices-bu/rhoai-nightly/blob/main/README.md

Applies the Image Content Source Policy to configure a registry mirror. This is part of the infrastructure setup phase.

```bash
make icsp
```

--------------------------------

### Install and Uninstall Eval Hub

Source: https://github.com/rh-aiservices-bu/rhoai-nightly/blob/main/CLAUDE.md

Commands to toggle the Eval Hub feature on and off.

```bash
make evalhub
make evalhub-uninstall
```

--------------------------------

### Example .gitignore Configuration

Source: https://github.com/rh-aiservices-bu/rhoai-nightly/blob/main/CLAUDE.md

This .gitignore configuration prevents common secret file types from being committed to the repository.

```gitignore
.env
.env.*
!.env.example
*.credentials
*credentials*
*secret*.txt
*secret*.json
*.key
*.pem
```

--------------------------------

### Initiate Full Deployment

Source: https://github.com/rh-aiservices-bu/rhoai-nightly/blob/main/README.md

Starts the complete RHOAI 3.x nightly build deployment process. This command orchestrates multiple phases including infrastructure, secrets, GitOps, and application deployment.

```bash
make
```

--------------------------------

### Dry Run MaaS Installation and Uninstallation

Source: https://github.com/rh-aiservices-bu/rhoai-nightly/blob/main/README.md

Allows previewing the actions of the `make maas` and `make maas-uninstall` commands without actually applying any changes to the cluster. This is useful for understanding the impact of the commands before execution.

```bash
make maas -- --dry-run        # Preview install without applying
make maas-uninstall -- --dry-run  # Preview uninstall
```

--------------------------------

### Uninstall Observability Stack

Source: https://github.com/rh-aiservices-bu/rhoai-nightly/blob/main/README.md

Reverses the installation of the observability stack, tearing down the monitoring cascade.

```bash
make observability-uninstall
```

--------------------------------

### Uninstall Observability Script Flow

Source: https://github.com/rh-aiservices-bu/rhoai-nightly/blob/main/CLAUDE.md

The uninstallation script reverses the installation process, tearing down observability components and reverting configurations.

```bash
install-observability.sh --uninstall:
  1. Flip instance-rhoai source -> overlays/maas (RHOAI operator's monitoring cascade tears down)
  2. Phase G-uninstall: Delete instance-maas-observability Application
     (resources-finalizer.argoproj.io cascades TelemetryPolicy + Istio Telemetry)
  3. Delete ServiceMonitors/Services labelled app.kubernetes.io/part-of=maas-observability
     and app.kubernetes.io/managed-by=maas-observability (kuadrant-system, openshift-ingress, llm)
  4. LEAVES bootstrap/cluster-monitoring-config ConfigMap in place (other workloads may depend on UWM)
  5. LEAVES Kuadrant CR spec.observability.enable=true (GitOps-owned; harmless without UWM scraping)
```

--------------------------------

### Configure Repository for Forks

Source: https://github.com/rh-aiservices-bu/rhoai-nightly/blob/main/CLAUDE.md

Permanently configures the repository for forks by mutating YAML files. Requires committing the changes. Use only for permanent fork setups.

```bash
GITOPS_REPO_URL=https://github.com/my-fork/rhoai-nightly \
GITOPS_BRANCH=my-default \
make configure-repo
```

--------------------------------

### Setup Pull-Secret Configuration

Source: https://github.com/rh-aiservices-bu/rhoai-nightly/blob/main/README.md

Configures the pull secret, automatically detecting whether to use External Secrets or manual credentials based on environment variables or Git access.

```bash
make secrets
```

--------------------------------

### Uninstall Observability Stack

Source: https://github.com/rh-aiservices-bu/rhoai-nightly/blob/main/README.md

Removes the observability stack. This action reverses the installation process by un-flipping the overlay, leading to the teardown of the monitoring cascade.

```bash
make observability-uninstall  # Reverse-flip; monitoring cascade tears down
```

--------------------------------

### Patching an Application Directly

Source: https://github.com/rh-aiservices-bu/rhoai-nightly/blob/main/CLAUDE.md

This command allows for direct patching of an existing Argo CD Application resource, for example, to update the target revision. This is useful when template changes in an ApplicationSet do not propagate to existing applications.

```bash
oc patch application.argoproj.io/<app-name> -n openshift-gitops \
  --type=merge -p '{"spec":{"source":{"targetRevision":"new-branch"}}}'
```

--------------------------------

### Deploy MaaS Models using Kustomize

Source: https://github.com/rh-aiservices-bu/rhoai-nightly/blob/main/CLAUDE.md

The setup-maas-model.sh script deploys or deletes MaaS models using 'oc kustomize'. It reads the MAAS_MODELS environment variable for default model selection.

```bash
setup-maas-model.sh deploys/deletes models using `oc kustomize`. It reads `MAAS_MODELS` from `.env` for default model selection (default: `gpt-oss-20b`).
```

--------------------------------

### Create GPU Machineset with CLI Arguments

Source: https://github.com/rh-aiservices-bu/rhoai-nightly/blob/main/CLAUDE.md

Use CLI arguments to specify instance type and replicas when creating a GPU machineset.

```bash
scripts/create-gpu-machineset.sh --instance-type g6e.4xlarge --replicas 2
```

--------------------------------

### Incremental Deployment: Cluster Preparation

Source: https://github.com/rh-aiservices-bu/rhoai-nightly/blob/main/CLAUDE.md

Commands for initial cluster readiness checks, environment configuration, and validation against cluster capabilities.

```bash
make preflight       # Quick cluster readiness check
cp .env.example .env # Configure credentials (edit .env)
make validate-config # Validate .env against cluster capabilities
```

--------------------------------

### Create GPU Machineset with Environment Variables

Source: https://github.com/rh-aiservices-bu/rhoai-nightly/blob/main/CLAUDE.md

Set environment variables for instance type and replicas to configure the GPU machineset creation script.

```bash
GPU_INSTANCE_TYPE=g6e.4xlarge GPU_REPLICAS=2 scripts/create-gpu-machineset.sh
```

--------------------------------

### Update Existing Application via Deletion and Sync

Source: https://github.com/rh-aiservices-bu/rhoai-nightly/blob/main/README.md

To apply template changes to existing applications, delete the application using 'oc delete' and then sync the recreated application using 'make sync-app'. This forces the ApplicationSet to recreate the application with the updated template.

```bash
# Delete the app - ApplicationSet will recreate it with new template
oc delete application.argoproj.io/<app-name> -n openshift-gitops

# Then sync the recreated app
make sync-app APP=<app-name>
```

--------------------------------

### Apply Kustomize Manifests Locally

Source: https://github.com/rh-aiservices-bu/rhoai-nightly/blob/main/CLAUDE.md

Test component manifests locally using `oc apply -k` before committing changes.

```bash
oc apply -k components/path/to/component
```

--------------------------------

### Check GPU Node Information

Source: https://github.com/rh-aiservices-bu/rhoai-nightly/blob/main/CLAUDE.md

List nodes with GPUs and describe their capacity and allocatable resources.

```bash
# Check GPU
oc get nodes -l nvidia.com/gpu.present=true
oc describe node -l node-role.kubernetes.io/gpu | grep -A5 "Capacity:|Allocatable:"
```

--------------------------------

### Correct Basic Authentication Format

Source: https://github.com/rh-aiservices-bu/rhoai-nightly/blob/main/CLAUDE.md

Use environment variables for sensitive credentials like tokens in basic authentication strings.

```bash
# CORRECT
--auth-basic="${QUAY_USER}:${QUAY_TOKEN}"
```

--------------------------------

### Create CPU MachineSet

Source: https://github.com/rh-aiservices-bu/rhoai-nightly/blob/main/README.md

Creates a MachineSet for CPU worker nodes using the m6a.4xlarge instance type. The command waits for the nodes to become Ready.

```bash
make cpu
```

--------------------------------

### Recreating an Application via ApplicationSet

Source: https://github.com/rh-aiservices-bu/rhoai-nightly/blob/main/CLAUDE.md

To apply template changes from an ApplicationSet to an existing application, you can delete the application. The ApplicationSet will then automatically recreate it using the updated template. This is often followed by `make sync-app`.

```bash
# Option 1: Delete and let ApplicationSet recreate
oc delete application.argoproj.io/<app-name> -n openshift-gitops
# ApplicationSet recreates it with new template
make sync-app APP=<app-name>
```

--------------------------------

### Create GPU MachineSet

Source: https://github.com/rh-aiservices-bu/rhoai-nightly/blob/main/README.md

Creates a MachineSet for GPU worker nodes using the g6e.2xlarge instance type. The command waits for the nodes to become Ready.

```bash
make gpu
```

--------------------------------

### Adding a New Component with Kustomization

Source: https://github.com/rh-aiservices-bu/rhoai-nightly/blob/main/CLAUDE.md

Steps to add a new operator component, including creating its directory, defining its Kustomization with remote resources, optionally adding patches, and committing the changes.

```bash
# 1. Create component directory
mkdir -p components/operators/my-operator

# 2. Create kustomization.yaml
cat > components/operators/my-operator/kustomization.yaml <<EOF
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - https://github.com/redhat-cop/gitops-catalog/path/to/operator?ref=main
EOF

# 3. (Optional) Add patches
cat > components/operators/my-operator/patch.yaml <<EOF
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: my-operator
spec:
  channel: stable
EOF

# Update kustomization.yaml to include patch
cat >> components/operators/my-operator/kustomization.yaml <<EOF
patches:
  - target:
      kind: Subscription
      name: my-operator
    path: patch.yaml
EOF

# 4. Commit and push
git add components/operators/my-operator/
git commit -m "Add my-operator"
git push

```

--------------------------------

### Deploy Models to MaaS

Source: https://github.com/rh-aiservices-bu/rhoai-nightly/blob/main/README.md

Deploys AI models to the MaaS platform. It supports autodetecting the best model based on GPU VRAM, or explicitly deploying specific models like simulator, gpt-oss-20b, or granite-tiny-gpu. Each model can be configured with free and premium subscription tiers.

```bash
make maas-model                         # Autodetect: picks by cluster GPU VRAM
                                        #   no GPU           -> simulator
                                        #   GPU VRAM >=40 Gi -> gpt-oss-20b
                                        #   otherwise        -> granite-tiny-gpu
make maas-model MODEL=auto              # Same; explicit
make maas-model MODEL=simulator         # Deploy simulator only (CPU)
make maas-model MODEL=gpt-oss-20b       # Deploy gpt-oss-20b (GPU)
make maas-model MODEL=granite-tiny-gpu  # Deploy Granite tiny (GPU)
```

--------------------------------

### Common Commands: Validation & Monitoring

Source: https://github.com/rh-aiservices-bu/rhoai-nightly/blob/main/CLAUDE.md

Commands for quick cluster readiness checks, validating configuration, checking ArgoCD status, and performing diagnostics.

```bash
make preflight       # Quick cluster readiness check (connection, nodes, basics)
make validate-config # Validate .env against cluster capabilities
make status          # Show ArgoCD application sync status
make diagnose        # Comprehensive cluster diagnosis (full health check)
```

--------------------------------

### Permanent Solution for ArgoCD Sync Issues

Source: https://github.com/rh-aiservices-bu/rhoai-nightly/blob/main/CLAUDE.md

Configure the GitOps repository URL and branch permanently, then commit the changes.

```bash
make configure-repo GITOPS_REPO_URL=your-repo-url GITOPS_BRANCH=your-branch-name
```

--------------------------------

### Quick Cluster Readiness Check

Source: https://github.com/rh-aiservices-bu/rhoai-nightly/blob/main/README.md

Performs a quick preflight check to assess the readiness of the cluster for RHOAI deployment.

```bash
make preflight
```

--------------------------------

### Deploy with .env Override

Source: https://github.com/rh-aiservices-bu/rhoai-nightly/blob/main/CLAUDE.md

Deploys the application using a persistent .env file override for the GitOps branch. This avoids direct file edits and persists across sessions.

```bash
echo "GITOPS_BRANCH=my-feature-branch" >> .env
make deploy
```

--------------------------------

### Check ImageContentSourcePolicy and MachineConfigPool

Source: https://github.com/rh-aiservices-bu/rhoai-nightly/blob/main/CLAUDE.md

Inspect ImageContentSourcePolicies and the status of MachineConfigPools.

```bash
# Check ICSP
oc get imagecontentsourcepolicy
oc get mcp  # MachineConfigPool status
```

--------------------------------

### Check ArgoCD Application Status

Source: https://github.com/rh-aiservices-bu/rhoai-nightly/blob/main/CLAUDE.md

Verifies that the `instance-maas-observability` ArgoCD application is created and running.

```bash
# ArgoCD app
oc get application.argoproj.io/instance-maas-observability -n openshift-gitops
```

--------------------------------

### Deploy a MaaS Model

Source: https://github.com/rh-aiservices-bu/rhoai-nightly/blob/main/README.md

Deploys a model to the MaaS platform. The default behavior is to automatically select a model based on the cluster's GPU VRAM.

```bash
make maas-model
```

--------------------------------

### ArgoCD Sync Options Annotation

Source: https://github.com/rh-aiservices-bu/rhoai-nightly/blob/main/CLAUDE.md

Use the `SkipDryRunOnMissingResource` sync option annotation for resources that may not exist initially.

```yaml
commonAnnotations:
  argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true
```

--------------------------------

### Deploy with Inline Environment Variable

Source: https://github.com/rh-aiservices-bu/rhoai-nightly/blob/main/CLAUDE.md

Deploys the application using an inline environment variable for the GitOps branch. This is ephemeral and suitable for test runs or feature branches.

```bash
GITOPS_BRANCH=my-feature-branch make deploy
```

--------------------------------

### Deploy ArgoCD Applications (Sync Disabled)

Source: https://github.com/rh-aiservices-bu/rhoai-nightly/blob/main/README.md

Deploys cluster-config and ApplicationSets using ArgoCD. Synchronization is disabled by default, allowing for manual review before syncing.

```bash
make deploy
```

--------------------------------

### Configure Default Models

Source: https://github.com/rh-aiservices-bu/rhoai-nightly/blob/main/README.md

Sets the default AI models to be used by the MaaS platform. This configuration is typically done in the `.env` file.

```bash
MAAS_MODELS=gpt-oss-20b granite-tiny-gpu
```

--------------------------------

### Deploy Models to MaaS

Source: https://github.com/rh-aiservices-bu/rhoai-nightly/blob/main/CLAUDE.md

Deploys models to the MaaS platform. Defaults to auto-detection of cluster GPU VRAM. Can be overridden by specifying MODEL or MAAS_MODELS.

```bash
make maas-model
# Override: make maas-model MODEL=simulator
# Or set MAAS_MODELS in .env: MAAS_MODELS=gpt-oss-20b granite-tiny-gpu
```

--------------------------------

### Verify MaaS Deployment

Source: https://github.com/rh-aiservices-bu/rhoai-nightly/blob/main/README.md

Runs a full end-to-end test of the MaaS platform. This includes deploying a temporary model, testing API and authentication, rate limits, and then cleaning up the test resources.

```bash
make maas-verify   # Full end-to-end test (deploys temp model, tests API/auth/rate limits, cleans up)
```

--------------------------------

### Incremental Deployment: Monitor Deployment

Source: https://github.com/rh-aiservices-bu/rhoai-nightly/blob/main/CLAUDE.md

Commands to check the status of ArgoCD applications and perform comprehensive cluster diagnostics.

```bash
make status          # Show ArgoCD application sync status
make diagnose        # Comprehensive cluster diagnosis
```

--------------------------------

### MaaS Debugging Commands

Source: https://github.com/rh-aiservices-bu/rhoai-nightly/blob/main/CLAUDE.md

A collection of oc and curl commands to check the health and status of MaaS infrastructure, operator-managed components, models, and Authorino SSL configuration.

```bash
# Check MaaS infrastructure
oc get application.argoproj.io/instance-maas -n openshift-gitops
oc get gateway maas-default-gateway -n openshift-ingress
oc get gatewayclass openshift-default

# Check operator-managed MaaS
oc get modelsasservice -A
oc get deployment maas-api maas-controller -n redhat-ods-applications
oc get pods -n redhat-ods-applications | grep -E "maas|postgres|payload"

# Check models
oc get llminferenceservice -n llm
oc get maasmodelref -n llm -o wide
oc get maassubscription -n models-as-a-service
oc get maasauthpolicy -n models-as-a-service
oc get pods -n llm

# Check health
curl -sk https://maas.<cluster-domain>/maas-api/health
# 200 = healthy, 401 = auth working (unauthenticated)

# Check Authorino SSL
oc get deployment authorino -n kuadrant-system -o jsonpath='{.spec.template.spec.containers[0].env}'
```

--------------------------------

### Check ArgoCD Status

Source: https://github.com/rh-aiservices-bu/rhoai-nightly/blob/main/CLAUDE.md

List ArgoCD pods and applications within the openshift-gitops namespace.

```bash
# Check ArgoCD
oc get pods -n openshift-gitops
oc get applications.argoproj.io -n openshift-gitops
oc get applicationsets.argoproj.io -n openshift-gitops
```

--------------------------------

### Kustomization with Remote References and Patches

Source: https://github.com/rh-aiservices-bu/rhoai-nightly/blob/main/CLAUDE.md

This Kustomization configuration pulls resources from a remote Git repository and applies local patches. It allows leveraging upstream catalog definitions while customizing them locally, with version pinning via git refs.

```yaml
# components/operators/nfd/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - https://github.com/redhat-cop/gitops-catalog/nfd/operator/overlays/stable?ref=main

patches:
  - target:
      kind: Subscription
      name: nfd
    path: patch-channel.yaml
```

--------------------------------

### Switching from Manual to External Secrets Mode

Source: https://github.com/rh-aiservices-bu/rhoai-nightly/blob/main/CLAUDE.md

To switch to External Secrets mode, ensure QUAY credentials are not set in `.env` and that you have bootstrap repo access. Running `make secrets` will then configure the system to use External Secrets.

```bash
# Clear credentials from .env, ensure bootstrap repo access
make secrets  # Detects no credentials, uses External Secrets
```

--------------------------------

### End-to-End Observability Diagnosis

Source: https://github.com/rh-aiservices-bu/rhoai-nightly/blob/main/CLAUDE.md

Provides commands for end-to-end verification of MaaS observability, including triggering inference and using the `make diagnose` command.

```bash
# End-to-end: fire inference, wait ~2 min, open RHOAI console -> Observability dashboard
# Or run:
make diagnose  # section 9 = Observability
```

--------------------------------

### Check RHOAI Components

Source: https://github.com/rh-aiservices-bu/rhoai-nightly/blob/main/CLAUDE.md

Verify the status of RHOAI DataScienceClusters and associated pods.

```bash
# Check RHOAI
oc get datascienceclusters -A
oc get pods -n redhat-ods-operator
oc get pods -n redhat-ods-applications
```

--------------------------------

### Switch to Manual Secrets Mode

Source: https://github.com/rh-aiservices-bu/rhoai-nightly/blob/main/README.md

To switch to manual mode, set QUAY_USER and QUAY_TOKEN in your .env file and run 'make secrets'. This command detects credentials, removes the ExternalSecret configuration, and enables manual secret management.

```bash
make secrets  # Detects credentials, deletes ExternalSecret, uses manual mode
```

--------------------------------

### Common Commands: Maintenance Operations - Refresh and Restart

Source: https://github.com/rh-aiservices-bu/rhoai-nightly/blob/main/CLAUDE.md

Commands for refreshing GitOps configurations, restarting catalog pods and operators, and managing ArgoCD sync settings.

```bash
make refresh         # GitOps refresh - pull latest from git (no sync)
                     # Use to see what changed in git vs cluster

make restart-catalog # Restart catalog pod and RHOAI operator
                     # Use after syncing new catalog image from git

make sync-disable    # Disable ArgoCD auto-sync (for manual changes)
make sync-enable     # Re-enable ArgoCD auto-sync
make refresh-apps    # Refresh from git AND sync all apps (one-time)
                     # Use when auto-sync is disabled to apply latest from git
```

--------------------------------

### Validate Kustomize Manifests

Source: https://github.com/rh-aiservices-bu/rhoai-nightly/blob/main/CLAUDE.md

Validate Kustomize manifests locally to check for potential issues.

```bash
oc kustomize components/path/to/component
```

--------------------------------

### Check Kuadrant Observability Enabled

Source: https://github.com/rh-aiservices-bu/rhoai-nightly/blob/main/CLAUDE.md

Confirms if the Kuadrant CR has observability enabled, which is a prerequisite for MaaS observability.

```bash
# Kuadrant observability enabled
oc get kuadrant -n kuadrant-system -o jsonpath='{.items[0].spec.observability.enable}'
```

--------------------------------

### Sync a Single ArgoCD Application

Source: https://github.com/rh-aiservices-bu/rhoai-nightly/blob/main/README.md

Synchronizes a specific ArgoCD application, identified by its name. Replace 'nfd' with the actual application name.

```bash
make sync-app APP=nfd
```

--------------------------------

### Staged Synchronization of All Apps

Source: https://github.com/rh-aiservices-bu/rhoai-nightly/blob/main/README.md

Performs a staged synchronization of all ArgoCD applications in dependency order. Auto-sync is enabled on each app after it becomes healthy. This is the recommended sync method.

```bash
make sync
```

--------------------------------

### Check ServiceMonitors and Services

Source: https://github.com/rh-aiservices-bu/rhoai-nightly/blob/main/CLAUDE.md

Verifies the existence of ServiceMonitors and related Services for various components, including those managed by MaaS observability.

```bash
# ServiceMonitors
oc get servicemonitor,podmonitor -n kuadrant-system
oc get servicemonitor,service -n openshift-ingress -l app.kubernetes.io/part-of=maas-observability
oc get servicemonitor kserve-llm-models -n llm  # scrapes vllm:* metrics from LLMInferenceService pods
```

--------------------------------

### Perform Full MaaS End-to-End Verification

Source: https://github.com/rh-aiservices-bu/rhoai-nightly/blob/main/README.md

Runs a comprehensive end-to-end verification process for the MaaS platform to ensure all components are functioning correctly.

```bash
make maas-verify
```

--------------------------------

### Incorrect Basic Authentication Format

Source: https://github.com/rh-aiservices-bu/rhoai-nightly/blob/main/CLAUDE.md

Avoid hardcoding sensitive credentials directly into authentication strings.

```bash
# WRONG - never do this
--auth-basic="user:actualpassword123"
```

--------------------------------

### Common Commands: Maintenance Operations - Scaling and Master Nodes

Source: https://github.com/rh-aiservices-bu/rhoai-nightly/blob/main/CLAUDE.md

Commands for scaling MachineSets and dedicating master nodes. Scaling requires specifying the MachineSet name and desired replica count.

```bash
make scale NAME=<machineset> REPLICAS=<N|+N|-N>
                     # Scale a MachineSet
                     # Examples:
                     #   make scale NAME=gpu-g6e-2xlarge REPLICAS=2
                     #   make scale NAME=gpu-g6e-2xlarge REPLICAS=+1
                     #   make scale NAME=cpu-m6a-4xlarge REPLICAS=-1

make dedicate-masters # Remove worker role from master nodes
                      # Must have Ready worker nodes first
```

--------------------------------

### Control Auto-Sync for Applications

Source: https://github.com/rh-aiservices-bu/rhoai-nightly/blob/main/README.md

Manages the auto-sync feature for applications managed by ArgoCD after a `make sync` operation. Commands allow disabling, enabling, syncing individual apps, or performing a one-time refresh and sync of all applications.

```bash
make sync-disable                        # Disable auto-sync (for manual changes)
make sync-enable                         # Re-enable auto-sync
make sync-app APP=<name>                 # Sync single app + enable auto-sync on it
make refresh-apps                        # Refresh from git AND sync all apps (one-time)
```

--------------------------------

### Check Cluster Connection

Source: https://github.com/rh-aiservices-bu/rhoai-nightly/blob/main/CLAUDE.md

Verify your connection to the OpenShift cluster and list available nodes.

```bash
# Check cluster connection
oc whoami --show-server
oc get nodes
```

--------------------------------

### Comprehensive Cluster Diagnosis

Source: https://github.com/rh-aiservices-bu/rhoai-nightly/blob/main/README.md

Runs a detailed diagnostic tool to identify and report potential issues within the OpenShift cluster.

```bash
make diagnose
```

--------------------------------

### Manage MaaS Models

Source: https://github.com/rh-aiservices-bu/rhoai-nightly/blob/main/README.md

Provides commands to manage deployed AI models within the MaaS platform. You can view the status of all deployed models or delete specific models, including all of them.

```bash
make maas-model-status                   # Show all deployed models
make maas-model-delete MODEL=simulator   # Delete one model
make maas-model-delete MODEL=all         # Delete all models
```

--------------------------------

### Verify GitOps Resources

Source: https://github.com/rh-aiservices-bu/rhoai-nightly/blob/main/CLAUDE.md

Checks for the presence of Kuadrant TelemetryPolicy and Istio Telemetry resources, which are managed via GitOps.

```bash
# GitOps resources
oc get telemetrypolicies.extensions.kuadrant.io maas-telemetry -n openshift-ingress
oc get telemetry.telemetry.istio.io latency-per-subscription -n openshift-ingress
```

--------------------------------

### Show ArgoCD Application Status

Source: https://github.com/rh-aiservices-bu/rhoai-nightly/blob/main/README.md

Displays the current synchronization status of all applications managed by ArgoCD.

```bash
make status
```

--------------------------------

### Ephemeral Feature Branch Deployment

Source: https://github.com/rh-aiservices-bu/rhoai-nightly/blob/main/README.md

Push a feature branch and deploy it using GITOPS_BRANCH environment variable. This method is recommended for pull requests and test runs as it avoids file edits and commits.

```bash
git push origin my-feature-branch
GITOPS_BRANCH=my-feature-branch make deploy
GITOPS_BRANCH=my-feature-branch make sync
```

```bash
GITOPS_BRANCH=my-feature-branch make refresh-apps
```

--------------------------------

### Login to OpenShift Cluster

Source: https://github.com/rh-aiservices-bu/rhoai-nightly/blob/main/README.md

Logs into the OpenShift cluster using the provided token and server URL. Ensure you have the correct token and API server address.

```bash
oc login --token=<token> --server=https://api.<cluster>:6443
```

--------------------------------

### Restart Catalog Pod and Operator

Source: https://github.com/rh-aiservices-bu/rhoai-nightly/blob/main/README.md

Restarts the catalog pod and its associated operator, forcing an image pull for the latest version.

```bash
make restart-catalog
```

--------------------------------

### Switching from External Secrets to Manual Mode

Source: https://github.com/rh-aiservices-bu/rhoai-nightly/blob/main/CLAUDE.md

To switch from External Secrets to manual credential management, delete the ExternalSecret resource and then run `make secrets`. This ensures the system uses the manually set QUAY credentials.

```bash
# ExternalSecret will be ignored when QUAY_USER/QUAY_TOKEN are set
# The script detects ExternalSecret and skips if present, or you can delete it:
oc delete externalsecret pull-secret -n openshift-config
make secrets  # Uses manual mode
```

--------------------------------

### Verify Eval Hub Resources

Source: https://github.com/rh-aiservices-bu/rhoai-nightly/blob/main/CLAUDE.md

Commands to check the reconciliation status of Eval Hub resources and their associated components.

```bash
# Eval-hub resources reconciled
oc get evalhub,mlflow,dspa -A
oc get evalhub evalhub -n redhat-ods-applications -o jsonpath='{.status.phase}'

# Pods Running
oc get pods -n redhat-ods-applications | grep -E 'evalhub|mlflow'
oc get pods -n evalhub-tenant

# Hook Job completed
oc get job update-secret-minio -n evalhub-tenant

# DSPA pipeline routes
oc get route -n evalhub-tenant

# ArgoCD Application status
oc get application.argoproj.io/instance-evalhub -n openshift-gitops
```

--------------------------------

### RHOAI Nightly Repository Structure

Source: https://github.com/rh-aiservices-bu/rhoai-nightly/blob/main/CLAUDE.md

Overview of the directory structure for the RHOAI Nightly repository, detailing the purpose of each directory and its sub-files.

```directory structure
rhoai-nightly/
├── scripts/                              # Pre-GitOps setup scripts
│   ├── add-pull-secret.sh               # Add quay.io/rhoai pull credentials
│   ├── create-icsp.sh                   # Create ImageContentSourcePolicy
│   ├── create-gpu-machineset.sh         # Create GPU MachineSet (g6e.2xlarge)
│   ├── create-cpu-machineset.sh         # Create CPU worker MachineSet (m6a.4xlarge)
│   ├── install-gitops.sh                # Install GitOps operator + ArgoCD
│   ├── deploy-apps.sh                   # Deploy root app (triggers GitOps)
│   ├── enable-uwm.sh                    # Enable User Workload Monitoring (part of `make infra`)
│   ├── install-maas.sh                  # Install MaaS platform ONLY (secrets, ArgoCD app, Authorino) — observability is separate
│   ├── uninstall-maas.sh               # Remove MaaS platform
│   ├── install-observability.sh        # Install/uninstall MaaS observability (Kuadrant, ServiceMonitors; requires UWM from make infra)
│   ├── install-evalhub.sh              # Install/uninstall eval-hub via dedicated instance-evalhub ArgoCD Application
│   ├── restart-catalog.sh              # Restart RHOAI catalog + re-resolve Subscription after catalog image flip
│   ├── setup-maas-model.sh             # Deploy/delete/status MaaS models
│   ├── verify-maas.sh                  # End-to-end MaaS verification
│   ├── status.sh                        # Show ArgoCD app status
│   ├── diagnose.sh                      # Comprehensive cluster diagnosis
│   ├── preflight-check.sh              # Quick cluster readiness check
│   ├── validate-config.sh              # Validate .env against cluster capabilities
│   ├── scale-machineset.sh              # Scale MachineSets
│   └── configure-repo.sh                # Update repo URLs for forks
│
├── bootstrap/                            # Bootstrap resources (applied by scripts)
│   ├── gitops-operator/                 # GitOps operator subscription
│   ├── argocd-instance/                 # ArgoCD instance config
│   ├── gpu-machineset/                  # GPU MachineSet templates
│   ├── cpu-machineset/                  # CPU MachineSet templates
│   ├── icsp/                            # ImageContentSourcePolicy
│   ├── external-secrets/                # ExternalSecret for pull-secret (Merge mode)
│   ├── cluster-monitoring-config/       # Enables user-workload monitoring (applied by scripts/enable-uwm.sh during `make infra`)
│   └── rhoaibu-cluster-nightly/         # Cluster-specific config
│
├── clusters/                             # Cluster definitions
│   ├── base/
│   │   └── kustomization.yaml           # Points to ArgoCD apps
│   └── overlays/
│       └── rhoaibu-cluster-nightly/     # Cluster-specific overlay
│
├── components/                           # GitOps-managed components
│   ├── argocd/
│   │   └── apps/
│   │       ├── cluster-operators-appset.yaml     # ApplicationSet for operators
│   │       ├── cluster-oper-instances-appset.yaml # ApplicationSet for instances
│   │       └── kustomization.yaml
│   │
│   ├── operators/                       # Operator subscriptions
│   │   ├── nfd/                         # Node Feature Discovery
│   │   ├── nvidia-operator/             # NVIDIA GPU Operator
│   │   ├── rhoai-operator/              # RHOAI operator + nightly catalog
│   │   ├── openshift-service-mesh/      # Service Mesh
│   │   ├── kueue-operator/              # Kueue for job queueing
│   │   ├── jobset-operator/             # JobSet for distributed workloads
│   │   ├── leader-worker-set/           # Leader-Worker pattern
│   │   ├── connectivity-link/           # Connectivity Link
│   │   └── cluster-observability-operator/ # COO (Perses CRDs for Observability dashboard)
│   │
│   └── instances/                       # Operator instances/configs
│       ├── nfd-instance/                # NFD NodeFeatureDiscovery CR
│       ├── nvidia-instance/             # ClusterPolicy for GPU
│       ├── rhoai-instance/              # base + overlays (maas, maas-observability)
│       ├── jobset-instance/             # JobSet config
│       ├── leader-worker-set-instance/  # Leader-Worker config
│       ├── connectivity-link-instance/  # Connectivity Link config
│       ├── maas-instance/               # MaaS Helm chart (PostgreSQL+PVC, Gateway)
│       ├── maas-observability/          # MaaS observability (TelemetryPolicy + Istio Telemetry)
│       └── maas-models/                # MaaS model manifests (kustomize)
│           ├── simulator/              # CPU-only mock model
│           ├── gpt-oss-20b/            # OpenAI gpt-oss-20b (GPU, vLLM CUDA)
│           └── granite-tiny-gpu/       # Granite 4.0-h-tiny FP8 (GPU, vLLM CUDA)
│
├── Makefile                             # Automation targets
├── .env.example                         # Configuration template
├── .gitignore                           # Git ignore rules
└── README.md                            # User documentation

```

--------------------------------

### Apply and Restart RHOAI Catalog

Source: https://github.com/rh-aiservices-bu/rhoai-nightly/blob/main/CLAUDE.md

Applies changes to ArgoCD if it's running and restarts the catalog pod to pull the new image.

```bash
# If ArgoCD is already running, apply changes
make refresh-apps
make restart-catalog  # Force catalog pod to pull new image
```

--------------------------------

### GitOps Refresh (No Sync)

Source: https://github.com/rh-aiservices-bu/rhoai-nightly/blob/main/README.md

Refreshes the GitOps configuration by pulling the latest changes from the Git repository without performing an automatic synchronization.

```bash
make refresh
```

--------------------------------

### Verify ArgoCD Application Status

Source: https://github.com/rh-aiservices-bu/rhoai-nightly/blob/main/CLAUDE.md

Check the status of ArgoCD applications in the openshift-gitops namespace. This helps verify if deployments are syncing correctly.

```bash
oc get applications.argoproj.io -n openshift-gitops
oc get application.argoproj.io/my-operator -n openshift-gitops
```

--------------------------------

### Validate .env Configuration

Source: https://github.com/rh-aiservices-bu/rhoai-nightly/blob/main/README.md

Validates the settings in the `.env` file against the capabilities of the current OpenShift cluster.

```bash
make validate-config
```

--------------------------------

### Check UWM Enabled Status

Source: https://github.com/rh-aiservices-bu/rhoai-nightly/blob/main/CLAUDE.md

Verifies if User Workload Monitoring (UWM) is enabled on the cluster by checking the cluster-monitoring-config ConfigMap and listing UWM pods.

```bash
# UWM enabled
oc get cm cluster-monitoring-config -n openshift-monitoring -o yaml | grep enableUserWorkload
oc get pods -n openshift-user-workload-monitoring
```

--------------------------------

### Scale MachineSet

Source: https://github.com/rh-aiservices-bu/rhoai-nightly/blob/main/README.md

Scales a specified MachineSet to a desired number of replicas. Replace NAME and N with the MachineSet name and replica count.

```bash
make scale NAME=<machineset> REPLICAS=N
```

--------------------------------

### Show Deployed MaaS Model Status

Source: https://github.com/rh-aiservices-bu/rhoai-nightly/blob/main/README.md

Displays the current status of the deployed model within the MaaS platform.

```bash
make maas-model-status
```

--------------------------------

### Scan for Potential Secrets in Scripts and YAML Files

Source: https://github.com/rh-aiservices-bu/rhoai-nightly/blob/main/CLAUDE.md

Use grep to recursively search for patterns indicating potential secrets like passwords or tokens within .sh and .yaml files.

```bash
grep -r -i "password=\|token=\|auth.*:" --include="*.sh" --include="*.yaml" .
```

--------------------------------

### Persistent GitOps Branch Override

Source: https://github.com/rh-aiservices-bu/rhoai-nightly/blob/main/README.md

Configure GITOPS_BRANCH and optionally GITOPS_REPO_URL in a .env file for persistent overrides. The Makefile automatically sources .env, applying these settings to all subsequent make commands.

```bash
cat >> .env <<EOF
GITOPS_BRANCH=my-feature-branch
GITOPS_REPO_URL=https://github.com/my-fork/rhoai-nightly
EOF
make deploy
```

--------------------------------

### Add Execute Permissions to Scripts

Source: https://github.com/rh-aiservices-bu/rhoai-nightly/blob/main/CLAUDE.md

Ensure scripts have execute permissions. The Makefile automatically handles this with `chmod +x`.

```bash
ls -la scripts/
```

--------------------------------

### Incremental Deployment: Dedicate Master Nodes

Source: https://github.com/rh-aiservices-bu/rhoai-nightly/blob/main/CLAUDE.md

Optional step to remove the worker role from master nodes after worker nodes are ready. Verification is required.

```bash
make dedicate-masters # Remove worker role from masters
                      # VERIFY: oc get nodes (masters no longer have worker role)
```

--------------------------------

### Deploy and Disable Auto-Sync

Source: https://github.com/rh-aiservices-bu/rhoai-nightly/blob/main/README.md

Performs a full deployment and then disables ArgoCD's auto-synchronization. Use this if you plan to make manual cluster changes after deployment.

```bash
make all sync-disable
```

--------------------------------

### Remove MaaS Platform

Source: https://github.com/rh-aiservices-bu/rhoai-nightly/blob/main/README.md

Uninstalls and removes the MaaS platform components from the cluster.

```bash
make maas-uninstall
```

--------------------------------

### Commit and Push Git Changes

Source: https://github.com/rh-aiservices-bu/rhoai-nightly/blob/main/CLAUDE.md

Commits and pushes changes after editing RHOAI configuration files. Includes diff, add, commit, and push commands.

```bash
# After editing, commit and push
git diff
git add -A && git commit -m "Update RHOAI catalog image"
git push
```

--------------------------------

### Uninstall MaaS Platform

Source: https://github.com/rh-aiservices-bu/rhoai-nightly/blob/main/README.md

Removes the MaaS platform from the cluster. This process cascade-deletes associated resources like the Gateway, PostgreSQL, and secrets.

```bash
make maas-uninstall   # Remove MaaS platform (cascade-deletes Gateway, PostgreSQL, secrets)
```

--------------------------------

### Check MachineSet and Machine Status

Source: https://github.com/rh-aiservices-bu/rhoai-nightly/blob/main/CLAUDE.md

List MachineSets and Machines managed by the OpenShift Machine API.

```bash
# Check MachineSets
oc get machinesets -n openshift-machine-api
oc get machines -n openshift-machine-api
```

--------------------------------

### MaaS Uninstall Flow Script

Source: https://github.com/rh-aiservices-bu/rhoai-nightly/blob/main/CLAUDE.md

The uninstall-maas.sh script handles the removal of MaaS components. It removes Authorino SSL environment variables, deletes the instance-maas ArgoCD Application, removes PostgreSQL secrets, and cleans up stale DNS records.

```bash
uninstall-maas.sh:
  1. Remove Authorino SSL env vars
  2. Delete instance-maas ArgoCD Application (cascade-deletes all Helm resources)
  3. Delete PostgreSQL secrets
  4. Clean up stale DNS records
```

--------------------------------

### Check Operator Status

Source: https://github.com/rh-aiservices-bu/rhoai-nightly/blob/main/CLAUDE.md

List ClusterServiceVersions (CSVs) and subscriptions for RHOAI, NVIDIA, and NFD operators.

```bash
# Check operators
oc get csv -A | grep -E "rhoai|nvidia|nfd"
oc get subscriptions -A
```

--------------------------------

### Ephemeral Fix for ArgoCD Sync Issues

Source: https://github.com/rh-aiservices-bu/rhoai-nightly/blob/main/CLAUDE.md

Temporarily patch ArgoCD with the correct branch for syncing. This is an in-place fix and does not require a YAML commit.

```bash
GITOPS_BRANCH=correct-branch make deploy
```

--------------------------------

### Delete Models from MaaS

Source: https://github.com/rh-aiservices-bu/rhoai-nightly/blob/main/CLAUDE.md

Deletes models from the MaaS platform, using the same MODEL= or MAAS_MODELS logic for selection.

```bash
make maas-model-delete
```

--------------------------------

### Check for Base64 Encoded Strings

Source: https://github.com/rh-aiservices-bu/rhoai-nightly/blob/main/CLAUDE.md

Scan for long base64 encoded strings in .sh and .yaml files, which might represent encoded secrets.

```bash
grep -r -E "[A-Za-z0-9+/]{40,}={0,2}" --include="*.sh" --include="*.yaml" .
```

--------------------------------

### Review Staged Git Changes

Source: https://github.com/rh-aiservices-bu/rhoai-nightly/blob/main/CLAUDE.md

Use 'git diff --staged' to review all changes that are about to be committed, ensuring no sensitive data is included.

```bash
git diff --staged
```