### Install MaaS Skill from Specific Phase Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/content/modules/ROOT/pages/claude-code.adoc Starts the MaaS installation process from a specified phase. Useful for resuming or targeting specific stages of the setup. ```bash /install-maas --from-phase 4 ``` -------------------------------- ### AI-Assisted Installation with Arguments Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/content/modules/ROOT/pages/quick-start.adoc Run the AI-assisted installation with specific arguments, such as model selection and observability. This allows customization of the AI-guided setup. ```bash /install-maas --model simulator --with-observability ``` -------------------------------- ### Clone and Run Automated Setup Script Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/content/modules/ROOT/pages/quick-start.adoc Clone the repository and execute the main setup script to install MaaS components from scratch. This script handles the entire lifecycle, skipping already completed phases. ```bash git clone https://github.com/rh-aiservices-bu/rhoai-maas-guide.git cd rhoai-maas-guide ./scripts/setup-maas.sh ``` -------------------------------- ### Deploy Only Models Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/content/modules/ROOT/pages/quick-start.adoc Use the setup script to deploy only models, skipping all prior installation phases. Assumes the MaaS platform is already set up. ```bash ./scripts/setup-maas.sh --from-phase 5 ``` -------------------------------- ### AI-Assisted Installation Command Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/content/modules/ROOT/pages/quick-start.adoc Initiate an AI-assisted installation workflow using the `/install-maas` command. This command wraps the `setup-maas.sh` script with an interactive AI process. ```bash /install-maas ``` -------------------------------- ### Install MaaS Skill with Model Argument Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/content/modules/ROOT/pages/claude-code.adoc Invokes the install-maas skill specifying a particular model to deploy. Use this to control which model is installed during the MaaS setup. ```bash /install-maas --model simulator ``` -------------------------------- ### Install MaaS Skill with Observability Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/content/modules/ROOT/pages/claude-code.adoc Includes observability components (COO and telemetry) during the MaaS installation. This ensures monitoring is set up as part of the deployment. ```bash /install-maas --with-observability ``` -------------------------------- ### Run MaaS Setup with Observability Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/README.md Include observability components, such as Cluster Observability Operator and Gateway telemetry, during the RHOAI Models-as-a-Service deployment. ```bash ./scripts/setup-maas.sh --with-observability ``` -------------------------------- ### Install Required Operator Subscriptions Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/content/modules/ROOT/pages/01-prerequisites.adoc Apply the Kubernetes configuration to install the five core operator subscriptions. OLM will handle the resolution and installation. ```bash oc apply -k manifests/01-prerequisites/operators/ ``` -------------------------------- ### Clone Repository and Navigate Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/content/modules/ROOT/pages/index.adoc Clone the RHOAI MaaS guide repository and change into its root directory. All subsequent commands and file paths are relative to this directory. ```bash git clone https://github.com/rh-aiservices-bu/rhoai-maas-guide.git cd rhoai-maas-guide ``` -------------------------------- ### Run Verification Only Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/content/modules/ROOT/pages/quick-start.adoc Execute only the verification phase (Phase 6) of the setup script. This is useful for testing a deployed MaaS environment. ```bash ./scripts/setup-maas.sh --from-phase 6 ``` -------------------------------- ### Resume MaaS Setup from Specific Phase Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/README.md Use the --from-phase flag to continue the RHOAI Models-as-a-Service deployment from a particular stage, useful after interruptions. ```bash ./scripts/setup-maas.sh --from-phase 4 ``` -------------------------------- ### Install Optional GPU Operators Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/content/modules/ROOT/pages/01-prerequisites.adoc If your cluster has GPU nodes, apply the configuration to install the necessary GPU operators for GPU workload support. ```bash oc apply -k manifests/01-prerequisites/gpu/ ``` -------------------------------- ### Run End-to-End MaaS Setup Script Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/README.md Execute the main automation script to deploy RHOAI Models-as-a-Service. This script is idempotent and can resume from specific phases. ```bash ./scripts/setup-maas.sh ``` -------------------------------- ### Install Cluster Observability Operator Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/content/modules/ROOT/pages/07-observability.adoc Apply the Cluster Observability Operator using its kustomization configuration. ```bash oc apply -k manifests/07-observability/coo/ ``` -------------------------------- ### Install MetalLB Operator for Non-Cloud Clusters Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/content/modules/ROOT/pages/01-prerequisites.adoc For baremetal, OpenStack, or RHPDS/SNO environments, install MetalLB to provide LoadBalancer IP addresses for the MaaS Gateway. ```bash oc apply -k manifests/01-prerequisites/metallb/ ``` -------------------------------- ### Check MetalLB Installation Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/content/modules/ROOT/pages/02-platform-config.adoc Verifies if MetalLB is installed by checking for its operator deployment. If MetalLB is not found, it indicates that a cloud load balancer is missing, which is required for the Gateway to become Programmed. ```bash oc get deployment metallb-operator-controller-manager -n metallb-system 2>/dev/null ``` -------------------------------- ### Verify Cluster Observability Operator Installation Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/content/modules/ROOT/pages/07-observability.adoc Check if the Cluster Observability Operator is installed and in a 'Succeeded' state by listing its CSV. ```bash oc get csv -n openshift-cluster-observability-operator | grep cluster-observability-operator # Expected: cluster-observability-operator Succeeded ``` -------------------------------- ### Run Verification Directly Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/content/modules/ROOT/pages/quick-start.adoc Execute the verification script directly. This bypasses the main setup script and runs only the verification checks. ```bash ./manifests/06-verification/verify.sh ``` -------------------------------- ### Apply DSC Initialization and DSC (Stage 1) Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/content/modules/ROOT/pages/04-rhoai-config.adoc Apply the DSCInitialization and DataScienceCluster resources to trigger operator reconciliation. This is the first stage for a fresh install. ```bash # Stage 1: DSC and DSCI (triggers operator reconciliation) oc apply -f manifests/04-rhoai-config/dscinitialization.yaml oc apply -f manifests/04-rhoai-config/datasciencecluster.yaml ``` -------------------------------- ### Install MaaS Skill Skipping Models and Verification Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/content/modules/ROOT/pages/claude-code.adoc Invokes the install-maas skill while skipping the model deployment and verification phases. Use this if models are already deployed or verification is not needed. ```bash /install-maas --skip-models --skip-verify ``` -------------------------------- ### Apply Gateway Telemetry Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/content/modules/ROOT/pages/07-observability.adoc Apply the Gateway Telemetry configuration after the Cluster Observability Operator is installed and the MaaS gateway is running. ```bash oc apply -k manifests/07-observability/telemetry/ ``` -------------------------------- ### Check MaaS Auth Policy and Subscription Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/content/modules/ROOT/pages/05-maas-models.adoc Use `oc get` to check the status of MaaS authentication policies and subscriptions in the `models-as-a-service` namespace. ```bash oc get maasauthpolicy -n models-as-a-service oc get maassubscription -n models-as-a-service ``` -------------------------------- ### Apply Dashboard Configuration (Stage 2) Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/content/modules/ROOT/pages/04-rhoai-config.adoc Apply the OdhDashboardConfig resource after the operator has reconciled and CRDs are available. This is the second stage for a fresh install. ```bash # Stage 2: Dashboard config (requires OdhDashboardConfig CRD to exist) oc apply -f manifests/04-rhoai-config/odh-dashboard-config.yaml ``` -------------------------------- ### Check MaaS CRDs Installation Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/content/modules/ROOT/pages/04-rhoai-config.adoc Verify that the necessary Custom Resource Definitions (CRDs) for MaaS have been installed by the operator. These are expected when modelsAsService is set to Managed. ```bash oc get crd | grep maas.opendatahub.io ``` -------------------------------- ### Get MaaS API Domain Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/content/modules/ROOT/pages/05-maas-models.adoc Retrieves the cluster domain to construct the MaaS API URL. ```bash CLUSTER_DOMAIN=$(oc get ingresses.config/cluster -o jsonpath='{.spec.domain}') MAAS_URL="maas.${CLUSTER_DOMAIN}" ``` -------------------------------- ### Get Model Endpoint URL Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/content/modules/ROOT/pages/05-maas-models.adoc Extracts the specific URL for a model from the list of available models. ```bash MODEL_URL=$(curl -sk "https://${MAAS_URL}/maas-api/v1/models" \ -H "Authorization: Bearer ${API_KEY}" | jq -r '.data[0].url') ``` -------------------------------- ### List Operator Subscriptions and CSVs Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/content/modules/ROOT/pages/01-prerequisites.adoc Lists all operator subscriptions across all namespaces and displays their current CSV and state. Useful for a quick status check of installed operators. ```bash oc get subscriptions.operators.coreos.com -A \ -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name,CSV:.status.currentCSV,STATE:.status.state' ``` -------------------------------- ### Wait for Required Operator CSVs to Succeed Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/content/modules/ROOT/pages/01-prerequisites.adoc Verify that the core RHOAI operator ClusterServiceVersions (CSVs) have reached the 'Succeeded' phase, indicating successful installation. ```bash oc wait csv -n redhat-ods-operator -l operators.coreos.com/rhods-operator.redhat-ods-operator="" \ --for=jsonpath='{.status.phase}'=Succeeded --timeout=600s oc wait csv -n openshift-operators -l operators.coreos.com/rhcl-operator.openshift-operators="" \ --for=jsonpath='{.status.phase}'=Succeeded --timeout=600s oc wait csv -n cert-manager-operator -l operators.coreos.com/openshift-cert-manager-operator.cert-manager-operator="" \ --for=jsonpath='{.status.phase}'=Succeeded --timeout=600s ``` -------------------------------- ### Check maas-api Deployment Status Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/content/modules/ROOT/pages/04-rhoai-config.adoc Confirm that the maas-api deployment has successfully rolled out. This deployment should start healthy if PostgreSQL and the maas-db-config secret are already configured. ```bash oc rollout status deployment/maas-api -n redhat-ods-applications --timeout=120s ``` -------------------------------- ### Wait for Authorino Deployment Availability Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/content/modules/ROOT/pages/02-platform-config.adoc Wait for the Authorino deployment to become available after configuring TLS. This confirms that the Authorino pods have started successfully with the new TLS settings. ```bash oc wait --for=condition=Available deployment/authorino -n kuadrant-system --timeout=300s ``` -------------------------------- ### Get Authorino TLS Secret Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/content/modules/ROOT/pages/02-platform-config.adoc Retrieve the authorino-server-cert TLS Secret, which was generated by the service-ca operator based on the service annotation. ```bash oc get secret authorino-server-cert -n kuadrant-system ``` -------------------------------- ### Get Infrastructure Platform Type Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/content/modules/ROOT/pages/02-platform-config.adoc Determines the platform type of the OpenShift cluster. This is crucial for deciding whether additional configuration for non-cloud platforms is needed. ```bash oc get infrastructure cluster -o jsonpath='{.status.platform}' ``` -------------------------------- ### Wait for NVIDIA GPU Operator CSV Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/content/modules/ROOT/pages/01-prerequisites.adoc Waits for the NVIDIA GPU Operator's CSV to reach the 'Succeeded' phase. Verifies the GPU operator is installed and ready. ```bash oc wait csv -n nvidia-gpu-operator -l operators.coreos.com/gpu-operator-certified.nvidia-gpu-operator="" \ --for=jsonpath='{.status.phase}'=Succeeded --timeout=600s ``` -------------------------------- ### Wait for Prometheus Operator Availability Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/content/modules/ROOT/pages/02-platform-config.adoc Wait for the prometheus-operator deployment in the user-workload-monitoring namespace to become available. This indicates that the User Workload Monitoring stack has started successfully. ```bash oc wait --for=condition=Available deployment/prometheus-operator \ -n openshift-user-workload-monitoring --timeout=300s ``` -------------------------------- ### Verify RHOAI Operator CSV Status Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/content/modules/ROOT/pages/04-rhoai-config.adoc Check if the Red Hat OpenShift AI operator's ClusterServiceVersion (CSV) is in a 'Succeeded' state. This confirms the operator is installed and ready. ```bash oc get csv -n redhat-ods-operator | grep rhods-operator # STATUS should be "Succeeded" ``` -------------------------------- ### Deploy PostgreSQL using Kustomize Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/content/modules/ROOT/pages/03-maas-platform.adoc Apply the PostgreSQL manifests using Kustomize for deployment. ```bash oc apply -k manifests/03-maas-platform/ ``` -------------------------------- ### Check Core MaaS Deployments Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/content/modules/ROOT/pages/06-verification.adoc Verifies the readiness of core MaaS deployments: PostgreSQL, maas-api, and maas-controller. ```bash # Core deployments oc get deployment postgres maas-api maas-controller -n redhat-ods-applications ``` -------------------------------- ### Deploy Simulator Model Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/content/modules/ROOT/pages/05-maas-models.adoc Deploy the simulator model, which is CPU-only and recommended for initial testing without requiring GPU infrastructure. ```bash oc apply -k manifests/05-maas-models/simulator/ ``` -------------------------------- ### Deploy PostgreSQL Individually Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/content/modules/ROOT/pages/03-maas-platform.adoc Apply individual PostgreSQL manifest files if not using Kustomize. ```bash oc apply -f manifests/03-maas-platform/postgres-pvc.yaml oc apply -f manifests/03-maas-platform/postgres-service.yaml oc apply -f manifests/03-maas-platform/postgres-deployment.yaml ``` -------------------------------- ### Troubleshoot PostgreSQL Pod Startup Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/content/modules/ROOT/pages/03-maas-platform.adoc Describe and log the PostgreSQL pod to diagnose startup issues. Common causes include missing secrets or PVC binding problems. ```bash oc describe pod -n redhat-ods-applications -l app=postgres oc logs -n redhat-ods-applications -l app=postgres ``` -------------------------------- ### Apply Kuadrant Manifests Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/content/modules/ROOT/pages/02-platform-config.adoc Apply the namespace, service annotation, and Kuadrant CR manifests sequentially. Wait for Kuadrant to become ready before proceeding. ```bash oc apply -f manifests/02-platform-config/kuadrant/namespace.yaml oc apply -f manifests/02-platform-config/kuadrant/service-annotation.yaml oc apply -f manifests/02-platform-config/kuadrant/kuadrant.yaml ``` -------------------------------- ### Verify Model Readiness Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/content/modules/ROOT/pages/05-maas-models.adoc Check the status of deployed models by verifying the LLMInferenceService, running pods, and MaaSModelRef registration. ```bash # Check LLMInferenceService status oc get llminferenceservice -n llm ``` ```bash # Check that model pods are running oc get pods -n llm ``` ```bash # Check MaaSModelRef registration oc get maasmodelref -n llm -o wide ``` -------------------------------- ### Verify PostgreSQL Pod Readiness Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/content/modules/ROOT/pages/03-maas-platform.adoc Wait for the PostgreSQL pod to become available and confirm it is running. ```bash # Wait for PostgreSQL pod to be ready oc wait --for=condition=Available deployment/postgres \ -n redhat-ods-applications --timeout=120s # Confirm the pod is running oc get pods -n redhat-ods-applications -l app=postgres ``` -------------------------------- ### Check Authorino Deployment Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/content/modules/ROOT/pages/06-verification.adoc Verifies the readiness of the Authorino deployment, which handles authorization. ```bash # Authorino oc get deployment authorino -n kuadrant-system ``` -------------------------------- ### Apply MetalLB Configuration Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/content/modules/ROOT/pages/01-prerequisites.adoc Apply the MetalLB instance configuration, including the IPAddressPool and L2Advertisement CRs, to enable LoadBalancer services. ```bash oc apply -f manifests/01-prerequisites/metallb/metallb.yaml ``` -------------------------------- ### Check Authorino Deployment and Certificate Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/content/modules/ROOT/pages/02-platform-config.adoc Verify that the Authorino deployment is running and its TLS certificate secret exists. ```bash oc get deployment authorino -n kuadrant-system oc get secret authorino-server-cert -n kuadrant-system ``` -------------------------------- ### Check DSCInitialization Status Conditions Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/content/modules/ROOT/pages/04-rhoai-config.adoc Inspect the status conditions of the default DSCInitialization resource to understand its current state and any potential issues. ```bash oc get dscinitializations default-dsci -o jsonpath='{.status.conditions}' | jq . ``` -------------------------------- ### Deploy GPU Models Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/content/modules/ROOT/pages/05-maas-models.adoc Deploy GPU-accelerated models. Choose the appropriate command based on your cluster's GPU VRAM capacity. ```bash # For clusters with modest GPU (8GB+ VRAM) oc apply -k manifests/05-maas-models/granite-tiny-gpu/ ``` ```bash # For clusters with larger GPU (16GB+ VRAM, A10G or better) oc apply -k manifests/05-maas-models/gpt-oss-20b/ ``` -------------------------------- ### Create LLM Namespace Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/content/modules/ROOT/pages/05-maas-models.adoc Create the `llm` namespace if it does not already exist before deploying models. ```bash oc create namespace llm ``` -------------------------------- ### Create Gateway with Dynamic Variables Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/manifests/03-maas-platform/openshift-gateway-setup/README.md Create the Gateway resource by substituting cluster domain and certificate name placeholders. This ensures the gateway is correctly configured for the specific OpenShift environment. ```bash export CLUSTER_DOMAIN=$(oc get ingresses.config.openshift.io cluster -o jsonpath='{.spec.domain}') export CERT_NAME=$(oc get ingresscontroller default -n openshift-ingress-operator \ -o jsonpath='{.spec.defaultCertificate.name}' 2>/dev/null) export CERT_NAME=${CERT_NAME:-router-certs-default} envsubst < manifests/04-maas-platform/openshift-gateway-setup/gateway.yaml.tmpl | oc apply -f - ``` -------------------------------- ### Troubleshoot DSC Conditions and Events Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/content/modules/ROOT/pages/04-rhoai-config.adoc If RHOAI components are not becoming ready, use 'oc describe' on the DataScienceCluster to inspect its conditions and associated events for clues. ```bash # Inspect DSC conditions and events oc describe datasciencecluster default-dsc ``` -------------------------------- ### Annotate Gateway for Authorino TLS Bootstrap Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/content/modules/ROOT/pages/02-platform-config.adoc Annotates the MaaS Gateway with `security.opendatahub.io/authorino-tls-bootstrap="true"` to enable Authorino TLS bootstrap. The `--overwrite` flag ensures the annotation is applied even if it already exists. ```bash oc annotate gateway maas-default-gateway -n openshift-ingress security.opendatahub.io/authorino-tls-bootstrap="true" --overwrite ``` -------------------------------- ### List Available Models via MaaS API Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/content/modules/ROOT/pages/05-maas-models.adoc Fetches a list of available models from the MaaS API using an authorization token. ```bash curl -sk "https://${MAAS_URL}/maas-api/v1/models" \ -H "Authorization: Bearer ${API_KEY}" ``` -------------------------------- ### Run Full MaaS Verification Script Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/content/modules/ROOT/pages/06-verification.adoc Executes the complete verification process, including deployment, testing, and cleanup of temporary resources. ```bash # Full verification (deploy, test, cleanup) ./manifests/06-verification/verify.sh ``` -------------------------------- ### Deploy Model Using Standalone Script Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/content/modules/ROOT/pages/quick-start.adoc Deploy a specific model using the standalone `deploy-model.sh` script. This offers more granular control for model deployment. ```bash ./scripts/deploy-model.sh --model simulator ``` -------------------------------- ### Create MaaS API Key Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/content/modules/ROOT/pages/05-maas-models.adoc Creates a new API key for accessing the MaaS API. Requires the `maas-api` to be running. ```bash API_KEY=$(curl -sk -X POST "https://${MAAS_URL}/maas-api/v1/api-keys" \ -H "Authorization: Bearer $(oc whoami -t)" \ -H "Content-Type: application/json" \ -d '{"name": "test-key"}' | jq -r '.key') ``` -------------------------------- ### Configure MetalLB IP Range using Environment Substitution Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/content/modules/ROOT/pages/01-prerequisites.adoc Dynamically set the MetalLB IP range using the node's internal IP for SNO clusters or a custom range for multi-node clusters, then apply the configuration. ```bash export METALLB_IP_RANGE="$(oc get nodes -o jsonpath='{.items[0].status.addresses[?(@.type=="InternalIP")].address}')-$(oc get nodes -o jsonpath='{.items[0].status.addresses[?(@.type=="InternalIP")].address}')" echo $METALLB_IP_RANGE envsubst < manifests/03-maas-platform/openshift-gateway-setup/metallb-config.yaml | oc apply -f - ``` -------------------------------- ### Verify Gateway, Service, and Route Status Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/manifests/03-maas-platform/openshift-gateway-setup/README.md Check the status of the Gateway, LoadBalancer service, and Route to ensure they have been successfully created and configured. This includes verifying the Gateway is Programmed, the service has an external IP, and the route is created. ```bash # Gateway should be Programmed oc get gateways -n openshift-ingress oc wait --for=condition=Programmed gateway/maas-default-gateway -n openshift-ingress --timeout=60s # LoadBalancer service should have an external IP oc get svc -n openshift-ingress | grep maas # Route should be created oc get route -n openshift-ingress | grep maas ``` -------------------------------- ### List Available MaaS Models Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/content/modules/ROOT/pages/06-verification.adoc Retrieves a list of available models from the MaaS API using a provided API key. ```bash # List models (use the key from the previous response) curl -sk -H "Authorization: Bearer " "${HOST}/maas-api/v1/models" ``` -------------------------------- ### Check Simulator Model Resources Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/content/modules/ROOT/pages/06-verification.adoc Verifies the existence and status of various Kubernetes resources related to the deployed simulator model. ```bash oc get llminferenceservice -n llm oc get maasmodelref -n llm -o wide oc get maasauthpolicy -n models-as-a-service oc get maassubscription -n models-as-a-service oc get pods -n llm ``` -------------------------------- ### Apply User Workload Monitoring Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/content/modules/ROOT/pages/02-platform-config.adoc Apply the User Workload Monitoring configuration using kustomize. This enables Prometheus to scrape metrics from user namespaces, including MaaS and Kuadrant. ```bash oc apply -k manifests/02-platform-config/uwm/ ``` -------------------------------- ### Configure MetalLB with IP Range Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/manifests/03-maas-platform/openshift-gateway-setup/README.md Apply the MetalLB configuration, replacing the IP range placeholder. This step is necessary when the Gateway API LoadBalancer Service remains in a Pending state due to the absence of a cloud load-balancer controller. ```bash export METALLB_IP_RANGE="192.168.1.240-192.168.1.250" envsubst < manifests/04-maas-platform/openshift-gateway-setup/metallb-config.yaml | oc apply -f - ``` -------------------------------- ### Set Authorino TLS Environment Variables Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/content/modules/ROOT/pages/02-platform-config.adoc Configure the Authorino deployment with environment variables for TLS certificate paths. This ensures Authorino can correctly load and use the generated TLS certificate. ```bash oc -n kuadrant-system set env deployment/authorino \ SSL_CERT_FILE=/etc/ssl/certs/openshift-service-ca/service-ca-bundle.crt \ REQUESTS_CA_BUNDLE=/etc/ssl/certs/openshift-service-ca/service-ca-bundle.crt ``` -------------------------------- ### RHOAI Manifests Directory Structure Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/content/modules/ROOT/pages/01-prerequisites.adoc Illustrates the typical directory structure for RHOAI manifests, including operators, GPU components, and MetalLB. ```text manifests/01-prerequisites/ operators/ kustomization.yaml # aggregates all operator subdirectories cert-manager/ connectivity-link/ leader-worker-set/ rhoai-operator/ service-mesh/ gpu/ kustomization.yaml # aggregates NFD + NVIDIA nfd/ nvidia-operator/ metallb/ kustomization.yaml # MetalLB namespace + operatorgroup + subscription namespace.yaml operatorgroup.yaml subscription.yaml metallb.yaml # MetalLB CR (applied after CSV succeeds) ``` -------------------------------- ### Verify GatewayClass Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/content/modules/ROOT/pages/02-platform-config.adoc Retrieves and displays the details of the `openshift-default` GatewayClass. Confirms that the controller is accepted and shows the age of the resource. ```bash oc get gatewayclass openshift-default ``` -------------------------------- ### Apply GatewayClass Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/content/modules/ROOT/pages/02-platform-config.adoc Applies the GatewayClass manifest to initialize OpenShift's built-in Gateway API controller. This is a prerequisite for using the Gateway API. ```bash oc apply -f manifests/02-platform-config/gatewayclass.yaml ``` -------------------------------- ### Create PostgreSQL and MaaS DB Secrets Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/content/modules/ROOT/pages/03-maas-platform.adoc Manually create the necessary PostgreSQL credentials and MaaS database connection URL secrets. A random password is generated for security. ```bash # Generate a random password POSTGRES_PASSWORD=$(openssl rand -base64 16 | tr -d '=+/') # Create postgres-creds (consumed by the PostgreSQL deployment) oc create secret generic postgres-creds \ -n redhat-ods-applications \ --from-literal=POSTGRES_USER=maas \ --from-literal=POSTGRES_PASSWORD="${POSTGRES_PASSWORD}" \ --from-literal=POSTGRES_DB=maas # Create maas-db-config (consumed by maas-api) oc create secret generic maas-db-config \ -n redhat-ods-applications \ --from-literal=DB_CONNECTION_URL="postgresql://maas:${POSTGRES_PASSWORD}@postgres:5432/maas?sslmode=disable" ``` -------------------------------- ### Create MaaS API Key Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/content/modules/ROOT/pages/06-verification.adoc Creates a new API key for accessing MaaS, specifying its name, expiration, and associated subscription. ```bash DOMAIN=$(oc get ingresses.config/cluster -o jsonpath='{.spec.domain}') HOST="https://maas.${DOMAIN}" # Create API key curl -sk -X POST \ -H "Authorization: Bearer $(oc whoami -t)" \ -H "Content-Type: application/json" \ -d '{"name":"test","expiresIn":"1h","subscription":"simulator-subscription"}' \ "${HOST}/maas-api/v1/api-keys" ``` -------------------------------- ### Verify Passthrough Route Creation Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/content/modules/ROOT/pages/02-platform-config.adoc Verifies that the passthrough Route for the MaaS Gateway has been created successfully. It displays the route details, including its host, path, services, and termination type. ```bash oc get route maas-default-gateway-https -n openshift-ingress ``` -------------------------------- ### Check Pods in Applications Namespace Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/content/modules/ROOT/pages/04-rhoai-config.adoc Use this command to verify that the pods within the redhat-ods-applications namespace are running correctly. ```bash oc get pods -n redhat-ods-applications ``` -------------------------------- ### Run MaaS Verification Script Without Cleanup Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/content/modules/ROOT/pages/06-verification.adoc Runs the verification tests but preserves all deployed test resources, including the simulator model. ```bash # Keep test resources after verification ./manifests/06-verification/verify.sh --no-cleanup ``` -------------------------------- ### Configure Authorino TLS Listener Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/content/modules/ROOT/pages/02-platform-config.adoc Patch the Authorino custom resource to enable the TLS listener and specify the certificate secret reference. This secures communication between the Gateway and Authorino. ```bash oc patch authorino authorino -n kuadrant-system --type=merge --patch '{ "spec": { "listener": { "tls": { "enabled": true, "certSecretRef": { "name": "authorino-server-cert" } } } } }' ``` -------------------------------- ### Create OpenShift Route Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/manifests/03-maas-platform/openshift-gateway-setup/README.md Apply the OpenShift Route configuration, substituting the cluster domain placeholder. This makes the MaaS Gateway accessible externally via an OpenShift Route. ```bash envsubst < manifests/04-maas-platform/openshift-gateway-setup/route.yaml.tmpl | oc apply -f - ``` -------------------------------- ### Check Gateway Programming Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/content/modules/ROOT/pages/02-platform-config.adoc Verify that the MaaS default gateway is programmed and has an assigned address. ```bash oc get gateway maas-default-gateway -n openshift-ingress ``` -------------------------------- ### Check Default Tenant CR Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/content/modules/ROOT/pages/04-rhoai-config.adoc Verify the existence of the default tenant custom resource created by the maas-controller in the models-as-a-service namespace. The status may initially show Ready=False. ```bash oc get tenant default-tenant -n models-as-a-service ``` -------------------------------- ### Check OdhDashboardConfig Flags Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/content/modules/ROOT/pages/04-rhoai-config.adoc Retrieve and inspect the dashboard configuration for the OdhDashboardConfig resource to verify that MaaS and related features are enabled. ```bash oc get odhdashboardconfig odh-dashboard-config -n redhat-ods-applications \ -o jsonpath='{.spec.dashboardConfig}' | jq . ``` -------------------------------- ### Apply Passthrough Route for Non-Cloud Platforms Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/content/modules/ROOT/pages/02-platform-config.adoc Applies a passthrough Route for non-cloud platforms (None, BareMetal, OpenStack). This Route directs external traffic through the OpenShift ingress controller to the Gateway's LoadBalancer service, making the Gateway accessible externally. ```bash export CLUSTER_DOMAIN=$(oc get ingresses.config.openshift.io cluster -o jsonpath='{.spec.domain}') envsubst '${CLUSTER_DOMAIN}' < manifests/03-maas-platform/openshift-gateway-setup/route.yaml.tmpl | oc apply -f - ``` -------------------------------- ### Verify Gateway Programmed Status Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/content/modules/ROOT/pages/03-maas-platform.adoc Confirm that the 'maas-default-gateway' has been programmed. ```bash oc wait --for=condition=Programmed gateway/maas-default-gateway \ -n openshift-ingress --timeout=60s ``` -------------------------------- ### Run MaaS Verification Script Cleanup Only Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/content/modules/ROOT/pages/06-verification.adoc Removes any leftover test resources from a previous, potentially interrupted, verification run. ```bash # Remove leftover test resources from a previous run ./manifests/06-verification/verify.sh --cleanup-only ``` -------------------------------- ### Apply MaaS Gateway Manifest Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/content/modules/ROOT/pages/02-platform-config.adoc Applies the MaaS Gateway manifest after rendering it with cluster-specific domain and certificate name. This command uses `envsubst` to substitute environment variables into the template. ```bash envsubst '${CLUSTER_DOMAIN} ${CERT_NAME}' < manifests/02-platform-config/gateway.yaml.tmpl | oc apply -f - ``` -------------------------------- ### Verify Istio Telemetry Existence Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/content/modules/ROOT/pages/07-observability.adoc Check for the existence of the 'latency-per-subscription' Istio Telemetry resource in the 'openshift-ingress' namespace. ```bash oc get telemetry.telemetry.istio.io latency-per-subscription -n openshift-ingress ``` -------------------------------- ### Wait for Model Controller (MaaS) to be Ready Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/content/modules/ROOT/pages/04-rhoai-config.adoc Poll the DataScienceCluster status until the Model Controller (MaaS) component is marked as ready. This check can take up to 5 minutes. ```bash # Wait for Model Controller (MaaS) to be ready (up to 5 minutes) oc wait --for=jsonpath='{.status.conditions[?(@.type=="ModelControllerReady")].status}'=True \ datasciencecluster/default-dsc --timeout=300s ``` -------------------------------- ### Cleanup Gateway Resources Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/manifests/03-maas-platform/openshift-gateway-setup/README.md Delete all gateway-related resources, including the Route, Gateway, and MetalLB configuration, using the cleanup manifest. ```bash oc delete -f manifests/04-maas-platform/openshift-gateway-setup/cleanup.yaml ``` -------------------------------- ### Check Gateway Status Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/content/modules/ROOT/pages/06-verification.adoc Verifies the 'Programmed' condition of the default MaaS gateway, indicating its readiness. ```bash # Gateway oc get gateway maas-default-gateway -n openshift-ingress \ -o jsonpath='{.status.conditions[?(@.type=="Programmed")].status}' ``` -------------------------------- ### Verify Prometheus Pods Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/content/modules/ROOT/pages/02-platform-config.adoc Checks if the necessary Prometheus pods are running in the specified namespace. Ensure `prometheus-user-workload-0` and `thanos-ruler-user-workload-0` are in a Running state. ```bash oc get pods -n openshift-user-workload-monitoring ``` -------------------------------- ### Check Passthrough Route Admission Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/content/modules/ROOT/pages/02-platform-config.adoc Verify that the passthrough HTTPS route for the gateway is admitted. This is relevant for bare metal, Open Stack, or None platform types. ```bash oc get route maas-default-gateway-https -n openshift-ingress -o jsonpath='{.status.ingress[0].conditions[?(@.type=="Admitted")].status}' ``` -------------------------------- ### Verify PostgreSQL Secrets Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/content/modules/ROOT/pages/03-maas-platform.adoc Check for the existence of the 'postgres-creds' and 'maas-db-config' secrets. ```bash # Verify both secrets exist oc get secret postgres-creds -n redhat-ods-applications oc get secret maas-db-config -n redhat-ods-applications ``` -------------------------------- ### Wait for KServe to be Ready Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/content/modules/ROOT/pages/04-rhoai-config.adoc Poll the DataScienceCluster status until the KServe component is marked as ready. This check can take up to 5 minutes. ```bash # Wait for KServe to be ready (up to 5 minutes) oc wait --for=jsonpath='{.status.conditions[?(@.type=="KserveReady")].status}'=True \ datasciencecluster/default-dsc --timeout=300s ``` -------------------------------- ### Check Kuadrant Status Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/content/modules/ROOT/pages/02-platform-config.adoc Verify that the Kuadrant operator is running in the cluster. ```bash oc get kuadrant -n kuadrant-system ``` -------------------------------- ### Remove a Model Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/content/modules/ROOT/pages/05-maas-models.adoc Deletes a model from the MaaS system using `oc delete` with a kustomization file. ```bash oc delete -k manifests/05-maas-models/simulator/ ``` -------------------------------- ### Wait for Kuadrant to be Ready Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/content/modules/ROOT/pages/02-platform-config.adoc Wait for the Kuadrant custom resource to reach the Ready condition. This ensures the Kuadrant operator has successfully deployed its components. ```bash oc wait --for=condition=Ready kuadrant/kuadrant -n kuadrant-system --timeout=120s ``` -------------------------------- ### Verify TLS Connection to Gateway Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/content/modules/ROOT/pages/02-platform-config.adoc Test the TLS connection to the MaaS gateway using curl to ensure it's properly secured. ```bash CLUSTER_DOMAIN=$(oc get ingresses.config.openshift.io cluster -o jsonpath='{.spec.domain}')n}') curl -vsk https://maas.${CLUSTER_DOMAIN} 2>&1 | grep -E "SSL connection|Connected" ``` -------------------------------- ### Wait for GatewayClass Acceptance Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/content/modules/ROOT/pages/02-platform-config.adoc Waits for the applied GatewayClass to be accepted by the controller. This ensures the GatewayClass is ready for use. The timeout is set to 120 seconds. ```bash oc wait --for=condition=Accepted gatewayclass/openshift-default --timeout=120s ``` -------------------------------- ### Wait for Gateway to be Programmed Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/content/modules/ROOT/pages/02-platform-config.adoc Waits for the MaaS Gateway to reach the `Programmed` condition. This indicates that the Gateway is fully configured and ready to receive traffic. The timeout is set to 120 seconds. ```bash oc wait --for=condition=Programmed gateway/maas-default-gateway -n openshift-ingress --timeout=120s ``` -------------------------------- ### Wait for Node Feature Discovery Operator CSV Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/content/modules/ROOT/pages/01-prerequisites.adoc Waits for the Node Feature Discovery (NFD) Operator's CSV to reach the 'Succeeded' phase. Essential for GPU operator prerequisites. ```bash oc wait csv -n openshift-nfd -l operators.coreos.com/nfd.openshift-nfd="" \ --for=jsonpath='{.status.phase}'=Succeeded --timeout=600s ``` -------------------------------- ### Test Rate Limiting Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/content/modules/ROOT/pages/06-verification.adoc Sends multiple rapid requests to a model endpoint to trigger rate limiting and checks for HTTP 429 responses. ```bash # Send rapid requests and watch for 429 for i in $(seq 1 16); do curl -sk -o /dev/null -w '%{http_code}\n' \ -H "Authorization: Bearer " \ -H "Content-Type: application/json" \ -d '{"model":"facebook/opt-125m","messages":[{"role":"user","content":"Hello, write a long essay"}],"max_tokens":50}' \ "/v1/chat/completions" done ``` -------------------------------- ### Test Authentication Enforcement Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/content/modules/ROOT/pages/06-verification.adoc Sends requests to a model endpoint without a token and with an invalid token to verify they are rejected with HTTP 401 or 403. ```bash # Should return 401 or 403 curl -sk -o /dev/null -w '%{http_code}' \ -H "Content-Type: application/json" \ -d '{"model":"facebook/opt-125m","messages":[{"role":"user","content":"Hello"}]}' \ "/v1/chat/completions" ``` -------------------------------- ### Verify TelemetryPolicy Existence Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/content/modules/ROOT/pages/07-observability.adoc Check for the existence of the 'maas-telemetry' TelemetryPolicy in the 'openshift-ingress' namespace. ```bash oc get telemetrypolicies.extensions.kuadrant.io maas-telemetry -n openshift-ingress ``` -------------------------------- ### Check Service Mesh 3 CRD Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/content/modules/ROOT/pages/01-prerequisites.adoc Verifies the existence of the Istio CRD for Service Mesh 3. ```bash oc get crd istios.sailoperator.io ``` -------------------------------- ### Send Chat Completion Request Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/content/modules/ROOT/pages/05-maas-models.adoc Sends a chat completion request to a specific model endpoint using the MaaS API. ```bash curl -sk "${MODEL_URL}/v1/chat/completions" \ -H "Authorization: Bearer ${API_KEY}" \ -H "Content-Type: application/json" \ -d '{ "model": "facebook/opt-125m", "messages": [{"role": "user", "content": "Hello!"}] }' ``` -------------------------------- ### Wait for Leader-Worker Set Operator CSV Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/content/modules/ROOT/pages/01-prerequisites.adoc Waits for the Leader-Worker Set Operator's CSV to reach the 'Succeeded' phase. This confirms the operator's successful deployment. ```bash oc wait csv -n openshift-lws-operator -l operators.coreos.com/leader-worker-set.openshift-lws-operator="" \ --for=jsonpath='{.status.phase}'=Succeeded --timeout=600s ``` -------------------------------- ### Check cert-manager CRD Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/content/modules/ROOT/pages/01-prerequisites.adoc Verifies the existence of the Certificate CRD from the cert-manager operator. ```bash oc get crd certificates.cert-manager.io ``` -------------------------------- ### Check Connectivity Link / Kuadrant CRD Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/content/modules/ROOT/pages/01-prerequisites.adoc Verifies the existence of the Kuadrant CRD, often used for connectivity link management. ```bash oc get crd kuadrants.kuadrant.io ``` -------------------------------- ### Restart Kuadrant Operator on Dependency Error Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/content/modules/ROOT/pages/02-platform-config.adoc If Kuadrant reports a MissingDependency error, restart the Kuadrant operator pod and wait again for Kuadrant to become ready. This can resolve Istio race conditions. ```bash oc delete pod -n openshift-operators \ $(oc get pods -n openshift-operators --no-headers | grep kuadrant-operator | awk '{print $1}') oc wait --for=condition=Ready kuadrant/kuadrant -n kuadrant-system --timeout=180s ``` -------------------------------- ### Check MaaS Health Endpoint Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/content/modules/ROOT/pages/04-rhoai-config.adoc Query the MaaS API health endpoint to ensure the service is running and responding correctly. This requires retrieving the cluster domain. ```bash CLUSTER_DOMAIN=$(oc get ingresses.config.openshift.io cluster -o jsonpath='{.spec.domain}') curl -sk "https://maas.${CLUSTER_DOMAIN}/maas-api/health" # Expected: {"status":"healthy"} ``` -------------------------------- ### Check MaaS API Health Endpoint Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/content/modules/ROOT/pages/06-verification.adoc Sends a request to the MaaS API health endpoint to confirm its availability. ```bash # Health endpoint DOMAIN=$(oc get ingresses.config/cluster -o jsonpath='{.spec.domain}') curl -sk "https://maas.${DOMAIN}/maas-api/health" ``` -------------------------------- ### Wait for Service Mesh Operator CSV Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/content/modules/ROOT/pages/01-prerequisites.adoc Waits for the Service Mesh Operator's ClusterServiceVersion (CSV) to reach the 'Succeeded' phase. Use this to confirm the operator is ready. ```bash oc wait csv -n openshift-operators -l operators.coreos.com/servicemeshoperator3.openshift-operators="" \ --for=jsonpath='{.status.phase}'=Succeeded --timeout=600s ``` -------------------------------- ### Wait for MetalLB Operator CSV to Succeed Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/content/modules/ROOT/pages/01-prerequisites.adoc Wait for the MetalLB operator's ClusterServiceVersion (CSV) to reach the 'Succeeded' phase before proceeding with its configuration. ```bash oc wait csv -n metallb-system -l operators.coreos.com/metallb-operator.metallb-system="" \ --for=jsonpath='{.status.phase}'=Succeeded --timeout=120s ``` -------------------------------- ### Export Cluster Domain Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/content/modules/ROOT/pages/02-platform-config.adoc Exports the cluster domain to the CLUSTER_DOMAIN environment variable. This value is used in templated manifests for configuration. ```bash export CLUSTER_DOMAIN=$(oc get ingresses.config.openshift.io cluster -o jsonpath='{.spec.domain}') echo $CLUSTER_DOMAIN ``` -------------------------------- ### Check RHOAI CRD Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/content/modules/ROOT/pages/01-prerequisites.adoc Verifies the existence of the DataScienceCluster CRD for RHOAI. ```bash oc get crd datascienceclusters.datasciencecluster.opendatahub.io ``` -------------------------------- ### Export Certificate Name Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/content/modules/ROOT/pages/02-platform-config.adoc Exports the default certificate name used by the ingress controller to the CERT_NAME environment variable. This is used for TLS configuration in templated manifests. ```bash export CERT_NAME=$(oc get ingresscontroller default -n openshift-ingress-operator -o jsonpath='{.spec.defaultCertificate.name}' 2>/dev/null) export CERT_NAME="${CERT_NAME:-router-certs-default}" echo $CERT_NAME ``` -------------------------------- ### Troubleshoot Operator Logs Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/content/modules/ROOT/pages/04-rhoai-config.adoc Examine the logs of the RHOAI operator (rhods-operator) to diagnose issues related to its reconciliation or component management. ```bash # Check operator logs oc logs -n redhat-ods-operator deployment/rhods-operator --tail=100 ``` -------------------------------- ### Check DSC ModelsAsServiceReady Condition Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/content/modules/ROOT/pages/06-verification.adoc Checks the 'ModelsAsServiceReady' status condition of the default DataScienceCluster, indicating MaaS readiness. ```bash # DSC condition oc get datasciencecluster default-dsc \ -o jsonpath='{.status.conditions[?(@.type=="ModelsAsServiceReady")].status}' ``` -------------------------------- ### Check Leader Worker Set CRD Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/content/modules/ROOT/pages/01-prerequisites.adoc Verifies the existence of the LeaderWorkerSetOperator CRD. ```bash oc get crd leaderworkersetoperators.operator.openshift.io ``` -------------------------------- ### Monitor Cluster Observability Operator CSV Status Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/content/modules/ROOT/pages/07-observability.adoc Watch the Cluster Observability Operator's Cluster Service Version (CSV) to ensure it reaches the 'Succeeded' state. ```bash oc get csv -n openshift-cluster-observability-operator -w ``` -------------------------------- ### Verify Perses CRD Registration Source: https://github.com/rh-aiservices-bu/rhoai-maas-guide/blob/main/content/modules/ROOT/pages/07-observability.adoc Check if the Perses Custom Resource Definition (CRD) has been successfully registered in the cluster. ```bash oc get crd perses.perses.dev ``` === COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.