### Install Multiple Ceph OSD Charts (vdc example) Source: https://github.com/openstack/openstack-helm/blob/master/doc/source/upgrade/multiple-osd-releases.rst Install new ceph-osd charts using Helm, specifying unique chart names and corresponding override values for data and journal locations. This example demonstrates the installation for 'ceph-osd-vdc'. ```bash example2: for CHART in ceph-osd-vdc; do helm upgrade --install ${CHART} ${OSH_PATH}/${CHART} \ --namespace=ceph \ --values=/tmp/ceph.yaml \ ${OSH_EXTRA_HELM_ARGS} \ ${OSH_EXTRA_HELM_ARGS_CEPH_DEPLOY} osd: - data: type: block-logical location: /dev/vdc journal: type: block-logical location: /dev/vda2 ``` -------------------------------- ### Install OpenStack Barbican Source: https://github.com/openstack/openstack-helm/blob/master/doc/source/install/openstack.rst Use this command to install or upgrade the Barbican service. Ensure Barbican is installed before Tacker. ```bash helm upgrade --install barbican openstack-helm/barbican \ --namespace=openstack \ $(helm osh get-values-overrides -p ${OVERRIDES_DIR} -c barbican ${FEATURES}) ``` ```bash helm osh wait-for-pods openstack ``` -------------------------------- ### Install OpenStack Tacker Source: https://github.com/openstack/openstack-helm/blob/master/doc/source/install/openstack.rst Deploy the Tacker service for NFV Orchestration. Barbican must be installed prior to Tacker. ```bash helm upgrade --install tacker openstack-helm/tacker \ --namespace=openstack \ $(helm osh get-values-overrides -p ${OVERRIDES_DIR} -c tacker ${FEATURES}) ``` ```bash helm osh wait-for-pods openstack ``` -------------------------------- ### Install MetalLB Source: https://github.com/openstack/openstack-helm/blob/master/doc/source/install/prerequisites.rst Adds the MetalLB Helm repository and installs MetalLB into the 'metallb-system' namespace. Ensure the Helm repository is accessible. ```bash helm repo add metallb https://metallb.github.io/metallb helm install metallb metallb/metallb -n metallb-system ``` -------------------------------- ### Install OpenStack Client Locally Source: https://github.com/openstack/openstack-helm/blob/master/doc/source/install/openstack.rst Installs the OpenStack client in a Python virtual environment using pip. Activate the environment before proceeding. ```bash python3 -m venv ~/openstack-client source ~/openstack-client/bin/activate pip install python-openstackclient ``` -------------------------------- ### CHANGELOG.md Example with Tag Source: https://github.com/openstack/openstack-helm/blob/master/doc/source/specs/2025.1/chart_versioning.rst Provides an example of the CHANGELOG.md file after setting a new git tag, showing how release notes are organized under the new tag. ```markdown ## 2024.2.0- - Some new update ## 2024.2.0 - Some update - Previous update ``` -------------------------------- ### Install Infrastructure Backends with Helm Source: https://context7.com/openstack/openstack-helm/llms.txt Commands to install core infrastructure services like RabbitMQ, MariaDB, and Memcached using `helm upgrade --install` and the `helm osh` plugin for values overrides and readiness checks. ```bash # Set environment variables for the target OpenStack release export OPENSTACK_RELEASE=2025.1 export FEATURES="${OPENSTACK_RELEASE} ubuntu_noble" export OVERRIDES_DIR=$(pwd)/overrides # Download pre-built values overrides for the target feature set OVERRIDES_URL=https://opendev.org/openstack/openstack-helm/raw/branch/master/values_overrides for chart in rabbitmq mariadb memcached keystone glance cinder placement nova neutron horizon; do helm osh get-values-overrides -d -u ${OVERRIDES_URL} \ -p ${OVERRIDES_DIR} -c ${chart} ${FEATURES} done # Install infrastructure backends helm upgrade --install rabbitmq openstack-helm/rabbitmq \ --namespace=openstack \ --set pod.replicas.server=1 \ --timeout=600s \ $(helm osh get-values-overrides -p ${OVERRIDES_DIR} -c rabbitmq ${FEATURES}) helm osh wait-for-pods openstack helm upgrade --install mariadb openstack-helm/mariadb \ --namespace=openstack \ --set pod.replicas.server=1 \ $(helm osh get-values-overrides -p ${OVERRIDES_DIR} -c mariadb ${FEATURES}) helm osh wait-for-pods openstack helm upgrade --install memcached openstack-helm/memcached \ --namespace=openstack \ $(helm osh get-values-overrides -p ${OVERRIDES_DIR} -c memcached ${FEATURES}) helm osh wait-for-pods openstack # Install Keystone (mandatory) helm upgrade --install keystone openstack-helm/keystone \ --namespace=openstack \ $(helm osh get-values-overrides -p ${OVERRIDES_DIR} -c keystone ${FEATURES}) helm osh wait-for-pods openstack ``` -------------------------------- ### Install OpenStack Freezer Source: https://github.com/openstack/openstack-helm/blob/master/doc/source/install/openstack.rst Install the Freezer component, providing disaster recovery and backup-as-a-service for OpenStack. ```bash helm upgrade --install freezer openstack-helm/freezer \ --namespace=openstack \ $(helm osh get-values-overrides -p ${OVERRIDES_DIR} -c freezer ${FEATURES}) ``` -------------------------------- ### Install Kube-State-Metrics Chart Source: https://github.com/openstack/openstack-helm/blob/master/doc/source/monitoring/prometheus.rst Installs the prometheus-kube-state-metrics chart to gather additional Kubernetes object metrics. The namespace should typically be 'kube-system'. ```bash helm install --namespace=kube-system local/prometheus-kube-state-metrics --name=prometheus-kube-state-metrics ``` -------------------------------- ### Install Prometheus Chart Source: https://github.com/openstack/openstack-helm/blob/master/doc/source/monitoring/prometheus.rst Installs the Prometheus chart for automatic service discovery and metric gathering. Ensure the namespace is set appropriately for your deployment. ```bash helm install --namespace=openstack local/prometheus --name=prometheus ``` -------------------------------- ### Install Ubuntu HWE Kernel Source: https://github.com/openstack/openstack-helm/blob/master/doc/source/troubleshooting/ubuntu-hwe-kernel.rst Run these commands on each node before deploying Kubernetes and OpenStack-Helm to install the HWE kernel and reboot. ```shell #!/bin/bash sudo -H apt-get update sudo -H apt-get install -y linux-generic-hwe-16.04 sudo -H reboot now ``` -------------------------------- ### Example Test Integration Script Source: https://github.com/openstack/openstack-helm/blob/master/doc/source/testing/helm-tests.rst An example script demonstrating how Helm tests are included as part of the deployment process for new service charts. ```bash #!/bin/bash -e # Copyright 2019 Red Hat, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # This script is used to deploy the Keystone service chart and run its tests. # It assumes that the necessary prerequisites and environment variables are set. # Source common functions # shellcheck source=./common.sh source ./common.sh # Deploy Keystone chart helm upgrade --install keystone "${ HELM_CHART_DIR }/keystone" # Run Keystone Helm tests helm test keystone # Verify test results by checking pod logs # The test pod name is typically 'keystone-test' # Replace with the actual namespace where keystone is deployed # kubectl logs keystone-test -n # Example of checking logs (uncomment and modify as needed) # TEST_POD_NAME=$(kubectl get pods -l app=keystone-test -o jsonpath='{.items[0].metadata.name}') # if [[ -n "$TEST_POD_NAME" ]]; then # kubectl logs "$TEST_POD_NAME" -n "${ OS_NAMESPACE }" # else # echo "Keystone test pod not found." # fi exit 0 ``` -------------------------------- ### Install OpenStack Blazar Source: https://github.com/openstack/openstack-helm/blob/master/doc/source/install/openstack.rst Deploy the Blazar service, which handles resource reservation for OpenStack. ```bash helm upgrade --install blazar openstack-helm/blazar \ --namespace=openstack \ $(helm osh get-values-overrides -p ${OVERRIDES_DIR} -c blazar ${FEATURES}) ``` ```bash helm osh wait-for-pods openstack ``` -------------------------------- ### Install OpenStack-Helm Helm Plugin Source: https://github.com/openstack/openstack-helm/blob/master/doc/source/install/before_starting.rst Install the OpenStack-Helm plugin to access helper commands. This plugin simplifies common tasks when working with OpenStack-Helm. ```bash helm plugin install https://opendev.org/openstack/openstack-helm-plugin ``` -------------------------------- ### Install OpenStack Trove Source: https://github.com/openstack/openstack-helm/blob/master/doc/source/install/openstack.rst Install the Trove service, which provides Database as a Service (DBaaS) for OpenStack. ```bash helm upgrade --install trove openstack-helm/trove \ --namespace=openstack \ --timeout=600s \ $(helm osh get-values-overrides -p ${OVERRIDES_DIR} -c trove ${FEATURES}) ``` ```bash helm osh wait-for-pods openstack ``` -------------------------------- ### Install Kibana Chart Source: https://github.com/openstack/openstack-helm/blob/master/doc/source/logging/kibana.rst Use this command to install the Kibana chart into your Kubernetes cluster. Replace with your target namespace. ```bash helm install --namespace= local/kibana --name=kibana ``` -------------------------------- ### Deploy MariaDB Service Source: https://github.com/openstack/openstack-helm/blob/master/doc/source/install/openstack.rst Install or upgrade the MariaDB service using Helm. ```bash helm upgrade --install mariadb openstack-helm/mariadb \ --namespace=openstack \ ``` -------------------------------- ### Install Rally Helm Chart Source: https://github.com/openstack/openstack-helm/blob/master/rally/README.rst Deploy the Rally chart to your Kubernetes cluster. This command installs Rally and initiates benchmarking jobs based on the configuration in values.yaml. ```bash helm install --name=rally ./rally --namespace=openstack ``` -------------------------------- ### Install Ansible Source: https://github.com/openstack/openstack-helm/blob/master/doc/source/install/kubernetes.rst Install the Ansible package using pip. This is required for running the Ansible playbooks to deploy the environment. ```bash pip install ansible ``` -------------------------------- ### Install Node Exporter Chart Source: https://github.com/openstack/openstack-helm/blob/master/doc/source/monitoring/prometheus.rst Installs the prometheus-node-exporter chart to collect hardware and OS metrics from nodes. Use the 'kube-system' namespace for this component. ```bash helm install --namespace=kube-system local/prometheus-node-exporter --name=prometheus-node-exporter ``` -------------------------------- ### Deploy RabbitMQ Service Source: https://github.com/openstack/openstack-helm/blob/master/doc/source/install/openstack.rst Install or upgrade the RabbitMQ service using Helm, applying specific configurations and waiting for pods to be ready. ```bash helm upgrade --install rabbitmq openstack-helm/rabbitmq \ --namespace=openstack \ --set pod.replicas.server=1 \ --timeout=600s \ $(helm osh get-values-overrides -p ${OVERRIDES_DIR} -c rabbitmq ${FEATURES}) helm osh wait-for-pods openstack ``` -------------------------------- ### Chart.yaml Example for Keystone Source: https://context7.com/openstack/openstack-helm/llms.txt Standard Helm chart descriptor for the Keystone chart, including app version, Helm release version, and dependencies. ```yaml apiVersion: v2 appVersion: 29.0.0 # upstream OpenStack service version name: keystone version: 2026.1.0 # OpenStack-Helm release version description: OpenStack-Helm Keystone dependencies: - name: helm-toolkit repository: file://../helm-toolkit version: ">= 0.1.0" ``` -------------------------------- ### Install OpenStack Exporter Chart Source: https://github.com/openstack/openstack-helm/blob/master/doc/source/monitoring/prometheus.rst Installs the prometheus-openstack-exporter chart for OpenStack service metrics. This requires a Keystone user for authentication, managed by the chart's job. ```bash helm install --namespace=openstack local/prometheus-openstack-exporter --name=prometheus-openstack-exporter ``` -------------------------------- ### Install OpenStack Zaqar Source: https://github.com/openstack/openstack-helm/blob/master/doc/source/install/openstack.rst Deploy the Zaqar messaging service for asynchronous communication between applications and services in OpenStack. ```bash helm upgrade --install zaqar openstack-helm/zaqar \ --namespace=openstack \ $(helm osh get-values-overrides -p ${OVERRIDES_DIR} -c zaqar ${FEATURES}) ``` ```bash helm osh wait-for-pods openstack ``` -------------------------------- ### Fluentd Nested Configuration Example Source: https://github.com/openstack/openstack-helm/blob/master/doc/source/logging/fluent-logging.rst Demonstrates how to define nested configurations for Fluentd filters, such as a record transformer, using a specific YAML structure. ```yaml conf: td_agent: - fluentbit_forward: header: source type: forward port: "#{ENV['FLUENTD_PORT']}" bind: 0.0.0.0 - log_transformer: header: filter type: record_transformer expression: "foo.bar" inner_def: - record_transformer: header: record hostname: my_host tag: my_tag ``` -------------------------------- ### Get Storage Classes Source: https://github.com/openstack/openstack-helm/blob/master/doc/source/specs/tenant-ceph.rst Displays available storage classes, including the 'general' class provisioned by 'ceph.com/rbd'. ```bash ubuntu@node1: kubectl get storageclasses ``` -------------------------------- ### Install Multiple Ceph OSD Charts Source: https://github.com/openstack/openstack-helm/blob/master/doc/source/upgrade/multiple-osd-releases.rst Install new ceph-osd charts using Helm, specifying unique chart names and corresponding override values for data and journal locations. This example demonstrates the installation for 'ceph-osd-vdb'. ```bash example1: for CHART in ceph-osd-vdb; do helm upgrade --install ${CHART} ${OSH_PATH}/${CHART} \ --namespace=ceph \ --values=/tmp/ceph.yaml \ ${OSH_EXTRA_HELM_ARGS} \ ${OSH_EXTRA_HELM_ARGS_CEPH_DEPLOY} osd: - data: type: block-logical location: /dev/vdb journal: type: block-logical location: /dev/vda1 ``` -------------------------------- ### Example Node and Label Configurations Source: https://github.com/openstack/openstack-helm/blob/master/doc/source/devref/node-and-label-specific-configurations.rst Demonstrates how to define node and label-specific configurations for Nova, including vCPU pinning and CPU allocation ratio. Note the precedence rules: host overrides supercede label overrides, and later labels in a list have greater precedence. ```yaml conf: nova: DEFAULT: vcpu_pin_set: "0-15" cpu_allocation_ratio: 3.0 ``` ```yaml conf: nova: DEFAULT: vcpu_pin_set: "16-31" cpu_allocation_ratio: 3.0 ``` ```yaml conf: nova: DEFAULT: vcpu_pin_set: "8-15" cpu_allocation_ratio: 3.0 ``` ```yaml conf: nova: DEFAULT: vcpu_pin_set: "16-23" cpu_allocation_ratio: 3.0 ``` ```yaml conf: nova: DEFAULT: vcpu_pin_set: "8-15" cpu_allocation_ratio: 3.0 ``` ```yaml conf: nova: DEFAULT: vcpu_pin_set: "16-23" cpu_allocation_ratio: 3.0 ``` ```yaml conf: nova: DEFAULT: vcpu_pin_set: "16-31" cpu_allocation_ratio: 3.0 ``` ```yaml conf: nova: DEFAULT: vcpu_pin_set: "0-15" cpu_allocation_ratio: 3.0 ``` ```yaml conf: nova: DEFAULT: vcpu_pin_set: "0-31" cpu_allocation_ratio: 3.0 ``` -------------------------------- ### Fluentd Forwarding Configuration Source: https://github.com/openstack/openstack-helm/blob/master/doc/source/logging/fluent-logging.rst Defines the Fluentd configuration for receiving logs forwarded from Fluentbit. This example shows a basic setup for the fluentbit_forward input. ```yaml conf: fluentd: - fluentbit_forward: ``` -------------------------------- ### Install New Ceph-OSD Charts Source: https://github.com/openstack/openstack-helm/blob/master/doc/source/upgrade/multiple-osd-releases.rst Execute these scripts to install new Ceph OSD releases. The output shows the listing of installed OSD charts after the installation. ```console ubuntu@k8smaster:/opt/openstack-helm$ ./tools/deployment/multinode/030-ceph-osd-vdb.sh ubuntu@k8smaster:/opt/openstack-helm$ ./tools/deployment/multinode/030-ceph-osd-vdc.sh ubuntu@k8smaster:/opt/openstack-helm# helm list | grep -i osd ceph-osd-vdb 1 Tue Mar 26 03:21:07 2019 DEPLOYED ceph-osd-vdb-0.1.0 ceph-osd-vdc 1 Tue Mar 26 03:22:13 2019 DEPLOYED ceph-osd-vdc-0.1.0 ``` -------------------------------- ### CRUSH Map Rules Example Source: https://github.com/openstack/openstack-helm/blob/master/doc/source/testing/ceph-resiliency/failure-domain.rst Example of a replicated rule definition within a CRUSH map. ```text rule replicated_rack { id 2 type replicated min_size 1 max_size 10 step take default step chooseleaf firstn 0 type rack step emit } ``` -------------------------------- ### Extended Configuration Schema for Software Source: https://github.com/openstack/openstack-helm/blob/master/doc/source/specs/multi-os.rst This example shows an extended configuration schema to include generic software information, such as binary locations and arguments, for applications like Apache2. ```yaml conf: software: apache2: #the apache2 binary location binary: apache2 start_args: -DFOREGROUND stop_args: -k graceful-stop #directory where to drop the config files for apache vhosts conf_dir: /etc/apache2/conf-enabled sites_dir: /etc/apache2/sites-enabled ``` -------------------------------- ### Download helm-docs Binary Source: https://context7.com/openstack/openstack-helm/llms.txt This command downloads the helm-docs binary, which is a prerequisite for generating documentation within the project. ```bash make helm-docs ``` -------------------------------- ### Install Envoy Gateway Source: https://github.com/openstack/openstack-helm/blob/master/doc/source/install/prerequisites.rst Installs the Envoy Gateway CRDs and the Envoy Gateway itself using Helm. Ensure the version matches your requirements. ```bash helm install eg oci://docker.io/envoyproxy/gateway-helm \ --version v1.7.0 \ --namespace envoy-gateway-system \ --create-namespace ``` -------------------------------- ### Install Ceph Adapter Rook Source: https://github.com/openstack/openstack-helm/blob/master/ceph-adapter-rook/README.md Install the ceph-adapter-rook chart to prepare your environment for Openstack-Helm charts. Ensure the namespace is set to 'openstack'. ```bash helm upgrade --install ceph-adapter-rook ./ceph-adapter-rook \ --namespace=openstack ``` -------------------------------- ### Keystone Configuration YAML Example Source: https://github.com/openstack/openstack-helm/blob/master/doc/source/devref/oslo-config.rst This YAML structure defines configuration options for a Keystone service. Keys at the 'DEFAULT' level are used for section headings in the generated config file. Options like 'max_token_size' are direct mappings, while 'oslo_messaging_notifications.driver' shows how multistring options are handled. List values, such as in 'security_compliance.password_expires_ignore_user_ids', are converted to comma-separated strings. ```yaml conf: keystone: DEFAULT: max_token_size: 255 token: provider: fernet fernet_tokens: key_repository: /etc/keystone/fernet-keys/ credential: key_repository: /etc/keystone/credential-keys/ database: max_retries: -1 cache: enabled: true backend: dogpile.cache.memcached oslo_messaging_notifications: driver: type: multistring values: - messagingv2 - log security_compliance: password_expires_ignore_user_ids: - "123" - "456" ``` -------------------------------- ### Deploy OpenStack Placement Service Source: https://github.com/openstack/openstack-helm/blob/master/doc/source/install/openstack.rst Use this command to deploy the OpenStack Placement service. Ensure dependencies are met before execution. ```bash helm upgrade --install placement openstack-helm/placement \ --namespace=openstack \ $(helm osh get-values-overrides -p ${OVERRIDES_DIR} -c placement ${FEATURES}) ``` -------------------------------- ### Build OpenStack-Helm Charts from Source Source: https://context7.com/openstack/openstack-helm/llms.txt Commands to build all chart tarballs or individual charts from source, and to lint a specific chart. ```bash # Build charts from source make all # build all chart tarballs make build-keystone # build a single chart make lint-nova # lint a single chart ``` -------------------------------- ### Render Kubernetes Init Container with kubernetes-entrypoint Source: https://context7.com/openstack/openstack-helm/llms.txt Renders an init container that uses `kubernetes-entrypoint` to gate a pod on the readiness of services, jobs, daemonsets, and custom resources before the main containers start. Ensure `dependencies.static.jobs` and `dependencies.static.services` are correctly defined in `values.yaml`. ```yaml # values.yaml dependency declaration dependencies: static: api: jobs: - keystone-db-sync - keystone-credential-setup - keystone-fernet-setup services: - endpoint: internal service: oslo_cache - endpoint: internal service: oslo_db ``` ```yaml # Template usage in a deployment template: initContainers: {{ tuple . "api" list | include "helm-toolkit.snippets.kubernetes_entrypoint_init_container" | indent 8 }} # Rendered output: initContainers: - name: init image: quay.io/airshipit/kubernetes-entrypoint:latest-ubuntu_noble imagePullPolicy: IfNotPresent securityContext: allowPrivilegeEscalation: false readOnlyRootFilesystem: true runAsUser: 65534 env: - name: POD_NAME valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.name - name: NAMESPACE valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.namespace - name: DEPENDENCY_SERVICE value: "openstack:memcached,openstack:mariadb" - name: DEPENDENCY_JOBS value: "keystone-db-sync,keystone-credential-setup,keystone-fernet-setup" command: - kubernetes-entrypoint ``` -------------------------------- ### Run Ansible Playbook to Deploy Kubernetes Source: https://github.com/openstack/openstack-helm/blob/master/doc/source/install/kubernetes.rst Execute the Ansible playbook to deploy the configured environment. Ensure you are in the correct directory and have the inventory file ready. ```bash cd ~/osh ansible-playbook -i inventory.yaml deploy-env.yaml ``` -------------------------------- ### Install Ceph Adapter Rook Chart Source: https://github.com/openstack/openstack-helm/blob/master/doc/source/install/prerequisites.rst Installs the ceph-adapter-rook chart into the 'openstack' namespace. This chart prepares Kubernetes secrets for interfacing with a Ceph cluster, assuming the Ceph cluster is deployed in the 'ceph' namespace. ```bash helm upgrade --install ceph-adapter-rook openstack-helm/ceph-adapter-rook \ --namespace=openstack ``` -------------------------------- ### Memory Usage Service Check Example Source: https://github.com/openstack/openstack-helm/blob/master/doc/source/monitoring/nagios.rst An example of a Nagios service check for memory usage, targeting the 'base-os' host group. This demonstrates how to apply general system checks to a broader set of hosts. ```yaml - check_memory_usage: use: notifying_service service_description: Memory_usage check_command: check_memory_usage hostgroup_name: base-os ``` -------------------------------- ### Ceph Quorum Service Check Example Source: https://github.com/openstack/openstack-helm/blob/master/doc/source/monitoring/nagios.rst An example of a Nagios service check for Ceph monitor quorum status, targeting the 'prometheus-hosts' group. It uses a specific check command and defines critical and OK states. ```yaml - check_ceph_monitor_quorum: use: notifying_service hostgroup_name: prometheus-hosts service_description: "CEPH_quorum" check_command: check_prom_alert!ceph_monitor_quorum_low!CRITICAL- ceph monitor quorum does not exist!OK- ceph monitor quorum exists check_interval: 60 ``` -------------------------------- ### List OpenStack Pods Source: https://github.com/openstack/openstack-helm/blob/master/doc/source/testing/ceph-node-resiliency.rst Use this command to list all OpenStack pods in the 'openstack' namespace. The --show-all=false flag filters out pods that are not ready. ```console ubuntu@mnode1:~$ kubectl get pods -n openstack --show-all=false -o wide Flag --show-all has been deprecated, will be removed in an upcoming release NAME READY STATUS RESTARTS AGE IP NODE cinder-api-66f4f9678-2lgwk 1/1 Unknown 0 32m 192.168.3.41 mnode3 cinder-api-66f4f9678-flvr5 1/1 Running 0 32m 192.168.0.202 mnode1 cinder-api-66f4f9678-w5xhd 1/1 Running 0 11m 192.168.4.45 mnode2 cinder-backup-659b68b474-582kr 1/1 Running 0 32m 192.168.4.39 mnode2 cinder-scheduler-6778f6f88c-mm9mt 1/1 Running 0 32m 192.168.0.201 mnode1 cinder-volume-79b9bd8bb9-qsxdk 1/1 Running 0 32m 192.168.4.40 mnode2 glance-api-676fd49d4d-4tnm6 1/1 Running 0 11m 192.168.0.212 mnode1 glance-api-676fd49d4d-j4bdb 1/1 Unknown 0 36m 192.168.3.37 mnode3 glance-api-676fd49d4d-wtxqt 1/1 Running 0 36m 192.168.4.31 mnode2 ingress-7b4bc84cdd-9fs78 1/1 Running 0 1h 192.168.5.3 mnode4 ingress-7b4bc84cdd-wztz7 1/1 Running 0 1h 192.168.1.4 mnode6 ingress-error-pages-586c7f86d6-2jl5q 1/1 Running 0 1h 192.168.2.4 mnode5 ingress-error-pages-586c7f86d6-455j5 1/1 Unknown 0 1h 192.168.3.3 mnode3 ingress-error-pages-586c7f86d6-55j4x 1/1 Running 0 11m 192.168.4.47 mnode2 keystone-api-5bcc7cb698-dzm8q 1/1 Running 0 45m 192.168.4.24 mnode2 keystone-api-5bcc7cb698-vvwwr 1/1 Unknown 0 45m 192.168.3.25 mnode3 keystone-api-5bcc7cb698-wx5l6 1/1 Running 0 11m 192.168.0.213 mnode1 mariadb-ingress-84894687fd-9lmpx 1/1 Running 0 11m 192.168.4.48 mnode2 mariadb-ingress-84894687fd-dfnkm 1/1 Unknown 2 1h 192.168.3.20 mnode3 mariadb-ingress-error-pages-78fb865f84-p8lpg 1/1 Running 0 1h 192.168.4.17 mnode2 mariadb-server-0 1/1 Running 0 1h 192.168.4.18 mnode2 memcached-memcached-5db74ddfd5-926ln 1/1 Running 0 11m 192.168.4.49 mnode2 memcached-memcached-5db74ddfd5-wfr9q 1/1 Unknown 0 48m 192.168.3.23 mnode3 rabbitmq-rabbitmq-0 1/1 Unknown 0 1h 192.168.3.21 mnode3 rabbitmq-rabbitmq-1 1/1 Running 0 1h 192.168.4.19 mnode2 rabbitmq-rabbitmq-2 1/1 Running 0 1h 192.168.0.195 mnode1 ```