IBM Storage Ceph CSI driver

https://www.ibm.com/docs/en/ceph-csi-driver/8.1.0
Admin
The IBM Storage Ceph CSI driver provides Container Storage Interface (CSI) support for IBM Storage...
Tokens:9,252
Snippets:103
Trust Score:10
Update:10 months ago
Show doc for...
Tokens:
Raw
### Ceph Configuration File Example

Source: https://www.ibm.com/docs/en/ceph-csi-driver/8.1.0/index_topic=configuring-ceph-csi-driver-ceph-file-systems

An example of a 'ceph.conf' file, showing global settings for a Ceph cluster, including fsid, monitor addresses, and authentication requirements. This file is used to create the ConfigMap.

```ini
[global]
fsid = e40b854e-d179-11ef-91a4-fa163e9eea52
mon_host = v2:10.0.64.187:3300,v1:10.0.64.187:6789
v2:10.0.65.157:3300,v1:10.0.65.157:6789
v2:10.0.64.13:3300,v1:10.0.64.13:6789
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
fuse_big_writes = true
```

--------------------------------

### Install Ceph CSI Helm Chart

Source: https://www.ibm.com/docs/en/ceph-csi-driver/8.1.0/index_topic=configuring-ceph-csi-driver-ceph-file-systems

Installs the Ceph CSI Helm chart for Ceph File Systems using a specified values file. This command deploys the driver into the 'ceph-csi-cephfs' namespace.

```bash
helm install --namespace ceph-csi-cephfs ceph-csi-cephfs ceph-csi/ceph-csi-cephfs --values ceph-csi-cephfs-values.yaml
```

--------------------------------

### Configure Ceph CSI Helm Chart Values

Source: https://www.ibm.com/docs/en/ceph-csi-driver/8.1.0/index_topic=configuring-ceph-csi-driver-ceph-file-systems

Example of a values.yaml file for configuring the Ceph CSI Helm chart. It specifies essential parameters like clusterID and monitors, along with provisioner settings. Users should only include necessary changes.

```yaml
csiConfig:
- clusterID: "CLUSTER_ID"
monitors:
- "MON_IP_01"
- "MON_IP_02"
- "MON_IP_03"
provisioner:
name: provisioner
replicaCount: 2
```

--------------------------------

### Check Helm Chart Status

Source: https://www.ibm.com/docs/en/ceph-csi-driver/8.1.0/index_topic=configuring-ceph-csi-driver-ceph-file-systems

Retrieves the status of the deployed Helm release for the Ceph CSI driver. This helps in verifying if the installation was successful.

```bash
helm status ceph-csi-cephfs -n ceph-csi-cephfs
```

--------------------------------

### Add Ceph CSI Helm Repository

Source: https://www.ibm.com/docs/en/ceph-csi-driver/8.1.0/index_topic=using-helm

This command adds the official `ceph-csi` Helm repository to your local Helm configuration. This is a prerequisite for installing or managing the Ceph CSI driver using Helm charts. It fetches the latest chart information from the specified URL.

```shell
helm repo add ceph-csi https://ceph.github.io/csi-charts
```

--------------------------------

### Configure Ceph CSI Driver Provisioner Image with Helm

Source: https://www.ibm.com/docs/en/ceph-csi-driver/8.1.0/index_topic=initial-configuration-storage-ceph

This snippet shows how to configure the provisioner image parameters for the Ceph-CSI driver using Helm. It sets the image repository, tag, pull policy, and extra arguments for the csi-provisioner. Adapt the namespace and chart name as needed.

```bash
helm install --namespace ceph-csi-rbd ceph-csi-rbd ceph-csi/ceph-csi-rbd --set provisioner.provisioner.image.repository=registry.k8s.io/sig-storage/csi-provisioner --set provisioner.provisioner.image.tag=v5.1.0 --set provisioner.provisioner.image.pullPolicy=IfNotPresent --set provisioner.provisioner.image.extraArgs=[]
```

--------------------------------

### Verify Ceph CSI Pods

Source: https://www.ibm.com/docs/en/ceph-csi-driver/8.1.0/index_topic=configuring-ceph-csi-driver-ceph-file-systems

Lists all pods running in the 'ceph-csi-cephfs' namespace to ensure that the Ceph CSI driver components are running as expected.

```bash
kubectl get pods -n ceph-csi-cephfs
```

--------------------------------

### Create Kubernetes Namespace for Ceph CSI

Source: https://www.ibm.com/docs/en/ceph-csi-driver/8.1.0/index_topic=configuring-ceph-csi-driver-ceph-file-systems

Creates a Kubernetes namespace named 'ceph-csi-cephfs' to host the Ceph CSI driver components. This is a prerequisite for deploying the driver.

```bash
kubectl create namespace ceph-csi-cephfs
```

--------------------------------

### Update Kubernetes CSI Provisioner Image Parameters with Helm

Source: https://www.ibm.com/docs/en/ceph-csi-driver/8.1.0/index_topic=configuring-ceph-csi-driver-ceph-block-devices

This snippet shows how to configure the image repository, tag, pull policy, and extra arguments for the Kubernetes CSI provisioner using Helm. These parameters are crucial for the proper functioning of the storage provisioner.

```bash
helm install --set provisioner.provisioner.image.repository=registry.k8s.io/sig-storage/csi-provisioner --set provisioner.provisioner.image.tag=v5.0.1 --set provisioner.provisioner.image.pullPolicy=IfNotPresent --set provisioner.provisioner.image.extraArgs=[]
```

--------------------------------

### Configure Ceph CSI Driver Nodeplugin Image with Helm

Source: https://www.ibm.com/docs/en/ceph-csi-driver/8.1.0/index_topic=initial-configuration-storage-ceph

This snippet demonstrates how to update the nodeplugin image parameters for the IBM Storage Ceph CSI driver using Helm. It specifies the image repository, tag, and pull policy. Ensure the correct namespace and chart name are used according to your deployment.

```bash
helm install --namespace ceph-csi-rbd ceph-csi-rbd ceph-csi/ceph-csi-rbd --set nodeplugin.plugin.image.repository=cp.icr.io/cp/ibm-ceph/cephcsi --set nodeplugin.plugin.image.tag=v4.16 --set nodeplugin.plugin.image.pullPolicy=IfNotPresent
```

--------------------------------

### Apply Kubernetes Pod Definition using kubectl

Source: https://www.ibm.com/docs/en/ceph-csi-driver/8.1.0/index_topic=devices-binding-persistentvolumeclaim-pod-resource

This command applies a Kubernetes resource definition from a YAML file to the cluster. It's a fundamental operation for deploying applications and configurations managed by Kubernetes. Ensure the FILENAME.YAML placeholder is replaced with the actual path to your configuration file.

```bash
kubectl apply -f FILENAME.YAML
```

--------------------------------

### Apply StorageClass using kubectl (Shell)

Source: https://www.ibm.com/docs/en/ceph-csi-driver/8.1.0/index_topic=devices-creating-storageclass

Applies a Kubernetes StorageClass definition to the cluster using the kubectl command-line tool. This command reads the configuration from the specified YAML file. Replace FILENAME.YAML with the actual path to your StorageClass definition file.

```shell
kubectl apply -f FILENAME.YAML

```

--------------------------------

### Create CephFS PVC with Kernel Mounting (YAML)

Source: https://www.ibm.com/docs/en/ceph-csi-driver/8.1.0/index_topic=systems-creating-persistentvolumeclaim-pvc

This YAML defines a PersistentVolumeClaim (PVC) for a Ceph File System (CephFS) using kernel mounting. It specifies the access mode, requested storage size, and the StorageClass. Ensure the 'storageClassName' matches the one created in the 'Creating a StorageClass' step.

```yaml
kind: PersistentVolumeClaim
metadata:
name: cephfs-pvc-kernel
spec:
accessModes:
  - ReadWriteOnce
resources:
  requests:
    storage: 10Gi
storageClassName: cephfs-stroageclass-kernel
```

--------------------------------

### Verify ceph-csi-cephfs ConfigMap using kubectl

Source: https://www.ibm.com/docs/en/ceph-csi-driver/8.1.0/index_topic=configuring-ceph-csi-driver-ceph-file-systems

This command allows you to inspect the 'ceph-config' ConfigMap within the 'ceph-csi-cephfs' namespace. It's crucial for verifying the Ceph cluster's network and authentication settings before proceeding with restarts.

```bash
kubectl describe configmap ceph-config -n ceph-csi-cephfs
```

--------------------------------

### Restart ceph-csi-cephfs-provisioner Deployment using kubectl

Source: https://www.ibm.com/docs/en/ceph-csi-driver/8.1.0/index_topic=configuring-ceph-csi-driver-ceph-file-systems

This command initiates a rolling restart of the 'ceph-csi-cephfs-provisioner' deployment in the 'ceph-csi-cephfs' namespace. This is a standard Kubernetes operation to apply configuration changes or refresh the deployment.

```bash
kubectl rollout restart deployment/ceph-csi-cephfs-provisioner -n ceph-csi-cephfs
```

--------------------------------

### Define Raw Block PersistentVolumeClaim (YAML)

Source: https://www.ibm.com/docs/en/ceph-csi-driver/8.1.0/index_topic=devices-creating-persistentvolumeclaim-pvc

This YAML defines a PersistentVolumeClaim (PVC) for raw block volume access. It specifies access modes, volume mode, requested storage size, and the storage class name. This is a prerequisite for dynamically provisioning block storage using the Ceph CSI driver.

```yaml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: raw-block-pvc
spec:
  accessModes:
    - ReadWriteOnce
  volumeMode: Block
  resources:
    requests:
      storage: 2Gi
  storageClassName: ceph-rbd-sc
```

--------------------------------

### Create CephFS StorageClass with Kernel Mounting (Kubernetes YAML)

Source: https://www.ibm.com/docs/en/ceph-csi-driver/8.1.0/index_topic=systems-creating-storageclass

Defines a Kubernetes StorageClass for Ceph File Systems using kernel mounting. It specifies the provisioner, Ceph cluster details, secret references for access, and mounting options. This is used for dynamic provisioning of CephFS volumes.

```yaml
kind: StorageClass
metadata:
name: ceph-cephfs-sc-kernel
provisioner: cephfs.csi.ceph.com
parameters
 clusterID: CLUSTER_ID
 fsName: cephfs
 csi.storage.k8s.io/provisioner-secret-name: cephfs-secret
 csi.storage.k8s.io/provisioner-secret-namespace: default
 csi.storage.k8s.io/node-stage-secret-name: cephfs-secret
 csi.storage.k8s.io/node-stage-secret-namespace: default
 mounter: kernel
reclaimPolicy: Delete
allowVolumeExpansion: true
```

--------------------------------

### Verify Kubernetes Secret Creation with kubectl

Source: https://www.ibm.com/docs/en/ceph-csi-driver/8.1.0/index_topic=systems-creating-secret

This command retrieves a list of all Kubernetes secrets in the current namespace. It helps verify that secrets like 'cephfs-secret' and 'dockerhub-secret' have been successfully created and applied, showing their type, data count, and age.

```shell
host01$ kubectl get secret
NAME             TYPE                           DATA AGE
cephfs-secret    Opaque                         4    7d1h
dockerhub-secret kubernetes.io/dockerconfigjson 1    7d
```

--------------------------------

### Create Ceph Configuration Map

Source: https://www.ibm.com/docs/en/ceph-csi-driver/8.1.0/index_topic=configuring-ceph-csi-driver-ceph-file-systems

Creates a Kubernetes ConfigMap named 'ceph-config' from a 'ceph.conf' file within the 'ceph-csi-cephfs' namespace. This ConfigMap stores the Ceph cluster configuration.

```bash
kubectl create configmap ceph-config --from-file=ceph.conf=ceph.conf -n ceph-csi-cephfs --dry-run=client -o yaml | kubectl apply -f
```

--------------------------------

### Update Ceph-CSI Node Plugin Image Parameters with Helm

Source: https://www.ibm.com/docs/en/ceph-csi-driver/8.1.0/index_topic=configuring-ceph-csi-driver-ceph-block-devices

This snippet demonstrates how to update the image repository, tag, and pull policy for the Ceph-CSI driver's node plugin using Helm commands. Ensure that the image details are correctly specified for your deployment.

```bash
helm install --set nodeplugin.plugin.image.repository=cp.icr.io/cp/ibm-ceph/cephcsi --set nodeplugin.plugin.image.tag=v4.16 --set nodeplugin.plugin.image.pullPolicy=IfNotPresent
```

--------------------------------

### Create CephFS PVC with FUSE Mounting (YAML)

Source: https://www.ibm.com/docs/en/ceph-csi-driver/8.1.0/index_topic=systems-creating-persistentvolumeclaim-pvc

This YAML defines a PersistentVolumeClaim (PVC) for a Ceph File System (CephFS) using FUSE mounting. It specifies the access mode, requested storage size, and the StorageClass. Ensure the 'storageClassName' matches the one created in the 'Creating a StorageClass' step.

```yaml
kind: PersistentVolumeClaim
metadata:
 name: cephfs-pvc-fuse
spec:
 accessModes:
   - ReadWriteOnce
 resources:
   requests:
     storage: 10Gi
 storageClassName: cephfs-stroageclass-fuse
```

--------------------------------

### Encode Ceph Credentials using Base64

Source: https://www.ibm.com/docs/en/ceph-csi-driver/8.1.0/index_topic=devices-creating-secret

Encodes Ceph user ID and user key into Base64 format for use in Kubernetes Secrets. This is a prerequisite for creating storage credentials for the Ceph-CSI driver. It requires the `echo` and `base64` commands.

```shell
echo -n USER_ID | base64
echo -n USER_KEY | base64
```

--------------------------------

### Create CephFS StorageClass with FUSE Mounting (Kubernetes YAML)

Source: https://www.ibm.com/docs/en/ceph-csi-driver/8.1.0/index_topic=systems-creating-storageclass

Defines a Kubernetes StorageClass for Ceph File Systems using FUSE mounting. It specifies the provisioner, Ceph cluster details, secret references for access, and mounting options. This is used for dynamic provisioning of CephFS volumes.

```yaml
kind: StorageClass
metadata:
name: ceph-cephfs-sc-fuse
provisioner: cephfs.csi.ceph.com
parameters
  clusterID: CLUSTER_ID
  fsName: cephfs
  csi.storage.k8s.io/provisioner-secret-name: cephfs-secret
  csi.storage.k8s.io/provisioner-secret-namespace: default
  csi.storage.k8s.io/node-stage-secret-name: cephfs-secret
  csi.storage.k8s.io/node-stage-secret-namespace: default
  mounter: fuse
reclaimPolicy: Delete
allowVolumeExpansion: true
```

--------------------------------

### Configure Ceph-CSI Cluster ID and Monitor Information

Source: https://www.ibm.com/docs/en/ceph-csi-driver/8.1.0/index_topic=configuring-ceph-csi-driver-ceph-block-devices

This YAML snippet defines essential configuration for the Ceph-CSI driver, including the cluster ID and a list of Ceph monitor endpoints. This information is typically provided by the Ceph administrator and is critical for driver operation.

```yaml
csiConfig:
 - clusterID: "79179d9d-98d8-4976-ab2e-58635caa7235"
   monitors:
     - "10.85.8.118:6789"
     - "10.85.8.123:6789"
     - "10.85.8.124:6789"
provisioner:
 name: provisioner
 replicaCount: 2
```

--------------------------------

### Apply Kubernetes Secret using kubectl

Source: https://www.ibm.com/docs/en/ceph-csi-driver/8.1.0/index_topic=systems-creating-secret

Applies a Kubernetes Secret definition from a YAML file to the cluster. This command is used after creating the Secret YAML, typically for configuring CSI drivers or other resources.

```bash
kubectl apply -fFILENAME.YAML
```

--------------------------------

### Verify PVC Status with kubectl

Source: https://www.ibm.com/docs/en/ceph-csi-driver/8.1.0/index_topic=systems-creating-persistentvolumeclaim-pvc

This command verifies that a Persistent Volume Claim (PVC) is in a 'Bound' state, indicating successful provisioning. It requires `kubectl` to be configured for the target Kubernetes cluster. The output shows PVC details like name, status, capacity, access modes, and storage class.

```bash
kubectl get pvc
```

--------------------------------

### Create StorageClass for Ceph Block Devices (RBD)

Source: https://www.ibm.com/docs/en/ceph-csi-driver/8.1.0/index_topic=devices-creating-storageclass

Defines a StorageClass YAML file for dynamic provisioning of Ceph Block Devices (RBD) using the Ceph-CSI driver. It specifies storage parameters like pool name, Secret reference, SpaceEfficiency, and fstype. Ensure Ceph Manager permissions and matching Secret details from the 'Creating a Secret' step.

```yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ceph-rbd
spec:
  provisioner: rook-ceph.rbd.csi.ceph.com
  parameters:
    # The cluster ID of your Ceph cluster. Found in your cluster's configmap.
    clusterID: <your-cluster-id>
    # The pool into which persistent volume snapshots will be created.
    snapshotPool: <your-snapshot-pool-name>
    # The pool into which new persistent volumes will be created.
    pool: <your-pool-name>
    # The namespace where the secret is located.
    csi.storage.k8s.io/controller-expand-secret-namespace: <your-secret-namespace>
    # The name of the secret that contains Ceph cluster credentials.
    csi.storage.k8s.io/controller-expand-secret-name: <your-secret-name>
    # The namespace where the secret is located.
    csi.storage.k8s.io/node-expand-secret-namespace: <your-secret-namespace>
    # The name of the secret that contains Ceph cluster credentials.
    csi.storage.k8s.io/node-expand-secret-name: <your-secret-name>
    # The namespace where the secret is located.
    csi.storage.k8s.io/provisioner-secret-namespace: <your-secret-namespace>
    # The name of the secret that contains Ceph cluster credentials.
    csi.storage.k8s.io/provisioner-secret-name: <your-secret-name>
    # The filesystem type for the provisioned volumes.
    fstype: ext4
    # Space efficiency settings for the volume. Options: "none", "zero", "reuse".
    SpaceEfficiency: "none"
  # The reclaim policy for volumes provisioned with this StorageClass.
  reclaimPolicy: Delete
  # Indicates whether the StorageClass supports volume expansion.
  allowVolumeExpansion: true
  # A list of topology keys that the provisioner can use for topology awareness.
  volumeBindingMode: Immediate
  # Optional: Specify image features for RBD volumes.
  # imageFeatures: "layer,journaling,exclusive-lock,object-map,fast-diff"

```

--------------------------------

### Setting Ceph CSI Driver Permissions for Secrets and StorageClass

Source: https://www.ibm.com/docs/en/ceph-csi-driver/8.1.0/index_topic=user-roles-permissions

This snippet outlines the specific Ceph permissions required for the provisioner, controller expansion, and node stage Secrets within a StorageClass. These commands define user capabilities for interacting with Ceph components.

```shell
mgr "profile rbd pool=csi"
osd "profile rbd pool=csi"
mon "profile rbd"
```

--------------------------------

### Create Ceph CSI RBD User with Capabilities

Source: https://www.ibm.com/docs/en/ceph-csi-driver/8.1.0/index_topic=user-roles-permissions

This command creates or retrieves an existing Ceph client user named 'csi-rbd' with specific capabilities required for Ceph Block Device operations within the 'csi' profile. It grants permissions for the manager (mgr), object storage daemon (osd), and monitor (mon) components, enabling the CSI driver to provision and manage RBD volumes.

```shell
ceph auth get-or-create client.csi-rbd \
    mgr "profile rbd pool=csi" \
    osd "profile rbd pool=csi" \
    mon "profile rbd"
```

--------------------------------

### Define Pod with Raw Block Volume using YAML

Source: https://www.ibm.com/docs/en/ceph-csi-driver/8.1.0/index_topic=devices-binding-persistentvolumeclaim-pod-resource

This YAML configuration defines a Kubernetes Pod that utilizes a raw block volume. It specifies the container image, command, and volume mounts, referencing a PersistentVolumeClaim named 'raw-block-pvc'. This is a common pattern for stateful applications requiring direct block device access.

```yaml
kind: Pod
metadata:
  name: pod-with-raw-block-volume
spec:
  containers:
  - name:  pod-with-raw-block-volume
    image: busybox
    command: ["sleep", "infinity"]
    volumeMounts:
    - mountPath: /mnt/ceph_rbd
      name: volume
  volumes:
  - name: volume
    persistentVolumeClaim:
      claimName: raw-block-pvc
```

--------------------------------

### Restart ceph-csi-cephfs-nodeplugin DaemonSet using kubectl

Source: https://www.ibm.com/docs/en/ceph-csi-driver/8.1.0/index_topic=configuring-ceph-csi-driver-ceph-file-systems

This command triggers a rolling restart for the 'ceph-csi-cephfs-nodeplugin' DaemonSet across all nodes in the 'ceph-csi-cephfs' namespace. DaemonSets ensure a copy of a pod runs on all nodes, making this restart critical for node-level driver updates.

```bash
kubectl rollout restart daemonset/ceph-csi-cephfs-nodeplugin -n ceph-csi-cephfs
```

--------------------------------

### Encode Ceph Credentials using Base64

Source: https://www.ibm.com/docs/en/ceph-csi-driver/8.1.0/index_topic=systems-creating-secret

This snippet demonstrates how to Base64 encode Ceph user ID and user key for use in Kubernetes Secrets. It requires the 'echo' and 'base64' commands.

```bash
echo -nUSER_ID| base64
echo -nUSER_KEY| base64
```

--------------------------------

### Verify Ceph Monitor Reachability

Source: https://www.ibm.com/docs/en/ceph-csi-driver/8.1.0/index_topic=configuring-ceph-csi-driver-ceph-block-devices

This command demonstrates how to verify network connectivity to Ceph monitors from the cluster. Replace MONITOR_IP with the actual IP address of a Ceph monitor. This step is important for ensuring the CSI driver can communicate with the Ceph cluster.

```bash
ping MONITOR_IP
```

```bash
ping 10.85.8.118
```

--------------------------------

### Define Ceph CSI StorageClass (Kubernetes YAML)

Source: https://www.ibm.com/docs/en/ceph-csi-driver/8.1.0/index_topic=devices-creating-storageclass

Defines a Kubernetes StorageClass for Ceph RBD volumes using the CSI driver. It specifies provisioner details, parameters like cluster ID and pool, and secret references for provisioning and expansion operations. Ensure secret names and namespaces match your existing secret configuration.

```yaml
kind: StorageClass
metadata:
  name: ceph-rbd-sc
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
provisioner: rbd.csi.ceph.com
parameters:
   clusterID: 79179d9d-98d8-4976-ab2e-58635caa7235
   pool: myPool
   imageFeatures: layering
   csi.storage.k8s.io/provisioner-secret-name: ceph-admin
   csi.storage.k8s.io/provisioner-secret-namespace: default
   csi.storage.k8s.io/controller-expand-secret-name: ceph-admin
   csi.storage.k8s.io/controller-expand-secret-namespace: default
   csi.storage.k8s.io/node-stage-secret-name: ceph-admin
   csi.storage.k8s.io/node-stage-secret-namespace: default
reclaimPolicy: Delete
allowVolumeExpansion: true

```

--------------------------------

### Create CephFS Secret YAML for Ceph-CSI Driver

Source: https://www.ibm.com/docs/en/ceph-csi-driver/8.1.0/index_topic=systems-creating-secret

Defines a Kubernetes Secret YAML file required for the Ceph-CSI driver to access Ceph File Systems. It includes Base64 encoded credentials for user and admin access. The secret is applied using kubectl.

```yaml
kind: Secret
metadata:
  name: cephfs-admin
  namespace: default
data:
  userID:USER_ID
  userKey:USER_KEY
  adminID:ADMIN_ID
  adminKey:ADMIN_KEY
```

--------------------------------

### Verify Ceph Subvolumes with ceph fs

Source: https://www.ibm.com/docs/en/ceph-csi-driver/8.1.0/index_topic=systems-creating-persistentvolumeclaim-pvc

This command checks for the existence of subvolumes within a specified Ceph filesystem. It requires access to the Ceph cluster's command-line interface. The output lists subvolumes, typically in JSON format, showing their names.

```bash
ceph fs subvolume ls <filesystem_name> <directory_name>
```

--------------------------------

### Create Ceph Admin Secret YAML

Source: https://www.ibm.com/docs/en/ceph-csi-driver/8.1.0/index_topic=devices-creating-secret

Defines a Kubernetes Secret of type 'kubernetes.io/rbd' to store Ceph user credentials (userID and userKey). The userID and userKey must be Base64 encoded. This secret is essential for the Ceph CSI driver to authenticate with the Ceph cluster.

```yaml
kind: Secret
metadata:
  name: ceph-admin
  namespace: default
type: kubernetes.io/rbd
data:
  userID: USER_ID
  userKey: USER_KEY
```
Tokens:
Raw