HwameiStor

https://hwameistor.io/docs
Admin
HwameiStor is an HA local storage system for cloud-native stateful workloads. It creates a local...
Tokens:42,703
Snippets:513
Trust Score:6.8
Update:4 months ago
Show doc for...
Tokens:
Raw
### Install dfget Client

Source: https://hwameistor.io/docs/volumes/cache

Download and install the dfget client command-line tool on each node.

```bash
wget https://github.com/dragonflyoss/Dragonfly2/releases/download/v2.1.44/dfget-2.1.44-linux-amd64.rpm
rpm -ivh dfget-2.1.44-linux-amd64.rpm
```

--------------------------------

### Install MySQL Client

Source: https://hwameistor.io/docs/use_cases/tidb

Install the MySQL client utility on the system, which is necessary for connecting to and interacting with the TiDB cluster.

```bash
yum -y install mysql-client  

```

--------------------------------

### Install Dragonfly using Helm

Source: https://hwameistor.io/docs/volumes/cache

Add the Dragonfly Helm repository and install Dragonfly components.

```bash
helm repo add dragonfly https://dragonflyoss.github.io/helm-charts/
helm install --create-namespace --namespace dragonfly-system dragonfly dragonfly/dragonfly --version 1.1.63
```

--------------------------------

### Install with Disk Reservation using helm install --set

Source: https://hwameistor.io/docs/install/operator

Configure disk reservation during installation using the --set flag to specify nodes and devices that should not be claimed by the operator.

```bash
helm install hwameistor-operator hwameistor-operator/hwameistor-operator -n hwameistor --create-namespace \
--set diskReserve[0].nodeName=node1 \
--set diskReserve[0].devices={/dev/sdc,/dev/sdd} \
--set diskReserve[1].nodeName=node2 \
--set diskReserve[1].devices={/dev/sdc,/dev/sde}
```

--------------------------------

### Install with Authentication Enabled

Source: https://hwameistor.io/docs/install/operator

Enable API server authentication during installation by setting authentication parameters like access ID and secret key via Helm values.

```bash
helm install hwameistor-operator hwameistor-operator/hwameistor-operator  -n hwameistor --create-namespace \
--set apiserver.authentication.enable=true \
--set apiserver.authentication.accessId={YourName} \
--set apiserver.authentication.secretKey={YourPassword}
```

--------------------------------

### Check and Install LVM2

Source: https://hwameistor.io/docs/faqs

Verify if LVM2 is installed on a node and install it if necessary. This is a prerequisite for certain Hwameistor operations.

```bash
rpm -qa | grep lvm2  # Check if LVM2 is installed  
yum install lvm2  # Install on each node  
```

--------------------------------

### HwameiStor LocalStorageNode Status Example

Source: https://hwameistor.io/docs/faqs

An example output of the `kubectl get localstoragenode` command, showing the status of storage pools and volumes on a node. Ensure volumes are empty before node removal.

```yaml
apiVersion: hwameistor.io/v1alpha1
kind: LocalStorageNode
metadata:
  name: NODE
spec:
  hostname: NODE
  storageIP: 10.6.113.22
  topogoly:
    region: default
    zone: default
status:
  ...
  pools:
    LocalStorage_PoolHDD:
      class: HDD
      disks:
      - capacityBytes: 17175674880
        devPath: /dev/sdb
        state: InUse
        type: HDD
      freeCapacityBytes: 16101933056
      freeVolumeCount: 999
      name: LocalStorage_PoolHDD
      totalCapacityBytes: 17175674880
      totalVolumeCount: 1000
      type: REGULAR
      usedCapacityBytes: 1073741824
      usedVolumeCount: 1
      volumeCapacityBytesLimit: 17175674880
      ## **** make sure volumes is empty **** ##
      volumes:    
  state: Ready  

```

--------------------------------

### Install with Disk Reservation using a YAML file

Source: https://hwameistor.io/docs/install/operator

Alternatively, define disk reservation configurations in a YAML file and apply it during Helm installation for better readability and management.

```yaml
diskReserve:
- nodeName: node1
  devices:
  - /dev/sdc
  - /dev/sdd
- nodeName: node2
  devices:
  - /dev/sdc
  - /dev/sde
```

```bash
helm install hwameistor-operator hwameistor-operator/hwameistor-operator -n hwameistor --create-namespace -f diskReserve.yaml
```

--------------------------------

### Install LVM2, linux-headers, and xfsprogs on Ubuntu

Source: https://hwameistor.io/docs/install/prereq

Use apt-get to install the LVM2 package for logical volume management, linux-headers for HA features, and xfsprogs for filesystem resizing on Ubuntu systems.

```bash
apt-get install -y lvm2

```

```bash
apt-get install -y linux-headers-$(uname -r)

```

```bash
apt-get install -y xfsprogs

```

--------------------------------

### Install using DaoCloud Image Registry

Source: https://hwameistor.io/docs/install/operator

Install the operator using images from a specific registry, such as DaoCloud's, by setting the global image registry configurations.

```bash
helm install hwameistor-operator hwameistor-operator/hwameistor-operator  -n hwameistor --create-namespace \
--set global.hwameistorImageRegistry=ghcr.m.daocloud.io \
--set global.k8sImageRegistry=m.daocloud.io/registry.k8s.io
```

--------------------------------

### Example LocalStorageNode YAML Output

Source: https://hwameistor.io/docs/nodes_and_disks/lvm_nodes

This is an example of the expected output when checking the LocalStorageNode CR, showing the configured storage pools and disk details.

```yaml
apiVersion: hwameistor.io/v1alpha1
kind: LocalStorageNode
metadata:
  name: k8s-worker-4
spec:
  hostname: k8s-worker-4
  storageIP: 10.6.182.103
  topogoly:
    region: default
    zone: default
status:
  pools:
    LocalStorage_PoolSSD:
      class: SSD
      disks:
      - capacityBytes: 214744170496
        devPath: /dev/sdb
        state: InUse
        type: SSD
      freeCapacityBytes: 214744170496
      freeVolumeCount: 1000
      name: LocalStorage_PoolSSD
      totalCapacityBytes: 214744170496
      totalVolumeCount: 1000
      type: REGULAR
      usedCapacityBytes: 0
      usedVolumeCount: 0
      volumeCapacityBytesLimit: 214744170496
      volumes:
  state: Ready

```

--------------------------------

### LocalStorageNode YAML Example

Source: https://hwameistor.io/docs/faqs

An example snippet of the LocalStorageNode YAML configuration, showing the 'pools' section with disk details. This is used to confirm successful disk addition to a storage pool.

```yaml
apiVersion: hwameistor.io/v1alpha1  
kind: LocalStorageNode  
...
pools:
  LocalStorage_PoolHDD:
  class: HDD
  disks:
  - capacityBytes: 17175674880
  devPath: /dev/sdb
...
```

--------------------------------

### Install HwameiStor with hwameistor-operator

Source: https://hwameistor.io/docs/install/operator

Install the HwameiStor system using the hwameistor-operator Helm chart. The operator will automatically claim disks and create StorageClasses if available clean disks are provided.

```bash
helm install hwameistor-operator hwameistor-operator/hwameistor-operator -n hwameistor --create-namespace
```

--------------------------------

### Configure IP Whitelist for User Access

Source: https://hwameistor.io/docs/use_cases/tidb

This example shows how to create a user with a specific IP address restriction and then test access from allowed and disallowed IP addresses.

```sql
mysql -u root -h 172.17.49.102 -P 4000
```

```sql
drop user tidb;
```

```sql
create user tidb@'127.0.0.1' IDENTIFIED by 'tidb';
```

```sql
flush privileges;
```

```sql
select * from mysql.user where user='tidb';
```

```sql
mysql -u tidb -h 127.0.0.1 -P 4000 -ptidb
```

```sql
mysql -u tidb -h 172.17.49.102 -P 4000 -ptidb
```

--------------------------------

### Example LocalDiskNode YAML Output

Source: https://hwameistor.io/docs/nodes_and_disks/disk_nodes

This is an example of the expected output when checking the LocalDiskNode resource after a storage pool has been created. It shows the pool details, including disk information and capacity.

```yaml
apiVersion: hwameistor.io/v1alpha1  
kind: LocalDiskNode  
metadata:  
  name: k8s-worker-2  
spec:  
  nodeName: k8s-worker-2  
status:  
  pools:  
    LocalDisk_PoolSSD:  
      class: SSD  
      disks:  
        - capacityBytes: 214744170496  
          devPath: /dev/sdb  
          state: Available  
          type: SSD  
      freeCapacityBytes: 214744170496  
      freeVolumeCount: 1  
      totalCapacityBytes: 214744170496  
      totalVolumeCount: 1  
      type: REGULAR  
      usedCapacityBytes: 0  
      usedVolumeCount: 0  
      volumeCapacityBytesLimit: 214744170496  
      volumes:  
  state: Ready  
```

--------------------------------

### Verify DataSet Configuration (MinIO Example)

Source: https://hwameistor.io/docs/volumes/cache

Defines a DataSet resource for MinIO, specifying endpoint, bucket, and credentials. Ensure to replace placeholders with your actual MinIO details.

```yaml
apiVersion: datastore.io/v1alpha1
kind: DataSet
metadata:
  name: dataset-test
spec:
  refresh: true
  type: minio
  minio:
    endpoint: Your service ip address:9000
    bucket: BucketName/Dir # Defined according to the directory level where your dataset is located
    secretKey: minioadmin
    accessKey: minioadmin
    region: ap-southeast-2


```

--------------------------------

### Create a PersistentVolumeClaim and Pod

Source: https://hwameistor.io/docs/volumes/volume_thin

This example shows how to create a PersistentVolumeClaim (PVC) with a thin-provisioned storage class and then create a Pod that mounts this PVC.

```yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: test-pvc
spec:
  volumeMode: Filesystem
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 3Gi
  storageClassName: hwameistor-storage-lvm-thin-hdd
---
apiVersion: v1
kind: Pod
metadata:
  name: test-pod
spec:
  containers:
  - name: busybox
    image: busybox:1.31.1
    command:
      - sleep
      - "360000000"
    imagePullPolicy: IfNotPresent
    volumeMounts:
    - name: temp-pvc
      mountPath: /mnt/temp-fs
  volumes:
  - name: temp-pvc
    persistentVolumeClaim:
      claimName: test-pvc
  terminationGracePeriodSeconds: 10

```

--------------------------------

### Volume Audit Event Example

Source: https://hwameistor.io/docs/system_audit

This example demonstrates a detailed audit record for a volume's lifecycle, including Create, Mount, Unmount, and Delete actions. It shows the specific content and timestamps for each operation.

```yaml
apiVersion: hwameistor.io/v1alpha1
kind: Event
metadata:
  creationTimestamp: "2023-08-08T15:52:55Z"
  generation: 5
  name: volume-pvc-34e3b086-2d95-4980-beb6-e175fd79a847
  resourceVersion: "10221888"
  uid: d3ebaffb-eddb-4c84-93be-efff350688af
spec:
  resourceType: Volume
  resourceName: pvc-34e3b086-2d95-4980-beb6-e175fd79a847
  records:
  - action: Create
    actionContent: '{"requiredCapacityBytes":5368709120,"volumeQoS":{},"poolName":"LocalStorage_PoolHDD","replicaNumber":2,"convertible":true,"accessibility":{"nodes":["k8s-node1","k8s-master"],"zones":["default"],"regions":["default"]},"pvcNamespace":"default","pvcName":"mysql-data-volume","volumegroup":"db890e34-a092-49ac-872b-f2a422439c81"}'
    time: "2023-08-08T15:52:55Z"
  - action: Mount
    actionContent: '{"allocatedCapacityBytes":5368709120,"replicas":["pvc-34e3b086-2d95-4980-beb6-e175fd79a847-krp927","pvc-34e3b086-2d95-4980-beb6-e175fd79a847-wm7p56"],"state":"Ready","publishedNode":"k8s-node1","fsType":"xfs","rawblock":false}'
    time: "2023-08-08T15:53:07Z"
  - action: Unmount
    actionContent: '{"allocatedCapacityBytes":5368709120,"usedCapacityBytes":33783808,"totalInode":2621120,"usedInode":3,"replicas":["pvc-34e3b086-2d95-4980-beb6-e175fd79a847-krp927","pvc-34e3b086-2d95-4980-beb6-e175fd79a847-wm7p56"],"state":"Ready","publishedNode":"k8s-node1","fsType":"xfs","rawblock":false}'
    time: "2023-08-08T16:03:03Z"
  - action: Delete
    actionContent: '{"requiredCapacityBytes":5368709120,"volumeQoS":{},"poolName":"LocalStorage_PoolHDD","replicaNumber":2,"convertible":true,"accessibility":{"nodes":["k8s-node1","k8s-master"],"zones":["default"],"regions":["default"]},"pvcNamespace":"default","pvcName":"mysql-data-volume","volumegroup":"db890e34-a092-49ac-872b-f2a422439c81","config":{"version":1,"volumeName":"pvc-34e3b086-2d95-4980-beb6-e175fd79a847","requiredCapacityBytes":5368709120,"convertible":true,"resourceID":2,"readyToInitialize":true,"initialized":true,"replicas":[{"id":1,"hostname":"k8s-node1","ip":"10.6.113.101","primary":true},{"id":2,"hostname":"k8s-master","ip":"10.6.113.100","primary":false}]},"delete":true}'
    time: "2023-08-08T16:03:38Z"

```

--------------------------------

### Create a Volume Snapshot

Source: https://hwameistor.io/docs/volumes/volume_thin

This example demonstrates how to create a VolumeSnapshot from an existing PersistentVolumeClaim using a specified snapshot class.

```yaml
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: my-snapshot
spec:
  volumeSnapshotClassName: hwameistor-storage-lvm-snapshot
  source:
    persistentVolumeClaimName: test-pvc

```

--------------------------------

### Localvolumereplica Status Metric Example

Source: https://hwameistor.io/docs/metrics

Example of a gauge metric showing the status of a local volume replica. A value of 1 typically indicates a 'Ready' status.

```text
# HELP hwameistor_localvolumereplica_status The status of the localvolumereplica.  
# TYPE hwameistor_localvolumereplica_status gauge  
hwameistor_localvolumereplica_status{nodeName="k8s-node2",poolName="HDD",status="Ready",volumeName="pvc-d1964bc9-9b0b-456d-be1a-6d0de9b47589"} 1  
```

--------------------------------

### Table Operations: Create, Delete, Modify, and Partition

Source: https://hwameistor.io/docs/use_cases/tidb

This script covers essential table operations including dropping and creating tables, inserting, updating, and deleting data. It also demonstrates modifying columns (changing type, adding, dropping) and setting default values, along with the initial setup for a partition table.

```sql
# Create and delete table  
drop table if exists t_test;  
create table if not exists t_test  
( id                   bigint default '0',  
  name                 varchar(200) default '' ,  
  sale_time            datetime default current_timestamp,  
  constraint pk_t_test primary key (id)  
);  
# Delete and modify  
insert into t_test(id,name) values (1,'a'),(2,'b'),(3,'c'),(4,'d'),(5,'e');  
update t_test set name='aaa' where id=1;  
update t_test set name='bbb' where id=2;  
delete from t_dml where id=5;  
# Modify, add, delete columns  
alter table t_test modify column name varchar(250);  
alter table t_test add column col varchar(255);  
insert into t_test(id,name,col) values(10,'test','new_col');       
alter table t_test add column colwithdefault varchar(255) default 'aaaa';  
insert into t_test(id,name) values(20,'testdefault');  
insert into t_test(id,name,colwithdefault ) values(10,'test','non-default ');       
alter table t_test drop column colwithdefault;  
# Type of partition table (only listed part of scripts)  
CREATE TABLE employees (  
    id INT NOT NULL,  
fname VARCHAR(30),  
lname VARCHAR(30),  
    hired DATE NOT NULL DEFAULT '1970-01-01',  
    separated DATE NOT NULL DEFAULT '9999-12-31',  
job_code INT NOT NULL,  
store_id INT NOT NULL  
)  


```

--------------------------------

### Localstoragenode Status Metric Example

Source: https://hwameistor.io/docs/metrics

Example of a gauge metric representing the status of a local storage node. A value of 1 typically indicates a 'Ready' status.

```text
# HELP hwameistor_localstoragenode_status The status of the localstoragenode.  
# TYPE hwameistor_localstoragenode_status gauge  
hwameistor_localstoragenode_status{nodeName="k8s-master",status="Ready"} 1  
```

--------------------------------

### Localdisk Capacity Metric Example

Source: https://hwameistor.io/docs/metrics

Example of a gauge metric representing the capacity of a local disk. It includes labels for device path, node name, owner, reservation status, and disk type.

```text
# HELP hwameistor_localdisk_capacity The capacity of the localdisk. 
# TYPE hwameistor_localdisk_capacity gauge  
hwameistor_localdisk_capacity{devPath="/dev/sda",nodeName="k8s-master",owner="system",reserved="true",status="Bound",type="HDD"} 1.37438953472e+11  
```

--------------------------------

### Localvolume Status Metric Example

Source: https://hwameistor.io/docs/metrics

Example of a gauge metric indicating the status of a local volume. A value of 1 typically signifies a 'Ready' status.

```text
# HELP hwameistor_localvolume_status The status summary of the localvolume.  
# TYPE hwameistor_localvolume_status gauge  
hwameistor_localvolume_status{mountedOn="k8s-node2",poolName="HDD",status="Ready",type="Convertible",volumeName="pvc-d1964bc9-9b0b-456d-be1a-6d0de9b47589"} 1  
```

--------------------------------

### Install LVM2, kernel-devel, and xfsprogs on CentOS/RHEL, Rocky, and Kylin

Source: https://hwameistor.io/docs/install/prereq

Use yum to install the LVM2 package for logical volume management, kernel-devel for HA features, and xfsprogs for filesystem resizing on compatible Linux distributions.

```bash
yum install -y lvm2

```

```bash
yum install -y kernel-devel-$(uname -r)

```

```bash
yum install -y xfsprogs

```

--------------------------------

### Localstoragenode Volume Count Metric Example

Source: https://hwameistor.io/docs/metrics

Example of a gauge metric showing the count of available volumes on a local storage node. Labels specify the kind (e.g., Free) and pool name.

```text
# HELP hwameistor_localstoragenode_volumecount The volume count of the localstoragenode.  
# TYPE hwameistor_localstoragenode_volumecount gauge  
hwameistor_localstoragenode_volumecount{kind="Free",nodeName="k8s-master",poolName="HDD"} 1000  
```

--------------------------------

### Localvolumereplica Capacity Metric Example

Source: https://hwameistor.io/docs/metrics

Example of a gauge metric for local volume replica capacity. It includes labels for volume name, node name, and pool name.

```text
# HELP hwameistor_localvolumereplica_capacity The capacity of the localvolumereplica.  
# TYPE hwameistor_localvolumereplica_capacity gauge  
hwameistor_localvolumereplica_capacity{nodeName="k8s-node2",poolName="HDD",volumeName="pvc-d1964bc9-9b0b-456d-be1a-6d0de9b47589"} 1.073741824e+09  
```

--------------------------------

### Localstoragenode Capacity Metric Example

Source: https://hwameistor.io/docs/metrics

Example of a gauge metric for local storage node capacity, indicating free space. Labels include node name and pool name.

```text
# HELP hwameistor_localstoragenode_capacity The storage capacity of the localstoragenode.  
# TYPE hwameistor_localstoragenode_capacity gauge  
hwameistor_localstoragenode_capacity{kind="Free",nodeName="k8s-master",poolName="HDD"} 1.07369988096e+11  
```

--------------------------------

### Localvolume Capacity Metric Example

Source: https://hwameistor.io/docs/metrics

Example of a gauge metric for local volume capacity, showing the allocated amount. Labels include volume name, pool, type, and mount point.

```text
# HELP hwameistor_localvolume_capacity The capacity of the localvolume.  
# TYPE hwameistor_localvolume_capacity gauge  
hwameistor_localvolume_capacity{kind="Allocated",mountedOn="k8s-node2",poolName="HDD",type="Convertible",volumeName="pvc-d1964bc9-9b0b-456d-be1a-6d0de9b47589"} 1.073741824e+09  
```

--------------------------------

### Create a PVC from a Snapshot

Source: https://hwameistor.io/docs/volumes/volume_thin

This example shows how to create a new PersistentVolumeClaim (PVC) using a previously created VolumeSnapshot as the data source.

```yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: test-pvc2
spec:
  volumeMode: Filesystem
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 3Gi
  storageClassName: hwameistor-storage-lvm-thin-hdd
  dataSource:
    name: my-snapshot
    kind: VolumeSnapshot
    apiGroup: snapshot.storage.k8s.io
---
apiVersion: v1
kind: Pod
metadata:
  name: test-pod2
spec:
  containers:
  - name: busybox
    image: busybox:1.31.1
    command:
      - sleep
      - "360000000"
    imagePullPolicy: IfNotPresent
    volumeMounts:
    - name: temp-pvc
      mountPath: /mnt/temp-fs
  volumes:
  - name: temp-pvc
    persistentVolumeClaim:
      claimName: test-pvc2

```

--------------------------------

### Check VolumeSnapshots with kubectl

Source: https://hwameistor.io/docs/volumes/volume_snapshot

Use `kubectl get vs` to list and check the status of created VolumeSnapshots. This command displays details such as readiness, source PVC, and creation time.

```bash
$ kubectl get vs
NAME                           READYTOUSE   SOURCEPVC               SOURCESNAPSHOTCONTENT   RESTORESIZE   SNAPSHOTCLASS                     SNAPSHOTCONTENT                                    CREATIONTIME   AGE
snapshot-local-storage-pvc-lvm   true         local-storage-pvc-lvm                           1Gi           hwameistor-storage-lvm-snapshot   snapcontent-0fc17697-68ea-49ce-8e4c-7a791e315110   53y            2m57s

```

--------------------------------

### Get MinIO Tenant StatefulSet YAML

Source: https://hwameistor.io/docs/use_cases/minio

Retrieve the YAML configuration for the minio-t1 pool 0 statefulset in the minio-tenant namespace.

```bash
kubectl get statefulset.apps/minio-t1-pool-0 -nminio-tenant -oyaml
```

--------------------------------

### Check Kubernetes Node Status

Source: https://hwameistor.io/docs/install/post_check

Verify that all nodes in the Kubernetes cluster are in a 'Ready' state after installation. This command lists nodes, their status, roles, age, and Kubernetes version.

```bash
$ kubectl get node  
NAME           STATUS   ROLES   AGE   VERSION  
10-6-234-40   Ready    control-plane,master   140d   v1.21.11  
10-6-234-41   Ready    <none>                 140d   v1.21.11  
10-6-234-42   Ready    <none>                 140d   v1.21.11  
```

--------------------------------

### Use Clone Volume with Nginx Pod

Source: https://hwameistor.io/docs/volumes/volume_clone

Apply a Kubernetes Pod definition to utilize the cloned data volume. This example demonstrates creating an Nginx pod that mounts the cloned volume at `/data`.

```yaml
cat << EOF | kubectl apply -f - 
apiVersion: v1
kind: Pod
metadata:
  name: nginx
  namespace: default
spec:
  containers:
  - name: nginx
    image: docker.io/library/nginx:latest
    imagePullPolicy: IfNotPresent
    volumeMounts:
    - name: data
      mountPath: /data
    ports:
    - containerPort: 80
  volumes:
  - name: data
    persistentVolumeClaim:
      claimName: hwameistor-lvm-volume-clone
EOF

```

--------------------------------

### Create Hello World Table

Source: https://hwameistor.io/docs/use_cases/tidb

Execute a SQL statement to create a simple table named 'hello_world' with an auto-incrementing ID and a VARCHAR column.

```sql
create table hello_world (id int unsigned not null auto_increment primary key, v varchar(32));   

```

--------------------------------

### Install TiDB CRDs

Source: https://hwameistor.io/docs/use_cases/tidb

Apply the Custom Resource Definitions (CRDs) required for TiDB Operator. This is a prerequisite for installing the operator.

```bash
kubectl apply -f https://raw.githubusercontent.com/pingcap/tidb-operator/master/manifests/crd.yaml  

```

--------------------------------

### View and Bind SQL Execution Plans

Source: https://hwameistor.io/docs/use_cases/tidb

Demonstrates how to view the current execution plan of SQL statements and then apply a binding to enforce a specific plan, such as hash join.

```sql
explain select * from employees3 a join employees4 b on a.id = b.id where a.lname='Johnson';
```

```sql
explain select /*+ hash_join(a,b) */ * from employees3 a join employees4 b on a.id = b.id where a.lname='Johnson';
```

--------------------------------

### Apply Tenant Configuration

Source: https://hwameistor.io/docs/use_cases/minio

Apply the tenant configuration using kubectl with Kustomize. Ensure you have navigated to the correct directory and modified the necessary YAML files beforehand.

```bash
kubectl apply –k .
```

--------------------------------

### Create and Query Table with Object Isolation

Source: https://hwameistor.io/docs/use_cases/tidb

This script demonstrates creating a database and table, inserting data, setting up user permissions for read-only access, and performing a select query. It is used to test object isolation by implementing different schemas.

```sql
create database if not exists testdb;  
use testdb  
create table if not exists t_test  
( id                   bigint,  
  name                 varchar(200),  
  sale_time            datetime default current_timestamp,  
  constraint pk_t_test primary key (id)  
);  
insert into t_test(id,name) values (1,'a'),(2,'b'),(3,'c');  
create user 'readonly'@'%' identified by "readonly";  
grant select on testdb.* to readonly@'%';  
select * from testdb.t_test;  
update testdb.t_test set name='aaa';  
create user 'otheruser'@'%' identified by "otheruser";  


```

--------------------------------

### Deploy TiDB Cluster and Monitor

Source: https://hwameistor.io/docs/use_cases/tidb

Create a new namespace 'tidb-cluster' and apply the YAML configurations for deploying a basic TiDB cluster and its monitoring components.

```bash
kubectl create namespace tidb-cluster && \
kubectl -n tidb-cluster apply -f https://raw.githubusercontent.com/pingcap/tidb-operator/master/examples/basic/tidb-cluster.yaml   
kubectl -n tidb-cluster apply -f https://raw.githubusercontent.com /pingcap/tidb-operator/master/examples/basic/tidb-monitor.yaml  

```

--------------------------------

### Find Corresponding LocalVolume CR

Source: https://hwameistor.io/docs/volumes/volume_provisioned_io

Use `kubectl get pvc` to identify the PVC and then `kubectl get localvolume` to find the associated LocalVolume CR. This helps in locating the correct resource to modify.

```bash
$ kubectl get pvc pvc-sample  
  
NAME             STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                        AGE  
demo             Bound     pvc-c354a56a-5cf4-4ff6-9472-4e24c7371e10   10Gi       RWO            hwameistor-storage-lvm-hdd          5d23h  
pvc-sample       Bound     pvc-cac82087-6f6c-493a-afcd-09480de712ed   10Gi       RWO            hwameistor-storage-lvm-hdd-sample   5d23h  


$ kubectl get localvolume  
  
NAME                                       POOL                   REPLICAS   CAPACITY      USED       STATE   RESOURCE   PUBLISHED   FSTYPE   AGE  
pvc-c354a56a-5cf4-4ff6-9472-4e24c7371e10   LocalStorage_PoolHDD   1          10737418240   33783808   Ready   -1         master      xfs      5d23h  
pvc-cac82087-6f6c-493a-afcd-09480de712ed   LocalStorage_PoolHDD   1          10737418240   33783808   Ready   -1         master      xfs      5d23h  

```

--------------------------------

### Create MySQL StatefulSet

Source: https://hwameistor.io/docs/apps/lvm

Deploy a MySQL StatefulSet using a local volume. Ensure the `storageClassName` in `volumeClaimTemplates` matches your HwameiStor StorageClass.

```yaml
spec:
  volumeClaimTemplates:
  - metadata:
      name: data
      labels:
        app: sts-mysql-local
        app.kubernetes.io/name: sts-mysql-local
    spec:
      storageClassName: hwameistor-storage-lvm-hdd
      accessModes: ["ReadWriteOnce"]
      resources:
        requests:
          storage: 1Gi

```

--------------------------------

### Get All Metrics

Source: https://hwameistor.io/docs/metrics

Use curl to fetch all exposed metrics from the hwameistor-exporter endpoint.

```bash
curl hwameistor-exporter:80/metrics
```

--------------------------------

### Add Disk to Volume Group

Source: https://hwameistor.io/docs/terms/lvm

Use `pvcreate` to initialize a new disk as a physical volume, then `vgextend` to add it to an existing volume group. This increases the VG's capacity for LV expansion.

```bash
pvcreate /dev/sdc  
vgextend vg-sdb1 /dev/sdb3  
```

--------------------------------

### List Block Devices

Source: https://hwameistor.io/docs/volumes/volume_provisioned_io

Use `lsblk` to list all block devices and their mount points. This helps identify the device mapper names for volumes.

```bash
$ ls  
NAME            MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT  
sda               8:0    0   160G  0 disk  
├─sda1            8:1    0     1G  0 part /boot  
└─sda2            8:2    0   159G  0 part  
  ├─centos-root 253:0    0   300G  0 lvm  /  
  ├─centos-swap 253:1    0   7.9G  0 lvm  
  └─centos-home 253:2    0 101.1G  0 lvm  /home  
sdb               8:16   0   100G  0 disk  
├─LocalStorage_PoolHDD-pvc--cac82087--6f6c--493a--afcd--09480de712ed  
                253:3    0    10G  0 lvm  /var/lib/kubelet/pods/3d6bc980-68ae-4a65-a1c8-8b410b7d240f/v  
└─LocalStorage_PoolHDD-pvc--c354a56a--5cf4--4ff6--9472--4e24c7371e10  
                253:4    0    10G  0 lvm  /var/lib/kubelet/pods/521fd7b4-3bef-415b-8720-09225f93f231/v  
sdc               8:32   0   300G  0 disk  
└─sdc1            8:33   0   300G  0 part  
  └─centos-root 253:0    0   300G  0 lvm  /  
sr0              11:0    1   973M  0 rom  
```

--------------------------------

### Get Specific PVC YAML

Source: https://hwameistor.io/docs/use_cases/minio

Retrieve the YAML configuration for the 'export-minio6-0' PVC in the 'minio-6' namespace.

```bash
kubectl get pvc export-minio6-0 -nminio-6 -oyaml
```

--------------------------------

### Clone a PersistentVolumeClaim

Source: https://hwameistor.io/docs/volumes/volume_thin

This example demonstrates how to clone an existing PersistentVolumeClaim (PVC) by specifying the source PVC in the dataSource field.

```yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: cloned-pvc
spec:
  storageClassName: hwameistor-storage-lvm-thin-hdd
  dataSource:
    name: test-pvc
    kind: PersistentVolumeClaim
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi
---
apiVersion: v1
kind: Pod
metadata:
  name: cloned-pod
spec:
  containers:
  - name: busybox
    image: busybox:1.31.1
    command:
      - sleep
      - "360000000"
    imagePullPolicy: IfNotPresent
    volumeMounts:
    - name: temp-pvc
      mountPath: /mnt/temp-fs
  volumes:
  - name: temp-pvc
    persistentVolumeClaim:
      claimName: cloned-pvc
  terminationGracePeriodSeconds: 10

```

--------------------------------

### Create StatefulSet for MySQL

Source: https://hwameistor.io/docs/apps/ha

Deploy a MySQL StatefulSet using the HA StorageClass. Ensure the volumeClaimTemplates specify the correct storageClassName.

```bash
$ kubectl apply -f exapmles/sts-mysql_ha.yaml  
```

```yaml
spec:  
  volumeClaimTemplates:  
  - metadata:  
      name: data  
      labels:  
        app: sts-mysql-ha  
        app.kubernetes.io/name: sts-mysql-ha  
    spec:  
      storageClassName: hwameistor-storage-lvm-hdd-ha  
      accessModes: ["ReadWriteOnce"]  
      resources:  
        requests:  
          storage: 1Gi  
```

--------------------------------

### Create DataSet using kubectl

Source: https://hwameistor.io/docs/volumes/cache

Applies a DataSet configuration to the cluster using kubectl. Ensure the YAML file is saved as dataset.yaml.

```bash
kubectl apply -f dataset.yaml

```

--------------------------------

### Check Pod Status

Source: https://hwameistor.io/docs/volumes/volume_encrypt

Use kubectl to get detailed information about pods, including their readiness and current status.

```bash
# kubectl get pod -o wide  
NAME                                      READY   STATUS    RESTARTS   AGE   IP               NODE        NOMINATED NODE   READINESS GATES  
nginx-local-storage-lvm-79886d9dd-44fsg   1/1     Running   0          20m   100.111.156.91   k8s-node1  
```

--------------------------------

### Create LocalVolumeMigrate Resource for Pod Eviction

Source: https://hwameistor.io/docs/volumes/volume_eviction

Use this method to proactively trigger volume migration by creating a LocalVolumeMigrate custom resource. This specifies the source node, suggested target nodes, and the volume to migrate. The pod is then deleted.

```yaml
cat << EOF | kubectl apply -f - 
apiVersion: hwameistor.io/v1alpha1
kind: LocalVolumeMigrate
metadata:
  name: migrate-pvc-6ca4c0d4-da10-4e2e-83b2-19cbf5c5e3e4
spec:
  sourceNode: k8s-node-1
  targetNodesSuggested:
  - k8s-node-2
  - k8s-node-3
  volumeName: pvc-6ca4c0d4-da10-4e2e-83b2-19cbf5c5e3e4
  migrateAllVols: true
EOF

$ kubectl delete pod mysql-pod
```

--------------------------------

### Check cgroupv1 IOPS and Throughput Limits

Source: https://hwameistor.io/docs/volumes/volume_provisioned_io

For cgroupv1, read the `blkio.throttle.read_iops_device`, `blkio.throttle.write_iops_device`, `blkio.throttle.read_bps_device`, and `blkio.throttle.write_bps_device` files to check IOPS and bandwidth limits for a specific device.

```bash
# if cgroup version is v1.  

$ cat /sys/fs/cgroup/blkio/blkio.throttle.read_iops_device  
253:3 100  

$ cat /sys/fs/cgroup/blkio/blkio.throttle.write_iops_device  
253:3 100  

$ cat /sys/fs/cgroup/blkio/blkio.throttle.read_bps_device  
253:3 1048576  

$ cat /sys/fs/cgroup/blkio/blkio.throttle.write_bps_device  
253:3 1048576  
```

--------------------------------

### Get LocalDiskClaim Information

Source: https://hwameistor.io/docs/modules/ldm

Retrieve information about a specific LocalDiskClaim resource. The system automatically cleans up successful claims.

```bash
kubectl get localdiskclaim <name>
```

--------------------------------

### Get Specific PVC YAML for MinIO Tenant Pool

Source: https://hwameistor.io/docs/use_cases/minio

Retrieve the YAML configuration for the 'data0-minio-t1-pool-0-0' PVC in the 'minio-tenant' namespace.

```bash
kubectl get pvc data0-minio-t1-pool-0-0 -nminio-tenant -oyaml
```

--------------------------------

### Check HwameiStor Pods on Node

Source: https://hwameistor.io/docs/nodes_and_disks/lvm_nodes

Verify that the necessary HwameiStor pods are running on the target node. This ensures the node is ready to be configured as a storage node.

```bash
kubectl get node  
NAME           STATUS   ROLES            AGE     VERSION  
k8s-master-1   Ready    master           96d     v1.24.3-2+63243a96d1c393  
k8s-worker-1   Ready    worker           96h     v1.24.3-2+63243a96d1c393  
k8s-worker-2   Ready    worker           96h     v1.24.3-2+63243a96d1c393  
k8s-worker-3   Ready    worker           96d     v1.24.3-2+63243a96d1c393  
k8s-worker-4   Ready    worker           1h      v1.24.3-2+63243a96d1c393  
  
$ kubectl -n hwameistor get pod -o wide | grep k8s-worker-4  
hwameistor-local-disk-manager-c86g5     2/2     Running   0     19h   10.6.182.105      k8s-worker-4   <none>  <none>  
hwameistor-local-storage-s4zbw          2/2     Running   0     19h   192.168.140.82    k8s-worker-4   <none>  <none>  
  
# check if LocalStorageNode exists  
$ kubectl get localstoragenode k8s-worker-4  
NAME                 IP           ZONE      REGION    STATUS   AGE  
k8s-worker-4   10.6.182.103       default   default   Ready    8d  
```

--------------------------------

### Kubernetes ExpandableVolumePlugin Interface

Source: https://hwameistor.io/docs/terms/csi

Illustrates the Go interface for an expandable volume plugin within Kubernetes, enabling the expansion of volume device capacity and filesystem resizing checks.

```go
type ExpandableVolumePlugin interface {
VolumePlugin
ExpandVolumeDevice(spec Spec, newSize resource.Quantity, oldSize resource.Quantity) (resource.Quantity, error)
RequiresFSResize() bool
}
```

--------------------------------

### Check Available Disks on Node

Source: https://hwameistor.io/docs/apps/disk

Use this command to verify if there are any free disks available on your storage nodes before proceeding with disk volume creation.

```bash
$ kubectl get localdisknodes  
NAME           FREECAPACITY   TOTALCAPACITY   TOTALDISK   STATUS   AGE  
k8s-worker-2   1073741824     1073741824      1           Ready    19d  
```

--------------------------------

### Add and Update hwameistor-operator Helm Repo

Source: https://hwameistor.io/docs/install/operator

Before installing the operator, add its Helm repository and update it to fetch the latest chart information.

```bash
helm repo add hwameistor-operator https://hwameistor.io/hwameistor-operator
helm repo update hwameistor-operator
```

--------------------------------

### Create LocalVolumeMigrate Task

Source: https://hwameistor.io/docs/volumes/migrate

Create a LocalVolumeMigrate task to initiate volume migration. Configure source node, suggested target nodes, volume name, and whether to migrate all volumes.

```yaml
apiVersion: hwameistor.io/v1alpha1
kind: LocalVolumeMigrate
metadata:  
  namespace: hwameistor  
  name: <localVolumeMigrateName>  
  annotations:  
    hwameistor.io/replica-affinity: <need/forbid>   
spec:  
  sourceNode: <sourceNodeName>  
  targetNodesSuggested:   
  - <targetNodesName1>  
  - <targetNodesName2>  
  volumeName: <volName>  
  migrateAllVols: <true/false>  
```

--------------------------------

### Check HwameiStor Component Pods

Source: https://hwameistor.io/docs/install/post_check

List all pods in the 'hwameistor' namespace to ensure that HwameiStor components are running correctly. This is crucial for verifying the health of the storage system.

```bash
$ kubectl -n hwameistor get pod  
NAME                                                       READY   STATUS    RESTARTS      AGE  
drbd-adapter-k8s-master-rhel7-gtk7t                        0/2     Completed 0             23m  
drbd-adapter-k8s-node1-rhel7-gxfw5                         0/2     Completed 0             23m  
drbd-adapter-k8s-node2-rhel7-lv768                         0/2     Completed 0             23m  
hwameistor-admission-controller-dc766f976-mtlvw            1/1     Running   0             23m  
hwameistor-apiserver-86d6c9b7c8-v67gg                      1/1     Running   0             23m  
hwameistor-auditor-54f46fcbc6-jb4f4                        1/1     Running   0             23m  
hwameistor-exporter-6498478c57-kr8r4                       1/1     Running   0             23m  
hwameistor-failover-assistant-cdc6bd665-56wbw              1/1     Running   0             23m  
hwameistor-local-disk-csi-controller-6587984795-fztcd      2/2     Running   0             23m  
hwameistor-local-disk-manager-7gg9x                        2/2     Running   0             23m  
hwameistor-local-disk-manager-kqkng                        2/2     Running   0             23m  
hwameistor-local-disk-manager-s66kn                        2/2     Running   0             23m  
hwameistor-local-storage-csi-controller-5cdff98f55-jj45w   6/6     Running   0             23m  
hwameistor-local-storage-mfqks                             2/2     Running   0             23m  
hwameistor-local-storage-pnfpx                             2/2     Running   0             23m  
hwameistor-local-storage-whg9t                             2/2     Running   0             23m  
hwameistor-pvc-autoresizer-86dc79d57-s2l68                 1/1     Running   0             23m  
hwameistor-scheduler-6db69957f-r58j6                       1/1     Running   0             23m  
hwameistor-ui-744cd78d84-vktjq                             1/1     Running   0             23m  
hwameistor-volume-evictor-5db99cf979-4674n                 1/1     Running   0             23m  
```

--------------------------------

### Get LocalDisk Information

Source: https://hwameistor.io/docs/modules/ldm

Retrieve a list of locally discovered disk resources. The output shows the disk name, the node it's on, and its current phase.

```bash
kubectl get localdisk
```

--------------------------------

### Create StorageClass for Disk Volumes

Source: https://hwameistor.io/docs/apps/disk

Define a StorageClass to provision raw disk volumes. This example creates a StorageClass named 'hwameistor-storage-disk-ssd' for SSD type disks.

```yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: hwameistor-storage-disk-ssd
parameters:
  diskType: SSD
provisioner: disk.hwameistor.io
allowVolumeExpansion: false
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
```

--------------------------------

### Deploy Multi-Volume Pod

Source: https://hwameistor.io/docs/volumes/migrate

Deploy a pod that utilizes multiple data volumes.

```bash
kubectl apply -f nginx-multiple-lvm.yaml  
```

--------------------------------

### Get TiDB PD Pod YAML

Source: https://hwameistor.io/docs/use_cases/tidb

Retrieve the full YAML configuration of the 'basic-pd-0' pod in the 'tidb-cluster' namespace. This is useful for inspecting pod details and configurations.

```bash
kubectl get po basic-pd-0 -oyaml  

```
Tokens:
Raw