Huawei Cloud Container Engine CCE

https://support.huaweicloud.com/cce/
Admin
Cloud Container Engine (CCE) provides highly scalable, high-performance enterprise-grade Kubernetes...
Tokens:127,351
Snippets:1,267
Trust Score:10
Update:2 months ago
Show doc for...
Tokens:
Raw
### Execute NVIDIA Driver Installation

Source: https://support.huaweicloud.com/cce_faq/cce_faq_00491.html

Commands for installing the NVIDIA driver, including the standard silent installation and the variant using the --no-drm flag to bypass DRM-KMS module loading errors.

```bash
./NVIDIA-Linux-x86_64-535.54.03.run --silent --accept-license
```

```bash
./NVIDIA-Linux-x86_64-535.54.03.run --silent --accept-license --no-drm
```

--------------------------------

### 查看Kubelet CPU管理器状态

Source: https://support.huaweicloud.com/cce_faq/cce_faq_00414.html

在节点上检查CPU管理器状态，确认策略是否为static。

```bash
cat /var/lib/kubelet/cpu_manager_state
```

```json
{"policyName":"static","defaultCpuSet":"0,2-3","entries":{"c1fcd22d-8a83-4aef-a27a-4c037e482b16":{"container-1":"1"}},"checksum":1500530529}
```

--------------------------------

### 查看节点状态

Source: https://support.huaweicloud.com/cce_faq/cce_faq_00098.html

使用kubectl命令检查集群中节点的Ready状态，以确认是否存在可用节点。

```bash
$ kubectl get node
NAME           STATUS   ROLES    AGE   VERSION
192.168.0.37   Ready    <none>   21d   v1.19.10-r1.0.0-source-121-gb9675686c54267
192.168.0.71   Ready    <none>   21d   v1.19.10-r1.0.0-source-121-gb9675686c54267
```

--------------------------------

### Install NVIDIA Driver Dependencies

Source: https://support.huaweicloud.com/cce_faq/cce_faq_00491.html

Install the necessary development and library packages required for successful NVIDIA driver compilation on EulerOS.

```bash
yum install -y gcc gcc-c++ perl make elfutils-libelf-devel libX11 libXext binutils
```

--------------------------------

### 扩容逻辑卷与文件系统

Source: https://support.huaweicloud.com/cce_faq/cce_faq_00018.html

在节点磁盘扩容后，执行LVM和文件系统调整操作。

```bash
# lsblk
NAME                MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda                   8:0    0   50G  0 disk 
└─sda1                8:1    0   50G  0 part /
sdb                   8:16   0  150G  0 disk      # 数据盘已扩容至150G，存在50G空间仍未分配
├─vgpaas-dockersys  253:0    0   90G  0 lvm  /var/lib/containerd
└─vgpaas-kubernetes 253:1    0   10G  0 lvm  /mnt/paas/kubernetes/kubelet
```

```bash
pvresize /dev/sdb
```

```bash
Physical volume "/dev/sdb" changed
1 physical volume(s) resized or updated / 0 physical volume(s) not resized
```

```bash
lvextend -l+100%FREE -n vgpaas/dockersys
```

```bash
Size of logical volume vgpaas/dockersys changed from <90.00 GiB (23039 extents) to 140.00 GiB (35840 extents).
Logical volume vgpaas/dockersys successfully resized.
```

```bash
resize2fs /dev/vgpaas/dockersys
```

```bash
Filesystem at /dev/vgpaas/dockersys is mounted on /var/lib/containerd; on-line resizing required
old_desc_blocks = 12, new_desc_blocks = 18
The filesystem on /dev/vgpaas/dockersys is now 36700160 blocks long.
```

--------------------------------

### Diagnose NVIDIA Driver Installation Errors

Source: https://support.huaweicloud.com/cce_faq/cce_faq_00491.html

Use these commands to inspect installation logs and identify missing dependencies or kernel source issues.

```bash
cat /var/log/nvidia-installer.log
```

--------------------------------

### 检查节点关键组件状态与日志

Source: https://support.huaweicloud.com/cce_faq/cce_faq_00468.html

通过systemctl和journalctl命令检查Kubelet、Docker、Containerd及NTP组件的运行状态和日志。

```bash
systemctl status kubelet
```

```bash
journalctl -u kubelet
```

```bash
systemctl status docker
```

```bash
journalctl -u docker
```

```bash
systemctl status containerd
```

```bash
journalctl -u containerd
```

```bash
systemctl status chronyd
```

```bash
journalctl -u chronyd
```

--------------------------------

### CCE Ingress Inconsistency Log Examples

Source: https://support.huaweicloud.com/cce_faq/cce_faq_00493.html

Examples of log messages indicating inconsistencies in forwarding policies, rules, backend pools, and members, along with their required configurations.

```text
inconsistent: ingress(xxx) need create l7policy(xxx) ...
```

```json
{"attention":"Auto-generated by CCE service, do not modify!","cluster_id":"5d4d44bd-0891-11f0-84d3-0255ac10003e","ingress_id":"8a79a17b-f9ab-4431-95f6-0e1125093aac"}
```

```text
inconsistent: ingress(default/nginx) need create l7policy(k8s_default_nginx_6666cd76) of listener(0bdabd92-a85b-4f9b-96df-0fde706ea028), the wanted configuration is {"name":"k8s_default_nginx_6666cd76","listener_id":"0bdabd92-a85b-4f9b-96df-0fde706ea028","action":"REDIRECT_TO_POOL","redirect_pool_id":"k8s_default_nginx-80_HTTP-8000"}
```

```text
inconsistent: ingress(xxx) need update l7policy(xxx) of listener(xxx) ...
```

```text
inconsistent: ingress(default/nginx) need update l7policy(c4beb313-0a32-4add-9d0e-68b8f0f90e01) of listener(0bdabd92-a85b-4f9b-96df-0fde706ea028), the wanted configuration is {"id":"c4beb313-0a32-4add-9d0e-68b8f0f90e01","redirect_pool_id":"k8s_default_nginx-80_HTTP-8000","redirect_pools_config":[]}
```

```text
inconsistent: ingress(xxx) need delete l7policy(id: xxx) ...
```

```text
inconsistent: need delete l7policy(id: f4eda0af-869a-47fb-a699-2f27f286b641)
```

```text
inconsistent: ingress(xxx) need create rule of l7policy(xxx) ...
```

```text
inconsistent: ingress(default/nginx) need create rule of l7policy(k8s_default_nginx_6666cd76), the wanted configuration is {"type":"PATH","compare_type":"STARTS_WITH","value":"/"}

inconsistent: need delete rule(6cf87a3e-373e-4378-bf7a-16a709feee63) of l7policy(c4beb313-0a32-4add-9d0e-68b8f0f90e01)
```

```text
inconsistent: need create pool(xxx)...
```

```text
inconsistent: ingress(default/nginx) need create pool(k8s_default_nginx-80_HTTP-8000), the wanted configuration is {"name":"k8s_default_nginx-80_HTTP-8000","protocol":"HTTP","loadbalancer_id":"4a090bf3-9d11-45dc-8d04-f9cf010ba2ec","lb_algorithm":"ROUND_ROBIN","slow_start":{"enable":false,"duration":30}}
```

```text
inconsistent: need create member(address: xxx) of pool(xxx) ...
```

```text
inconsistent: need create member(address: 172.16.0.54) of pool(dab7b0c1-623e-4499-8c70-7477f36fc2ef), the wanted configuration is {"name":"2c2405d1abc17da48c70e9edc9a340fc","subnet_cidr_id":"77b9ad29-e30f-451b-92e7-949b83220b0f","address":"172.16.0.54","protocol_port":30838,"weight":6}
```

```text
inconsistent: need update member(address: xxx) of pool(xxx) ...
```

```text
inconsistent: need update member(9a65b96c-891a-439e-a692-508c3e095095) of pool(dab7b0c1-623e-4499-8c70-7477f36fc2ef), the wanted configuration is {"id":"9a65b96c-891a-439e-a692-508c3e095095","name":"2c2405d1abc17da48c70e9edc9a340fc","weight":6}
```

```text
inconsistent: need delete member(address: xxx) of pool(xxx) ...
```

```text
inconsistent: need delete members([d0891296-daf9-496b-9cac-ad1a1101b303]) of pool(dab7b0c1-623e-4499-8c70-7477f36fc2ef)
```

--------------------------------

### 检查device-plugin组件状态

Source: https://support.huaweicloud.com/cce_faq/cce_faq_00501.html

使用kubectl命令检查集群中nvidia相关组件的Pod状态。

```bash
kubectl get po -A -owide|grep nvidia
```

--------------------------------

### 查看节点Pod列表输出示例

Source: https://support.huaweicloud.com/cce_faq/cce_faq_00326.html

展示执行kubectl get pod命令后的输出结果。

```text
NAMESPACE     NAME                                      READY   STATUS    RESTARTS   AGE     IP              NODE            NOMINATED NODE   READINESS GATES
default       nginx-5bcc57c74b-lgcvh                    1/1     Running   0          7m25s   10.0.0.140      192.168.0.160   <none>           <none>
kube-system   coredns-6fcd88c4c-97p6s                   1/1     Running   0          3h16m   10.0.0.138      192.168.0.160   <none>           <none>
kube-system   everest-csi-controller-56796f47cc-99dtm   1/1     Running   0          3h16m   10.0.0.139      192.168.0.160   <none>           <none>
kube-system   everest-csi-driver-dpfzl                  2/2     Running   2          12d     192.168.0.160   192.168.0.160   <none>           <none>
kube-system   icagent-tpfpv                             1/1     Running   1          12d     192.168.0.160   192.168.0.160   <none>           <none>
```

--------------------------------

### 检查PCIE总线设备数量

Source: https://support.huaweicloud.com/cce_faq/cce_faq_00500.html

通过统计PCIE总线上的加速器数量来判断是否发生PCIE断链。

```bash
lspci | grep -i accelerator | wc -l
```

--------------------------------

### Check PLEG Health Status

Source: https://support.huaweicloud.com/cce_faq/cce_faq_00468.html

Example of a PLEG health error message found in Kubelet logs.

```text
skipping pod synchronization - PLEG is not healthy: pleg was last seen active 3m17.028393648s ago; threshold is 3m0s.
```

--------------------------------

### ELB Ingress Error Response

Source: https://support.huaweicloud.com/cce_faq/cce_faq_00483.html

Example of the error message returned by the ELB API when the rule condition limit is exceeded.

```json
Update elb(*****) listener(*****)error: status_code: 400, resp_body:{"error_msg":"Invalid input for rules. Reason: the number of condition for per policy must be no larger than 10.","error_code":"ELB.8902"."request_id": *****"}
```

--------------------------------

### 重启Kubelet服务

Source: https://support.huaweicloud.com/cce_faq/cce_faq_00468.html

在节点异常时尝试重启Kubelet进程，重启操作不会影响已运行的容器。

```bash
systemctl restart kubelet
```

```bash
systemctl status kubelet
```

```bash
journalctl -u kubelet
```

--------------------------------

### Node metrics JSON response

Source: https://support.huaweicloud.com/cce_faq/cce_faq_00426.html

Example of the raw JSON output returned by the metrics API containing usage statistics.

```json
{\n\t"kind": "NodeMetricsList",\n\t"apiVersion": "metrics.k8s.io/v1beta1",\n\t"metadata": {},\n\t"items": [\n\t\t{\n\t\t\t...\n\t\t\t"timestamp": "2023-08-15T14:09:38Z",\n\t\t\t"window": "1m0.177s",\n\t\t\t"usage": {\n\t\t\t\t"cpu": "78528126n",\n\t\t\t\t"memory": "2413824Ki"\n\t\t\t}\n\t\t}\n\t]\n}
```

--------------------------------

### 检查Docker服务与容器状态

Source: https://support.huaweicloud.com/cce_faq/cce_faq_00120.html

确认Docker服务运行状态并统计节点上的容器数量，以排查是否存在异常容器堆积。

```bash
systemctl status docker
```

```bash
docker ps -a | wc -l
```

--------------------------------

### HSTS Response Header Example

Source: https://support.huaweicloud.com/cce_faq/cce_faq_00489.html

The HTTP response header sent by the server that instructs browsers to enforce HTTPS for a specified duration.

```http
Strict-Transport-Security: max-age=31536000; includeSubDomains
```

--------------------------------

### 删除cpu_manager_state文件

Source: https://support.huaweicloud.com/cce_faq/cce_faq_00140.html

在变更节点规格后，删除该状态文件以解决CPU亲和性冲突。

```bash
rm -rf /mnt/paas/kubernetes/kubelet/cpu_manager_state
```

--------------------------------

### 检查组件运行状态

Source: https://support.huaweicloud.com/cce_faq/cce_faq_00120.html

用于查看kubelet等关键组件的运行状态，请根据实际组件名称替换命令中的kubelet。

```bash
systemctl status kubelet
```

--------------------------------

### Example Ingress configuration before property injection

Source: https://support.huaweicloud.com/cce_faq/cce_faq_00234.html

Initial Ingress definition containing the property annotation before the CCE controller processes the rules.

```yaml
kind: Ingress
apiVersion: extensions/v1beta1
metadata:
  name: test
  namespace: default
  resourceVersion: '2904229'
  generation: 1
  labels:
    isExternal: 'true'
    zone: data
  annotations:
    kubernetes.io/ingress.class: cce
    kubernetes.io/ingress.property: '[{"host":"test.com","path":"/test","matchmode":"STARTS_WITH"},{"Path":"/dw","MatchMode":"EQUAL_TO"}]'
spec:
  rules:
     - host: test.com
       http:
        paths:
          - path: /ss
            backend:
              serviceName: zlh-test
              servicePort: 80
          - path: /dw
            backend:
              serviceName: zlh-test
              servicePort: 80
```

--------------------------------

### Verify NVIDIA Driver on GPU Node

Source: https://support.huaweicloud.com/cce_faq/cce_faq_00109.html

Commands to check the NVIDIA driver status on GPU nodes based on the installed plugin version.

```bash
cd /opt/cloud/cce/nvidia/bin && ./nvidia-smi
```

```bash
cd /usr/local/nvidia/bin && ./nvidia-smi
```

--------------------------------

### 重启Docker服务

Source: https://support.huaweicloud.com/cce_faq/cce_faq_00317.html

应用配置更改后重启Docker服务。

```bash
systemctl restart docker
```

--------------------------------

### 查看驱动安装日志

Source: https://support.huaweicloud.com/cce_faq/cce_faq_00501.html

查看nvidia-driver-installer容器的安装日志以确认安装过程是否成功。

```bash
cat /usr/local/nvidia/nvidia-installer.log
```

--------------------------------

### Identify Pod Write Failure Event

Source: https://support.huaweicloud.com/cce_faq/cce_faq_00302.html

Example of the error message observed in Pod events when the node file system is in a read-only state.

```text
Message： Pod Update Plugin resources failed due to failed to write checkpoint file "kubelet_internal_checkpoint": open /var/lib/kubelet/device-plugins/.xxxxxx: read-only file system, which is unexpected.
```

--------------------------------

### Example Ingress configuration after property injection

Source: https://support.huaweicloud.com/cce_faq/cce_faq_00234.html

The resulting Ingress structure after the CCE controller has matched the annotation rules and injected the property fields into the spec.

```yaml
kind: Ingress
apiVersion: extensions/v1beta1
metadata:
  name: test
  namespace: default
  resourceVersion: '2904229'
  generation: 1
  labels:
    isExternal: 'true'
    zone: data
  annotations:
    kubernetes.io/ingress.class: cce
    kubernetes.io/ingress.property: '[{"host":"test.com","path":"/ss","matchmode":"STARTS_WITH"},{"host":"","path":"/dw","matchmode":"EQUAL_TO"}]'
spec:
  rules:
     - host: test.com
       http:
        paths:
          - path: /ss
            backend:
              serviceName: zlh-test
              servicePort: 80
            property:
              ingress.beta.kubernetes.io/url-match-mode: STARTS_WITH
          - path: /dw
            backend:
              serviceName: zlh-test
              servicePort: 80
```

--------------------------------

### 查看Pod状态与Kubelet日志

Source: https://support.huaweicloud.com/cce_faq/cce_faq_00209.html

使用kubectl查看Pod状态，并通过grep搜索kubelet日志中关于Evicted的记录。

```bash
kubectl get pods
```

```bash
cat /var/log/cce/kubernetes/kubelet.log | grep -i Evicted -C3
```

--------------------------------

### 扩容逻辑卷物理卷

Source: https://support.huaweicloud.com/cce_faq/cce_faq_00015.html

在磁盘扩容后，通过pvresize和lvextend命令调整逻辑卷大小。

```bash
pvresize /dev/sdb
```

```bash
Physical volume "/dev/sdb" changed
1 physical volume(s) resized or updated / 0 physical volume(s) not resized
```

```bash
lvextend -l+100%FREE -n vgpaas/dockersys
```

--------------------------------

### 查看节点污点信息

Source: https://support.huaweicloud.com/cce_faq/cce_faq_00098.html

使用kubectl describe命令查看特定节点上是否存在阻止Pod调度的污点。

```bash
$ kubectl describe node 192.168.0.37
Name:               192.168.0.37
...
Taints:             key1=value1:NoSchedule
...
```

--------------------------------

### 恢复openvswitch组件脚本

Source: https://support.huaweicloud.com/cce_faq/cce_faq_00182.html

在异常节点上使用root用户执行此脚本以重新安装并更新openvswitch组件。

```bash
#!/bin/bash
function upgrade_openvswitch()
{
    cd /opt/cloud/cce/package/node-package
    openvswitch_pkg=$(ls canal-agent*.tgz | tail -n 1)
    tar -xzf $openvswitch_pkg
    cd canal-agent/package
    tar -xzf openvswitch*
    cd openvswitch
    bash can_ovs.sh update
}
upgrade_openvswitch
```

--------------------------------

### 检查节点CPU限制

Source: https://support.huaweicloud.com/cce_faq/cce_faq_00018.html

查看节点CPU核心数限制，用于排查Go程序因核心数超过255导致的崩溃问题。

```bash
cat /sys/fs/cgroup/cpuset/cpuset.cpus
```

--------------------------------

### 处理节点运行时异常

Source: https://support.huaweicloud.com/cce_faq/cce_faq_00468.html

针对Docker或Containerd运行时异常进行重启和状态检查。

```bash
# Docker
systemctl restart docker
# Containerd
systemctl restart containerd
```

```bash
# Docker
systemctl status docker
# Containerd
systemctl status containerd
```

```bash
# Docker
journalctl -u docker
# containerd
journalctl -u containerd
```

--------------------------------

### 删除kernel.printk配置

Source: https://support.huaweicloud.com/cce_faq/cce_faq_00444.html

从配置文件中移除现有的kernel.printk设置。

```bash
sed -i '/^kernel.printk/d' /etc/sysctl.conf
```

--------------------------------

### 查询Pod挂载路径

Source: https://support.huaweicloud.com/cce_faq/cce_faq_00499.html

通过kubectl命令查看Pod的Volumes挂载信息，以排查hostPath或emptyDir导致的磁盘占用。

```bash
kubectl describe pods | grep -Pz  Volumes
```

--------------------------------

### 查询CentOS节点列表

Source: https://support.huaweicloud.com/cce_faq/cce_faq_00303.html

通过kubectl遍历集群节点并筛选出操作系统为CentOS的节点。

```bash
for node_item in $(kubectl get nodes --no-headers | awk '{print $1}') ; do kubectl get node ${node_item} -o yaml | grep CentOS >/dev/null; if [[ "$?" == "0" ]];then echo "${node_item} is CentOS node";fi;done
```

--------------------------------

### 查询Containerd日志配置

Source: https://support.huaweicloud.com/cce_faq/cce_faq_00499.html

查看kubelet配置文件以确认Containerd的日志大小和文件数量限制。

```bash
cat /opt/cloud/cce/kubernetes/kubelet/kubelet_config.yaml
```

```yaml
containerLogMaxSize: 50Mi
containerLogMaxFiles: 20
```

--------------------------------

### 配置hosts文件格式

Source: https://support.huaweicloud.com/cce_faq/cce_faq_00509.html

在Pod或节点内编辑/etc/hosts文件时使用的配置格式。

```text
127.0.x.x example.com
```

--------------------------------

### 验证nvidia-smi状态

Source: https://support.huaweicloud.com/cce_faq/cce_faq_00501.html

进入bin目录并执行nvidia-smi以检查驱动是否正常工作。

```bash
cd /usr/local/nvidia/bin
./nvidia-smi
```
Tokens:
Raw