### Install Helm Chart with Device Memory Scaling Source: https://github.com/project-hami/hami/blob/master/docs/config.md Example of installing the HAMi Helm chart and setting the device memory scaling parameter. ```bash helm install hami hami-charts/hami --set devicePlugin.deviceMemoryScaling=5 ... ``` -------------------------------- ### Install Dependencies and Tools Source: https://github.com/project-hami/hami/blob/master/docs/how-to-profiling-scheduler.md Prepare your profiling environment by installing Go, fetching HAMi source code, and installing necessary dependencies and tools. ```shell cd /k8s-vgpu make tidy go install github.com/NVIDIA/mig-parted/cmd/nvidia-mig-parted@v0.10.0 ``` -------------------------------- ### Device Registration Example Source: https://github.com/project-hami/hami/blob/master/docs/develop/protocol.md Example of node annotations for registered Nvidia and MLU devices. ```text hami.io/node-handshake-nvidia: Reported 2024-01-23 04:30:04.434037031 +0000 UTC m=+1104711.777756895 hami.io/node-handshake-mlu: Requesting_2024.01.10 04:06:57 hami.io/node-mlu-register: MLU-45013011-2257-0000-0000-000000000000,10,23308,0,MLU-MLU370-X4,0,false:MLU-54043011-2257-0000-0000-000000000000,10,23308,0, hami.io/node-nvidia-register: GPU-00552014-5c87-89ac-b1a6-7b53aa24b0ec,10,32768,100,NVIDIA-Tesla V100-PCIE-32GB,0,true:GPU-0fc3eda5-e98b-a25b-5b0d-cf5c855d1448,10,32768,100,NVIDIA-Tesla V100-PCIE-32GB,0,true: ``` -------------------------------- ### Schedule Decision Example Source: https://github.com/project-hami/hami/blob/master/docs/develop/protocol.md Example of pod annotations for a multi-container device allocation request. ```text hami.io/devices-to-allocate: GPU-0fc3eda5-e98b-a25b-5b0d-cf5c855d1448,NVIDIA,3000,0:GPU-0fc3eda5-e98b-a25b-5b0d-cf5c855d1448,NVIDIA,5000,0: hami.io/vgpu-node: node67-4v100 hami.io/vgpu-time: 1705054796 ``` -------------------------------- ### Applying an Example Workload Source: https://github.com/project-hami/hami/blob/master/README.md Command to apply a sample Kubernetes workload configuration to test HAMi's functionality. This assumes you have the example YAML file available. ```bash kubectl apply -f examples/nvidia/default_use.yaml ``` -------------------------------- ### Install HAMi with Iluvatar support Source: https://github.com/project-hami/hami/blob/master/docs/iluvatar-gpu-support.md Use Helm to install HAMi with the iluvatar device flag enabled. ```bash helm install hami hami-charts/hami --set scheduler.kubeScheduler.imageTag={your kubernetes version} --set devices.iluvatar.enabled=true ``` -------------------------------- ### Install HAMi via Helm Source: https://github.com/project-hami/hami/blob/master/docs/general-technical-review.md Commands to label GPU nodes, add the repository, and install the HAMi chart into the kube-system namespace. ```bash # 1) Label GPU nodes for HAMi management kubectl label nodes gpu=on # 2) Add HAMi Helm repository helm repo add hami-charts https://project-hami.github.io/HAMi/ helm repo update # 3) Install HAMi into kube-system helm install hami hami-charts/hami -n kube-system # 4) (Optional) Customize values during install # helm install hami hami-charts/hami -n kube-system -f values.yaml # 5) Verify core components are running kubectl get pods -n kube-system | grep -E "hami-scheduler|hami-device-plugin" ``` -------------------------------- ### Example Allocation Profile Analysis Source: https://github.com/project-hami/hami/blob/master/docs/how-to-profiling-scheduler.md This is an example of the output from analyzing the allocation profile using `go tool pprof`. It shows the top memory-consuming functions and their breakdown. The analysis was performed with a 120-second duration. ```bash /usr/local/go/bin/go tool pprof --seconds 120 https+insecure://10.42.0.24/debug/pprof/allocs ``` ```text Fetching profile over HTTP from https+insecure://10.42.0.24/debug/pprof/allocs?seconds=120 Saved profile in /root/pprof/pprof.scheduler.alloc_objects.alloc_space.inuse_objects.inuse_space.041.pb.gz File: scheduler Type: alloc_space Time: 2025-04-01 07:03:05 UTC Entering interactive mode (type "help" for commands, "o" for options) (pprof) top Showing nodes accounting for 4383.93MB, 69.18% of 6336.84MB total Dropped 376 nodes (cum <= 31.68MB) Showing top 10 nodes out of 164 flat flat% sum% cum cum% 1114.44MB 17.59% 17.59% 1114.94MB 17.59% io.ReadAll 980.52MB 15.47% 33.06% 980.52MB 15.47% sync.(*Pool).pinSlow 606.88MB 9.58% 42.64% 606.88MB 9.58% golang.org/x/net/http2.init.func5 357.15MB 5.64% 48.27% 357.15MB 5.64% k8s.io/apimachinery/pkg/runtime.(*RawExtension).UnmarshalJSON 293.20MB 4.63% 52.90% 293.20MB 4.63% reflect.mapassign_faststr0 265.58MB 4.19% 57.09% 265.58MB 4.19% reflect.unsafe_NewArray 234.07MB 3.69% 60.78% 461.59MB 7.28% sigs.k8s.io/json/internal/golang/encoding/json.(*decodeState).literalStore 210.54MB 3.32% 64.11% 3409.63MB 53.81% github.com/Project-HAMi/HAMi/pkg/scheduler.(*Scheduler).RegisterFromNodeAnnotations 162.02MB 2.56% 66.66% 331.76MB 5.24% github.com/Project-HAMi/HAMi/pkg/scheduler.(*Scheduler).getNodesUsage 159.52MB 2.52% 69.18% 225.53MB 3.56% encoding/json.Unmarshal (pprof) list RegisterFromNodeAnnotations Total: 6.21GB ROUTINE ======================== github.com/Project-HAMi/HAMi/pkg/scheduler.(*Scheduler).RegisterFromNodeAnnotations in /k8s-vgpu/pkg/scheduler/scheduler.go 210.54MB 3.33GB (flat, cum) 53.73% of Total . . 158:func (s *Scheduler) RegisterFromNodeAnnotations() { . . 159: klog.InfoS("Entering RegisterFromNodeAnnotations") . . 160: defer klog.InfoS("Exiting RegisterFromNodeAnnotations") . . 161: ticker := time.NewTicker(time.Second * 15) . . 162: defer ticker.Stop() . . 163: printedLog := map[string]bool{} . . 164: for { 512.05kB 512.05kB 165: select { . . 166: case <-s.nodeNotify: . . 167: klog.V(5).InfoS("Received node notification") . . 168: case <-ticker.C: . 46.84MB 169: klog.InfoS("Ticker triggered") . . 170: case <-s.stopCh: . . 171: klog.InfoS("Received stop signal, exiting RegisterFromNodeAnnotations") . . 172: return . . 173: } . . 174: labelSelector := labels.Everything() . . 175: if len(config.NodeLabelSelector) > 0 { . . 176: labelSelector = (labels.Set)(config.NodeLabelSelector).AsSelector() . . 177: klog.InfoS("Using label selector", "selector", labelSelector.String()) . . 178: } . . 179: rawNodes, err := s.nodeLister.List(labelSelector) . . 180: if err != nil { . . 181: klog.ErrorS(err, "Failed to list nodes with selector", "selector", labelSelector.String()) . . 182: continue . . 183: } 1MB 1MB 184: klog.V(5).InfoS("Listed nodes", "nodeCount", len(rawNodes)) . . 185: var nodeNames []string . . 186: for _, val := range rawNodes { 1.50MB 1.50MB 187: nodeNames = append(nodeNames, val.Name) 5.50MB 5.50MB 188: klog.V(5).InfoS("Processing node", "nodeName", val.Name) . . 189: . . 190: for devhandsk, devInstance := range device.GetDevices() { 36.50MB 36.50MB 191: klog.V(5).InfoS("Checking device health", "nodeName", val.Name, "deviceVendor", devhandsk) . . 192: . . 193: health, needUpdate := devInstance.CheckHealth(devhandsk, val) 72.51MB 72.51MB 194: klog.V(5).InfoS("Device health check result", "nodeName", val.Name, "deviceVendor", devhandsk, "health", health, "needUpdate", needUpdate) . . 195: . . 196: if !health { . . 197: klog.Warning("Device is unhealthy, cleaning up node", "nodeName", val.Name, "deviceVendor", devhandsk) . . 198: err := devInstance.NodeCleanUp(val.Name) ``` -------------------------------- ### Install HAMi using Helm Source: https://github.com/project-hami/hami/blob/master/docs/offline-install.md Execute this Helm command to install HAMi after configuring the chart values. Ensure you are in the HAMi chart directory. ```bash helm install hami hami --set scheduler.kubeScheduler.imageTag={your k8s server version} -n kube-system ``` -------------------------------- ### Install HAMi with Topology-Aware Scheduling Source: https://github.com/project-hami/hami/blob/master/docs/proposals/nvidia-gpu-topology-scheduler.md Install HAMi using Helm, setting the gpuSchedulerPolicy to topology-aware. This is the initial setup for enabling topology-aware scheduling. ```bash helm install hami hami-charts/hami --set scheduler.defaultSchedulerPolicy.gpuSchedulerPolicy=topology-aware -n kube-system ``` -------------------------------- ### Example pprof CPU Profile Output Source: https://github.com/project-hami/hami/blob/master/docs/how-to-profiling-scheduler.md An example of the interactive pprof output after capturing a CPU profile, showing top functions by CPU usage. ```bash root@hami-pprof-76cfcb66f6-jpjnm:/# go tool pprof --seconds 120 https+insecure:://10.42.0.24/debug/pprof/profile Fetching profile over HTTP from https+insecure://10.42.0.24/debug/pprof/profile?seconds=120 Please wait... (2m0s) Saved profile in /root/pprof/pprof.scheduler.samples.cpu.002.pb.gz File: scheduler Type: cpu Time: 2025-04-01 07:08:42 UTC Duration: 120s, Total samples = 10ms (0.0083%) Entering interactive mode (type "help" for commands, "o" for options) (pprof) top Showing nodes accounting for 10ms, 100% of 10ms total Showing top 10 nodes out of 12 flat flat% sum% cum cum% 10ms 100% 100% 10ms 100% sigs.k8s.io/json/internal/golang/encoding/json.unquoteBytes 0 0% 100% 10ms 100% k8s.io/apimachinery/pkg/runtime.Decode (inline) 0 0% 100% 10ms 100% k8s.io/apimachinery/pkg/runtime.WithoutVersionDecoder.Decode 0 0% 100% 10ms 100% k8s.io/apimachinery/pkg/runtime/serializer/json.(*Serializer).Decode 0 0% 100% 10ms 100% k8s.io/apimachinery/pkg/runtime/serializer/json.(*Serializer).unmarshal 0 0% 100% 10ms 100% k8s.io/apimachinery/pkg/watch.(*StreamWatcher).receive 0 0% 100% 10ms 100% k8s.io/client-go/rest/watch.(*Decoder).Decode 0 0% 100% 10ms 100% sigs.k8s.io/json.UnmarshalCaseSensitivePreserveInts (inline) 0 0% 100% 10ms 100% sigs.k8s.io/json/internal/golang/encoding/json.(*decodeState).object 0 0% 100% 10ms 100% sigs.k8s.io/json/internal/golang/encoding/json.(*decodeState).unmarshal ``` -------------------------------- ### HAMi Scheduler Log Example Source: https://github.com/project-hami/hami/blob/master/docs/proposals/nvidia-gpu-topology-scheduler.md Example log output from hami-scheduler indicating a successful device allocation for a pod. Ensure the log level is greater than 5 to view these messages. ```bash I0703 08:34:27.032644 1 device.go:708] "device allocate success" pod="default/testpod" best device combination={"NVIDIA":[{"Idx":7,"UUID":"GPU-dsaf","Type":"NVIDIA","Usedmem":1024,"Usedcores":0,"CustomInfo":null},{"Idx":5,"UUID":"GPU-gads","Type":"NVIDIA","Usedmem":1024,"Usedcores":0,"CustomInfo":null}]} ``` -------------------------------- ### Installing HAMi with Helm Source: https://github.com/project-hami/hami/blob/master/README.md Command to install HAMi into the 'kube-system' namespace using Helm. This deploys the HAMi scheduler and device plugin. ```bash helm install hami hami-charts/hami -n kube-system ``` -------------------------------- ### Install HAMi with Enflame GCU support Source: https://github.com/project-hami/hami/blob/master/docs/enflame-gcu-support.md Use Helm to install HAMi with the Enflame device support enabled. ```bash helm install hami hami-charts/hami --set devices.enflame.enabled=true -n kube-system ``` -------------------------------- ### Example v5 Device Detail Logs Source: https://github.com/project-hami/hami/blob/master/docs/scheduler-event-log.md Sample logs demonstrating the v5-level logging format for various device allocation failure scenarios. ```text (v=5) I0422 02:15:42.349712 1 score.go:210] NodeInsufficientDevice pod="llm/llm/deepseek-5996b8569d-kgwgx" node="node2" request devices nums=2 node device nums=1 (v=5) I0422 02:15:42.349712 1 score.go:99] CardTypeMismatch pod="llm/deepseek-5996b8569d-kgwgx" node="node1" device="GPU-0fc3eda5-e98b-a25b-5b0d-cf5c855d1448" DCU="DCU-f4502784-0000-0000-0000-000000000000" (v=5) I0422 02:15:42.349712 1 score.go:499] CardInsufficientMemory pod="llm/deepseek-5996b8569d-kgwgx" node="node3" device="GPU-62b7408e-edb2-41d1-bc91-f46165c61130" device index=0 device total memory=50 device used memory=0 request memory=1000 (v=5) I0422 02:15:42.349712 1 score.go:99] CardTypeMismatch pod="llm/deepseek-5996b8569d-kgwgx" node="node2" device="GPU-006aa3c3-b59a-4fc1-8480-c5676c7bedbe" DCU="DCU-7b8fb3af-fa73-4717-8ce9-2798684431b0" (v=5) I0422 02:15:42.349712 1 score.go:205] NumaNotFit pod="llm/deepseek-5996b8569d-kgwgx" node="node1" device="GPU-cc244283-5652-4c35-81b0-0f54d75c0a56", k.nums=1 numa=true prevnuma=1 device numa=0 (v=5) I0422 02:15:42.349712 1 score.go:205] NumaNotFit pod="llm/deepseek-5996b8569d-kgwgx" node="node2" device="GPU-acaecde2-cfed-4240-8d86-3155a7648a8b", k.nums=1 numa=true prevnuma=1 device numa=1 (v=5) I0422 02:15:42.349712 1 score.go:137] CardInsufficientMemory pod="llm/deepseek-5996b8569d-kgwgx" node="node2" device="GPU-c12ea36b-23c1-400a-a18f-df9d25b956f0" device index=0 device total memory=45912 device used memory=41645 request memory=20480 (v=5) I0422 02:15:42.349712 1 score.go:137] CardInsufficientMemory pod="llm/deepseek-5996b8569d-kgwgx" node="node1" device="GPU-4972a185-507a-4e0b-82b4-ef5af46fa229" device index=0 device total memory=45912 device used memory=39645 request memory=20480 (v=5) I0422 02:15:42.349712 1 score.go:137] CardInsufficientMemory pod="llm/deepseek-5996b8569d-kgwgx" node="node3" device="GPU-0fs12a5-e98b-a25b-s454-cf5c855d1448" device index=0 device total memory=45912 device used memory=27645 request memory=20480 (v=5) I0422 02:15:42.349712 1 score.go:137] CardInsufficientMemory pod="llm/deepseek-5996b8569d-kgwgx" node="node4" device="GPU-f5fa8195-324f-47be-a514-c3f856b4fef2" device index=0 device total memory=45912 device used memory=36145 request memory=20480 (v=5) I0422 02:15:42.349712 1 score.go:137] CardInsufficientMemory pod="llm/deepseek-5996b8569d-kgwgx" node="node1" device="GPU-9e83b1fc-a3a6-4b7b-920a-66c344f0955e" device index=0 device total memory=45912 device used memory=37645 request memory=20480 (v=5) I0422 02:15:42.349712 1 score.go:137] CardInsufficientMemory pod="llm/deepseek-5996b8569d-kgwgx" node="node2" device="GPU-9518dec4-e81e-44b5-973f-567be552fd4c" device index=0 device total memory=45912 device used memory=37645 request memory=20480 (v=5) I0422 02:15:42.349712 1 score.go:137] CardInsufficientMemory pod="llm/deepseek-5996b8569d-kgwgx" node="node4" device="GPU-acaecde2-cfed-4240-8d86-3155a7648a8b" device index=0 device total memory=45912 device used memory=37645 request memory=20480 (v=5) I0422 02:15:42.349712 1 score.go:142] CardInsufficientCore pod="llm/deepseek-5996b8569d-kgwgx" node="node3" device="GPU-f4c45e36-2860-4050-99bd-17a437c3e53c" device index=4 device total core=100 device used core=90 request cores=20 (v=5) I0422 02:15:42.349712 1 score.go:142] CardInsufficientCore pod="llm/deepseek-5996b8569d-kgwgx" node="node2" device="GPU-477b6dba-1cb6-4de4-9aa8-14ec4cf4db8a" device index=0 device total core=100 device used core=85 request cores=20 (v=5) I0422 02:15:42.349712 1 score.go:142] CardInsufficientCore pod="llm/deepseek-5996b8569d-kgwgx" node="node1" device="GPU-05ba43a8-1eb5-4645-8bdd-7fab4276ed87" device index=7 device total core=100 device used core=95 request cores=20 (v=5) I0422 02:15:42.349712 1 score.go:142] CardInsufficientCore pod="llm/deepseek-5996b8569d-kgwgx" node="node1" device="GPU-41ee1864-ebdc-4846-a304-f49ca3105f9d" device index=3 device total core=100 device used core=89 request cores=20 (v=5) I0422 02:15:42.349712 1 score.go:148] ExclusiveDeviceAllocateConflict pod="llm/deepseek-5996b8569d-kgwgx" node="node1" device index=0 used=9 ``` -------------------------------- ### Install Volcano via Helm Source: https://github.com/project-hami/hami/blob/master/docs/how-to-use-volcano-ascend.md Adds the Volcano Helm repository and installs the scheduler into the cluster. ```shell helm repo add volcano-sh https://volcano-sh.github.io/helm-charts helm install volcano volcano-sh/volcano -n volcano-system --create-namespace ``` -------------------------------- ### Example vDCU device output Source: https://github.com/project-hami/hami/blob/master/docs/hygon-dcu-support.md Sample output indicating successful vDCU configuration inside a container. ```text Device 0: Actual Device: 0 Compute units: 60 Global memory: 2097152000 bytes ``` -------------------------------- ### Example: Pod with Specific Ascend 910B3 UUID Source: https://github.com/project-hami/hami/blob/master/docs/ascend910b-support_cn.md This complete example demonstrates using the `hami.io/use-Ascend910B-uuid` annotation to ensure a Pod runs only on Ascend910B3 devices with specified UUIDs. Ensure the container image is compatible. ```yaml apiVersion: v1 kind: Pod metadata: name: ascend-pod annotations: hami.io/use-Ascend910B-uuid: "device-uuid-1,device-uuid-2" spec: containers: - name: ubuntu-container image: ascendhub.huawei.com/public-ascendhub/ascend-mindspore:23.0.RC3-centos7 command: ["bash", "-c", "sleep 86400"] resources: limits: huawei.com/Ascend910B3: 1 huawei.com/Ascend910B3-memory: 2000 ``` -------------------------------- ### Run XPU Tasks in Pod Source: https://github.com/project-hami/hami/blob/master/docs/kunlun-vxpu-support.md Example Pod configuration requesting a specific VXPU count and memory allocation. ```yaml apiVersion: v1 kind: Pod metadata: name: vxpu-pod-demo spec: containers: - name: vxpu-pod-demo image: pytorch:resnet50 workingDir: /root command: ["sleep","infinity"] resources: limits: kunlunxin.com/vxpu: 1 # requesting a VXPU kunlunxin.com/vxpu-memory: 24576 # requesting a virtual XPU that requires 24576 MiB of device memorymemory ``` -------------------------------- ### Build Profiling Docker Image Source: https://github.com/project-hami/hami/blob/master/docs/how-to-profiling-scheduler.md Build a Docker image with Go and HAMi source code, including dependency installation, for profiling. ```docker FROM golang:1.24.4-bullseye ADD . /k8s-vgpu RUN cd /k8s-vgpu && make tidy RUN go install github.com/NVIDIA/mig-parted/cmd/nvidia-mig-parted@v0.10.0 ``` -------------------------------- ### Configure Kunlunxin VXPU Resources Source: https://github.com/project-hami/hami/blob/master/docs/kunlun-vxpu-support.md Examples of valid and invalid resource configurations for Kunlunxin chips, noting the 32-handle limit. ```yaml # valid kunlunxin.com/vxpu: 8 # valid kunlunxin.com/vxpu: 6 kunlunxin.com/vxpu-memory: 24576 # valid kunlunxin.com/vxpu: 8 kunlunxin.com/vxpu-memory: 49152 # invalid kunlunxin.com/vxpu: 8 # not support kunlunxin.com/vxpu-memory: 24576 ``` -------------------------------- ### Get Pods Overview Source: https://github.com/project-hami/hami/blob/master/skill/k8s-debug-gpu-pod/SKILL.md Retrieves a wide view of all pods in a namespace to categorize their states. ```bash kubectl get pods -n -o wide ``` -------------------------------- ### Install Hami with Mthreads GPU Sharing Enabled Source: https://github.com/project-hami/hami/blob/master/docs/mthreads-support.md Use this Helm command to install the hami chart with mthreads GPU sharing enabled. Ensure you replace {your kubernetes version} with your actual Kubernetes version. ```bash helm install hami hami-charts/hami --set scheduler.kubeScheduler.imageTag={your kubernetes version} --set device.mthreads.enabled=true -n kube-system ``` -------------------------------- ### Define a GPU-enabled Pod Source: https://github.com/project-hami/hami/blob/master/docs/scheduler-event-log.md Example configuration for a Pod requesting specific NVIDIA GPU resources. ```yaml apiVersion: v1 kind: Pod metadata: name: gpu-pod spec: containers: - name: worker01 image: ubuntu:22.04 command: ["bash", "-c", "sleep 86400"] resources: limits: nvidia.com/gpu: "1" nvidia.com/gpumem: "3000" nvidia.com/gpucores: "30" ``` -------------------------------- ### Deploy dcgm-exporter with Helm Source: https://github.com/project-hami/hami/blob/master/docs/dashboard.md Commands to clone the repository, update the serviceMonitor configuration in values.yaml, and install the exporter. ```shell #Clone dcgm-exporter code repository(the compatibility matrix for dcgm-export and kubernetes is not been founded in its official website, using "tag 3.2.5-3.1.7" here) git clone -b 3.2.5-3.1.7 https://github.com/NVIDIA/dcgm-exporter.git cd dcgm-exporter #Modified file "deployment/values.yaml", add configurations under relabelings in serviceMonitor section, these configurations are necessary for dashboard page display #There is only one serviceMonitor in file "deployment/values.yaml", and the contents of the modified file are as follows vi deployment/values.yaml ... serviceMonitor: apiVersion: "monitoring.coreos.com/v1" enabled: true interval: 15s honorLabels: false additionalLabels: {} #monitoring: prometheus relabelings: - sourceLabels: [__meta_kubernetes_endpoints_name] regex: dcgm-exporter replacement: $1 action: keep - sourceLabels: [__meta_kubernetes_pod_node_name] regex: (.*) targetLabel: node_name replacement: ${1} action: replace - sourceLabels: [__meta_kubernetes_pod_host_ip] regex: (.*) targetLabel: ip replacement: $1 action: replace ... #Install dcgm-exporter under monitoring namespace with helm helm install dcgm-exporter deployment/ -n monitoring #Check the result of running "helm install ..." helm list -n monitoring NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION dcgm-exporter monitoring 1 2024-10-02 16:32:35.691073696 +0800 CST deployed dcgm-exporter-3.1.7 3.1.7 #Make sure pod(s)/dcgm-exporter-xxx is(are) in running status kubectl -n monitoring get pods | egrep "NAME|dcgm-exporter" NAME READY STATUS RESTARTS AGE dcgm-exporter-p4fdp 1/1 Running 0 4h14m ``` -------------------------------- ### Define Ascend Pod Resource Request Source: https://github.com/project-hami/hami/blob/master/docs/how-to-use-volcano-ascend.md Example Pod specification requesting Ascend 310P resources. ```yaml apiVersion: v1 kind: Pod metadata: name: ascend-pod spec: schedulerName: volcano containers: - name: ubuntu-container image: swr.cn-south-1.myhuaweicloud.com/ascendhub/ascend-pytorch:24.0.RC1-A2-1.11.0-ubuntu20.04 command: ["sleep"] args: ["100000"] resources: limits: huawei.com/Ascend310P: "1" huawei.com/Ascend310P-memory: "4096" ``` -------------------------------- ### Get Container Resource Configuration Source: https://github.com/project-hami/hami/blob/master/skill/k8s-debug-gpu-pod/SKILL.md Fetches and formats the resource requests and limits for a pod's containers using jq. ```bash kubectl get pod -n -o jsonpath='{.spec.containers[*].resources}' | jq . ``` -------------------------------- ### Request GPU Resources with MIG Mode Source: https://github.com/project-hami/hami/blob/master/docs/develop/dynamic-mig.md Example of a Pod requesting GPU resources and explicitly setting the vGPU mode to 'mig'. ```yaml apiVersion: v1 kind: Pod metadata: name: gpu-pod1 annotations: nvidia.com/vgpu-mode: "mig" spec: containers: - name: ubuntu-container1 image: ubuntu:20.04 command: ["bash", "-c", "sleep 86400"] resources: limits: nvidia.com/gpu: 2 # requesting 2 vGPUs nvidia.com/gpumem: 8000 # Each vGPU contains 8000m device memory （Optional,Integer ``` -------------------------------- ### Run Benchmark Locally with Docker Source: https://github.com/project-hami/hami/blob/master/benchmarks/README.md Execute the benchmark locally using a Docker container. The script starts a vLLM container, waits for it to be ready, runs the benchmark, and then stops the container. ```bash cd ai-benchmark sh run_bench.sh ``` -------------------------------- ### Enable Enflame Support in HAMi Source: https://github.com/project-hami/hami/blob/master/docs/enflame-vgcu-support.md Install HAMi using Helm, enabling Enflame support with the specified configuration. Ensure you are in the correct namespace. ```bash helm install hami hami-charts/hami --set devices.enflame.enabled=true -n kube-system ``` -------------------------------- ### Request Iluvatar GPU resources in a Pod Source: https://github.com/project-hami/hami/blob/master/docs/iluvatar-gpu-support.md Example Pod specification requesting specific vGPU, vCore, and vMem resources for an Iluvatar BI-V150 card. ```yaml apiVersion: v1 kind: Pod metadata: name: BI-V150-poddemo spec: restartPolicy: Never containers: - name: BI-V150-poddemo image: registry.iluvatar.com.cn:10443/saas/mr-bi150-4.3.0-x86-ubuntu22.04-py3.10-base-base:v1.0 command: - bash args: - -c - | set -ex echo "export LD_LIBRARY_PATH=/usr/local/corex/lib64:$LD_LIBRARY_PATH">> /root/.bashrc cp -f /usr/local/iluvatar/lib64/libcuda.* /usr/local/corex/lib64/ cp -f /usr/local/iluvatar/lib64/libixml.* /usr/local/corex/lib64/ source /root/.bashrc sleep 360000 resources: requests: iluvatar.ai/BI-V150-vgpu: 1 iluvatar.ai/BI-V150.vCore: 50 iluvatar.ai/BI-V150.vMem: 64 limits: iluvatar.ai/BI-V150-vgpu: 1 iluvatar.ai/BI-V150.vCore: 50 iluvatar.ai/BI-V150.vMem: 64 ``` -------------------------------- ### Accessing HAMi Metrics Source: https://github.com/project-hami/hami/blob/master/README.md Example of how to access HAMi's monitoring metrics endpoint. The default port is 31993, but this can be configured during Helm installation. ```text http://:/metrics ``` -------------------------------- ### Build and Push Server and Client Images Source: https://github.com/project-hami/hami/blob/master/benchmarks/README.md Navigate to the ai-benchmark directory and execute the build script to build and push both the server and client images. Override the registry and tag as needed. ```bash cd ai-benchmark # Build and push both server and client images REGISTRY=docker.io/your-org TAG=v0.1.0 sh build.sh ``` -------------------------------- ### Build All Binaries Source: https://github.com/project-hami/hami/blob/master/CLAUDE.md Use this command to build all project binaries, including the scheduler, vGPUmonitor, and nvidia-device-plugin. Binaries are placed in the bin/ directory. ```bash make build # Build all binaries (scheduler, vGPUmonitor, nvidia-device-plugin) to bin/ ``` -------------------------------- ### Initialize NodeInfo Structure Source: https://github.com/project-hami/hami/blob/master/docs/how-to-profiling-scheduler.md Initializes a NodeInfo structure with basic details like ID and the node itself. ```Go nodeInfo := &device.NodeInfo{} nodeInfo.ID = val.Name nodeInfo.Node = val ``` -------------------------------- ### Configure Device-Share Prestart Command Source: https://github.com/project-hami/hami/blob/master/docs/iluvatar-gpu-support.md This script must be set as the prestart command for device-share to function correctly. It configures library paths and copies necessary CUDA and ixml libraries. ```sh set -ex echo "export LD_LIBRARY_PATH=/usr/local/corex/lib64:$LD_LIBRARY_PATH">> /root/.bashrc cp -f /usr/local/iluvatar/lib64/libcuda.* /usr/local/corex/lib64/ cp -f /usr/local/iluvatar/lib64/libixml.* /usr/local/corex/lib64/ source /root/.bashrc ``` -------------------------------- ### Build Benchmark Images Source: https://github.com/project-hami/hami/blob/master/docs/benchmark.md Builds the necessary Docker images for running AI benchmarks. Navigate to the benchmark directory before executing. ```bash cd benchmarks/ai-benchmark sh build.sh ``` -------------------------------- ### Find Ascend Device UUIDs using kubectl get node -o yaml Source: https://github.com/project-hami/hami/blob/master/docs/ascend910b-support_cn.md Retrieve Ascend device UUIDs by inspecting the node's YAML configuration using `kubectl get node -o yaml`. Search for annotations like `hami.io/node-register-Ascend910B`. ```bash kubectl get node -o yaml | grep -A 10 "annotations:" ``` -------------------------------- ### Deploy HAMi Device Plugin Components Source: https://github.com/project-hami/hami/blob/master/docs/how-to-use-volcano-ascend.md Applies the configuration map and device plugin manifests required for Ascend vNPU support. ```shell kubectl apply -f https://raw.githubusercontent.com/Project-HAMi/ascend-device-plugin/refs/heads/main/ascend-device-configmap.yaml ``` ```shell kubectl apply -f https://raw.githubusercontent.com/Project-HAMi/ascend-device-plugin/refs/heads/main/ascend-device-plugin.yaml ``` -------------------------------- ### Deploy Benchmark on Native NVIDIA Device Plugin Source: https://github.com/project-hami/hami/blob/master/benchmarks/README.md Apply the Kubernetes job configuration to run the benchmark on the native NVIDIA device plugin. This serves as a baseline for comparison against the HAMi vGPU results. ```bash kubectl apply -f deployments/job-on-nvidia-device-plugin.yml ``` -------------------------------- ### Create ServiceMonitor for hami-device-plugin Source: https://github.com/project-hami/hami/blob/master/docs/dashboard.md Commands to create and apply the ServiceMonitor YAML for the hami-device-plugin. ```shell #Create the file hami-device-plugin-svc-monitor.yaml root@controller01:~# touch hami-device-plugin-svc-monitor.yaml #The content of the file hami-device-plugin-svc-monitor.yaml root@controller01:~# cat hami-device-plugin-svc-monitor.yaml apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: hami-device-plugin-svc-monitor namespace: kube-system spec: selector: matchLabels: app.kubernetes.io/component: hami-device-plugin namespaceSelector: matchNames: - "kube-system" endpoints: - path: /metrics port: monitorport interval: "15s" honorLabels: false relabelings: - sourceLabels: [__meta_kubernetes_endpoints_name] regex: hami-.* replacement: $1 action: keep - sourceLabels: [__meta_kubernetes_pod_node_name] regex: (.*) targetLabel: node_name replacement: ${1} action: replace - sourceLabels: [__meta_kubernetes_pod_host_ip] regex: (.*) targetLabel: ip replacement: $1 action: replace #apply the file hami-device-plugin-svc-monitor.yaml root@controller01:~# kubectl apply -f hami-device-plugin-svc-monitor.yaml ``` -------------------------------- ### Get Container Restart Count Source: https://github.com/project-hami/hami/blob/master/skill/k8s-debug-gpu-pod/SKILL.md Retrieves the restart count for all containers within a pod's status. ```bash kubectl get pod -n -o jsonpath='{.status.containerStatuses[*].restartCount}' ``` -------------------------------- ### Get Pod Logs Source: https://github.com/project-hami/hami/blob/master/skill/k8s-debug-gpu-pod/SKILL.md Retrieves the last 100 lines of logs for a pod's current container. ```bash kubectl logs -n --tail=100 ``` -------------------------------- ### Configure Namespace Selector with Match Labels Source: https://github.com/project-hami/hami/blob/master/docs/config.md Example of configuring the admission webhook to apply only to namespaces with a specific label. ```yaml scheduler: admissionWebhook: namespaceSelector: matchLabels: app.kubernetes.io/part-of: kubeflow-profile ``` -------------------------------- ### Deploy vxpu-device-plugin Source: https://github.com/project-hami/hami/blob/master/docs/kunlun-vxpu-support.md Deploys the necessary RBAC roles, service account, and DaemonSet to enable XPU sharing in the cluster. ```yaml apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: vxpu-device-plugin rules: - apiGroups: [""] resources: ["pods"] verbs: ["get", "list", "update", "watch", "patch"] - apiGroups: [""] resources: ["nodes"] verbs: ["get", "list", "watch", "update", "patch"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: vxpu-device-plugin subjects: - kind: ServiceAccount name: vxpu-device-plugin namespace: kube-system roleRef: kind: ClusterRole name: vxpu-device-plugin apiGroup: rbac.authorization.k8s.io --- apiVersion: v1 kind: ServiceAccount metadata: name: vxpu-device-plugin namespace: kube-system labels: app.kubernetes.io/component: vxpu-device-plugin --- apiVersion: apps/v1 kind: DaemonSet metadata: name: vxpu-device-plugin namespace: kube-system labels: app.kubernetes.io/component: vxpu-device-plugin spec: selector: matchLabels: app.kubernetes.io/component: vxpu-device-plugin template: metadata: labels: app.kubernetes.io/component: vxpu-device-plugin hami.io/webhook: ignore spec: priorityClassName: "system-node-critical" serviceAccountName: vxpu-device-plugin containers: - image: projecthami/vxpu-device-plugin:v1.0.0 name: device-plugin resources: requests: memory: 500Mi cpu: 500m limits: memory: 500Mi cpu: 500m args: - --memory-unit=MiB - --resource-name=kunlunxin.com/vxpu - -logtostderr securityContext: privileged: true capabilities: add: [ "ALL" ] volumeMounts: - name: device-plugin mountPath: /var/lib/kubelet/device-plugins - name: xre mountPath: /usr/local/xpu - name: dev mountPath: /dev env: - name: NODE_NAME valueFrom: fieldRef: fieldPath: spec.nodeName - name: KUBECONFIG value: /etc/kubernetes/kubelet.conf volumes: - name: device-plugin hostPath: path: /var/lib/kubelet/device-plugins - name: xre hostPath: path: /usr/local/xpu - name: dev hostPath: path: /dev nodeSelector: xpu: "on" ``` -------------------------------- ### Build Docker Image Source: https://github.com/project-hami/hami/blob/master/CLAUDE.md Command to build the Docker image for the project. ```bash make docker # Build Docker image ``` -------------------------------- ### CPU Profiling with pprof Source: https://github.com/project-hami/hami/blob/master/docs/how-to-profiling-scheduler.md Capture a 120-second CPU profile of the scheduler using `go tool pprof`. Ensure HAMi source code is accessible to avoid 'no such file or directory' errors. ```bash go tool pprof --seconds 120 https+insecure:///debug/pprof/profile ``` -------------------------------- ### Get Warning Events Source: https://github.com/project-hami/hami/blob/master/skill/k8s-debug-gpu-pod/SKILL.md Fetches the last 50 warning events in a namespace, sorted by timestamp, to identify immediate issues. ```bash kubectl get events -n --field-selector type=Warning --sort-by=.lastTimestamp | tail -50 ```