Intel Gaudi

https://docs.habana.ai/en/v1.23.0/
Admin
Intel Gaudi is an AI accelerator platform designed for high-performance deep learning training and...
Tokens:363,952
Snippets:8,708
Trust Score:9.9
Update:6 months ago
Show doc for...
Tokens:
Raw
### Getting Started with Intel Gaudi

Source: https://docs.habana.ai/en/v1.23.0/Management_and_Monitoring/HLML_API/index.html

Information on installing and setting up Intel Gaudi hardware and software.

```APIDOC
## Installation

### Description
Guides for installing Gaudi drivers, software, and firmware.

### Method
N/A

### Endpoint
N/A

### Parameters
N/A

### Request Body
N/A

### Response
N/A

## Quick Start Guides

### Description
Guides to quickly start running workloads on Intel Gaudi.

### Method
N/A

### Endpoint
N/A

### Parameters
N/A

### Request Body
N/A

### Response
N/A
```

--------------------------------

### Getting Started with Intel Gaudi

Source: https://docs.habana.ai/en/v1.23.0/Management_and_Monitoring/Network_Configuration/Configuration_Files_Example_L3.html

Information on installing and setting up Intel Gaudi hardware and software, including driver and firmware updates.

```APIDOC
## Installation

### Description
Guides for installing Gaudi software, drivers, and firmware, including options for bare metal, Docker, and Kubernetes.

### Method
N/A

### Endpoint
N/A

### Parameters
N/A

### Request Body
N/A

### Response
N/A

## Quick Start Guides

### Description
Quick start guides for various cloud platforms and deployment methods.

### Method
N/A

### Endpoint
N/A

### Parameters
N/A

### Request Body
N/A

### Response
N/A
```

--------------------------------

### Quick Start Guides

Source: https://docs.habana.ai/en/v1.23.0/Management_and_Monitoring/Kernel_Module_Diagnostics/index.html

Concise guides to help users quickly set up and run workloads on Intel Gaudi across different platforms.

```APIDOC
## Intel Gaudi Quick Start Guides

### Description
These guides offer a streamlined approach to getting started with Intel Gaudi on various cloud and bare-metal environments.

### Available Guides
- Intel Tiber AI Cloud Quick Start Guide
- IBM Cloud Quick Start Guide
- Running Workloads on Bare Metal
- Running Workloads on Docker
- Running Workloads on Kubernetes
```

--------------------------------

### Installation Guides

Source: https://docs.habana.ai/en/v1.23.0/Management_and_Monitoring/Kernel_Module_Diagnostics/index.html

Detailed instructions for installing and setting up the Intel Gaudi software and drivers.

```APIDOC
## Intel Gaudi Installation

### Description
This section outlines the necessary steps and requirements for installing the Intel Gaudi software suite, drivers, and firmware.

### Installation Topics
- Hardware and Network Requirements
- Driver and Software Installation
- Firmware Upgrade and Platform Level Components
  - Firmware Upgrade
  - HL-325 In-Band (IB) CPLD Programming
  - HL-338 In-Band (IB) CPLD Programming
- Additional Installation Methods
  - Bare Metal Installation
  - Docker Installation
  - Kubernetes Installation (Intel Gaudi Base Operator, Device Plugin)
  - OpenShift Installation
- System Verifications and Final Tests
```

--------------------------------

### Getting Started with Training on Intel Gaudi

Source: https://docs.habana.ai/en/v1.23.0/Management_and_Monitoring/Kernel_Module_Diagnostics/index.html

An introductory guide to begin training machine learning models using PyTorch on Intel Gaudi accelerators.

```APIDOC
## Getting Started with Training on Intel Gaudi

### Description
This guide provides the initial steps and essential information required to start training PyTorch models on Intel Gaudi hardware.
```

--------------------------------

### Installation Guides for Intel Gaudi

Source: https://docs.habana.ai/en/v1.23.0/Media_Pipeline/Media_Operator_BasicCrop.html

Comprehensive guides for installing the necessary software and drivers for Intel Gaudi.

```APIDOC
## Installation Guides for Intel Gaudi

### Description
This section provides detailed instructions for installing Intel Gaudi software, drivers, and firmware, including requirements and different installation methods.

### Topics
- Hardware and Network Requirements
- Driver and Software Installation
- Firmware Upgrade and Platform Level Components
  - Firmware Upgrade
  - HL-325 In-Band (IB) CPLD Programming
  - HL-338 In-Band (IB) CPLD Programming
- Additional Installation
  - Bare Metal Installation
  - Docker Installation
  - Kubernetes Installation
    - Intel Gaudi Base Operator for Kubernetes
    - Intel Gaudi Device Plugin for Kubernetes
  - OpenShift Installation
- System Verifications and Final Tests
```

--------------------------------

### Gaudi Installation Guide

Source: https://docs.habana.ai/en/v1.23.0/Management_and_Monitoring/PYHLML_APIs/HLML_PY_API.html

Instructions for installing Gaudi software, drivers, and firmware, including options for bare metal, Docker, and Kubernetes.

```APIDOC
## Installation

### Description
Guides users through the process of installing Gaudi software, drivers, and firmware. Covers hardware requirements, driver installation, firmware upgrades, and various deployment methods.

### Methods
- **Driver and Software Installation**
- **Firmware Upgrade**
- **Bare Metal Installation**
- **Docker Installation**
- **Kubernetes Installation**
- **OpenShift Installation**

### Endpoints
N/A (This section describes installation procedures, not API endpoints)

### Parameters
N/A

### Request Body
N/A

### Response
N/A
```

--------------------------------

### Installation

Source: https://docs.habana.ai/en/v1.23.0/Management_and_Monitoring/Kernel_Module_Diagnostics/index.html

Comprehensive guide covering all aspects of installing Intel Gaudi software and drivers.

```APIDOC
## Intel Gaudi Installation Guide

### Description
This guide provides detailed instructions for installing the necessary drivers, software, and firmware for Intel Gaudi accelerators.
```

--------------------------------

### Getting Started with Inference on Intel Gaudi

Source: https://docs.habana.ai/en/v1.23.0/Management_and_Monitoring/Kernel_Module_Diagnostics/index.html

An introductory guide to performing inference tasks using PyTorch models on Intel Gaudi.

```APIDOC
## Getting Started with Inference on Intel Gaudi

### Description
This guide provides the essential steps to begin running inference workloads with PyTorch models on Intel Gaudi accelerators.
```

--------------------------------

### Example Docker Daemon Configuration

Source: https://docs.habana.ai/en/v1.23.0/Installation_Guide/Additional_Installation/Docker_Installation.html

A complete example of a configured /etc/docker/daemon.json file.

```json
{
   "default-runtime": "habana",
   "runtimes": {
      "habana": {
            "path": "/usr/bin/habana-container-runtime",
            "runtimeArgs": []
      }
   }
}
```

--------------------------------

### Install Base Driver and Software

Source: https://docs.habana.ai/en/v1.23.0/Installation_Guide/Driver_Installation.html

Downloads and executes the installer script to set up the base driver and software environment.

```bash
wget -nv https://vault.habana.ai/artifactory/gaudi-installer/1.23.0/habanalabs-installer.sh
chmod +x habanalabs-installer.sh
./habanalabs-installer.sh install --type base
```

--------------------------------

### Full Switch Interface Configuration Examples

Source: https://docs.habana.ai/en/v1.23.0/Management_and_Monitoring/Network_Configuration/Configure_E2E_Test_in_L3.html

Complete interface configuration examples for Ethernet1/1 and Ethernet2/1.

```text
interface Ethernet1/1
mtu 9198
speed 400g-8
error-correction encoding reed-solomon
no switchport
ip address 10.208.128.1/30
qos trust dscp
priority-flow-control on
priority-flow-control priority 0 no-drop
priority-flow-control priority 1 no-drop
priority-flow-control priority 2 no-drop
priority-flow-control priority 3 no-drop
uc-tx-queue 2
   no priority
uc-tx-queue 3
   no priority
```

```text
interface Ethernet2/1
mtu 9198
speed 100g-2
error-correction encoding reed-solomon
no switchport
ip address 10.208.0.2/30
qos trust dscp
priority-flow-control on
priority-flow-control priority 0 no-drop
priority-flow-control priority 1 no-drop
priority-flow-control priority 2 no-drop
priority-flow-control priority 3 no-drop
uc-tx-queue 2
   no priority
uc-tx-queue 3
   no priority
```

--------------------------------

### Test Plan YAML Example

Source: https://docs.habana.ai/en/v1.23.0/Management_and_Monitoring/Qualification_Library/Diagnostic_Tool/Test_Plan_Automation.html

Defines the general setup for a test plan, including paths to test-specific configuration files and test execution parameters. Ensure paths and flags are correctly set for your environment.

```yaml
test_plan_yaml_dir: /opt/habanalabs/qual/diag_tool/test_plans/gaudi3/qual_test_plan
test-plan-name: qual_test_plan
bin-Folder: /opt/habanalabs/qual/gaudi3/bin
enable-vuart: true
tests:
- test-name: E2E
  test-yaml-path: E2E.yaml
  test-repeat-no: 3
  pre-run: 'sudo dmesg -C'
  post-run: N/A
- test-name: FunctionalTest_extreme
  test-yaml-path: FunctionalTest_extreme.yaml
  test-repeat-no: 6
  pre-run: 'driver_load_unload.sh'
  post-run: N/A

```

--------------------------------

### Install Intel Gaudi PyTorch Dependencies and Package

Source: https://docs.habana.ai/en/v1.23.0/Installation_Guide/Additional_Installation/Bare_Metal_Installation.html

Run these commands sequentially to install the necessary dependencies and the Intel Gaudi PyTorch package on a bare metal system. Ensure you have sudo permissions for dependency installation.

```bash
./habanalabs-installer.sh install -t dependencies
```

```bash
./habanalabs-installer.sh install --type pytorch
```

--------------------------------

### Guides

Source: https://docs.habana.ai/en/v1.23.0/Management_and_Monitoring/Kernel_Module_Diagnostics/index.html

A collection of guides for various functionalities and tools available with Intel Gaudi.

```APIDOC
## Intel Gaudi Guides

### Description
This section contains various guides and tutorials for utilizing Intel Gaudi accelerators with different frameworks and tools.
```

--------------------------------

### List Installed Habana Labs Software Components

Source: https://docs.habana.ai/en/v1.23.0/Installation_Guide/System_Verification_and_Final_Tests.html

Use `apt list --installed` and grep for 'habana' to verify that all necessary Habana Labs software components are installed correctly. This confirms the software environment is set up as expected.

```bash
$ apt list --installed | grep habana
habanalabs-container-runtime/focal,now 1.23.0-695 amd64 [installed]
habanalabs-dkms/focal,focal,now 1.23.0-695 all [installed]
habanalabs-firmware-tools/focal,now 1.23.0-695 amd64 [installed]
habanalabs-firmware/focal,now 1.23.0-695 amd64 [installed]
habanalabs-graph/focal,now 1.23.0-695 amd64 [installed]
habanalabs-qual/focal,now 1.23.0-695 amd64 [installed]
habanalabs-thunk/focal,focal,now 1.23.0-695 all [installed]
habanalabs-tools/focal,now 1.23.0-695 amd64 [installed]
```

--------------------------------

### MediaPipe Guides

Source: https://docs.habana.ai/en/v1.23.0/Management_and_Monitoring/Kernel_Module_Diagnostics/index.html

Guides for using MediaPipe for creating and executing media processing pipelines on Intel Gaudi.

```APIDOC
## MediaPipe Guides for Intel Gaudi

### Description
This section provides guides on utilizing MediaPipe for building and running efficient media processing pipelines on Intel Gaudi accelerators.

### MediaPipe Topics
- Creating and Executing Media Pipeline
- MediaPipe for PyTorch ResNet
- MediaPipe for PyTorch ResNet3d
```

--------------------------------

### TPC Programming Guide

Source: https://docs.habana.ai/en/v1.23.0/API_Reference_Guides/PYHLML_APIs/Per_Device_APIs.html

Guides for TPC programming, including getting started, tool installation, and detailed user guides covering the TPC language, architecture, and built-in functions.

```APIDOC
## TPC Programming

### Description
Comprehensive guides for TPC (presumably a domain-specific language or framework) programming. This section covers initial setup, tool usage, and in-depth explanations of the programming language, processor architecture, and available functions.

### Endpoints

- **TPC Getting Started Guide**: Initial steps to begin TPC programming.
- **TPC Tools Installation Guide**: Instructions for installing necessary TPC development tools.
- **TPC User Guide**: Detailed guide on TPC programming.
  - **TPC Programming Language**: Syntax and semantics of the TPC language.
  - **Processor Architectural Overview**: Understanding the underlying processor architecture.
  - **TPC Programming Model**: How to structure TPC programs.
  - **TPC-C Language**: Specifics of the TPC-C variant.
  - **Built-in Functions**: Usage of pre-defined functions.
  - **Implementing and Integrating New lib**: Adding custom libraries.
  - **TPC Coherency**: Concepts related to data coherency.
  - **Multiple Kernel Libraries**: Working with multiple libraries.
  - **Abbreviations**: Glossary of terms.
```

--------------------------------

### TPC Tools Debugger

Source: https://docs.habana.ai/en/v1.23.0/API_Reference_Guides/PYHLML_APIs/Per_Device_APIs.html

Guide for using the TPC Tools Debugger, covering installation, session management, and debugging views.

```APIDOC
## TPC Tools Debugger

### Description
Guide for utilizing the TPC Tools Debugger. This section details the installation process, how to initiate and manage debugging sessions, and the various views and operations available during debugging.

### Endpoints

- **Installation**: Steps to install the debugger.
- **Starting a Debug Session**: How to begin a debugging session.
- **TPC-C Source or Disassembly Level Debugging**: Debugging at source code or assembly level.
- **Debug Session Views and Operations**: Overview of the debugger's interface and functionalities.
```

--------------------------------

### Get Total ECC Errors

Source: https://docs.habana.ai/en/v1.23.0/API_Reference_Guides/PYHLML_APIs/Per_Device_APIs.html

Returns the number of ECC errors for a specific device since the last reset or driver installation.

```APIDOC
## hlmlDeviceGetTotalECCErrors

### Description
Returns the number of ECC errors for a specific device, since the last device reset, or since the driver was installed. Only the number of uncorrected errors is supported.

### Method
GET

### Endpoint
/device/ecc/errors

### Parameters
#### Path Parameters
- **device** (hlml_t.HLML_DEVICE.TYPE) - Required - The identifier of the target AIP.
- **error_type** (hlml_t.HLML_MEMORY_ERROR.TYPE) - Required - Flag that specifies the type of the errors.
- **counter_type** (hlml_t.HLML_ECC_COUNTER) - Required - Flag that specifies the countertype of the errors.

### Return Value
- **error_count** (int) - Specified ECC errors.

### Raises
- HLMLError_Uninitialized: If the library has not been successfully initialized.
- HLMLError_InvalidArgument: If device, error type or counter type is invalid, or ecc counts is NULL.
- HLMLError_NotSupported: If the device does not support this feature.
- HLMLError_Unknown: If error occurred during ECC error retrieval.
```

--------------------------------

### Expected deployment output

Source: https://docs.habana.ai/en/v1.23.0/Installation_Guide/Additional_Installation/Kubernetes_Installation/Intel_Gaudi_Kubernetes_Device_Plugin.html

Example output showing a successfully running device plugin daemonset.

```text
NAME                                       READY   STATUS    RESTARTS   AGE
habanalabs-device-plugin-daemonset-qtpnh   1/1     Running   0          2d11h
```

--------------------------------

### Gaudi 3 PCI Bandwidth Test Execution Examples

Source: https://docs.habana.ai/en/v1.23.0/Management_and_Monitoring/Qualification_Library/Bandwidth_Tests_Plugin.html

Specific command examples for executing the PCI bandwidth test on Gaudi 3 devices.

```bash
./hl_qual -gaudi3 -c all -rmod serial -t 20 -p
-b -size 2048000000
```

```bash
./hl_qual -gaudi3 -c all -rmod serial -dis_mon -t 120 -p -b
```

--------------------------------

### Get Device Module ID

Source: https://docs.habana.ai/en/v1.23.0/Management_and_Monitoring/HLML_API/Per_Device_APIs.html

Retrieves the module ID configured on the device. The module ID is an identifier for the specific module installed on the device.

```c
hlml_return_t hlml_device_get_module_id(hlml_device_t device, unsigned int *module_id)
```

--------------------------------

### vLLM Fork for Intel Gaudi

Source: https://docs.habana.ai/en/v1.23.0/Gaudi_Overview/Intel_Gaudi_Software_Suite.html

Guides for using the vLLM fork with Intel Gaudi for efficient large language model inference, including quick start and advanced usage.

```APIDOC
## vLLM Fork for Intel Gaudi

### Description
This section provides documentation for using the vLLM fork specifically optimized for Intel Gaudi. It covers quick start guides, inference procedures, FP8 calibration, managing warmup time, deployable containers, and profiling.

### Topics
- vLLM Quick Start Guide
- Inference Using vLLM
- FP8 Calibration and Inference with vLLM
- Managing and Reducing vLLM Warmup Time
- Deployable vLLM Containers Tutorial
- Profiling with vLLM
- vLLM with Intel Gaudi FAQs
```

--------------------------------

### Build and Install MSV Tool from Source

Source: https://docs.habana.ai/en/v1.23.0/Management_and_Monitoring/Hypervisor_Tools/Hpervisor_Tools_Installation.html

Navigate to the MSV source directory and use make to build and install the custom MSV tool.

```bash
cd /opt/habanalabs/hypervisor-msv/src
make
make install

```

--------------------------------

### OpenSSL Configuration for Certificate Generation

Source: https://docs.habana.ai/en/v1.23.0/Management_and_Monitoring/Embedded_System_Tools_Guide/Enable_Secure_Boot.html

This is an example OpenSSL configuration file used for generating certificates. Customize fields like countryName, stateOrProvinceName, localityName, organizationName, commonName, and emailAddress according to your setup.

```openssl.cnf
# This definition stops the following lines checking if $HOME isn't defined.
HOME                    = .
RANDFILE                = $ENV::HOME/.rnd
[ req ]
distinguished_name      = req_distinguished_name
x509_extensions         = v3
string_mask             = utf8only
prompt                  = no

[ req_distinguished_name ]
countryName             = CA  # Update according to your configuration
stateOrProvinceName     = Intel Ave  # Update according to your configuration
localityName            = Folsom  # Update according to your configuration
0.organizationName      = Intel  # Update according to your configuration
commonName              = Secure Boot Signing  # Update according to your configuration
emailAddress            = abc.xyz@intel.com  # Update according to your configuration

[ v3 ]
subjectKeyIdentifier    = hash
authorityKeyIdentifier    = keyid:always,issuer
basicConstraints        = critical,CA:FALSE
extendedKeyUsage        = codeSigning,1.3.6.1.4.1.311.10.3.6,1.3.6.1.4.1.2312.16.1.2
nsComment               = "OpenSSL Generated Certificate"

```

--------------------------------

### Gaudi 2 PCI Bandwidth Test Execution Examples

Source: https://docs.habana.ai/en/v1.23.0/Management_and_Monitoring/Qualification_Library/Bandwidth_Tests_Plugin.html

Specific command examples for executing the PCI bandwidth test on Gaudi 2 devices.

```bash
./hl_qual -gaudi3 -c all -rmod serial -t 20 -p
-b -size 2048000000
```

```bash
./hl_qual -gaudi2 -c all -rmod serial -dis_mon -t 120 -p -b
```

--------------------------------

### Install Driver and Software Dependencies

Source: https://docs.habana.ai/en/v1.23.0/Management_and_Monitoring/Embedded_System_Tools_Guide/Enable_Secure_Boot.html

Execute the Habanalabs installer script to install necessary dependencies for the driver and software. Skip if already installed.

```bash
./habanalabs-installer.sh install -t dependencies
```

--------------------------------

### Run Diagnostic Tool in Rack-Scale Environment

Source: https://docs.habana.ai/en/v1.23.0/Management_and_Monitoring/Qualification_Library/Diagnostic_Tool/Rack_Scale_Script.html

Full example demonstrating environment variable configuration and execution of the rack-scale diagnostic tool.

```bash
export PDSH_SSH_ARGS="-p <ssh_port>"  # (Optional) Set SSH port (default: 22)
export REMOTE_URI="remote_uri"
export DIAG_TOOL_COMMAND="python /opt/habanalabs/qual/diag_tool/diag_tool_automation.py \
  --exec run_test_plan \
  --input_path /opt/habanalabs/qual/diag_tool/test_plans/E2E.yaml \
  --output_path $REMOTE_URI:/var/log/habanalabs \
  --core gaudi2 \
  --uri_key_path ~/.ssh/id_ed25519"

cd <QUAL_SRC>/diag_tool
python diag_tool_automation_rack_scale.py -c "$DIAG_TOOL_COMMAND" -f hostfile.txt
```

--------------------------------

### Host File Example

Source: https://docs.habana.ai/en/v1.23.0/Management_and_Monitoring/RDMA_PerfTest_Tool/RDMA_PerfTest_Tool.html

Example format for a host file listing nodes with their SSH connection details (IP and port) for the PerfTest tool.

```text
kuku-kvm12-lake:22
kuku-kvm13-lake:22
kuku-kvm14-lake:22
kuku-kvm15-lake:22
```

--------------------------------

### Install Hypervisor Tools Package on RHEL

Source: https://docs.habana.ai/en/v1.23.0/Management_and_Monitoring/Hypervisor_Tools/Hpervisor_Tools_Installation.html

Install the hypervisor utilities .rpm package on RHEL 9.4 using dnf. This installs both hl-smi-async and MSV tools.

```bash
sudo dnf install ./habanalabs-hypervisor-utils_1.23.0-695_el9.x86_64.rpm -y

```

--------------------------------

### Install Hypervisor Tools Package on Ubuntu

Source: https://docs.habana.ai/en/v1.23.0/Management_and_Monitoring/Hypervisor_Tools/Hpervisor_Tools_Installation.html

Install the hypervisor utilities .deb package on Ubuntu 22.04.5 using apt. This installs both hl-smi-async and MSV tools.

```bash
sudo apt install ./habanalabs-hypervisor-utils_1.23.0-695_amd64.deb -y

```

--------------------------------

### Install libpcap-dev

Source: https://docs.habana.ai/en/v1.23.0/Management_and_Monitoring/Network_Configuration/Configure_E2E_Test_in_L3.html

Installs the libpcap-dev package, which is a prerequisite for network operations.

```bash
sudo apt install libpcap-dev

```

--------------------------------

### gaudi3_cn_macro_sw_init

Source: https://docs.habana.ai/en/v1.23.0/Management_and_Monitoring/Kernel_Module_Diagnostics/Dmesg_Error_Causes.html

Initializes the switch macro.

```APIDOC
## gaudi3_cn_macro_sw_init

### Description
Initializes the switch macro, including allocation of host memory for DRAM emulation and creation of a pool for the door-bell FIFO.

### Method
Not specified (likely internal function)

### Endpoint
Not applicable

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Request Example
None

### Response
#### Success Response (0)
- **No error** - Indicates successful execution.

#### Error Responses
- **ENOMEM (12)** - Failed to allocate host memory to emulate HBM when HBM is not present.
- **Failed to create gen_pool** - Gen pool creation for door-bell FIFO failed.

#### Response Example
None
```

--------------------------------

### Install Intel Gaudi Base Operator using CLI

Source: https://docs.habana.ai/en/v1.23.0/Installation_Guide/Additional_Installation/OpenShift_Installation/index.html

Apply this YAML configuration to install the Intel Gaudi Base Operator on OpenShift. Ensure the `oc` CLI is installed and configured.

```yaml
---
apiVersion: v1
kind: Namespace
metadata:
   name: habana-ai-operator
---
apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
   name: habana-ai-operator
   namespace: habana-ai-operator
spec:
   targetNamespaces:
   - habana-ai-operator
---
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
   name: habana-ai-operator
   namespace: habana-ai-operator
spec:
   channel: stable
   installPlanApproval: Automatic
   name: habana-ai-operator
   source: certified-operators
   sourceNamespace: openshift-marketplace

```
Tokens:
Raw