### InfiniBand Burn-in Workload Generator - Configure Fabric Topology Source: https://context7.com/imbue-ai/cluster-health/llms.txt Generates comprehensive burn-in scripts to exercise all InfiniBand fabric links. Requires three input files defining fabric topology: ib_server.list (nodes), ib_switch.list (switches), and ib_switch.link (link mappings). Configuration includes throughput thresholds for link validation. ```json { "ib_burn": { "min_link_throughput": 50.0, "max_link_throughput": 100.0 } } ``` ```bash # Create IB fabric topology files # ib_server.list: GUID host hca (one per line) # ib_switch.list: GUID switch_name (one per line) # ib_switch.link: GUID1 GUID2 port1 port2 (one per line) ``` -------------------------------- ### Execute InfiniBand Burn-in Script Source: https://context7.com/imbue-ai/cluster-health/llms.txt This bash script executes a pre-generated InfiniBand burn-in test. It creates a log directory and runs staged bandwidth tests (e.g., `ib_write_bw`) using SSH to connect to remote nodes. The script assumes the `ib_burn.sh` file has been generated. ```bash # Execute the generated script: bash ib_burn.sh ``` -------------------------------- ### Generate InfiniBand Burn-in Script Source: https://context7.com/imbue-ai/cluster-health/llms.txt This script generates an InfiniBand burn-in test script (`ib_burn.sh`) which performs staged bandwidth tests across specified nodes. It requires Python and utilizes SSH to execute bandwidth tests between servers and writers. The output is a bash script designed for InfiniBand communication validation. ```python # Generate burn-in script python ib_burn/ib_burn.py ``` -------------------------------- ### Compound Health Check Creation Source: https://context7.com/imbue-ai/cluster-health/llms.txt This Python code defines how to create and manage compound health checks for GPU and related hardware components. It allows combining checks like NVIDIA SMI, NVLink, and InfiniBand health into a single execution command. The `health_checks` module is required. ```python from health_checks import ( CompoundHealthCheck, NVIDIA_SMI_HEALTH_CHECK, IB_HEALTH_CHECK, NVLINK_HEALTH_CHECK, make_compound_error ) # Create compound check gpu_health = CompoundHealthCheck(( NVIDIA_SMI_HEALTH_CHECK, NVLINK_HEALTH_CHECK, )) # Generate parallel command command = gpu_health.create_command() # Creates bash script that runs checks in parallel using temp files ``` -------------------------------- ### Group InfiniBand Tests - Test Multi-Node IB Communication Across Rails Source: https://context7.com/imbue-ai/cluster-health/llms.txt Tests InfiniBand communication between groups of nodes across all 8 rails. Supports configurable group sizes and multiple iterations with per-rail performance tracking. Results are organized by iteration and group size with separate scores for total throughput and individual rail performance. ```python from communication_validation_tests import run_group_ib_tests, get_worker_connections from pathlib import Path connections = get_worker_connections() # Run tests with different group sizes results = run_group_ib_tests( connections=connections, output_file=Path("ib_test_results.txt"), rail_aligned=True, # Each rail tested separately group_sizes=(2, 4, 8), # Test 2-node, 4-node, 8-node groups max_iterations=3 ) # Results structure: {(iteration, group_size): {(connection, rail): scores}} # Example: # { # (0, 2): { # (RemoteCommandRunner(ip='10.0.1.5'), 'total'): 45.23, # (RemoteCommandRunner(ip='10.0.1.5'), '0'): (5.67, 5.68, 5.66), # (RemoteCommandRunner(ip='10.0.1.6'), 'total'): 44.98, # (RemoteCommandRunner(ip='10.0.1.6'), '0'): (5.65, 5.64, 5.67) # } # } for (iteration, group_size), scores in results.items(): print(f"Iteration {iteration}, Group size {group_size}:") for (conn, rail), score in scores.items(): if rail == 'total': print(f" {conn.ip}: Total={score}") ``` -------------------------------- ### InfiniBand Health Check - Validate Adapter Status and Drivers Source: https://context7.com/imbue-ai/cluster-health/llms.txt Validates InfiniBand adapter status, driver versions, and error counters using hca_self_test. Parses output to detect driver mismatches and hardware errors, returning specific error types with remediation suggestions. Requires sudo access to execute hca_self_test.ofed. ```python from health_checks import IB_HEALTH_CHECK, InfinibandHealthCheckError, InfinibandDriverHealthCheckError check = IB_HEALTH_CHECK command = check.create_command() # Runs: sudo flock -w 90 -F /usr/bin/hca_self_test.ofed /usr/bin/hca_self_test.ofed # Example successful output format: output = """ Host Driver Version..................PASS (MLNX_OFED_LINUX-23.10-1.1.9.0) Firmware on CA mlx5_0................v28.39.1002 Port State on mlx5_0.................UP 4X NDR (InfiniBand) Node GUID on mlx5_0..................94:6d:ae:03:00:af:de:7c Error Counter Check on mlx5_0........PASS """ result = check.validate_result(output, 0) # Check for driver version mismatch if isinstance(result, InfinibandDriverHealthCheckError): print(f"Driver issue: {result.message}") print(f"Fix: {result.suggested_remediation}") # Check for error counters if isinstance(result, InfinibandHealthCheckError): print(f"Hardware issue: {result.message}") ``` -------------------------------- ### Health Checks Library Usage Source: https://context7.com/imbue-ai/cluster-health/llms.txt Demonstrates the usage of the core health checks library for automated host issue detection and remediation. Includes running individual checks, validating results, and converting outcomes to health statuses. Requires importing necessary components from the 'health_checks' module. ```python from health_checks import ( NVIDIA_SMI_HEALTH_CHECK, IB_HEALTH_CHECK, ALL_HEALTH_CHECKS, ComputeHostHealth, outcome_to_health_check_result ) # Run individual health check health_check = NVIDIA_SMI_HEALTH_CHECK command = health_check.create_command() # Execute: nvidia-smi --query-gpu=index,ecc.errors.uncorrected.volatile.total,ecc.mode.current,ecc.mode.pending --format=csv,noheader,nounits # Validate results output = """ 0, 0, Enabled, Enabled 1, 0, Enabled, Enabled 2, 5, Enabled, Enabled 3, 0, Enabled, Enabled""" result = health_check.validate_result(output, returncode=0) # Check result type if isinstance(result, HealthCheckError): print(f"Error: {result.message}") print(f"Remediation: {result.suggested_remediation}") # Output: Error: Expected 0 ECC errors, but found 5 on GPU 2. # Convert to health status health_status = outcome_to_health_check_result(result) # Returns: ComputeHostHealth.CRITICAL ``` -------------------------------- ### Programmatic UFM Event Log Parsing Source: https://context7.com/imbue-ai/cluster-health/llms.txt This Python code demonstrates programmatic usage of the UFM event log parsing functionality. It shows how to read log entries, identify problematic events, and generate actions to disable ports. Dependencies include `find_problematic_events` module and its sub-modules like `PortId` and `DisablePortAction`. ```python from find_problematic_events import ( read_entries, get_actions_from_logs, PortId, DisablePortAction ) # Parse event log entries = read_entries(prune_consecutive=True) # Returns: (Entry(...), Entry(...), ...) # Get recommended actions actions = get_actions_from_logs(entries) # Returns: (DisablePortAction(port=PortId(switch='T3-E21-N-U47', port='17/1'), cause=...), ...) for action in actions: if isinstance(action, DisablePortAction): print(f"Disable {action.port.switch}:{action.port.port}") print(f"Reason: {action.cause.original_line}") ``` -------------------------------- ### Communication Validation Tests - Run Single-Node and Multi-Node GPU Tests Source: https://context7.com/imbue-ai/cluster-health/llms.txt Executes distributed GPU communication tests across InfiniBand rails and NVLink. Supports four test modes: all_single_node (nvlink + p2p IB), group_ib (multi-node InfiniBand), nvlink (all-node NVLink), and p2p_ib (point-to-point InfiniBand). Returns latency statistics including min, max, mean, and percentile metrics. ```bash # Run all single-node tests (nvlink + p2p IB) python host_validation/communication_validation_tests.py --test all_single_node # Run group InfiniBand tests across multiple nodes python host_validation/communication_validation_tests.py --test group_ib # Run nvlink tests on all nodes python host_validation/communication_validation_tests.py --test nvlink # Run point-to-point InfiniBand tests python host_validation/communication_validation_tests.py --test p2p_ib ``` ```python # Example programmatic usage: from communication_validation_tests import run_nvlink_tests, get_worker_connections connections = get_worker_connections() results = run_nvlink_tests(connections, dims=1_000_000_000, loops=20) # Results format: # { # "10.0.1.5": { # "count": 20, # "min": 0.342, # "max": 0.389, # "mean": 0.356, # "25_percentile": 0.348, # "50_percentile": 0.355, # "75_percentile": 0.364 # } # } for ip, stats in results.items(): if isinstance(stats.get("mean"), str): print(f"Host {ip} failed: {stats['mean']}") else: print(f"Host {ip}: {stats['mean']:.3f}s mean latency") ``` -------------------------------- ### Parse UFM Event Logs for Problematic Ports Source: https://context7.com/imbue-ai/cluster-health/llms.txt This Python script parses InfiniBand network event logs from UFM (Unified Fabric Manager) to identify problematic ports that should be disabled. It supports processing log files or `iblinkinfo` output. Configuration requires setting UFM connection details in the script. ```python # Configure UFM connection in find_problematic_events.py: # UFM_LOCAL_IP = "10.0.0.100" # UFM_AUTH = ("username", "password") # Process UFM event logs to find problematic ports python ufm_events/find_problematic_events.py process-logs ~/actions.jsonl ``` ```python # Find bad ports from iblinkinfo output sudo iblinkinfo > iblinkinfo.txt python ufm_events/find_problematic_events.py get-bad-from-iblinkinfo \ --infile iblinkinfo.txt \ --outfile bad_ports.txt ``` ```python # Find bad ports from traffic counters after IB burn python ufm_events/find_problematic_events.py get-bad-from-counters \ --bad-ports-file counters_bad.txt \ --iblinkinfo-file iblinkinfo.txt ``` -------------------------------- ### Run GPU Stress Test Source: https://context7.com/imbue-ai/cluster-health/llms.txt Executes a GPU stress test to evaluate memory allocation and computation across all GPUs on a host, including matrix operations and data transfers via NVLink. Supports custom timeout durations. Outputs status messages indicating success or failure. ```python # Run GPU stress test with default 5 minute runtime python gpu_stress_test/gpu_stress_test.py # Run with custom timeout (in seconds) python gpu_stress_test/gpu_stress_test.py 600 # Example output on success: # hostname-abc123: All okay for 342 loops # Example output on failure: # hostname-xyz456: Issue with GPUS, values don't match # hostname-def789: out of memory CUDA out of memory. Tried to allocate 40.00 GiB ``` -------------------------------- ### Dmesg Health Check - Scan Kernel Messages for Hardware Errors Source: https://context7.com/imbue-ai/cluster-health/llms.txt Scans kernel dmesg output for unrecognized hardware errors and issues using whitelist filtering. Normalizes and redacts variable data from dmesg entries, then identifies anomalies. Returns warnings with suggested remediation when unrecognized messages are detected. ```python from health_checks import DMESG_WHITELIST_HEALTH_CHECK, DmesgHealthCheckWarning check = DMESG_WHITELIST_HEALTH_CHECK command = check.create_command() # Complex pipeline that normalizes dmesg output, redacts variable data, sorts by content # Process dmesg output output = """1 nvidia-nvswitch2: SXid (PCI:ADDR): 12028, Non-fatal, Link 61 egress error 2 mlx5_core ADDR: 252.056 Gb/s available PCIe bandwidth, limited 3 Out of memory: Killed process PID (python3)""" result = check.validate_result(output, 0) if isinstance(result, DmesgHealthCheckWarning): print(result.message) # Output: 3 unrecognized dmesg line(s) detected # 1: nvidia-nvswitch2: SXid (PCI:ADDR): 12028, Non-fatal, Link 61 egress error # 2: mlx5_core ADDR: 252.056 Gb/s available PCIe bandwidth, limited # ...3 more... print(result.suggested_remediation) ``` -------------------------------- ### Compound Health Check Result Aggregation Source: https://context7.com/imbue-ai/cluster-health/llms.txt This Python code demonstrates how to process and aggregate results from multiple health checks. It shows how to obtain individual check outcomes and combine them into a single compound result, indicating overall cluster health. The `health_checks` module, including `HealthCheckError`, `HealthCheckWarning`, and `HealthyResult`, is necessary. ```python # Get individual check results # health_map = gpu_health.get_health_check_map(output, returncode) # { # NvidiaSmiHealthCheck(): HealthyResult(), # NvlinkHealthCheck(): HealthCheckWarning(message="5 error(s) detected") # } # Create compound result from multiple outcomes from health_checks import HealthCheckError, HealthCheckWarning, HealthyResult outcomes = [ HealthCheckError(message="GPU 3 has ECC errors", suggested_remediation="Replace GPU"), HealthCheckWarning(message="NVLink error rate elevated"), HealthyResult() ] compound_result = make_compound_error(outcomes) # Returns CompoundHealthCheckError with aggregated messages print(compound_result.display(indentation=2)) ``` -------------------------------- ### NVIDIA SMI Health Check Validation Source: https://context7.com/imbue-ai/cluster-health/llms.txt Validates GPU status, specifically ECC errors and configuration, using the 'nvidia-smi' command. This Python code demonstrates creating the command, executing it, and validating the output for both successful (no errors) and failed (ECC errors present) scenarios. It utilizes the 'NvidiaSmiHealthCheck' class from the 'health_checks' module. ```python from health_checks import NvidiaSmiHealthCheck, GpuHealthCheckError, HealthyResult check = NvidiaSmiHealthCheck() command = check.create_command() # Returns: nvidia-smi --query-gpu=index,ecc.errors.uncorrected.volatile.total,ecc.mode.current,ecc.mode.pending --format=csv,noheader,nounits # Successful validation output = """ 0, 0, Enabled, Enabled 1, 0, Enabled, Enabled 2, 0, Enabled, Enabled 3, 0, Enabled, Enabled 4, 0, Enabled, Enabled 5, 0, Enabled, Enabled 6, 0, Enabled, Enabled 7, 0, Enabled, Enabled""" result = check.validate_result(output, 0) assert isinstance(result, HealthyResult) # Failed validation - ECC errors present output_with_errors = """ 0, 0, Enabled, Enabled 1, 3, Enabled, Enabled""" result = check.validate_result(output_with_errors, 0) assert isinstance(result, GpuHealthCheckError) assert "Expected 0 ECC errors, but found 3 on GPU 1" in result.message ``` -------------------------------- ### Run Health Checks on Multiple Nodes Source: https://context7.com/imbue-ai/cluster-health/llms.txt Enables parallel execution of health checks across multiple cluster nodes with automated reporting. Configuration is managed via 'health_checks/config.json'. The script can run on nodes specified in the config or on a custom list of nodes provided as arguments. Outputs a summary of node health statuses. ```python # Configure nodes in health_checks/config.json: # { # "node_info": { # "nodes": ["host-1", "host-2", "host-3"], # "port": "22", # "user": "admin" # }, # "leaf_health_checks": [] # Empty = run all checks # } # Run on nodes from config python health_checks/run_health_checks.py # Run on specific nodes python health_checks/run_health_checks.py host-1,host-2,host-3 # Example output: # Running health checks on nodes: ['host-1', 'host-2', 'host-3'] # Health ComputeHostHealth.OK # Health ComputeHostHealth.CRITICAL # host-2: # 8 error(s) detected # Port State:DOWN # Remove the host for multi-node training. # Flag as it may need a support ticket from Dell. # # ----------------------------SUMMARY:---------------------------- # Nodes with ComputeHostHealth.OK: ['host-1', 'host-3'] # Nodes with ComputeHostHealth.CRITICAL: ['host-2'] ``` === COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.