nv-monitor

https://github.com/wentbackward/nv-monitor
Admin
A lightweight terminal monitoring TUI, CSV logger, and Prometheus exporter for NVIDIA GPU systems...
Tokens:5,511
Snippets:83
Trust Score:8.6
License:MIT
Update:4 months ago
Show doc for...
Tokens:
Raw
### Example: Multi-node test setup

Source: https://github.com/wentbackward/nv-monitor/blob/main/DEMO-LOAD.md

Demonstrates how to run demo-load with GPU enabled on each node and start nv-monitor to export Prometheus metrics. This setup is for end-to-end testing of monitoring pipelines.

```bash
# On each node:
./demo-load --gpu &
./nv-monitor -n -p 9101
```

--------------------------------

### Install nv-monitor System-Wide

Source: https://github.com/wentbackward/nv-monitor/blob/main/README.md

Install the nv-monitor binary system-wide after building from source.

```bash
sudo make install
```

--------------------------------

### Prometheus Exporter Setup

Source: https://github.com/wentbackward/nv-monitor/blob/main/CLAUDE.md

Details the opt-in Prometheus exporter, including port configuration, HTTP server, and metrics format.

```c
// Prometheus exporter: Opt-in via `-p PORT`. Runs a minimal HTTP server on a dedicated pthread, serving OpenMetrics-formatted metrics at `/metrics`. Uses POSIX sockets with `poll()` for clean shutdown. Zero overhead when not enabled. Enables multi-machine monitoring via Prometheus/Grafana.
```

--------------------------------

### Install nv-monitor on Arch Linux

Source: https://github.com/wentbackward/nv-monitor/blob/main/README.md

Install nv-monitor directly using pacman on Arch-based Linux distributions.

```bash
pacman -S nv-monitor
```

--------------------------------

### Expose Prometheus Metrics

Source: https://github.com/wentbackward/nv-monitor/blob/main/README.md

Start nv-monitor to expose Prometheus metrics on a specified port. This can be used with the TUI or in headless mode for pure metrics export.

```bash
./nv-monitor -p 9101              # TUI + metrics at http://localhost:9101/metrics
./nv-monitor -n -p 9101           # Pure headless exporter
curl -s localhost:9101/metrics     # Check it works
```

--------------------------------

### Prometheus Scrape Configuration

Source: https://github.com/wentbackward/nv-monitor/blob/main/README.md

Example Prometheus scrape configuration to collect metrics from nv-monitor. Ensure the job name and target address are correctly set.

```yaml
scrape_configs:
  - job_name: 'nv-monitor'
    authorization:
      credentials: 'my-secret-token'
    static_configs:
      - targets: ['dgx-spark:9101']
```

--------------------------------

### nv-monitor Allocation Table

Source: https://github.com/wentbackward/nv-monitor/blob/main/PERFORMANCE.md

Lists the buffers allocated by nv-monitor at startup or when the Prometheus thread starts, detailing their size and allocation timing. Emphasizes zero allocations during normal operation.

```text
Buffer | Sized to | When allocated | When freed |
|---|---|---|---|
`prev_ticks` (CPU tick snapshots) | `num_cpus + 1` | Program start | Program exit |
`cur_ticks` (current frame ticks) | `num_cpus + 1` | Program start | Program exit |
`cpu_pct` (per-core utilization) | `num_cpus + 1` | Program start | Program exit |
`cpu_part` (ARM core type IDs) | `num_cpus` | Program start | Program exit |
`prom_body` (HTTP response buffer) | `gpu_count + num_cpus` | Prometheus thread start | Thread exit |
`prom_gpus` (GPU snapshot array) | `gpu_count` | Prometheus thread start | Thread exit |
```

--------------------------------

### Soak Test Results Summary

Source: https://github.com/wentbackward/nv-monitor/blob/main/PERFORMANCE.md

Example output from a soak test showing memory usage over a short period under high load. Note the minimal RSS delta after initial kernel memory mapping.

```text
DGX Spark — 2 minutes, 1215 CSV rows, ~240 Prometheus scrapes:
  RSS start: 20920 KB
  RSS end:   20932 KB
  RSS delta: +12 KB (one-time kernel page table expansion, stabilised in <5s)
```

--------------------------------

### Build demo-load with Make

Source: https://github.com/wentbackward/nv-monitor/blob/main/DEMO-LOAD.md

Use 'make' to build the demo-load executable. This is the recommended build method.

```bash
make demo-load
```

--------------------------------

### Run demo-load with CPU and GPU load

Source: https://github.com/wentbackward/nv-monitor/blob/main/DEMO-LOAD.md

Enable both CPU and GPU load generation by using the --gpu flag.

```bash
./demo-load --gpu
```

--------------------------------

### Build demo-load directly with GCC

Source: https://github.com/wentbackward/nv-monitor/blob/main/DEMO-LOAD.md

Compile demo-load directly using GCC. This requires linking against pthread, dl, and m libraries.

```bash
gcc -O2 -o demo-load demo-load.c -lpthread -ldl -lm
```

--------------------------------

### Build nv-monitor from Source

Source: https://github.com/wentbackward/nv-monitor/blob/main/README.md

Build the nv-monitor binary from source. Requires gcc and libncurses-dev.

```bash
sudo apt install build-essential libncurses-dev
make
```

--------------------------------

### Run demo-load with CPU load only

Source: https://github.com/wentbackward/nv-monitor/blob/main/DEMO-LOAD.md

Execute demo-load to generate CPU load across all available cores. This is the default behavior.

```bash
./demo-load
```

--------------------------------

### NVML Dynamic Loading

Source: https://github.com/wentbackward/nv-monitor/blob/main/CLAUDE.md

Demonstrates dynamic loading of libnvidia-ml.so.1 using dlopen/dlsym and a macro for versioned symbol loading.

```c
// NVML dynamic loading (line ~115): Loads `libnvidia-ml.so.1` via `dlopen`/`dlsym` at runtime. Uses a variadic LOAD macro to try versioned symbols first (e.g. `nvmlInit_v2` before `nvmlInit`). All NVML function pointers are prefixed with `p` (e.g. `pNvmlInit`).
```

--------------------------------

### Run nv-monitor with TUI

Source: https://github.com/wentbackward/nv-monitor/blob/main/README.md

Launch the nv-monitor TUI. Default refresh rate is 1 second.

```bash
./nv-monitor
```

--------------------------------

### Build and Run Synthetic Load Test

Source: https://github.com/wentbackward/nv-monitor/blob/main/README.md

Build the demo-load tool and run it to generate sinusoidal CPU and GPU loads. This is useful for visual testing and multi-node validation.

```bash
make demo-load
./demo-load --gpu          # CPU + GPU sinusoidal load
```

--------------------------------

### Build nv-monitor

Source: https://github.com/wentbackward/nv-monitor/blob/main/CLAUDE.md

Standard make targets for building the nv-monitor binary, including a portable version and testing. Also shows direct compilation command and dependencies.

```bash
make          # builds nv-monitor binary (-O3 -march=native)
make portable # builds without -march=native (for CI/distribution)
make test     # builds and runs unit tests
make clean    # removes binaries
```

```bash
Direct compilation: `gcc -O2 -Wall -Wextra -std=gnu11 -o nv-monitor nv-monitor.c -lncursesw -ldl -lpthread`
```

--------------------------------

### Run demo-load with GPU load only

Source: https://github.com/wentbackward/nv-monitor/blob/main/DEMO-LOAD.md

Generate GPU load exclusively, without any CPU threads, by using the --gpu-only flag.

```bash
./demo-load --gpu-only
```

--------------------------------

### Run demo-load with increased GPU blocks

Source: https://github.com/wentbackward/nv-monitor/blob/main/DEMO-LOAD.md

Customize the GPU load by specifying a higher number of CUDA blocks per kernel launch using the --blocks flag. This is useful for larger GPUs.

```bash
./demo-load --gpu --blocks 4096
```

--------------------------------

### Binary Sizes and Resource Usage on Jetson Orin Nano Super

Source: https://github.com/wentbackward/nv-monitor/blob/main/PERFORMANCE.md

Compares binary sizes and resource usage (RSS, Private Memory, CPU%) of nv-monitor against 'top' on a Jetson Orin Nano Super. Highlights nv-monitor's efficiency on edge devices.

```text
Binary sizes:
  nv-monitor: 64K
  top:        118K
  htop:       265K

Measurements (10s sample):
  nv-monitor (headless)            RSS: 15312 KB  Private: 12932 KB  CPU: 0.20%
  nv-monitor (headless+prometheus) RSS: 15328 KB  Private: 12940 KB  CPU: 0.20%
  top (batch mode)                 RSS:  3424 KB  Private:   788 KB  CPU: 2.10%

NVML + Tegra libraries: ~1.7 MB (split across multiple Tegra shared objects)
```

--------------------------------

### Binary Sizes and Resource Usage on DGX Spark

Source: https://github.com/wentbackward/nv-monitor/blob/main/PERFORMANCE.md

Compares the binary sizes and resource consumption (RSS, Private Memory, CPU%) of nv-monitor against 'top' on a DGX Spark system. nv-monitor shows significantly lower CPU usage while offering more features.

```text
Binary sizes:
  nv-monitor: 76K
  top:        136K
  htop:       391K

Measurements (10s sample):
  nv-monitor (headless)            RSS: 20908 KB  Private: 15088 KB  CPU: 0.10%
  nv-monitor (headless+prometheus) RSS: 20920 KB  Private: 15100 KB  CPU: 0.10%
  top (batch mode)                 RSS:  5508 KB  Private:  2080 KB  CPU: 0.70%

NVML library: 2.2 MB (shared with other GPU processes)
```

--------------------------------

### CSV Logging Implementation

Source: https://github.com/wentbackward/nv-monitor/blob/main/CLAUDE.md

Explains the opt-in CSV logging feature, including file output and sharing data with the TUI.

```c
// CSV logging: Opt-in via `-l FILE`, writes timestamped rows with all CPU/memory/GPU metrics. Shares derived calculations with the TUI via `meminfo_calc()`.
```

--------------------------------

### Run nv-monitor with Custom Refresh Rate

Source: https://github.com/wentbackward/nv-monitor/blob/main/README.md

Run nv-monitor with the TUI and set a custom refresh rate in milliseconds.

```bash
./nv-monitor -r 2000
```

--------------------------------

### Run nv-monitor with TUI and CSV Logging

Source: https://github.com/wentbackward/nv-monitor/blob/main/README.md

Run nv-monitor with the TUI and log statistics to a CSV file. The interval can be specified.

```bash
./nv-monitor -l stats.csv
```

```bash
./nv-monitor -l stats.csv -i 5000
```

--------------------------------

### CPU Sampling Logic

Source: https://github.com/wentbackward/nv-monitor/blob/main/CLAUDE.md

Explains how CPU usage percentages are computed by reading /proc/stat deltas between frames.

```c
// CPU sampling: Reads `/proc/stat` delta between frames to compute per-core usage percentages.
```

--------------------------------

### Run nv-monitor with Prometheus Metrics

Source: https://github.com/wentbackward/nv-monitor/blob/main/README.md

Run nv-monitor with the TUI and expose Prometheus metrics on a specified port.

```bash
./nv-monitor -p 9101
```

--------------------------------

### GPU Process Information

Source: https://github.com/wentbackward/nv-monitor/blob/main/CLAUDE.md

Explains how GPU process details, including PID to command name resolution, are obtained via NVML and /proc.

```c
// GPU process info: Queries both compute and graphics process lists via NVML, resolves PID to command name and user via `/proc/<pid>/`.
```

--------------------------------

### Run nv-monitor Headless with Prometheus Exporter

Source: https://github.com/wentbackward/nv-monitor/blob/main/README.md

Run nv-monitor in headless mode to act as a Prometheus metrics exporter on a specified port.

```bash
./nv-monitor -n -p 9101
```

--------------------------------

### Systemd Environment for Locale

Source: https://github.com/wentbackward/nv-monitor/blob/main/CLAUDE.md

Shows the systemd service file configuration to safeguard locale settings for Prometheus.

```bash
Environment="LC_ALL=C"
```

--------------------------------

### TUI Rendering with ncursesw

Source: https://github.com/wentbackward/nv-monitor/blob/main/CLAUDE.md

Details the TUI rendering approach using ncursesw for wide character support and color pairs for different UI elements.

```c
// TUI rendering: ncursesw (wide character support) with color pairs (1=red/critical, 2=green/normal, 3=yellow/medium, 6=cyan/headers). `draw_screen()` is the main render function.
```

--------------------------------

### Run nv-monitor in Headless Mode with CSV Logging

Source: https://github.com/wentbackward/nv-monitor/blob/main/README.md

Run nv-monitor without the TUI (headless) and log statistics to a CSV file with a specified interval.

```bash
./nv-monitor -n -l stats.csv -i 500
```

--------------------------------

### CPU Thermals and Frequency

Source: https://github.com/wentbackward/nv-monitor/blob/main/CLAUDE.md

Describes the sources for reading CPU thermal and frequency information.

```c
// CPU thermals/freq: Reads from `/sys/class/thermal/` and `/sys/devices/system/cpu/`.
```

--------------------------------

### Execute Performance Benchmarking Scripts

Source: https://github.com/wentbackward/nv-monitor/blob/main/PERFORMANCE.md

Scripts used for performance testing. `bench.sh` provides a resource usage snapshot, while `soak-test.sh` performs a memory stability test.

```bash
./bench.sh              # Resource usage snapshot vs top/htop
```

```bash
./soak-test.sh 10       # 10-minute memory stability soak test
```

--------------------------------

### nv-monitor Memory Breakdown Comparison

Source: https://github.com/wentbackward/nv-monitor/blob/main/PERFORMANCE.md

Details the unique memory cost of nv-monitor on DGX Spark and Orin Nano systems, differentiating between code, private heap, and shared library code.

```text
Component | DGX Spark | Orin Nano |
-----------|-----------|-----------|
nv-monitor code + stack | ~3 MB | ~3 MB |
NVML private heap | ~12 MB | ~10 MB |
NVML shared library code | 2.2 MB (shared) | 1.7 MB (shared) |
Prometheus thread stack | +128 KB | +128 KB |
**Total unique to nv-monitor** | **~15 MB** | **~13 MB** |
```

--------------------------------

### History Chart Rendering

Source: https://github.com/wentbackward/nv-monitor/blob/main/CLAUDE.md

Describes the implementation of a history chart using Unicode block elements and a ring buffer.

```c
// History chart: Ring buffer of last 20 CPU/GPU samples, rendered as full-width vertical bar chart using Unicode block elements (▁▂▃▄▅▆▇█).
```

--------------------------------

### Secure Prometheus Metrics with Token

Source: https://github.com/wentbackward/nv-monitor/blob/main/README.md

Enable token authentication for the Prometheus metrics endpoint using a CLI flag or environment variable. Using an environment variable is preferred for security.

```bash
./nv-monitor -p 9101 -t my-secret-token           # token via CLI flag
NV_MONITOR_TOKEN=my-secret-token ./nv-monitor -p 9101  # token via env var (preferred)
```

--------------------------------

### Run Soak Test Script

Source: https://github.com/wentbackward/nv-monitor/blob/main/PERFORMANCE.md

Command to run the memory stability soak test. The default duration is 10 minutes, but can be increased for production validation.

```bash
./soak-test.sh [minutes]
```

--------------------------------

### Memory Parsing Logic

Source: https://github.com/wentbackward/nv-monitor/blob/main/CLAUDE.md

Details the method for obtaining memory statistics by parsing /proc/meminfo.

```c
// Memory: Parses `/proc/meminfo` for used/available/buffers/cached/swap.
```

--------------------------------

### Prometheus Overhead Analysis

Source: https://github.com/wentbackward/nv-monitor/blob/main/PERFORMANCE.md

Shows the minimal CPU and memory overhead added by the Prometheus export feature in nv-monitor. The Prometheus thread is designed for low activity between scrapes.

```text
Mode | Threads | Additional CPU | Additional memory |
------|---------|---------------|-------------------|
Headless only | 2 | baseline | baseline |
Headless + Prometheus | 3 | +0.00% | +128 KB (thread stack) |
```

--------------------------------

### Locale Handling for Prometheus

Source: https://github.com/wentbackward/nv-monitor/blob/main/CLAUDE.md

Addresses the critical issue of decimal separators for Prometheus by setting the LC_NUMERIC locale to 'C'.

```c
// Fix in code (line ~1757): After `setlocale(LC_ALL, "")`, the code calls `setlocale(LC_NUMERIC, "C")` to force POSIX decimal formatting while preserving Unicode support for the TUI. This is the authoritative fix — it works regardless of the system locale or systemd environment.
```
Tokens:
Raw