### Install Kubernetes Controller Runtime

Source: https://github.com/nvidia/nvsentinel/blob/main/DEVELOPMENT.md

Use 'go get' to install the controller-runtime library, which is recommended for implementing Kubernetes controllers.

```bash
# Use controller-runtime for Kubernetes controllers
go get sigs.k8s.io/controller-runtime
```

--------------------------------

### Setup KIND Cluster and Install NVSentinel

Source: https://github.com/nvidia/nvsentinel/blob/main/demos/local-fault-injection-demo/README.md

Initializes a Kubernetes cluster using KIND and installs NVSentinel. This is the first step in the interactive demo.

```bash
./scripts/00-setup.sh
```

--------------------------------

### EventProcessor Example

Source: https://github.com/nvidia/nvsentinel/blob/main/store-client/README.md

Illustrates creating and starting an EventProcessor for unified event handling with retry logic and metrics. Requires a watcher, database client, and configuration.

```go
processor := client.NewEventProcessor(watcher, dbClient, config)
processor.SetEventHandler(func(ctx, event) error {
    // Your processing logic
    return nil
})
processor.Start(ctx)
```

--------------------------------

### Test URL Construction

Source: https://github.com/nvidia/nvsentinel/blob/main/DEVELOPMENT.md

Source the setup script to inspect generated download URLs and verify their accessibility without running the full installation.

```bash
# Source the script functions
source scripts/setup-dev-env.sh

# Test specific tool URLs
echo "yq URL: $YQ_URL"
echo "kubectl URL: $KUBECTL_URL"
echo "protoc URL: $PROTOC_URL"
echo "shellcheck URL: $SHELLCHECK_URL"

# Test URL accessibility
curl -I "$SHELLCHECK_URL"
```

--------------------------------

### Automated Development Environment Setup

Source: https://github.com/nvidia/nvsentinel/blob/main/DEVELOPMENT.md

Execute the development environment setup script with AUTO_MODE=true to automatically install all dependencies without interactive prompts.

```bash
make dev-env-setup AUTO_MODE=true
```

--------------------------------

### Setup Environment with Make

Source: https://github.com/nvidia/nvsentinel/blob/main/demos/local-slinky-drain-demo/README.md

Runs the make setup command to create a KIND cluster with all necessary components for the Slinky Drainer demo.

```bash
make setup
```

--------------------------------

### Install Helm Chart with --set Arguments

Source: https://github.com/nvidia/nvsentinel/blob/main/distros/kubernetes/nvsentinel/charts/mongodb-store/charts/mongodb/README.md

Use the --set key=value argument to specify configuration parameters during Helm chart installation. This example demonstrates setting authentication credentials and database details for MongoDB.

```console
helm install my-release \
    --set auth.rootPassword=secretpassword,auth.username=my-user,auth.password=my-password,auth.database=my-database \
    oci://REGISTRY_NAME/REPOSITORY_NAME/mongodb
```

--------------------------------

### Install NVSentinel via Helm

Source: https://context7.com/nvidia/nvsentinel/llms.txt

Commands to install prerequisites like cert-manager and Prometheus, followed by the NVSentinel chart installation and verification.

```bash
# Install prerequisites
# 1. Install cert-manager for TLS certificate management
helm repo add jetstack https://charts.jetstack.io --force-update
helm upgrade --install cert-manager jetstack/cert-manager \
  --namespace cert-manager --create-namespace \
  --version v1.19.1 --set installCRDs=true \
  --wait

# 2. Install Prometheus for metrics (optional but recommended)
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts --force-update
helm upgrade --install prometheus prometheus-community/kube-prometheus-stack \
  --namespace monitoring --create-namespace \
  --set prometheus.enabled=true \
  --set alertmanager.enabled=false \
  --set grafana.enabled=false \
  --wait

# 3. Install NVSentinel
NVSENTINEL_VERSION=v1.0.0
helm upgrade --install nvsentinel oci://ghcr.io/nvidia/nvsentinel \
  --namespace nvsentinel --create-namespace \
  --version "$NVSENTINEL_VERSION" \
  --timeout 15m \
  --wait

# Verify installation
kubectl get pods -n nvsentinel
kubectl get nodes
```

--------------------------------

### Install Prerequisites

Source: https://github.com/nvidia/nvsentinel/blob/main/distros/kubernetes/README.md

Install cert-manager and Prometheus using Helm repositories to satisfy NVSentinel dependencies.

```bash
# Install cert-manager
helm repo add jetstack https://charts.jetstack.io --force-update
helm install cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --create-namespace \
  --version v1.19.1 \
  --set installCRDs=true

# Install Prometheus (optional, for metrics collection)
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm install prometheus prometheus-community/kube-prometheus-stack \
  --namespace monitoring \
  --create-namespace
```

--------------------------------

### Install Sigstore Policy Controller

Source: https://github.com/nvidia/nvsentinel/blob/main/distros/kubernetes/nvsentinel/policies/README.md

Commands to install the Sigstore Policy Controller via kubectl or Helm.

```bash
# Install Policy Controller using the latest release
kubectl apply -f https://github.com/sigstore/policy-controller/releases/latest/download/policy-controller.yaml

# Verify installation
kubectl -n cosign-system get pods
```

```bash
helm repo add sigstore https://sigstore.github.io/helm-charts
helm repo update

# Get the version from .versions.yaml
POLICY_CONTROLLER_VERSION=$(yq eval '.cluster.policy_controller' .versions.yaml)

# Install specific version
helm install policy-controller sigstore/policy-controller \
  -n cosign-system \
  --create-namespace \
  --version "${POLICY_CONTROLLER_VERSION}"
```

--------------------------------

### Install Development Dependencies

Source: https://github.com/nvidia/nvsentinel/blob/main/CONTRIBUTING.md

Install the necessary dependencies for development using the provided make target.

```bash
make dev-env-setup
```

--------------------------------

### Enable Debug Mode for Setup Scripts

Source: https://github.com/nvidia/nvsentinel/blob/main/DEVELOPMENT.md

Run setup scripts with detailed output to diagnose architecture detection, URL construction, and HTTP response issues.

```bash
# Run with detailed debugging output
DEBUG=true make dev-env-setup AUTO_MODE=true

# Or run the script directly
DEBUG=true AUTO_MODE=true ./scripts/setup-dev-env.sh
```

--------------------------------

### Install Helm Chart with Parameters

Source: https://github.com/nvidia/nvsentinel/blob/main/distros/kubernetes/nvsentinel/charts/postgresql/README.md

Use the `--set` argument to specify parameters when installing a Helm chart. Replace placeholders with your registry and repository names.

```console
helm install my-release \
    --set auth.postgresPassword=secretpassword
    oci://REGISTRY_NAME/REPOSITORY_NAME/postgresql
```

--------------------------------

### Install MongoDB Helm Chart

Source: https://github.com/nvidia/nvsentinel/blob/main/distros/kubernetes/nvsentinel/charts/mongodb-store/charts/mongodb/README.md

Use this command to install the MongoDB Helm chart for a quick deployment.

```console
helm install my-release oci://registry-1.docker.io/bitnamicharts/mongodb
```

--------------------------------

### NVSentinel Configuration Example

Source: https://github.com/nvidia/nvsentinel/blob/main/README.md

Example YAML configuration for enabling or disabling NVSentinel modules and global settings.

```yaml
global:
  dryRun: false  # Test mode - log actions without executing
  
  # Health Monitors (enabled by default)
  gpuHealthMonitor:
    enabled: true
  syslogHealthMonitor:
    enabled: true

  # Core Modules (disabled by default - enable for production)
  faultQuarantine:
    enabled: false
  nodeDrainer:
    enabled: false
  faultRemediation:
    enabled: false
  janitor:
    enabled: false
  mongodbStore:
    enabled: false 
```

--------------------------------

### DCGM Service Connection Examples

Source: https://github.com/nvidia/nvsentinel/blob/main/docs/configuration/gpu-health-monitor.md

Examples for connecting to standard GPU Operator services or custom namespace services.

```yaml
dcgm:
  dcgmK8sServiceEnabled: true
  service:
    endpoint: "nvidia-dcgm.gpu-operator.svc"
    port: 5555
```

```yaml
dcgm:
  dcgmK8sServiceEnabled: true
  service:
    endpoint: "dcgm-service.custom-namespace.svc.cluster.local"
    port: 5555
```

--------------------------------

### Install Helm Chart with Values File

Source: https://github.com/nvidia/nvsentinel/blob/main/distros/kubernetes/nvsentinel/charts/postgresql/README.md

Install a Helm chart using a YAML file to specify parameters. Replace placeholders with your registry and repository names.

```console
helm install my-release -f values.yaml oci://REGISTRY_NAME/REPOSITORY_NAME/postgresql
```

--------------------------------

### Fast Retry Configuration Example

Source: https://github.com/nvidia/nvsentinel/blob/main/docs/configuration/event-exporter.md

An example of configuring failure handling with faster retry attempts and shorter backoff periods.

```yaml
failureHandling:
  maxRetries: 10
  initialBackoff: "100ms"
  maxBackoff: "10s"
  backoffMultiplier: 1.5
```

--------------------------------

### Start PostgreSQL Docker Container

Source: https://github.com/nvidia/nvsentinel/blob/main/store-client/POSTGRESQL_IMPLEMENTATION.md

Command to start a PostgreSQL 15 Docker container for integration testing. It sets up the database and exposes the port.

```bash
# Start PostgreSQL
docker run -d \
  --name nvsentinel-postgres \
  -e POSTGRES_PASSWORD=password \
  -e POSTGRES_DB=nvsentinel \
  -p 5432:5432 \
  postgres:15
```

--------------------------------

### Example Metadata Augmentor Configuration

Source: https://github.com/nvidia/nvsentinel/blob/main/docs/configuration/platform-connectors.md

An example demonstrating how to configure the MetadataAugmentor with custom cache settings and allowed labels, including a custom company-specific label.

```yaml
platformConnector:
  transformers:
    MetadataAugmentor:
      cacheSize: 100
      cacheTTLSeconds: 3600
      allowedLabels:
        - "topology.kubernetes.io/zone"
        - "topology.kubernetes.io/region"
        - "custom.company.com/rack-id"
```

--------------------------------

### Example Multi-Tier Application Eviction Configuration

Source: https://github.com/nvidia/nvsentinel/blob/main/docs/INTEGRATIONS.md

Define specific eviction modes for different application tiers within user namespaces. This example shows configurations for 'database', 'batch-jobs', and 'frontend' namespaces, with a wildcard '*' for all other namespaces.

```yaml
userNamespaces:
  # Critical database - wait for graceful shutdown
  - name: "database"
    mode: "AllowCompletion"
  
  # Batch processing - allow time for checkpoint
  - name: "batch-jobs"
    mode: "DeleteAfterTimeout"
  
  # Web frontend - fast failover
  - name: "frontend"
    mode: "Immediate"
  
  # Default for everything else
  - name: "*"
    mode: "AllowCompletion"

```

--------------------------------

### Run Step-by-Step Demo

Source: https://github.com/nvidia/nvsentinel/blob/main/demos/local-custom-remediation-demo/README.md

Executes the manual demonstration workflow, including cluster setup, triggering memory pressure, and verifying remediation.

```bash
# Step 0: Create cluster, install NVSentinel, build and deploy custom components
make setup

# Step 1: View the healthy cluster
make show-cluster

# Step 2: Deploy a memory-hungry pod to trigger pressure
make trigger

# Step 3: Verify the full remediation pipeline
make verify

# Clean up
make cleanup
```

--------------------------------

### Debug Development Environment Setup Issues

Source: https://github.com/nvidia/nvsentinel/blob/main/DEVELOPMENT.md

Enable debug mode by setting DEBUG=true when running the setup script to get detailed output for troubleshooting installation problems.

```bash
DEBUG=true make dev-env-setup AUTO_MODE=true
```

--------------------------------

### Initialize DataStore

Source: https://github.com/nvidia/nvsentinel/blob/main/store-client/POSTGRESQL_IMPLEMENTATION.md

Instantiate the DataStore using the provided context and configuration.

```go
ds, err := datastore.NewDataStore(ctx, *config)
if err != nil {
    return err
}
```

--------------------------------

### Install Prometheus for Metrics

Source: https://github.com/nvidia/nvsentinel/blob/main/README.md

Installs the kube-prometheus-stack using Helm for metrics collection and monitoring. This setup enables Prometheus but disables Alertmanager, Grafana, kube-state-metrics, and nodeExporter.

```bash
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts --force-update
helm upgrade --install prometheus prometheus-community/kube-prometheus-stack \
  --namespace monitoring --create-namespace \
  --set prometheus.enabled=true \
  --set alertmanager.enabled=false \
  --set grafana.enabled=false \
  --set kubeStateMetrics.enabled=false \
  --set nodeExporter.enabled=false \
  --wait
```

--------------------------------

### Install Fern and Run Local Preview

Source: https://github.com/nvidia/nvsentinel/blob/main/fern/README.md

Install the Fern CLI globally and run the local development server. Bind to a specific host IP for remote access if needed. Navigate to the provided URL to view the documentation locally.

```bash
npm install -g fern-api@4.42.1
fern check
HOST=0.0.0.0 fern docs dev   # bind to host IP for remote access
# navigate to http://<host>:3000/nvsentinel/dev
```

--------------------------------

### Initialize Legacy DatabaseClient in Go

Source: https://github.com/nvidia/nvsentinel/blob/main/store-client/README.md

Use the helper package for backward compatibility with existing modules.

```go
import (
    "github.com/nvidia/nvsentinel/store-client/pkg/helper"
    _ "github.com/nvidia/nvsentinel/store-client/pkg/datastore/providers/mongodb"
)

// Create database client only (no change streams)
dbClient, err := helper.NewDatabaseClientOnly(ctx, "my-module")
if err != nil {
    return err
}
defer dbClient.Close(ctx)

// Use MongoDB-style filters
filter := map[string]interface{}{"nodename": "node-1"}
cursor, err := dbClient.Find(ctx, filter, nil)
```

--------------------------------

### Implement GpuService Go Client

Source: https://context7.com/nvidia/nvsentinel/llms.txt

Example client implementation for listing GPUs and watching for lifecycle events.

```go
// Example Go client for GpuService
package main

import (
    "context"
    "log"

    pb "github.com/nvidia/nvsentinel/api/gen/go/device/v1alpha1"
    "google.golang.org/grpc"
)

func main() {
    conn, err := grpc.Dial("localhost:50051", grpc.WithInsecure())
    if err != nil {
        log.Fatalf("Failed to connect: %v", err)
    }
    defer conn.Close()

    client := pb.NewGpuServiceClient(conn)

    // List all GPUs
    resp, err := client.ListGpus(context.Background(), &pb.ListGpusRequest{})
    if err != nil {
        log.Fatalf("ListGpus failed: %v", err)
    }

    for _, gpu := range resp.GpuList.Items {
        log.Printf("GPU: %s (UUID: %s)", gpu.Name, gpu.Spec.Uuid)
        for _, condition := range gpu.Status.Conditions {
            log.Printf("  Condition: %s = %s (%s)",
                condition.Type, condition.Status, condition.Reason)
        }
    }

    // Watch for GPU changes
    stream, err := client.WatchGpus(context.Background(), &pb.WatchGpusRequest{})
    if err != nil {
        log.Fatalf("WatchGpus failed: %v", err)
    }

    for {
        event, err := stream.Recv()
        if err != nil {
            log.Fatalf("Watch error: %v", err)
        }
        log.Printf("Event: %s - GPU %s", event.Type, event.Object.Name)
    }
}
```

--------------------------------

### AWS SendRebootSignal gRPC Server in Go

Source: https://github.com/nvidia/nvsentinel/blob/main/docs/designs/017-remediation-plugins.md

An example of a gRPC server implementation for the CSPPluginService, specifically handling the SendRebootSignal request. This server interacts with the AWS EC2 API to perform the reboot operation. It includes the main function to start the gRPC server.

```go
package main

import (
	"context"
	"fmt"
	"log"
	"net"

	"google.golang.org/grpc"
	pb "github.com/nvidia/nvsentinel/janitor/protos/v1"
)

type cspPluginServer struct {
	pb.UnimplementedCSPPluginServiceServer
}

func (s *cspPluginServer) SendRebootSignal(ctx context.Context, req *pb.SendRebootSignalRequest) (*pb.SendRebootSignalResponse, error) {
	log.Printf("Received reboot request for node: %s", req.NodeName)

	_, err = c.ec2.RebootInstances(ctx, &ec2.RebootInstancesInput{
		InstanceIds: []string{req.NodeName},
	})

	requestID := fmt.Sprintf("reboot-%s-123456", req.NodeName)
	return &pb.SendRebootSignalResponse{RequestId: requestID}, nil
}

func main() {
	lis, err := net.Listen("tcp", ":50051")
	if err != nil {
		log.Fatalf("Failed to listen: %v", err)
	}

	grpcServer := grpc.NewServer()
	pb.RegisterCSPPluginServiceServer(grpcServer, &cspPluginServer{})

	log.Printf("CSP Plugin Server listening on :50051")
	if err := grpcServer.Serve(lis); err != nil {
		log.Fatalf("Failed to serve: %v", err)
	}
}
```

--------------------------------

### Configure Manual Development Environment

Source: https://github.com/nvidia/nvsentinel/blob/main/DEVELOPMENT.md

Commands to set up the Go environment and install necessary development tools for module-specific work.

```bash
# Set up Go environment
export GOPATH=$(go env GOPATH)
export GO_CACHE_DIR=$(go env GOCACHE)

# Install development dependencies
go install github.com/golangci/golangci-lint/v2/cmd/golangci-lint@latest
go install gotest.tools/gotestsum@latest
go install github.com/boumenot/gocover-cobertura@latest

# For controller modules
go install sigs.k8s.io/controller-runtime/tools/setup-envtest@latest
```

--------------------------------

### Run Python GPU Health Monitor Tests

Source: https://github.com/nvidia/nvsentinel/blob/main/DEVELOPMENT.md

Use the Makefile for recommended commands to run lint-test, setup, linting, testing, or formatting for the GPU health monitor module. Alternatively, use manual Poetry commands for installation and execution of tests and linters.

```bash
# Using the module's Makefile (recommended)
make -C health-monitors/gpu-health-monitor lint-test  # Full lint-test
make -C health-monitors/gpu-health-monitor setup      # Just Poetry setup
make -C health-monitors/gpu-health-monitor lint       # Just Black check
make -C health-monitors/gpu-health-monitor test       # Just tests
make -C health-monitors/gpu-health-monitor format     # Run Black formatter

# Manual Poetry commands
cd health-monitors/gpu-health-monitor
poetry install
poetry run pytest -v
poetry run black --check .
poetry run coverage run --source=gpu_health_monitor -m pytest
```

--------------------------------

### Initialize Legacy Database Client

Source: https://github.com/nvidia/nvsentinel/blob/main/store-client/DEVELOPER_GUIDE.md

Use this for legacy code requiring backward compatibility. Helper packages automatically load configuration for a database client.

```go
import "github.com/nvidia/nvsentinel/store-client/pkg/helper"

// Helper packages automatically load configuration
dbClient, err := helper.NewDatabaseClientOnly(ctx, "my-module")
if err != nil {
    return err
}
defer dbClient.Close(ctx)
```

--------------------------------

### Add GKE Installer Pod Indexer in Labeler

Source: https://github.com/nvidia/nvsentinel/blob/main/docs/designs/018-syslog-monitor-preinstalled-driver-support.md

Defines an indexer for GKE installer pods, used to identify nodes where drivers are installed by GKE. It checks for both 'k8s-app' and 'name' labels matching the GKE installer application.

```go
// Add to indexers in NewLabeler
const NodeGKEInstallerIndex = "nodeGKEInstaller"

NodeGKEInstallerIndex: func(obj any) ([]string, error) {
    pod, ok := obj.(*v1.Pod)
    if !ok {
        return nil, fmt.Errorf("object is not a pod")
    }
    // Check for "k8s-app" label (primary)
    if app, exists := pod.Labels["k8s-app"]; exists && app == gkeInstallerApp {
        return []string{pod.Spec.NodeName}, nil
    }
    // Also check for "name" label (used by some GKE installer variants)
    if name, exists := pod.Labels["name"]; exists && name == gkeInstallerApp {
        return []string{pod.Spec.NodeName}, nil
    }
    return []string{}, nil
}
```

--------------------------------

### Clone NVSentinel and Set Up Development Environment

Source: https://github.com/nvidia/nvsentinel/blob/main/DEVELOPMENT.md

Clone the repository and run the make command to install all necessary dependencies for local development.

```bash
git clone https://github.com/nvidia/nvsentinel.git
cd nvsentinel
make dev-env-setup
```

--------------------------------

### Install cert-manager for TLS

Source: https://github.com/nvidia/nvsentinel/blob/main/README.md

Installs cert-manager using Helm, which is required for TLS certificate management in Kubernetes. This command adds the Jetstack Helm repository and upgrades or installs cert-manager.

```bash
helm repo add jetstack https://charts.jetstack.io --force-update
helm upgrade --install cert-manager jetstack/cert-manager \
  --namespace cert-manager --create-namespace \
  --version v1.19.1 --set installCRDs=true \
  --wait
```

--------------------------------

### Initialize Go Module for Preflight Check

Source: https://github.com/nvidia/nvsentinel/blob/main/DEVELOPMENT.md

Initialize a new Go module for your preflight check. Replace the example path with your module's specific path.

```bash
go mod init github.com/nvidia/nvsentinel/preflight-checks/my-check
```

--------------------------------

### Install NVSentinel from GitHub Container Registry

Source: https://github.com/nvidia/nvsentinel/blob/main/README.md

Installs NVSentinel using Helm. Ensure you have Kubernetes 1.25+ and Helm 3.0+ installed. The version can be specified using the NVSENTINEL_VERSION variable.

```bash
NVSENTINEL_VERSION=v1.0.0
# Install from GitHub Container Registry
helm install nvsentinel oci://ghcr.io/nvidia/nvsentinel \
  --version "$NVSENTINEL_VERSION" \
  --namespace nvsentinel \
  --create-namespace

# View chart information
helm show chart oci://ghcr.io/nvidia/nvsentinel --version "$NVSENTINEL_VERSION"
```

--------------------------------

### Get and Use PipelineBuilder

Source: https://github.com/nvidia/nvsentinel/blob/main/store-client/POSTGRESQL_IMPLEMENTATION.md

Demonstrates how to obtain the correct PipelineBuilder based on the DATASTORE_PROVIDER environment variable and use it to build a pipeline for a change stream watcher.

```go
import "github.com/nvidia/nvsentinel/store-client/pkg/client"

// GetPipelineBuilder() automatically returns the correct builder
// based on DATASTORE_PROVIDER environment variable
builder := client.GetPipelineBuilder()

// Build database-appropriate pipeline
pipeline := builder.BuildNodeQuarantineStatusPipeline()

// Use with change stream watcher
watcher, err := factory.CreateChangeStreamWatcher(ctx, dbClient, "my-service", pipeline)
```

--------------------------------

### Install NVSentinel with Dry Run Enabled

Source: https://context7.com/nvidia/nvsentinel/llms.txt

Install NVSentinel using Helm with the global dry-run option enabled. After installation, check module logs for 'DRY RUN' messages to verify intended actions.

```bash
# Install with dry-run enabled
helm upgrade --install nvsentinel oci://ghcr.io/nvidia/nvsentinel \
  --namespace nvsentinel --create-namespace \
  --version v1.0.0 \
  --set global.dryRun=true

# Check logs to see what WOULD happen
kubectl logs -n nvsentinel deployment/fault-quarantine | grep "DRY RUN"
kubectl logs -n nvsentinel deployment/node-drainer | grep "DRY RUN"
```

--------------------------------

### Local NVSentinel Demo with KIND

Source: https://context7.com/nvidia/nvsentinel/llms.txt

Set up and run a local NVSentinel demo using KIND (Kubernetes in Docker). This allows testing NVSentinel's fault injection and quarantine mechanisms without requiring physical GPU hardware.

```bash
# Clone the repository
git clone https://github.com/NVIDIA/NVSentinel.git
cd NVSentinel/demos/local-fault-injection-demo

# Run automated demo (5-10 minutes)
make demo

# Or step-by-step for learning:
# Step 0: Create cluster and install NVSentinel
./scripts/00-setup.sh

# Step 1: View the healthy cluster
./scripts/01-show-cluster.sh
# Expected: All nodes Ready, all pods Running

# Step 2: Inject a GPU fault (simulates hardware failure)
./scripts/02-inject-error.sh
# Uses fake DCGM to simulate corrupt InfoROM error

# Step 3: Verify node was cordoned
./scripts/03-verify-cordon.sh
# Expected: Worker node shows SchedulingDisabled
kubectl get nodes
# NAME                            STATUS                     ROLES           AGE   VERSION
# nvsentinel-demo-control-plane   Ready                      control-plane   12m   v1.31.0
# nvsentinel-demo-worker          Ready,SchedulingDisabled   <none>          12m   v1.31.0

# Cleanup
./scripts/99-cleanup.sh
# Or: make cleanup
```

--------------------------------

### PostgreSQL Generated SQL Example

Source: https://github.com/nvidia/nvsentinel/blob/main/store-client/POSTGRESQL_IMPLEMENTATION.md

Example of a generated SQL query for PostgreSQL health events.

```sql
SELECT id, data, createdAt, updatedAt
FROM health_events
WHERE (data->'healtheventstatus'->'userpodsevictionstatus'->>'status' = $1) OR
      ((data->'healtheventstatus'->>'nodequarantined' = $2) AND
       (data->'healtheventstatus'->'userpodsevictionstatus'->>'status' IN ($3, $4)))
ORDER BY createdAt DESC
```

--------------------------------

### Load Datastore Configuration and Create DataStore

Source: https://github.com/nvidia/nvsentinel/blob/main/store-client/DEVELOPER_GUIDE.md

Use this for new code. It automatically detects MongoDB or PostgreSQL from environment variables and creates a database-agnostic DataStore client.

```go
import "github.com/nvidia/nvsentinel/store-client/pkg/datastore"

// Load from environment - automatically detects MongoDB or PostgreSQL
config, err := datastore.LoadDatastoreConfig()
if err != nil {
    return err
}

// Create datastore - works with both databases
ds, err := datastore.NewDataStore(ctx, *config)
if err != nil {
    return err
}
defer ds.Close(ctx)
```

--------------------------------

### Use gRPC Client Connection

Source: https://github.com/nvidia/nvsentinel/blob/main/docs/designs/030-grpc-tls-authentication.md

Demonstrates how to use the `dialProvider` function to establish a gRPC connection, execute a client call, and ensure the connection is closed afterwards using `defer`.

```go
client, cleanup, err := r.dialProvider(ctx)
if err != nil { ... }
defer cleanup()

resp, err := client.RebootNode(ctx, req)
```

--------------------------------

### Register MongoDB Provider and Create Database Client

Source: https://github.com/nvidia/nvsentinel/blob/main/store-client/DEVELOPER_GUIDE.md

Import the MongoDB provider to enable its registration. Then, create a client factory from configuration and use it to obtain a database-agnostic client.

```go
// Register provider (done automatically by importing)
import _ "github.com/nvidia/nvsentinel/store-client/pkg/datastore/providers/mongodb"

// Create factory from environment
factory, err := helper.CreateClientFactory(config)
if err != nil {
    return err
}

// Get database-agnostic client
dbClient, err := factory.CreateDatabaseClient(ctx)
if err != nil {
    return err
}
```

--------------------------------

### Troubleshoot Download Failures

Source: https://github.com/nvidia/nvsentinel/blob/main/DEVELOPMENT.md

Verify URLs manually when encountering 404 errors during the setup process.

```bash
# Enable debug mode to see the exact URL
DEBUG=true ./scripts/setup-dev-env.sh

# Verify the URL manually
curl -I "https://github.com/koalaman/shellcheck/releases/download/v0.11.0/shellcheck-v0.11.0.linux.x86_64.tar.xz"

# Check if the release exists on GitHub
# Visit: https://github.com/<owner>/<repo>/releases
```

--------------------------------

### Install PSMDB Operator via Helm

Source: https://github.com/nvidia/nvsentinel/blob/main/distros/kubernetes/nvsentinel/charts/mongodb-store/charts/psmdb-operator/README.md

Adds the Percona Helm repository and installs the operator into a specific namespace.

```sh
helm repo add percona https://percona.github.io/percona-helm-charts/
helm install my-operator percona/psmdb-operator --version 1.21.0 --namespace my-namespace
```

--------------------------------

### Or Query Operator Example

Source: https://github.com/nvidia/nvsentinel/blob/main/store-client/POSTGRESQL_IMPLEMENTATION.md

Example of using the Or operator to combine multiple conditions. In PostgreSQL, this corresponds to `(cond1) OR (cond2)`.

```go
query.Or(cond1, cond2)
```

--------------------------------

### Initialize New Go Module

Source: https://github.com/nvidia/nvsentinel/blob/main/DEVELOPMENT.md

Initialize a new Go module for your health monitor. Replace the example path with your module's specific path.

```bash
go mod init github.com/nvidia/nvsentinel/health-monitors/my-monitor
```

--------------------------------

### And Query Operator Example

Source: https://github.com/nvidia/nvsentinel/blob/main/store-client/POSTGRESQL_IMPLEMENTATION.md

Example of using the And operator to combine multiple conditions. In PostgreSQL, this corresponds to `(cond1) AND (cond2)`.

```go
query.And(cond1, cond2)
```

--------------------------------

### Initialize Database-Agnostic Datastore in Go

Source: https://github.com/nvidia/nvsentinel/blob/main/store-client/README.md

Use the datastore package to initialize a connection that automatically adapts to the configured provider.

```go
import (
    "github.com/nvidia/nvsentinel/store-client/pkg/datastore"
    "github.com/nvidia/nvsentinel/store-client/pkg/query"
    _ "github.com/nvidia/nvsentinel/store-client/pkg/datastore/providers"
)

// Load configuration from environment
config, err := datastore.LoadDatastoreConfig()
if err != nil {
    return err
}

// Create datastore (automatically uses MongoDB or PostgreSQL based on DATASTORE_PROVIDER)
ds, err := datastore.NewDataStore(ctx, *config)
if err != nil {
    return err
}
defer ds.Close(ctx)

// Get health event store
healthStore := ds.HealthEventStore()

// Build database-agnostic query
q := query.New().Build(
    query.And(
        query.Eq("healthevent.nodename", "node-1"),
        query.Eq("healthevent.isfatal", true),
    ),
)

// Execute query - works with both MongoDB and PostgreSQL!
events, err := healthStore.FindHealthEventsByQuery(ctx, q)
if err != nil {
    return err
}

// Process events
for _, event := range events {
    fmt.Printf("Event: %s\n", event.HealthEvent.NodeName)
}
```

--------------------------------

### Perform CI-like Builds

Source: https://github.com/nvidia/nvsentinel/blob/main/DEVELOPMENT.md

Environment setup and build commands to simulate GitHub Actions CI workflows.

```bash
# Set up environment like GitHub Actions
export CONTAINER_REGISTRY="ghcr.io"
export CONTAINER_ORG="your-github-username"
export CI_COMMIT_REF_NAME="main"

# Build all images with full CI features (standardized)
make docker-all
```

--------------------------------

### Example ConfigMap

Source: https://github.com/nvidia/nvsentinel/blob/main/distros/kubernetes/nvsentinel/charts/mongodb-store/charts/mongodb/charts/common/README.md

An example of a ConfigMap resource that can be rendered by the common library chart. It includes a placeholder for dynamic naming.

```yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: {{ include "common.names.fullname" . }}
data:
  myvalue: "Hello World"
```

--------------------------------

### Install PSMDB Operator with YAML values

Source: https://github.com/nvidia/nvsentinel/blob/main/distros/kubernetes/nvsentinel/charts/mongodb-store/charts/psmdb-operator/README.md

Use a YAML file to specify configuration parameters during the Helm installation of the operator.

```sh
helm install psmdb-operator -f values.yaml percona/psmdb-operator
```

--------------------------------

### Select Preflight Checks for Workloads

Source: https://github.com/nvidia/nvsentinel/blob/main/docs/designs/034-preflight-check-selection.md

Examples of Pod specifications using different preflight check configurations.

```yaml
# Interactive notebook — fast check only (default, no annotation needed)
apiVersion: v1
kind: Pod
metadata:
  name: notebook
spec: ...
```

```yaml
# Batch training — medium DCGM + NCCL loopback
apiVersion: v1
kind: Pod
metadata:
  name: training-job
  annotations:
    nvsentinel.nvidia.com/preflight-checks: "preflight-dcgm-medium,preflight-nccl-loopback"
spec: ...
```

```yaml
# Post-maintenance validation — full stress test, no NCCL
apiVersion: v1
kind: Pod
metadata:
  name: gpu-validation
  annotations:
    nvsentinel.nvidia.com/preflight-checks: "preflight-dcgm-full"
spec: ...
```

--------------------------------

### Janitor Controller Integration Example

Source: https://github.com/nvidia/nvsentinel/blob/main/docs/INTEGRATIONS.md

Example of a RebootNode custom resource used by the Janitor controller to manage node maintenance.

```yaml
apiVersion: janitor.dgxc.nvidia.com/v1alpha1
kind: RebootNode
metadata:
  name: maintenance-gpu-node-01-673bac8e9f1234567890abcd
  namespace: nvsentinel
spec:
  nodeName: gpu-node-01
  reason: "Health event 673bac8e9f1234567890abcd"
  force: false
status:
  conditions:
    - type: NodeReady
      status: "False"
      reason: "RebootInProgress"
```

--------------------------------

### Install PSMDB Helm Chart

Source: https://github.com/nvidia/nvsentinel/blob/main/distros/kubernetes/nvsentinel/charts/mongodb-store/charts/psmdb-db/README.md

Use these commands to add the Percona Helm repository and install the PSMDB chart into a specific namespace.

```sh
helm repo add percona https://percona.github.io/percona-helm-charts/
helm install my-db percona/psmdb-db --version 1.21.1 --namespace my-namespace
```

--------------------------------

### Install PSMDB Operator with Helm

Source: https://github.com/nvidia/nvsentinel/blob/main/distros/kubernetes/nvsentinel/charts/mongodb-store/charts/psmdb-operator/README.md

This snippet shows how to add the Percona Helm repository and install the PSMDB Operator in a dedicated namespace.

```APIDOC
## Installing the chart

To install the chart with the `psmdb` release name using a dedicated namespace (recommended):

```sh
helm repo add percona https://percona.github.io/percona-helm-charts/
helm install my-operator percona/psmdb-operator --version 1.21.0 --namespace my-namespace
```

The chart can be customized using the following configurable parameters:

| Parameter                    | Description                                                                                                  | Default                                   |
| ---------------------------- | ------------------------------------------------------------------------------------------------------------ | ----------------------------------------- |
| `image.repository`           | PSMDB Operator Container image name                                                                          | `percona/percona-server-mongodb-operator` |
| `image.tag`                  | PSMDB Operator Container image tag                                                                           | `1.21.0`                                  |
| `image.pullPolicy`           | PSMDB Operator Container pull policy                                                                         | `Always`                                  |
| `image.pullSecrets`          | PSMDB Operator Pod pull secret                                                                               | `[]`                                      |
| `replicaCount`               | PSMDB Operator Pod quantity                                                                                  | `1`                                       |
| `tolerations`                | List of node taints to tolerate                                                                              | `[]`                                      |
| `annotations`                | PSMDB Operator Deployment annotations                                                                        | `{}`                                      |
| `podAnnotations`             | PSMDB Operator Pod annotations                                                                               | `{}`                                      |
| `labels`                     | PSMDB Operator Deployment labels                                                                             | `{}`                                      |
| `podLabels`                  | PSMDB Operator Pod labels                                                                                    | `{}`                                      |
| `resources`                  | Resource requests and limits                                                                                 | `{}`                                      |
| `nodeSelector`               | Labels for Pod assignment                                                                                    | `{}`                                      |
| `podAnnotations`             | Annotations for pod                                                                                          | `{}`                                      |
| `podSecurityContext`         | Pod Security Context                                                                                         | `{}`                                      |
| `watchNamespace`             | Set when a different from default namespace is needed to watch (comma separated if multiple needed)          | `""`                                    |
| `createNamespace`            | Set if you want to create watched namespaces with helm                                                       | `false`                                   |
| `rbac.create`                | If false RBAC will not be created. RBAC resources will need to be created manually                           | `true`                                    |
| `securityContext`            | Container Security Context                                                                                   | `{}`                                      |
| `serviceAccount.create`      | If false the ServiceAccounts will not be created. The ServiceAccounts must be created manually               | `true`                                    |
| `serviceAccount.annotations` | PSMDB Operator ServiceAccount annotations                                                                    | `{}`                                      |
| `logStructured`              | Force PSMDB operator to print JSON-wrapped log messages                                                      | `false`                                   |
| `logLevel`                   | PSMDB Operator logging level                                                                                 | `INFO`                                    |
```

--------------------------------

### XID 13 Journal Entry Example

Source: https://github.com/nvidia/nvsentinel/blob/main/docs/designs/014-workflow-XID-13-and-31.md

Example of a raw XID 13 journal entry containing hardware location data.

```text
NVRM: Xid (PCI:0009:01:00): 13, Graphics SM Warp Exception on (GPC 0, TPC 3, SM 0): Out Of Range Address
```

--------------------------------

### Troubleshoot Cluster Creation

Source: https://github.com/nvidia/nvsentinel/blob/main/demos/local-fault-injection-demo/README.md

Reset the demo environment by deleting the cluster and restarting the setup script.

```bash
# Clean up and retry
kind delete cluster --name nvsentinel-demo
./scripts/00-setup.sh
```

--------------------------------

### Environment-based Configuration

Source: https://github.com/nvidia/nvsentinel/blob/main/store-client/DEVELOPER_GUIDE.md

Initializes client factories using environment variables for configuration.

```go
// Automatic configuration from environment variables
config := &datastore.Config{
    ModuleName: "my-module",
    // DatabaseClientCertMountPath will be loaded from MONGODB_CLIENT_CERT_MOUNT_PATH
}

factory, err := helper.CreateClientFactory(config)
```

--------------------------------

### Kubernetes Node Condition YAML Example

Source: https://github.com/nvidia/nvsentinel/blob/main/docs/DATA_FLOW.md

Example of a Kubernetes Node Condition in YAML format, reflecting the status of a GPU node.

```yaml
conditions:
- type: GPUHealthy
  status: "False"
  reason: XID_ERROR_48
  message: "GPU 0 reported XID 48"
  lastTransitionTime: "2025-10-28T10:15:30Z"
```

--------------------------------

### Conservative Retry Configuration Example

Source: https://github.com/nvidia/nvsentinel/blob/main/docs/configuration/event-exporter.md

An example of configuring failure handling with more retries and longer backoff periods for conservative retry behavior.

```yaml
failureHandling:
  maxRetries: 30
  initialBackoff: "5s"
  maxBackoff: "15m"
  backoffMultiplier: 2.5
```

--------------------------------

### Aggressive Backfill Example

Source: https://github.com/nvidia/nvsentinel/blob/main/docs/configuration/event-exporter.md

An example configuration for an aggressive backfill strategy, covering a longer historical period and higher processing limits.

```yaml
backfill:
  enabled: true
  maxAge: "2160h"     # 90 days
  maxEvents: 5000000
  batchSize: 1000
  rateLimit: 5000
```

--------------------------------

### Go Module Configuration

Source: https://github.com/nvidia/nvsentinel/blob/main/DEVELOPMENT.md

Note on dependency management for the project.

```bash
# Dependencies managed via go.mod files with replace directives for local development
# No manual GOPRIVATE configuration needed
# Private repository authentication handled via SSH keys
```

--------------------------------

### DataStore API Usage with Query Builder

Source: https://github.com/nvidia/nvsentinel/blob/main/store-client/POSTGRESQL_IMPLEMENTATION.md

Demonstrates the modern, database-agnostic approach to interacting with the DataStore API. It shows loading configuration, creating a DataStore instance, and executing a query using the query builder.

```go
import (
    "github.com/nvidia/nvsentinel/store-client/pkg/datastore"
    "github.com/nvidia/nvsentinel/store-client/pkg/query"
    _ "github.com/nvidia/nvsentinel/store-client/pkg/datastore/providers"
)

// Load configuration
config, err := datastore.LoadDatastoreConfig()
if err != nil {
    return err
}

// Create datastore (works with both MongoDB and PostgreSQL)
ds, err := datastore.NewDataStore(ctx, *config)
if err != nil {
    return err
}
defer ds.Close(ctx)

// Get health event store
healthStore := ds.HealthEventStore()

// Build database-agnostic query
q := query.New().Build(
    query.And(
        query.Eq("healthevent.nodename", "node-1"),
        query.In("healtheventstatus.nodequarantined", []interface{}{"Quarantined", "AlreadyQuarantined"}),
    ),
)

// Execute query (works with MongoDB and PostgreSQL!)
events, err := healthStore.FindHealthEventsByQuery(ctx, q)
if err != nil {
    return err
}

// Process events
for _, event := range events {
    fmt.Printf("Event: %s\n", event.HealthEvent.NodeName)
}
```

--------------------------------

### Configure preflight-nccl-loopback

Source: https://github.com/nvidia/nvsentinel/blob/main/docs/configuration/preflight.md

Example configuration for the loopback preflight container, demonstrating an override for PCIe interconnect bandwidth thresholds.

```yaml
initContainers:
  - name: preflight-nccl-loopback
    image: ghcr.io/nvidia/nvsentinel/preflight-nccl-loopback:latest
    env:
      - name: BW_THRESHOLD_GBPS
        value: "15"           # PCIe interconnect
      - name: TEST_SIZE_MB
        value: "512"
```

--------------------------------

### Install NVSentinel with Helm

Source: https://github.com/nvidia/nvsentinel/blob/main/README.md

Installs or upgrades NVSentinel using Helm, specifying the namespace, version, and a timeout for the operation. It waits for the release to be deployed.

```bash
NVSENTINEL_VERSION=v1.0.0

helm upgrade --install nvsentinel oci://ghcr.io/nvidia/nvsentinel \
  --namespace nvsentinel --create-namespace \
  --version "$NVSENTINEL_VERSION" \
  --timeout 15m \
  --wait
```

--------------------------------

### CloudEvents v1.0 Format Example

Source: https://github.com/nvidia/nvsentinel/blob/main/docs/event-exporter.md

An example of a health event transformed into the CloudEvents v1.0 format, including metadata and health event details.

```json
{
  "specversion": "1.0",
  "type": "com.nvidia.nvsentinel.health.v1",
  "source": "nvsentinel://production-us-west/healthevents",
  "id": "550e8400-e29b-41d4-a716-446655440000",
  "time": "2025-11-27T10:30:00Z",
  "data": {
    "metadata": {
      "cluster": "production-us-west",
      "environment": "production",
      "region": "us-west-2"
    },
    "healthEvent": {
      "version": "v1",
      "agent": "gpu-health-monitor",
      "componentClass": "GPU",
      "checkName": "XIDError",
      "nodeName": "gpu-node-01",
      "message": "GPU XID error detected",
      "isFatal": true,
      "isHealthy": false,
      "recommendedAction": 2,
      "errorCode": ["XID_79"],
      "entitiesImpacted": [
        {
          "entityType": "GPU",
          "entityValue": "GPU-abc123"
        }
      ],
      "generatedTimestamp": "2025-11-27T10:30:00Z"
    }
  }
}

```