### Verify Environment Setup

Source: https://github.com/nvidia/cuda-checkpoint/blob/main/_autodocs/quick-reference.md

Verify the CUDA driver version, CUDA toolkit, header file locations, library locations, and CRIU installation.

```bash
# Verify driver version
nvidia-smi --query | grep "Driver Version"

# Verify CUDA toolkit
nvcc --version

# Locate CUDA header files
ls /usr/local/cuda/include/cuda.h

# Locate CUDA libraries
ls /usr/local/cuda/lib64/libcuda.so*

# Verify CRIU is installed
which criu
criu --version
```

--------------------------------

### Example Device Map with UUIDs

Source: https://github.com/nvidia/cuda-checkpoint/blob/main/_autodocs/quick-reference.md

An example illustrating the device map format using actual GPU UUIDs for migration.

```text
GPU-12345678-1234-5678-1234-567812345678=GPU-87654321-4321-8765-4321-876543218765
```

--------------------------------

### Container Partial Passthrough Example

Source: https://github.com/nvidia/cuda-checkpoint/blob/main/_autodocs/driver-version-features.md

Illustrates checkpointing and migrating a containerized GPU application using `cuda-checkpoint`. This example shows how to obtain the container's PID and then use the CLI to lock, checkpoint, restore with a device map, and unlock the container's process.

```bash
# Run container with GPU access
docker run -d --gpus all myimage ./app &
CONTAINER_PID=$(docker inspect -f '{{.State.Pid}}' <container_id>)

# Checkpoint and migrate to different GPU
cuda-checkpoint --action lock --pid $CONTAINER_PID
cuda-checkpoint --action checkpoint --pid $CONTAINER_PID
cuda-checkpoint --action restore --pid $CONTAINER_PID --device-map "<device_map>"
cuda-checkpoint --action unlock --pid $CONTAINER_PID
```

--------------------------------

### Install CRIU and Verify Tools

Source: https://github.com/nvidia/cuda-checkpoint/blob/main/_autodocs/integration-guide.md

Installs CRIU on Ubuntu/Debian and verifies the installation of CRIU, NVIDIA driver, and CUDA toolkit.

```bash
# Install CRIU (example for Ubuntu/Debian)
sudo apt-get install criu

# Verify CRIU is installed and supports dump/restore
criu --version

# Verify nvidia-smi and CUDA driver
nvidia-smi
nvcc --version

# Verify CRIU CUDA plugin (comes with CUDA toolkit)
ls -la /usr/local/cuda/lib64/criu/cuda_plugin.so
```

--------------------------------

### CUDA Checkpoint CLI Success Example

Source: https://github.com/nvidia/cuda-checkpoint/blob/main/_autodocs/error-handling.md

Demonstrates a successful execution of the cuda-checkpoint command and checks the exit code.

```bash
$ cuda-checkpoint --get-state --pid 12345
CU_PROCESS_STATE_RUNNING
$ echo $?
```

--------------------------------

### Get Help and Debugging Information

Source: https://github.com/nvidia/cuda-checkpoint/blob/main/_autodocs/quick-reference.md

Access CLI help, check driver and CUDA toolkit versions, verify CRIU availability, and enable verbose logging.

```bash
# CLI help
cuda-checkpoint --help

# Check driver version
nvidia-smi

# Check CUDA toolkit version
nvcc --version

# Verify CRIU is available
criu --version

# Enable verbose logging (if available)
CUDA_DEBUG=1 cuda-checkpoint --get-state --pid $PID
```

--------------------------------

### Build and Run Counter Example

Source: https://github.com/nvidia/cuda-checkpoint/blob/main/_autodocs/example-programs.md

This script builds a counter application, launches it, and then uses cuda-checkpoint and CRIU to perform a full checkpoint and restore cycle, verifying GPU state.

```bash
#!/bin/bash
# Builds counter, launches it, checkpoints with CRIU, restores, and verifies GPU state
```

--------------------------------

### Run CUDA Counter Application

Source: https://github.com/nvidia/cuda-checkpoint/blob/main/README.md

Launch the counter application in the background and wait for it to start listening.

```bash
localhost# ./counter &
[1] 298027
localhost# sleep 1

```

--------------------------------

### Minimal CUDA Application Example

Source: https://github.com/nvidia/cuda-checkpoint/blob/main/_autodocs/integration-guide.md

A basic C application that initializes CUDA, retains a primary context, and stays running to allow for checkpointing.

```c
#include <cuda.h>
#include <stdio.h>

int main() {
    cuInit(0);
    
    CUcontext ctx;
    CUdevice dev;
    cuDeviceGet(&dev, 0);
    cuDevicePrimaryCtxRetain(&ctx, dev);
    cuCtxSetCurrent(ctx);
    
    // Your CUDA computations here
    // ...
    
    // Stay running so we can checkpoint
    sleep(1000);
    
    return 0;
}
```

--------------------------------

### Migrate GPU State to Different GPU

Source: https://github.com/nvidia/cuda-checkpoint/blob/main/_autodocs/example-programs.md

This example shows how to checkpoint a process on its original GPU and then restore it on a different GPU by mapping the old UUID to the new UUID. CUDA must be locked and unlocked around the operations.

```bash
# Checkpoint on original GPU
cuda-checkpoint --action lock --pid $PID
cuda-checkpoint --action checkpoint --pid $PID

# Restore on different GPU
OLD_UUID=$(cuDeviceGetUuid(0))
NEW_UUID=$(cuDeviceGetUuid(1))
cuda-checkpoint --action restore --pid $PID --device-map "GPU-$OLD_UUID=GPU-$NEW_UUID"

cuda-checkpoint --action unlock --pid $PID
```

--------------------------------

### CUDA Checkpoint GPU Migration Example

Source: https://github.com/nvidia/cuda-checkpoint/blob/main/_autodocs/quick-reference.md

This C code demonstrates how to perform a GPU migration during the restore process by creating a mapping of old GPU UUIDs to new ones. Ensure the target_pid is set.

```c
int dev_count;
cuDeviceGetCount(&dev_count);

// Create mapping: GPU i -> GPU (i+1) % dev_count
CUcheckpointGpuPair *pairs = malloc(dev_count * sizeof(pairs[0]));
for (int i = 0; i < dev_count; i++) {
    cuDeviceGetUuid(&pairs[i].oldUuid, i);
    cuDeviceGetUuid(&pairs[i].newUuid, (i + 1) % dev_count);
}

// Lock and checkpoint
CUcheckpointLockArgs lock_args = {0};
cuCheckpointProcessLock(target_pid, &lock_args);

CUcheckpointCheckpointArgs checkpoint_args = {0};
cuCheckpointProcessCheckpoint(target_pid, &checkpoint_args);

// Restore with mapping
CUcheckpointRestoreArgs restore_args = {0};
restore_args.gpuPairs = pairs;
restore_args.gpuPairsCount = dev_count;
cuCheckpointProcessRestore(target_pid, &restore_args);

// Unlock
CUcheckpointUnlockArgs unlock_args = {0};
cuCheckpointProcessUnlock(target_pid, &unlock_args);

free(pairs);
```

--------------------------------

### Process Tree Checkpoint with NVML

Source: https://github.com/nvidia/cuda-checkpoint/blob/main/_autodocs/DELIVERY.txt

Example of a process tree checkpoint using NVML. This is relevant for Display Driver 570 and higher.

```c
/*
 * r570-features.c
 * Process tree checkpoint with NVML
 */

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <nvml.h>

// Assume functions like checkpoint_process_tree() and nvml_init() are defined elsewhere

int main() {
    nvmlInit();
    printf("NVML initialized.\n");

    // Placeholder for actual checkpointing logic
    printf("Performing process tree checkpoint...\n");
    // checkpoint_process_tree();

    printf("Checkpoint complete.\n");
    nvmlShutdown();
    return 0;
}

```

--------------------------------

### CLI Checkpoint with Timeout and CRIU Integration (Driver 570)

Source: https://github.com/nvidia/cuda-checkpoint/blob/main/_autodocs/driver-version-features.md

Illustrates using the cuda-checkpoint CLI utility with a timeout for lock operations and subsequent CRIU checkpointing. This example requires Display Driver 570+ and CRIU 4.0+.

```bash
# CLI with timeout
cuda-checkpoint --action lock --pid 12345 --timeout 5000
cuda-checkpoint --action checkpoint --pid 12345
# Run CRIU checkpoint
cuda-checkpoint --action unlock --pid 12345
```

--------------------------------

### Check System Driver Version

Source: https://github.com/nvidia/cuda-checkpoint/blob/main/_autodocs/driver-version-features.md

Use the nvidia-smi command to query the installed NVIDIA driver version on your system. This is useful for a quick check of the base driver.

```bash
nvidia-smi --query | grep "Driver Version"
# Output: Driver Version: 570.86.10
```

--------------------------------

### Build CUDA Counter Program

Source: https://github.com/nvidia/cuda-checkpoint/blob/main/_autodocs/example-programs.md

Compile the CUDA C++ source file using nvcc to create an executable. Ensure the CUDA Toolkit is installed and accessible.

```bash
nvcc counter.cu -o counter
```

--------------------------------

### Get Process Info

Source: https://github.com/nvidia/cuda-checkpoint/blob/main/_autodocs/quick-reference.md

Retrieve detailed information about a specific process, including PID, user, state, command, and memory usage (VSZ, RSS). Replace `<pid>` with the actual process ID.

```bash
ps -p <pid> -o pid,user,stat,cmd,vsz,rss
```

--------------------------------

### Integration Test CUDA Checkpoint Workflow

Source: https://github.com/nvidia/cuda-checkpoint/blob/main/_autodocs/best-practices.md

An integration test script that builds a test application, checkpoints it using CUDA checkpoint tools and `criu`, restores it, and verifies recovery. It requires `gcc`, `cuda-checkpoint`, and `criu` to be installed.

```bash
#!/bin/bash
# Integration test with application and checkpoint

set -e

# Build test application
gcc test_app.c -o test_app -lcuda

# Start application
./test_app & 
APP_PID=$!

# Give application time to initialize
sleep 2

# Verify application is running
kill -0 $APP_PID || { echo "App failed to start"; exit 1; }

# Checkpoint
cuda-checkpoint --action lock --pid $APP_PID
cuda-checkpoint --action checkpoint --pid $APP_PID
criu dump --shell-job --images-dir ./test-checkpoint \
    --libdir /usr/local/cuda/lib64/criu --tree $APP_PID

# Restore
criu restore --shell-job --restore-detached \
    --images-dir ./test-checkpoint \
    --libdir /usr/local/cuda/lib64/criu

NEW_PID=$(pgrep -f test_app)
cuda-checkpoint --action lock --pid $NEW_PID
cuda-checkpoint --action restore --pid $NEW_PID
cuda-checkpoint --action unlock --pid $NEW_PID

# Verify application recovered
if kill -0 $NEW_PID 2>/dev/null; then
    echo "SUCCESS: Application recovered"
    kill $NEW_PID
else
    echo "FAILURE: Application did not recover"
    exit 1
fi

# Cleanup
rm -rf ./test-checkpoint

```

--------------------------------

### Build r580-migration-cli.c

Source: https://github.com/nvidia/cuda-checkpoint/blob/main/_autodocs/example-programs.md

Compile the C program for GPU migration using the command-line interface. Ensure CUDA Toolkit include paths are correctly specified.

```bash
gcc -I /usr/local/cuda-13.0/include r580-migration-cli.c -o r580-migration-cli -lcuda
```

--------------------------------

### CUDA Checkpoint CLI Invalid Argument Example

Source: https://github.com/nvidia/cuda-checkpoint/blob/main/_autodocs/error-handling.md

Shows an example of an invalid argument being passed to the cuda-checkpoint CLI, resulting in a usage error.

```bash
$ cuda-checkpoint --unknown-option
Usage: cuda-checkpoint ...
$ echo $?
```

--------------------------------

### Initialize CUcheckpointLockArgs

Source: https://github.com/nvidia/cuda-checkpoint/blob/main/_autodocs/cuda-driver-api.md

Example of zero-initializing the CUcheckpointLockArgs structure for the cuCheckpointProcessLock() function.

```c
CUcheckpointLockArgs lock_args = {0};
```

--------------------------------

### GPU Migration using CLI

Source: https://github.com/nvidia/cuda-checkpoint/blob/main/_autodocs/DELIVERY.txt

Illustrates GPU migration using a command-line interface (CLI) utility. This is a convenient way to manage GPU reassignment for Display Driver 580 and higher.

```bash
# r580-migration-cli.c
# GPU migration using CLI

# Assume 'cuda-migrate' is a hypothetical CLI tool
# Example usage:
# cuda-migrate --source-pid <pid> --from-gpu <gpu_id_src> --to-gpu <gpu_id_dst>

echo "Performing GPU migration using CLI..."
# cuda-migrate --source-pid 12345 --from-gpu 0 --to-gpu 1
echo "GPU migration via CLI initiated."

```

--------------------------------

### Build r580-migration-api.c

Source: https://github.com/nvidia/cuda-checkpoint/blob/main/_autodocs/example-programs.md

Compile the C program using GCC, linking against CUDA and NVML libraries. Ensure the CUDA Toolkit include path is specified.

```bash
gcc -I /usr/local/cuda-13.0/include r580-migration-api.c -o r580-migration-api -lcuda -lnvidia-ml
```

--------------------------------

### Get Restore Thread ID CLI Command

Source: https://github.com/nvidia/cuda-checkpoint/blob/main/_autodocs/quick-reference.md

Retrieves the restore thread ID for a given process.

```bash
cuda-checkpoint --get-restore-tid --pid <pid>
```

--------------------------------

### CUDA Checkpoint CLI Process Not Found Example

Source: https://github.com/nvidia/cuda-checkpoint/blob/main/_autodocs/error-handling.md

Illustrates the error when the specified PID is not found for the cuda-checkpoint operation.

```bash
$ cuda-checkpoint --get-state --pid 999999
Error: Process 999999 not found
$ echo $?
```

--------------------------------

### CUDA Application with Checkpoint Signal Handling

Source: https://github.com/nvidia/cuda-checkpoint/blob/main/_autodocs/integration-guide.md

This C code demonstrates how to initialize CUDA, allocate device memory, and set up a signal handler to gracefully respond to checkpoint requests. The application enters a loop, performs GPU work, and waits for a SIGUSR1 signal to initiate a checkpoint.

```c
#include <cuda.h>
#include <stdio.h>
#include <signal.h>
#include <stdlib.h>
#include <unistd.h>

volatile sig_atomic_t checkpoint_requested = 0;

void handle_signal(int sig) {
    if (sig == SIGUSR1) {
        checkpoint_requested = 1;
    }
}

int main() {
    // Initialize CUDA
    cuInit(0);
    CUdevice dev;
    CUcontext ctx;
    cuDeviceGet(&dev, 0);
    cuDevicePrimaryCtxRetain(&ctx, dev);
    cuCtxSetCurrent(ctx);
    
    // Allocate device memory
    CUdeviceptr d_data;
    size_t data_size = 1024 * 1024;
    cuMemAlloc(&d_data, data_size);
    
    // Set up signal handler for checkpoint requests
    signal(SIGUSR1, handle_signal);
    
    printf("Application ready, PID=%d\n", getpid());
    
    // Main loop
    int iteration = 0;
    while (1) {
        // Do GPU work
        printf("Iteration %d\n", iteration++);
        sleep(1);
        
        // Check if checkpoint was requested
        if (checkpoint_requested) {
            printf("Checkpoint requested\n");
            // Application can perform cleanup here if needed
            checkpoint_requested = 0;
            
            // Wait for external checkpoint/restore process
            // (cuda-checkpoint + CRIU handle the actual work)
            sleep(100);  // Will be interrupted by restore
        }
    }
    
    return 0;
}
```

--------------------------------

### CUDA Checkpoint CLI Lock Timeout Example

Source: https://github.com/nvidia/cuda-checkpoint/blob/main/_autodocs/error-handling.md

Demonstrates the error scenario where a lock operation exceeds the specified timeout.

```bash
$ cuda-checkpoint --action lock --pid 12345 --timeout 1000
Error: Lock timeout after 1000ms
$ echo $?
```

--------------------------------

### Display Help Message

Source: https://github.com/nvidia/cuda-checkpoint/blob/main/_autodocs/cli-reference.md

Use the --help or -h option to print the help message and exit the application.

```bash
cuda-checkpoint --help
```

--------------------------------

### Get GPU Memory Info

Source: https://github.com/nvidia/cuda-checkpoint/blob/main/_autodocs/quick-reference.md

Query specific GPU memory details, including total and free memory, formatted as CSV.

```bash
nvidia-smi --query-gpu=memory.total,memory.free --format=csv
```

--------------------------------

### Create Checkpoint Directory

Source: https://github.com/nvidia/cuda-checkpoint/blob/main/README.md

Create a directory to store the checkpoint image.

```bash
localhost# mkdir -p demo

```

--------------------------------

### Get CUDA Process State

Source: https://github.com/nvidia/cuda-checkpoint/blob/main/_autodocs/cuda-driver-api.md

Queries the current checkpoint state of a specified process. Use this to determine if a process is running or has been checkpointed.

```c
CUresult cuCheckpointProcessGetState(pid_t pid, CUprocessState *state);

```

```c
pid_t target_pid = 12345;
CUprocessState state;
CUresult result = cuCheckpointProcessGetState(target_pid, &state);
if (result == CUDA_SUCCESS) {
    if (state == CU_PROCESS_STATE_RUNNING) {
        printf("Process CUDA is running\n");
    } else {
        printf("Process CUDA is checkpointed\n");
    }
}

```

--------------------------------

### Get Current State CLI Command

Source: https://github.com/nvidia/cuda-checkpoint/blob/main/_autodocs/quick-reference.md

Use this command to retrieve the current state of a CUDA process. Replace <pid> with the actual process ID.

```bash
cuda-checkpoint --get-state --pid <pid>
```

--------------------------------

### Test Recovery Procedures

Source: https://github.com/nvidia/cuda-checkpoint/blob/main/_autodocs/DELIVERY.txt

Demonstrates how to test recovery procedures by restoring applications from checkpoints. This ensures that the checkpointing and restore mechanisms are reliable.

```bash
# Example of testing recovery procedures

# Create a checkpoint, then simulate a failure and restore

echo "Testing recovery procedures..."
# 1. Create checkpoint:
#    cuda-checkpoint --pid <pid> --operation checkpoint --output /tmp/test_checkpoint
# 2. Simulate failure (e.g., kill process)
# 3. Restore:
#    cuda-checkpoint --operation restore --input /tmp/test_checkpoint --pid <new_pid>

echo "Recovery procedure testing simulation complete."

```

--------------------------------

### CUDA Process Checkpoint with Device Migration

Source: https://github.com/nvidia/cuda-checkpoint/blob/main/_autodocs/cuda-driver-api.md

Illustrates how to checkpoint a process and then restore it, migrating its associated GPUs to different devices. This is useful in environments where GPU availability or numbering changes between checkpoint and restore.

```c
pid_t target_pid = 12345;
int dev_count;
cuDeviceGetCount(&dev_count);

// Build mapping: rotate each GPU to the next
CUcheckpointGpuPair *pairs = malloc(dev_count * sizeof(pairs[0]));
for (int i = 0; i < dev_count; i++) {
    cuDeviceGetUuid(&pairs[i].oldUuid, i);
    cuDeviceGetUuid(&pairs[i].newUuid, (i + 1) % dev_count);
}

// Lock and checkpoint
CUcheckpointLockArgs lock_args = {0};
cuCheckpointProcessLock(target_pid, &lock_args);

CUcheckpointCheckpointArgs checkpoint_args = {0};
cuCheckpointProcessCheckpoint(target_pid, &checkpoint_args);

// Restore with migration
CUcheckpointRestoreArgs restore_args = {0};
restore_args.gpuPairs = pairs;
restore_args.gpuPairsCount = dev_count;
cuCheckpointProcessRestore(target_pid, &restore_args);

CUcheckpointUnlockArgs unlock_args = {0};
cuCheckpointProcessUnlock(target_pid, &unlock_args);

free(pairs);
```

--------------------------------

### Build r570-features.c Program

Source: https://github.com/nvidia/cuda-checkpoint/blob/main/_autodocs/example-programs.md

Compile the r570-features.c program using gcc, linking against CUDA and NVML libraries. Ensure the CUDA include path is correctly specified.

```bash
gcc -I /usr/local/cuda-12.8/include r570-features.c -o r570-features -lcuda -lnvidia-ml
```

--------------------------------

### Minimal Feature Set CLI Command

Source: https://github.com/nvidia/cuda-checkpoint/blob/main/_autodocs/driver-version-features.md

Use this command for simple checkpoint/restore without integration. It requires only the driver.

```bash
cuda-checkpoint --toggle --pid $PID
```

--------------------------------

### Get Restore Thread ID (CLI)

Source: https://github.com/nvidia/cuda-checkpoint/blob/main/_autodocs/DELIVERY.txt

Use the --get-restore-tid CLI operation to retrieve the thread ID associated with a restore operation. This can be useful for debugging.

```bash
cuda-checkpoint --get-restore-tid
```

--------------------------------

### Get CUDA Error String

Source: https://github.com/nvidia/cuda-checkpoint/blob/main/_autodocs/cuda-driver-api.md

Use `cuGetErrorString` to convert a `CUresult` error code into a human-readable string. This is useful for logging or displaying error information to the user.

```c
CUresult result = cuCheckpointProcessCheckpoint(pid, &args);
if (result != CUDA_SUCCESS) {
    const char *error_str = cuGetErrorString(result);
    fprintf(stderr, "Checkpoint failed: %s\n", error_str);
}
```

--------------------------------

### GPU Migration using CLI

Source: https://github.com/nvidia/cuda-checkpoint/blob/main/_autodocs/driver-version-features.md

Shows how to perform GPU migration operations (lock, checkpoint, restore, unlock) using the `cuda-checkpoint` command-line interface. The restore action supports a `--device-map` parameter for specifying old-to-new GPU UUID mappings.

```bash
cuda-checkpoint --action lock --pid 12345
cuda-checkpoint --action checkpoint --pid 12345

# Restore with device map: old_uuid=new_uuid,old_uuid2=new_uuid2,...
cuda-checkpoint --action restore --pid 12345 \
    --device-map "GPU-aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee=GPU-ffffffff-gggg-hhhh-iiii-jjjjjjjjjjjj"

cuda-checkpoint --action unlock --pid 12345
```

--------------------------------

### Check CUDA Toolkit Version

Source: https://github.com/nvidia/cuda-checkpoint/blob/main/_autodocs/driver-version-features.md

Verify the installed CUDA Toolkit version using the nvcc command. This indicates the version of the CUDA compiler and associated libraries available.

```bash
nvcc --version
# Output: Cuda compilation tools, release 12.8, V12.8.123
```

--------------------------------

### Container Checkpoint/Restore

Source: https://github.com/nvidia/cuda-checkpoint/blob/main/_autodocs/DELIVERY.txt

Shows how to checkpoint and restore entire containers using the provided tools. This integrates with containerization platforms like Docker.

```bash
# Example of container checkpoint and restore

# Assume 'docker' and 'cuda-checkpoint' are used

echo "Performing container checkpoint..."
# docker checkpoint create --checkpoint-dir /path/to/checkpoints <container_id> my_container_checkpoint

echo "Performing container restore..."
# docker start --checkpoint /path/to/checkpoints/my_container_checkpoint <new_container_name>

echo "Container checkpoint/restore simulation complete."

```

--------------------------------

### Checkpoint CUDA Application in Container

Source: https://github.com/nvidia/cuda-checkpoint/blob/main/_autodocs/integration-guide.md

Checkpoint a running CUDA application within a Docker container. This involves getting the container PID, locking, checkpointing, and dumping with CRIU.

```bash
# Container must be running CUDA application
CONTAINER_ID=$(docker ps -q)
CONTAINER_PID=$(docker inspect -f '{{.State.Pid}}' $CONTAINER_ID)

# Checkpoint within the container
docker exec $CONTAINER_ID mkdir -p /checkpoint-images

cuda-checkpoint --action lock --pid $CONTAINER_PID
cuda-checkpoint --action checkpoint --pid $CONTAINER_PID

criu dump \
    --shell-job \
    --images-dir /tmp/container-checkpoint \
    --libdir /usr/local/cuda/lib64/criu \
    --tree $CONTAINER_PID

cuda-checkpoint --action unlock --pid $CONTAINER_PID 2>/dev/null || true

# Container can be stopped; images saved
docker stop $CONTAINER_ID
```

--------------------------------

### Get Driver and Runtime Versions in C

Source: https://github.com/nvidia/cuda-checkpoint/blob/main/_autodocs/driver-version-features.md

Retrieve the CUDA driver and runtime versions programmatically within a C application. This is essential for runtime checks and compatibility assessments.

```c
#include <cuda.h>
#include <stdio.h>

int main() {
    int driver_version = 0;
    int runtime_version = 0;
    
    cuDriverGetVersion(&driver_version);
    cudaRuntimeGetVersion(&runtime_version);
    
    printf("Driver version: %d\n", driver_version);
    printf("Runtime version: %d\n", runtime_version);
    
    // Driver version returned as integer: major*1000 + minor*10
    // Example: 570.86 → 570086
    // Example: 580.00 → 580000
    
    int major = driver_version / 1000;
    int minor = (driver_version % 1000) / 10;
    
    printf("Driver: %d.%d\n", major, minor);
    
    return 0;
}
```

--------------------------------

### CUDA Checkpoint CLI Reference

Source: https://github.com/nvidia/cuda-checkpoint/blob/main/_autodocs/INDEX.md

Complete command-line interface documentation for the `cuda-checkpoint` binary, including command syntax, operations, parameters, options, return codes, and typical workflows.

```APIDOC
## CLI Reference

### Description
Provides documentation for the `cuda-checkpoint` command-line interface.

### Usage
`cuda-checkpoint [options] --action <action>`

### Operations
- `--get-state`: Retrieves the current state of a process.
- `--action lock`: Locks a process to prevent checkpointing.
- `--action checkpoint`: Performs a checkpoint of a process.
- `--action restore`: Restores a process from a checkpoint.
- `--action unlock`: Unlocks a process.
- `--get-restore-tid`: Retrieves the thread ID for restore operations.
- `--toggle`: Toggles checkpointing for a process.

### Global Options
- `--pid <pid>`: Specifies the process ID.
- `--timeout <seconds>`: Sets a timeout for the operation.
- `--help`: Displays help information.

### Return Codes
Details on exit codes and error handling are provided in the full documentation.
```

--------------------------------

### Query Process CUDA State

Source: https://github.com/nvidia/cuda-checkpoint/blob/main/_autodocs/cli-reference.md

Use this command to get the current CUDA checkpoint state of a specified process. Ensure the process ID is correct and you have sufficient permissions.

```bash
cuda-checkpoint --get-state --pid 12345
```

--------------------------------

### Checkpoint/Restore Workflow with CUDA

Source: https://github.com/nvidia/cuda-checkpoint/blob/main/_autodocs/example-programs.md

Demonstrates the full checkpoint and restore process for the CUDA counter application. This includes suspending CUDA, dumping the process state with CRIU, restoring it, and resuming CUDA to continue execution.

```bash
# Build and run counter in background
nvcc counter.cu -o counter
./counter &
PID=$!
sleep 1

# Send packet and verify it works
echo hello | nc -u localhost 10000 -W 1  # Should respond with 101

# Suspend CUDA
cuda-checkpoint --toggle --pid $PID

# Checkpoint with CRIU (CUDA is now suspended)
mkdir -p demo
criu dump --shell-job --images-dir demo --tree $PID

# Restore with CRIU
criu restore --shell-job --restore-detached --images-dir demo

# Resume CUDA
cuda-checkpoint --toggle --pid $PID

# Send another packet; counter continues from where it left off
echo hello | nc -u localhost 10000 -W 1  # Should respond with 102
```

--------------------------------

### Display Help Message for cuda-checkpoint

Source: https://github.com/nvidia/cuda-checkpoint/blob/main/README.md

Use this command to view all available operations and options for the cuda-checkpoint utility. It displays version information and copyright details.

```bash
localhost$ cuda-checkpoint --help
CUDA checkpoint and restore utility.
Version 570.86.10. Copyright (C) 2025 NVIDIA Corporation. All rights reserved.

Operations:
--get-state --pid <pid>
        Prints the current checkpoint state of the process specified by <pid>

--action lock | checkpoint | restore | unlock --pid <pid> [--timeout <ms>]
        Performs the specified action on <pid>.
        For the lock action a timeout can be provided, the lock operation will wait up to <ms> milliseconds for the operation to succeed.

--toggle --pid <pid>
        Toggles the CUDA state in the specified process between the running and checkpointed states

--get-restore-tid --pid <pid>
        Retrieves the CUDA restore thread ID of the process specified by <pid>

Options:
--pid|-p <pid>
        The pid upon which to perform the operation

--timeout|-t <timeout>
        Optional timeout that can be specified for the lock action in milliseconds

--help|-h
        Print this help message
```

--------------------------------

### Restore CUDA Application in New Container

Source: https://github.com/nvidia/cuda-checkpoint/blob/main/_autodocs/integration-guide.md

Restore a checkpointed CUDA application in a new Docker container. This involves starting a new container, restoring CRIU images, and then using cuda-checkpoint.

```bash
# Start new container
docker run -d --gpus all myimage sleep 3600

NEW_CONTAINER_ID=$(docker ps -q | head -1)
NEW_CONTAINER_PID=$(docker inspect -f '{{.State.Pid}}' $NEW_CONTAINER_ID)

# Restore CUDA and process
criu restore \
    --shell-job \
    --restore-detached \
    --images-dir /tmp/container-checkpoint \
    --libdir /usr/local/cuda/lib64/criu

cuda-checkpoint --action lock --pid $NEW_CONTAINER_PID
cuda-checkpoint --action restore --pid $NEW_CONTAINER_PID
cuda-checkpoint --action unlock --pid $NEW_CONTAINER_PID
```

--------------------------------

### GPU Migration using CUDA Driver API

Source: https://github.com/nvidia/cuda-checkpoint/blob/main/_autodocs/driver-version-features.md

Demonstrates how to perform GPU migration by reassigning checkpointed workloads to different GPUs using the CUDA Driver API. Requires setting up GPU UUID mappings and using `cuCheckpointRestoreArgs`.

```c
int dev_count;
cuDeviceGetCount(&dev_count);

CUcheckpointGpuPair *pairs = malloc(dev_count * sizeof(pairs[0]));

// Map GPU i to GPU (i+1) % dev_count
for (int i = 0; i < dev_count; i++) {
    cuDeviceGetUuid(&pairs[i].oldUuid, i);
    cuDeviceGetUuid(&pairs[i].newUuid, (i + 1) % dev_count);
}

CUcheckpointLockArgs lock_args = {0};
cuCheckpointProcessLock(target_pid, &lock_args);

CUcheckpointCheckpointArgs checkpoint_args = {0};
cuCheckpointProcessCheckpoint(target_pid, &checkpoint_args);

CUcheckpointRestoreArgs restore_args = {0};
restore_args.gpuPairs = pairs;
restore_args.gpuPairsCount = dev_count;
cuCheckpointProcessRestore(target_pid, &restore_args);

CUcheckpointUnlockArgs unlock_args = {0};
cuCheckpointProcessUnlock(target_pid, &unlock_args);

free(pairs);
```

--------------------------------

### Load Balancing Strategies

Source: https://github.com/nvidia/cuda-checkpoint/blob/main/_autodocs/DELIVERY.txt

Discusses load balancing strategies in a GPU cluster, potentially involving migrating workloads or checkpoints to distribute load evenly.

```bash
# Conceptual example of load balancing strategy

# This would typically involve monitoring cluster load and making decisions

echo "Implementing load balancing strategy..."
# Monitor GPU utilization across nodes
# If node A is overloaded and node B is underutilized:
#   Migrate a workload from A to B using cuda-migrate or similar tools
#   Or, migrate a checkpoint to B and start a new instance there

echo "Load balancing strategy simulation complete."

```

--------------------------------

### List Processes Using GPU

Source: https://github.com/nvidia/cuda-checkpoint/blob/main/_autodocs/quick-reference.md

Display a list of processes currently utilizing the GPU, including their PID and name, formatted as CSV.

```bash
nvidia-smi --query-compute-apps=pid,name --format=csv
```

--------------------------------

### Basic CUDA Process Checkpoint and Restore

Source: https://github.com/nvidia/cuda-checkpoint/blob/main/_autodocs/cuda-driver-api.md

Demonstrates the fundamental sequence of locking a process, performing a checkpoint, and then restoring it. This is the most basic usage pattern for the CUDA Checkpoint API.

```c
pid_t target_pid = fork(); // or attach to running process
CUresult result;

// Lock the process
CUcheckpointLockArgs lock_args = {0};
result = cuCheckpointProcessLock(target_pid, &lock_args);
assert(result == CUDA_SUCCESS);

// Checkpoint
CUcheckpointCheckpointArgs checkpoint_args = {0};
result = cuCheckpointProcessCheckpoint(target_pid, &checkpoint_args);
assert(result == CUDA_SUCCESS);

// Process is now checkpointed; can be saved by CRIU or other tools
// ...

// Restore
CUcheckpointRestoreArgs restore_args = {0};
restore_args.gpuPairs = NULL;
restore_args.gpuPairsCount = 0;
result = cuCheckpointProcessRestore(target_pid, &restore_args);
assert(result == CUDA_SUCCESS);

// Unlock
CUcheckpointUnlockArgs unlock_args = {0};
result = cuCheckpointProcessUnlock(target_pid, &unlock_args);
assert(result == CUDA_SUCCESS);
```

--------------------------------

### Orchestration Patterns

Source: https://github.com/nvidia/cuda-checkpoint/blob/main/_autodocs/DELIVERY.txt

Discusses orchestration patterns for managing checkpointed applications, particularly in containerized environments. This involves coordinating checkpointing and restoration with orchestrators like Kubernetes.

```bash
# Conceptual example of orchestration patterns

# This would involve custom controllers or operators

echo "Implementing orchestration patterns for checkpointing..."
# Example workflow:
# 1. Orchestrator detects need for checkpoint (e.g., node maintenance)
# 2. Sends signal to application or uses CLI to trigger checkpoint
# 3. Application state saved
# 4. Pod/container rescheduled on another node
# 5. Orchestrator triggers restore operation on the new node

echo "Orchestration pattern simulation complete."

```

--------------------------------

### Migrate GPU Assignment

Source: https://github.com/nvidia/cuda-checkpoint/blob/main/_autodocs/cli-reference.md

This workflow demonstrates migrating a process's GPU assignment using CUDA Checkpoint, starting from driver version 580. It involves checkpointing on the original GPU and restoring on different GPUs using a device map.

```bash
# Checkpoint on original GPU
cuda-checkpoint --action lock --pid $PID
cuda-checkpoint --action checkpoint --pid $PID

# Restore on different GPU(s) via device map
cuda-checkpoint --action restore --pid $PID \
  --device-map "GPU-old-uuid-1=GPU-new-uuid-1,GPU-old-uuid-2=GPU-new-uuid-2"

cuda-checkpoint --action unlock --pid $PID
```

--------------------------------

### Compile and Run CUDA Application

Source: https://github.com/nvidia/cuda-checkpoint/blob/main/_autodocs/integration-guide.md

This bash script shows how to compile the CUDA application using nvcc and then run it in the background. It captures the application's PID and demonstrates how to send a SIGUSR1 signal to initiate a checkpoint.

```bash
nvcc -arch sm_80 app.cu -o app -lcuda
./app &
APP_PID=$!

# Checkpoint when ready
kill -SIGUSR1 $APP_PID

# Then run: cuda-checkpoint + criu commands
# After restore, application continues from where it left off
```

--------------------------------

### Basic Checkpoint/Restore (Driver 550)

Source: https://github.com/nvidia/cuda-checkpoint/blob/main/_autodocs/DELIVERY.txt

Covers the fundamental checkpoint and restore functionality available in Display Driver 550. This is the baseline for more advanced features.

```c
// r550-basic.c
// Basic checkpoint/restore for Display Driver 550

#include <stdio.h>
#include <cuda.h> // Assuming CUDA Driver API

int main() {
    // Initialize CUDA Driver API and create context (omitted for brevity)

    printf("Performing basic checkpoint and restore operations...\n");

    // Placeholder for basic checkpoint and restore logic
    // cuCheckpoint();
    // cuRestore();

    printf("Basic checkpoint/restore simulation complete.\n");

    // Clean up (omitted for brevity)

    return 0;
}

```

--------------------------------

### CLI Utility for Basic Operations

Source: https://github.com/nvidia/cuda-checkpoint/blob/main/_autodocs/DELIVERY.txt

Shows the usage of a CLI utility for basic checkpoint and restore operations. This utility is part of the Display Driver 550 features.

```bash
# Example CLI usage for basic checkpoint/restore

# Assume 'cuda-checkpoint-cli' is the utility

echo "Performing basic checkpoint using CLI..."
# cuda-checkpoint-cli --pid 12345 --operation checkpoint --output /path/to/checkpoint

echo "Performing basic restore using CLI..."
# cuda-checkpoint-cli --pid 67890 --operation restore --input /path/to/checkpoint

echo "CLI operations simulated."

```

--------------------------------

### View All GPUs with nvidia-smi

Source: https://github.com/nvidia/cuda-checkpoint/blob/main/_autodocs/quick-reference.md

Use this command to display detailed information about all detected NVIDIA GPUs in the system.

```bash
nvidia-smi
```

--------------------------------

### Restore Process with GPU Remapping

Source: https://github.com/nvidia/cuda-checkpoint/blob/main/_autodocs/integration-guide.md

Restore a process on different GPUs by mapping old UUIDs to new ones. Ensure all visible GPUs are included in the mapping.

```bash
# After CRIU restore, but before CUDA unlock

# Build device mapping
OLD_UUID=$(cuDeviceGetUuid(0))
NEW_UUID=$(cuDeviceGetUuid(1))

# Restore with GPU remapping
cuda-checkpoint --action restore --pid $NEW_APP_PID \
    --device-map "GPU-$OLD_UUID=GPU-$NEW_UUID"

# Unlock and resume
cuda-checkpoint --action unlock --pid $NEW_APP_PID
```

--------------------------------

### Checkpoint Process with CUDA Driver API

Source: https://github.com/nvidia/cuda-checkpoint/blob/main/_autodocs/README.md

This C code demonstrates how to checkpoint a process using the CUDA Driver API. Ensure CUDA headers are included and the correct process ID is provided.

```c
CUcheckpointCheckpointArgs checkpoint_args = {0};
cuCheckpointProcessCheckpoint(12345, &checkpoint_args);
```

--------------------------------

### Checkpoint State (CLI)

Source: https://github.com/nvidia/cuda-checkpoint/blob/main/_autodocs/DELIVERY.txt

Use the --action checkpoint CLI operation to initiate a checkpoint of the current CUDA state. Ensure the APIs are locked prior to execution.

```bash
cuda-checkpoint --action checkpoint
```

--------------------------------

### Checkpoint and Restore CUDA Process with CRIU

Source: https://github.com/nvidia/cuda-checkpoint/blob/main/_autodocs/quick-reference.md

This bash script demonstrates the full workflow of locking, checkpointing, and restoring a CUDA process using `cuda-checkpoint` and CRIU. Ensure the `myapp` process is running before executing.

```bash
#!/bin/bash

PID=$1
CHECKPOINT_DIR=$2

# Step 1: Lock CUDA
cuda-checkpoint --action lock --pid $PID

# Step 2: Checkpoint CUDA
cuda-checkpoint --action checkpoint --pid $PID

# Step 3: Checkpoint with CRIU
criu dump \
    --shell-job \
    --images-dir $CHECKPOINT_DIR \
    --libdir /usr/local/cuda/lib64/criu \
    --tree $PID

# Step 4: Restore with CRIU
criu restore \
    --shell-job \
    --restore-detached \
    --images-dir $CHECKPOINT_DIR \
    --libdir /usr/local/cuda/lib64/criu

# Step 5: Get restored PID and restore CUDA
NEW_PID=$(pgrep -f myapp)
cuda-checkpoint --action lock --pid $NEW_PID
cuda-checkpoint --action restore --pid $NEW_PID
cuda-checkpoint --action unlock --pid $NEW_PID

```

--------------------------------

### Sample Stateful CUDA Application

Source: https://github.com/nvidia/cuda-checkpoint/blob/main/_autodocs/DELIVERY.txt

A sample CUDA application designed to demonstrate stateful operations. This can be used with checkpointing mechanisms to save and restore application state.

```cu
// counter.cu
// Sample stateful CUDA application

#include <stdio.h>
#include <cuda_runtime.h>

__global__ void increment_counter(int *counter) {
    atomicAdd(counter, 1);
}

int main() {
    int *counter_d;
    int counter_h = 0;

    cudaMalloc(&counter_d, sizeof(int));
    cudaMemset(counter_d, 0, sizeof(int));

    // Launch kernel to increment counter
    increment_counter<<<1, 1>>>(counter_d);
    cudaDeviceSynchronize();

    // Read counter value
    cudaMemcpy(&counter_h, counter_d, sizeof(int), cudaMemcpyDeviceToHost);
    printf("Counter value: %d\n", counter_h);

    cudaFree(counter_d);
    return 0;
}

```

--------------------------------

### Docker Integration

Source: https://github.com/nvidia/cuda-checkpoint/blob/main/_autodocs/DELIVERY.txt

Illustrates the integration of CUDA checkpointing with Docker containers. This allows stateful applications within Docker to be checkpointed.

```bash
# Example of CUDA checkpointing integration with Docker

# Ensure CUDA drivers and toolkit are available within the Docker image

echo "Integrating CUDA checkpointing with Docker..."
# Run application inside Docker, then use docker checkpoint commands
# Or, use cuda-checkpoint CLI with container-specific options if available

echo "Docker integration example simulated."

```

--------------------------------

### Build CUDA Counter Application

Source: https://github.com/nvidia/cuda-checkpoint/blob/main/README.md

Compile the CUDA counter application using nvcc.

```bash
localhost$ nvcc counter.cu -o counter

```

--------------------------------

### Multi-Checkpoint Rotation

Source: https://github.com/nvidia/cuda-checkpoint/blob/main/_autodocs/DELIVERY.txt

Illustrates a strategy for managing multiple checkpoints by rotating them. This ensures that a history of checkpoints is available while managing storage space.

```bash
# Example of multi-checkpoint rotation strategy

# Assume a script manages checkpoint creation and deletion

echo "Implementing multi-checkpoint rotation..."
# Script logic:
# 1. Create new checkpoint
# 2. If number of checkpoints > max_allowed:
#      Delete oldest checkpoint

echo "Multi-checkpoint rotation strategy simulation complete."

```

--------------------------------

### CLI Commands

Source: https://github.com/nvidia/cuda-checkpoint/blob/main/_autodocs/quick-reference.md

Command-line interface commands for interacting with CUDA Checkpoint.

```APIDOC
## CLI Commands

### Get Current State
```bash
cuda-checkpoint --get-state --pid <pid>
```

### Lock CUDA APIs
```bash
cuda-checkpoint --action lock --pid <pid> [--timeout <ms>]
```

### Checkpoint CUDA State
```bash
cuda-checkpoint --action checkpoint --pid <pid>
```

### Restore CUDA State
```bash
cuda-checkpoint --action restore --pid <pid> [--device-map "<map>"]
```

### Unlock CUDA APIs
```bash
cuda-checkpoint --action unlock --pid <pid>
```

### Toggle (Checkpoint or Restore)
```bash
cuda-checkpoint --toggle --pid <pid>
```

### Get Restore Thread ID
```bash
cuda-checkpoint --get-restore-tid --pid <pid>
```
```

--------------------------------

### Container Partial Passthrough (Driver 580+)

Source: https://github.com/nvidia/cuda-checkpoint/blob/main/_autodocs/DELIVERY.txt

Explains container partial passthrough, a feature in Display Driver 580 and higher that allows specific parts of a container's state to be passed through during checkpointing.

```bash
# Example of container partial passthrough in Driver 580+

# This command assumes a hypothetical tool or integration

echo "Performing container checkpoint with partial passthrough..."
# cuda-checkpoint --container-id <container_id> --checkpoint-dir /path/to/checkpoints \
#                 --partial-passthrough --include-devices

echo "Container partial passthrough simulation complete."

```

--------------------------------

### Main Function for Unit Tests

Source: https://github.com/nvidia/cuda-checkpoint/blob/main/_autodocs/best-practices.md

Executes the defined unit tests for CUDA checkpoint functionality and reports success.

```c
int main() {
    test_lock_unlock();
    test_state_transitions();
    printf("All tests passed\n");
    return 0;
}

```

--------------------------------

### Periodic Checkpoint Strategy

Source: https://github.com/nvidia/cuda-checkpoint/blob/main/_autodocs/DELIVERY.txt

Illustrates a periodic checkpoint strategy for fault tolerance. Applications can be configured to checkpoint their state at regular intervals.

```bash
# Example of setting up a periodic checkpoint strategy

# Assume a scheduler or cron job is used to trigger checkpoints

echo "Configuring periodic checkpointing..."
# cron job entry example:
# 0 * * * * /usr/local/bin/cuda-checkpoint --pid <pid> --operation checkpoint --interval 3600

echo "Periodic checkpoint strategy configuration simulated."

```

--------------------------------

### NVML Support in Checkpoint/Restore

Source: https://github.com/nvidia/cuda-checkpoint/blob/main/_autodocs/DELIVERY.txt

Highlights the support for NVML (NVIDIA Management Library) during checkpoint and restore operations. This ensures that GPU state managed by NVML is preserved.

```c
// Example demonstrating NVML support during checkpoint/restore

#include <stdio.h>
#include <nvml.h>

int main() {
    nvmlInit();
    printf("NVML initialized.\n");

    // Assume application state related to NVML is managed here
    // e.g., querying GPU utilization, power limits, etc.

    printf("Performing checkpoint that includes NVML state...\n");
    // checkpoint_application_state_with_nvml();

    printf("Checkpoint complete. NVML state should be preserved.\n");

    nvmlShutdown();
    return 0;
}

```

--------------------------------

### Restore Process with CUDA Driver API

Source: https://github.com/nvidia/cuda-checkpoint/blob/main/_autodocs/README.md

This C code demonstrates how to restore a process from a checkpoint using the CUDA Driver API. Ensure CUDA headers are included and the correct process ID is provided. The gpuPairs argument can be set to NULL if not specifying specific GPU pairs.

```c
CUcheckpointRestoreArgs restore_args = {0};
restore_args.gpuPairs = NULL;
cuCheckpointProcessRestore(12345, &restore_args);
```

--------------------------------

### Listing GPU UUIDs with nvidia-smi

Source: https://github.com/nvidia/cuda-checkpoint/blob/main/_autodocs/error-handling.md

Shows how to use nvidia-smi to query and display GPU UUIDs, useful for verifying device maps.

```bash
# List valid GPU UUIDs
nvidia-smi --query-gpu=gpu_uuid --format=csv,noheader

# Verify format: GPU-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
# Use exact UUIDs from nvidia-smi output in device map
```

--------------------------------

### Verify Counter Process on GPU

Source: https://github.com/nvidia/cuda-checkpoint/blob/main/README.md

Use nvidia-smi to confirm that the counter application is running on a GPU by checking its PID.

```bash
localhost# nvidia-smi --query --display=PIDS | grep $PID
        Process ID                        : 298027

```

--------------------------------

### Restore Process from Checkpoint (With GPU Migration)

Source: https://github.com/nvidia/cuda-checkpoint/blob/main/_autodocs/cuda-driver-api.md

Restores a process from a checkpoint while remapping GPUs according to the provided `gpuPairs` mapping. All accessible GPUs must be included in the mapping.

```c
CUcheckpointGpuPair pairs[2];

// Get original GPU UUIDs
cuDeviceGetUuid(&pairs[0].oldUuid, 0);
cuDeviceGetUuid(&pairs[1].oldUuid, 1);

// Specify target GPUs (e.g., rotate by one)
cuDeviceGetUuid(&pairs[0].newUuid, 1);
cuDeviceGetUuid(&pairs[1].newUuid, 0);

CUcheckpointRestoreArgs restore_args = {0};
restore_args.gpuPairs = pairs;
restore_args.gpuPairsCount = 2;

CUresult result = cuCheckpointProcessRestore(target_pid, &restore_args);
if (result == CUDA_SUCCESS) {
    printf("Restore completed with device migration\n");
}
```

--------------------------------

### Checkpoint a Process with CRIU

Source: https://github.com/nvidia/cuda-checkpoint/blob/main/_autodocs/cli-reference.md

This workflow demonstrates how to checkpoint a process using CUDA Checkpoint and CRIU. It includes locking CUDA, performing the checkpoint, dumping the process with CRIU, and unlocking CUDA.

```bash
# Lock CUDA
cuda-checkpoint --action lock --pid $PID

# Checkpoint CUDA
cuda-checkpoint --action checkpoint --pid $PID

# Checkpoint the process with CRIU (CUDA is now idle)
criu dump --shell-job --images-dir ./checkpoint-images --tree $PID

# Unlock CUDA (if process is restored)
cuda-checkpoint --action unlock --pid $PID
```

--------------------------------

### Test Counter Application

Source: https://github.com/nvidia/cuda-checkpoint/blob/main/README.md

Send a UDP packet to the counter application and observe the returned value, verifying GPU memory modification.

```bash
localhost# echo hello | nc -u localhost 10000 -W 1
101

```

--------------------------------

### Compile and Run CUDA Application

Source: https://github.com/nvidia/cuda-checkpoint/blob/main/_autodocs/integration-guide.md

Compiles the minimal CUDA application and runs it in the background, capturing its process ID.

```bash
# Compile the application
gcc -I /usr/local/cuda/include myapp.c -o myapp -lcuda

# Run it in the background
./myapp &
APP_PID=$!

# Give it time to initialize
sleep 1
```

--------------------------------

### Checkpoint Process with CRIU

Source: https://github.com/nvidia/cuda-checkpoint/blob/main/_autodocs/example-programs.md

Use the CRIU command-line tool to dump (checkpoint) a process. This command requires CUDA to be suspended first and specifies directories for images and plugins, as well as socket information.

```bash
criu dump --shell-job --images-dir <images-dir> --libdir <plugin-dir> --ext-unix-sk=<socket-inode> --tree <pid>
```

--------------------------------

### CRIU Integration with Process Trees

Source: https://github.com/nvidia/cuda-checkpoint/blob/main/_autodocs/DELIVERY.txt

Illustrates how CUDA checkpointing integrates with CRIU for checkpointing process trees. This is a key feature for Display Driver 570 and higher.

```bash
# Example of CRIU integration for process tree checkpointing

# Assume 'criu' is installed and configured
# Assume 'cuda-checkpoint' tool is available

echo "Integrating CUDA checkpointing with CRIU..."
# criu dump -t <pid> -o dump.img --ext-unix-sk --shell-job
# cuda-checkpoint --pid <pid> --criu-dump dump.img

echo "CRIU integration example simulated."

```

--------------------------------

### Initialize CUDA Driver API

Source: https://github.com/nvidia/cuda-checkpoint/blob/main/_autodocs/quick-reference.md

Initializes the CUDA driver API. The 'flags' parameter is used for initialization options.

```c
CUresult cuInit(unsigned int flags);
```

--------------------------------

### Trace System Calls

Source: https://github.com/nvidia/cuda-checkpoint/blob/main/_autodocs/quick-reference.md

Attach to a running process and trace its system calls in real-time. Replace `<pid>` with the actual process ID.

```bash
strace -p <pid>
```

--------------------------------

### Migrate Checkpoints Between Nodes

Source: https://github.com/nvidia/cuda-checkpoint/blob/main/_autodocs/DELIVERY.txt

Illustrates the process of migrating checkpoints from one node to another in a GPU cluster. This is essential for load balancing and fault tolerance.

```bash
# Example of migrating checkpoints between nodes

# Assume checkpoint data is transferred using tools like rsync or scp

echo "Migrating checkpoints between nodes..."
# On source node:
# rsync -avz /path/to/checkpoint user@destination_node:/path/to/destination/

echo "Checkpoint migration between nodes simulated."

```

--------------------------------

### CUDA Checkpoint CUDA Driver API Workflow

Source: https://github.com/nvidia/cuda-checkpoint/blob/main/_autodocs/quick-reference.md

This C code demonstrates the typical workflow for process checkpointing using the CUDA Driver API. It includes initialization, locking, checkpointing, optional restore, and unlocking.

```c
#include <cuda.h>

int main() {
    // Initialize
    cuInit(0);
    
    pid_t target_pid = 12345;
    
    // Lock
    CUcheckpointLockArgs lock_args = {0};
    cuCheckpointProcessLock(target_pid, &lock_args);
    
    // Checkpoint
    CUcheckpointCheckpointArgs checkpoint_args = {0};
    cuCheckpointProcessCheckpoint(target_pid, &checkpoint_args);
    
    // Perform other operations (CRIU, save state, etc.)
    // ...
    
    // Restore
    CUcheckpointRestoreArgs restore_args = {0};
    restore_args.gpuPairs = NULL;
    restore_args.gpuPairsCount = 0;
    cuCheckpointProcessRestore(target_pid, &restore_args);
    
    // Unlock
    CUcheckpointUnlockArgs unlock_args = {0};
    cuCheckpointProcessUnlock(target_pid, &unlock_args);
    
    return 0;
}
```