### Example Entry Structure (JSON)

Source: https://github.com/deepreinforce-ai/cuda-l1/blob/main/README.md

This JSON structure defines an example entry for the CUDA-L1 project, including various scores related to different optimization techniques like torch.compile and CUDA graphs. It serves as a template for data logging during experiments.

```json
{
  "level_id": 1,
  "task_id": 1,
  "ref_code": "import torch...",
  "custom_code": "import torch...",
  "cuda_graph_code": "import torch...",
  "score_default": 1.762,
  "score_torch_compile_default": 1.958,
  "score_torch_compile_reduce_overhead": 2.118,
  "score_cuda_graph": 1.566
}
```

--------------------------------

### Benchmark CUDA Kernels Against PyTorch Backends

Source: https://context7.com/deepreinforce-ai/cuda-l1/llms.txt

Compares the performance of a custom CUDA kernel against several PyTorch backends including vanilla PyTorch, torch.compile, torch.compile with reduce-overhead mode, and manual CUDA graph implementations. It utilizes the `eval_kernel_against_ref` function to measure speedup and execution time. Dependencies include PyTorch and potentially CUDA libraries. It takes model source code, seed, number of trials, verbosity, build directory, device, and evaluation setup as input.

```python
score_vanilla, time_vanilla, msg_vanilla = eval_kernel_against_ref(
    warmup_src=ref_code,
    original_model_src=ref_code,
    custom_model_src=custom_code,
    seed_num=42,
    num_perf_trials=10,                       # Number of timing measurements
    verbose=True,                             # Print benchmark results
    build_dir=None,
    device=device,
    info_string="Vanilla-Benchmark",
    original_eval_setup="vanilla"             # Baseline evaluation mode
)

print(f"Speedup vs vanilla PyTorch: {score_vanilla:.2f}x")
print(f"GPU execution time: {time_vanilla:.3f}s")

# Compare against torch.compile
score_compile, time_compile, msg_compile = eval_kernel_against_ref(
    warmup_src=ref_code,
    original_model_src=ref_code,
    custom_model_src=custom_code,
    seed_num=42,
    num_perf_trials=10,
    verbose=True,
    build_dir=None,
    device=device,
    info_string="TorchCompile-Benchmark",
    original_eval_setup="torch_compile"       # Compare against torch.compile
)

print(f"Speedup vs torch.compile: {score_compile:.2f}x")

# Compare against torch.compile with reduce-overhead mode (CUDA graphs)
score_reduce, time_reduce, msg_reduce = eval_kernel_against_ref(
    warmup_src=ref_code,
    original_model_src=ref_code,
    custom_model_src=custom_code,
    seed_num=42,
    num_perf_trials=10,
    verbose=True,
    build_dir=None,
    device=device,
    info_string="ReduceOverhead-Benchmark",
    original_eval_setup="torch_compile_reduce_overhead"
)

print(f"Speedup vs torch.compile (reduce-overhead): {score_reduce:.2f}x")

# Compare against manual CUDA graph implementation
if cuda_graph_code is not None:
    score_graph, time_graph, msg_graph = eval_kernel_against_ref(
        warmup_src=ref_code,
        original_model_src=cuda_graph_code,   # Use CUDA graph baseline
        custom_model_src=custom_code,
        seed_num=42,
        num_perf_trials=10,
        verbose=True,
        build_dir=None,
        device=device,
        info_string="CUDAGraph-Benchmark",
        original_eval_setup="cuda_graph"
    )
    print(f"Speedup vs CUDA graph: {score_graph:.2f}x")
```

--------------------------------

### Load and Compare Custom Optimized Model in Python

Source: https://context7.com/deepreinforce-ai/cuda-l1/llms.txt

This snippet demonstrates loading a custom optimized model using `load_custom_model`, initializing it, and comparing its output against an original model's output. It includes error handling for cases where the model might not load and uses `torch.cuda.synchronize` for accurate timing. The comparison uses `torch.allclose` to check for similarity within a specified tolerance.

```python
context_custom = {}
ModelNew = load_custom_model(
    model_custom_src=custom_code,
    context=context_custom,
    build_directory="/tmp/cuda_build",
    timeout=300.0,
    info_string="CustomModel"
)

if ModelNew is not None:
    with torch.no_grad():
        custom_model = ModelNew(*init_inputs).to(device)
        custom_output = custom_model(*inputs)
        torch.cuda.synchronize(device)

    # Compare outputs
    if torch.allclose(original_output, custom_output, atol=1e-2, rtol=1e-2):
        print("✅ Outputs match within tolerance")
    else:
        max_diff = torch.max(torch.abs(original_output - custom_output)).item()
        print(f"❌ Outputs differ: max_diff={max_diff}")
```

--------------------------------

### Load and Evaluate CUDA Kernels with Python

Source: https://context7.com/deepreinforce-ai/cuda-l1/llms.txt

Loads pre-optimized CUDA kernels from JSON files for a specific GPU architecture and evaluates them against baseline implementations using `eval_pipeline`. This function requires `torch` and `json` libraries, takes the path to the CUDA kernel file as input, and returns performance metrics.

```python
import json
import torch
from eval.eval_cuda import load_cuda_file, eval_pipeline

# Load pre-optimized CUDA kernels for specific GPU architecture
PATH_TO_CUDA_FILE = "/path/to/CUDA-L1/optimized_cuda_code/a100.json"
cuda_dict = load_cuda_file(PATH_TO_CUDA_FILE)

# Access specific task by level and task ID
level_id = 1  # Difficulty level (1, 2, or 3)
task_id = 1   # Task index
task_data = cuda_dict[level_id][task_id]

# Extract code variants
ref_code = task_data["ref_code"]              # Original PyTorch reference
custom_code = task_data["custom_code"]        # CUDA-L1 optimized code
cuda_graph_code = task_data["cuda_graph_code"]  # CUDA graph variant
cudnn_code = task_data["cudnn_code"]          # cuDNN variant

# Evaluate optimized kernel against vanilla PyTorch baseline
result = eval_pipeline(
    warmup_src=ref_code,                      # Warmup code for GPU
    original_model_src=ref_code,              # Reference implementation
    custom_model_src=custom_code,             # Optimized implementation
    num_correct_trials=10,                    # Correctness check iterations
    num_perf_trials=10,                       # Performance measurement iterations
    global_n_trials=7,                        # Number of full evaluation runs
    gpu_index=0,                              # CUDA device index
    verbose=False,                            # Print detailed logs
    log_path="/path/to/output.json",         # Results output path
    max_time=1800,                            # Timeout in seconds
    original_eval_setup="vanilla"             # Baseline: vanilla/torch_compile/torch_compile_reduce_overhead/cuda_graph/cudnn
)
```

--------------------------------

### End-to-End Evaluation Pipeline in Python

Source: https://context7.com/deepreinforce-ai/cuda-l1/llms.txt

This Python script outlines a complete evaluation pipeline for CUDA-L1, from loading optimized CUDA kernels to generating performance reports. It configures GPU architecture, specifies CUDA file paths, and iterates through different levels and tasks to run evaluations using `eval_pipeline`. The script logs results, parses them, and summarizes performance metrics like speedup, standard deviation, and aggregate statistics.

```python
import json
import torch
from eval.eval_cuda import load_cuda_file, eval_pipeline

# Configuration
GPU_ARCHITECTURE = "a100"  # Options: a100, h100, 3090, 3090, h20, l40
CUDA_FILE = f"/path/to/CUDA-L1/optimized_cuda_code/{GPU_ARCHITECTURE}.json"
OUTPUT_DIR = "/path/to/results"

# Load all optimized kernels
cuda_dict = load_cuda_file(CUDA_FILE)

# Iterate through all levels and tasks
results_summary = []

for level_id in [1, 2, 3]:  # Three difficulty levels
    for task_id in cuda_dict[level_id].keys():
        task_data = cuda_dict[level_id][task_id]

        # Skip tasks without optimized code
        if task_data["custom_code"] is None:
            print(f"Level {level_id} Task {task_id}: No optimized code available")
            continue

        # Define log path for this task
        log_path = f"{OUTPUT_DIR}/level{level_id}_task{task_id}.json"

        print(f"\n{'='*60}")
        print(f"Evaluating Level {level_id} Task {task_id}")
        print(f"{('='*60)}")

        # Run comprehensive evaluation pipeline
        result = eval_pipeline(
            warmup_src=task_data["ref_code"],
            original_model_src=task_data["ref_code"],
            custom_model_src=task_data["custom_code"],
            num_correct_trials=10,
            num_perf_trials=10,
            global_n_trials=7,
            gpu_index=0,
            verbose=True,
            log_path=log_path,
            max_time=1800,
            original_eval_setup="vanilla"
        )

        # Parse results from log file
        with open(log_path, 'r') as f:
            lines = f.readlines()
            final_result = json.loads(lines[-1])

            if final_result.get("done") and not final_result.get("error"):
                results_summary.append({
                    "level": level_id,
                    "task": task_id,
                    "score": final_result["score"],
                    "std": final_result["std"],
                    "completed_trials": final_result["completed_trials"]
                })
                print(f"✅ Score: {final_result['score']:.3f}x speedup")
            else:
                print(f"❌ Evaluation failed: {final_result.get('error_msg', 'Unknown error')}")

# Generate summary report
print(f"\n{'='*60}")
print("EVALUATION SUMMARY")
print(f"{('='*60)}")

for result in results_summary:
    print(f"Level {result['level']} Task {result['task']}: "
          f"{result['score']:.3f}x ± {result['std']:.3f}")

# Calculate aggregate statistics
all_scores = [r['score'] for r in results_summary]
print(f"\nAggregate Statistics:")
print(f"  Mean speedup: {sum(all_scores)/len(all_scores):.3f}x")
print(f"  Max speedup: {max(all_scores):.3f}x")
print(f"  Tasks with speedup > 1.01x: {sum(1 for s in all_scores if s > 1.01)}/{len(all_scores)}")
```

--------------------------------

### Load Original and Custom PyTorch Models (Python)

Source: https://context7.com/deepreinforce-ai/cuda-l1/llms.txt

Loads PyTorch models from source code strings, along with their input generation functions, for evaluation and comparison purposes. This utility is essential for setting up benchmarks and ensuring consistent testing environments. Dependencies include PyTorch. Inputs include the model source code, an execution context, timeout value, and an info string. Outputs are the model class, input initialization function, and input generation function.

```python
from eval.eval_cuda import load_original_model_and_inputs, load_custom_model
import torch

# Load reference model with input generators
context_original = {}
Model, get_init_inputs, get_inputs = load_original_model_and_inputs(
    model_original_src=ref_code,
    context=context_original,
    timeout=300.0,
    info_string="OriginalModel"
)

# Initialize reference model
device = torch.device('cuda:0')
init_inputs = get_init_inputs()
init_inputs = [x.cuda(device) if isinstance(x, torch.Tensor) else x for x in init_inputs]

with torch.no_grad():
    original_model = Model(*init_inputs).to(device)

# Generate test inputs
inputs = get_inputs()
inputs = [x.cuda(device) if isinstance(x, torch.Tensor) else x for x in inputs]

# Run inference
original_output = original_model(*inputs)
torch.cuda.synchronize(device)
```

--------------------------------

### Execute Model Code with Timeout Protection (Python)

Source: https://context7.com/deepreinforce-ai/cuda-l1/llms.txt

Safely executes PyTorch model code from a source string with built-in timeout protection and error handling, crucial for testing untrusted Reinforcement Learning-generated code. It supports both threading and process isolation for enhanced security and stability. Dependencies include PyTorch and CUDA libraries. Inputs include model source, execution context, timeout duration, build directory, process isolation flag, and an info string.

```python
from eval.eval_cuda import execute_model_with_timeout

# Prepare execution context
context = {}

# Execute model code with timeout protection
success, error_msg, execution_time = execute_model_with_timeout(
    model_src=custom_code,                    # Source code to execute
    context=context,                          # Execution namespace dictionary
    timeout=100.0,                            # Timeout in seconds
    build_directory="/tmp/cuda_extensions",   # CUDA extension build path
    use_process_isolation=False,              # Use threading (True for multiprocessing)
    info_string="ModelExecution"             # Logging identifier
)

if success:
    print(f"✅ Model executed successfully in {execution_time:.2f}s")
    # Access loaded model class from context
    ModelNew = context.get("ModelNew")
    if ModelNew is not None:
        # Instantiate and use the model
        model = ModelNew().cuda()
        inputs = [torch.randn(2048, 2048).cuda()]
        output = model(*inputs)
        print(f"Model output shape: {output.shape}")
else:
    print(f"❌ Execution failed: {error_msg}")

# Handle potentially blocking code with process isolation
risky_code = """
import torch
import time
time.sleep(10)  # This will timeout
"""

success_risky, error_risky, time_risky = execute_model_with_timeout(
    model_src=risky_code,
    context={},
    timeout=5.0,
    use_process_isolation=True,               # Force process isolation for risky code
    info_string="RiskyCode"
)

if not success_risky:
    print(f"Expected timeout: {error_risky}")
```

--------------------------------

### Check CUDA Kernel Correctness with Python

Source: https://context7.com/deepreinforce-ai/cuda-l1/llms.txt

Verifies that custom CUDA kernels produce numerically identical results to the reference implementation using `check_kernel_correctness`. This function takes the reference and custom kernel source codes, a random seed, and the number of trials as input, returning a boolean indicating success and an error message if validation fails. It requires the `torch` library.

```python
from eval.eval_cuda import check_kernel_correctness
import torch

# Set up GPU device
device = torch.device('cuda:0')

# Run correctness validation with multiple trials
correctness_passed, error_msg, metadata = check_kernel_correctness(
    warmup_src=ref_code,                      # Warmup code
    original_model_src=ref_code,              # Reference model
    custom_model_src=custom_code,             # Custom optimized model
    seed_num=42,                              # Random seed for reproducibility
    num_correct_trials=5,                     # Number of validation trials
    verbose=True,                             # Print trial details
    build_dir=None,                           # CUDA extension build directory
    device=device,                            # GPU device
    timeout=300.0,                            # Code execution timeout
    info_string="Level1-Task1"               # Logging identifier
)

if correctness_passed:
    print(f"✅ Correctness validation passed!")
    print(f"Trials passed: {metadata['trials_passed']}/{metadata['num_trials']}")
    print(f"Hardware: {metadata['hardware']}")
else:
    print(f"❌ Correctness validation failed: {error_msg}")
    print(f"Max difference: {metadata.get('max_difference', 'N/A')}")
    print(f"Avg difference: {metadata.get('avg_difference', 'N/A')}")
```

--------------------------------

### Benchmark CUDA Kernel Performance with Python

Source: https://context7.com/deepreinforce-ai/cuda-l1/llms.txt

Measures the execution time of custom CUDA kernels against baseline implementations using `eval_kernel_against_ref`. This function supports multiple evaluation modes for benchmarking and requires the `torch` library for GPU device configuration.

```python
from eval.eval_cuda import eval_kernel_against_ref
import torch

# Configure GPU device
device = torch.device('cuda:0')


```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.