### Python Beginner Tutorial

Source: https://github.com/openai/mle-bench/blob/main/mlebench/competitions/spooky-author-identification/kernels.txt

This snippet offers a foundational guide to Python programming. It typically includes examples of variable assignment, data types, control flow (if/else, loops), and function definitions.

```python
# Assign a value to a variable
name = "Alice"
 age = 30

# Print a formatted string
print(f"My name is {name} and I am {age} years old.")

# Conditional statement
if age >= 18:
    print("I am an adult.")
else:
    print("I am a minor.")

# For loop
for i in range(5):
    print(f"Iteration {i}")

# Define a function
def greet(person_name):
    return f"Hello, {person_name}!"

print(greet("Bob"))

```

--------------------------------

### Support Vector Machine (SVM) for Beginners

Source: https://github.com/openai/mle-bench/blob/main/mlebench/competitions/statoil-iceberg-classifier-challenge/kernels.txt

An introductory example of using a Support Vector Machine (SVM) for classification tasks, likely with scikit-learn. This snippet would cover the basic setup and usage of an SVM classifier, potentially including data preparation and evaluation metrics like ROC AUC. Dependencies include scikit-learn.

```python
from sklearn.svm import SVC
from sklearn.model_selection import train_test_split
from sklearn.metrics import roc_auc_score
from sklearn.preprocessing import StandardScaler
import numpy as np

# Assume 'X' is your feature data and 'y' is your target labels
# X = ...
# y = ...

# Split data into training and testing sets
# X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Scale features (important for SVM)
# scaler = StandardScaler()
# X_train = scaler.fit_transform(X_train)
# X_test = scaler.transform(X_test)

# Initialize and train the SVM classifier
# svm_classifier = SVC(kernel='rbf', probability=True) # probability=True for roc_auc_score
# svm_classifier.fit(X_train, y_train)

# Predict probabilities on the test set
# y_pred_proba = svm_classifier.predict_proba(X_test)[:, 1] # Probability of the positive class

# Calculate ROC AUC score
# roc_auc = roc_auc_score(y_test, y_pred_proba)
# print(f"ROC AUC Score: {roc_auc}")
```

--------------------------------

### Run-Length Encoding Example

Source: https://github.com/openai/mle-bench/blob/main/mlebench/competitions/vesuvius-challenge-ink-detection/description.md

This example demonstrates how to format predicted pixel data using run-length encoding for submission. The format requires pairs of start position and run length, representing binary pixel values (0 for 'no ink', 1 for 'ink'). The metric validates sorted, positive pairs and non-duplicated decoded pixel values.

```text
Id,Predicted
a,1 1 5 1 etc.
b,10 20 etc.
```

--------------------------------

### Example Submission Format

Source: https://github.com/openai/mle-bench/blob/main/mlebench/competitions/mlsp-2013-birds/description.md

An example of the required submission file format for the MLSP 2013 Bird Classification Challenge. Each line should contain an 'Id' and a 'Probability' prediction.

```text
Id,Probability
0,0
1,0
2,0
3,0
4,0
5,0
6,0
7,0
8,0
9,0
10,0
11,0
12,0
13,0
14,0
15,0
16,0
17,0
18,0
100,0
101,0
102,0
etc...

```

--------------------------------

### CNN and VGG16 Implementation

Source: https://github.com/openai/mle-bench/blob/main/mlebench/competitions/aerial-cactus-identification/kernels.txt

Getting started with Convolutional Neural Networks (CNNs) and the VGG16 architecture. This snippet likely demonstrates the basic setup and usage of VGG16 for image classification tasks.

```python
from tensorflow.keras.applications import VGG16
from tensorflow.keras.layers import Dense, Flatten
from tensorflow.keras.models import Model

# Load VGG16 base model without the top classification layer
base_model = VGG16(weights='imagenet', include_top=False, input_shape=(224, 224, 3))

# Add custom classification layers
x = base_model.output
x = Flatten()(x)
x = Dense(512, activation='relu')(x)
predictions = Dense(num_classes, activation='softmax')(x)

# Create the final model
model = Model(inputs=base_model.input, outputs=predictions)

# Freeze the layers of the base model (optional for transfer learning)
for layer in base_model.layers:
    layer.trainable = False

# Compile the model
model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])
```

--------------------------------

### Prepare MLE-bench datasets via CLI

Source: https://github.com/openai/mle-bench/blob/main/README.md

Commands to download and prepare Kaggle competition datasets for evaluation. Supports full, lite, or specific competition-based preparation.

```console
mlebench prepare --all
mlebench prepare --lite
mlebench prepare -c <competition-id>
```

--------------------------------

### Ink Detection Tutorial Reference

Source: https://github.com/openai/mle-bench/blob/main/mlebench/competitions/vesuvius-challenge-ink-detection/description.md

A link to a Kaggle tutorial demonstrating how to output files in the required format for the Ink Detection competition. This tutorial provides practical guidance on data preparation and submission.

```python
https://www.kaggle.com/code/jpposma/vesuvius-challenge-ink-detection-tutorial
```

--------------------------------

### Bash: Running ML Agents on Competitions with MLE-Bench

Source: https://context7.com/openai/mle-bench/llms.txt

This section provides bash commands for setting up and running ML agents on competitions using MLE-Bench. It includes building the base environment Docker image, building an agent's Docker image with specified directories, and executing the agent on a competition set for testing. Key commands involve `docker build` and `python run_agent.py`.

```bash
# Build the base environment image first
docker build --platform=linux/amd64 -t mlebench-env -f environment/Dockerfile .

# Build an agent image (e.g., AIDE)
export SUBMISSION_DIR=/home/submission
export LOGS_DIR=/home/logs
export CODE_DIR=/home/code
export AGENT_DIR=/home/agent

docker build --platform=linux/amd64 -t aide agents/aide/ \
    --build-arg SUBMISSION_DIR=$SUBMISSION_DIR \
    --build-arg LOGS_DIR=$LOGS_DIR \
    --build-arg CODE_DIR=$CODE_DIR \
    --build-arg AGENT_DIR=$AGENT_DIR

# Run agent on a single competition (for testing)
python run_agent.py \
    --agent-id aide \
    --competition-set experiments/splits/spaceship-titanic.txt

```

--------------------------------

### PyTorch Starter for CNNs

Source: https://github.com/openai/mle-bench/blob/main/mlebench/competitions/statoil-iceberg-classifier-challenge/kernels.txt

A basic PyTorch starter template for building and training Convolutional Neural Networks (CNNs). This snippet likely includes setting up the model, defining the loss function, and an optimizer. It serves as a starting point for PyTorch-based deep learning projects. Dependencies include PyTorch.

```python
import torch
import torch.nn as nn
import torch.optim as optim

# Define a simple CNN model
class SimpleCNN(nn.Module):
    def __init__(self, num_classes=10):
        super(SimpleCNN, self).__init__()
        self.conv1 = nn.Conv2d(3, 32, kernel_size=3, padding=1)
        self.relu = nn.ReLU()
        self.pool = nn.MaxPool2d(kernel_size=2, stride=2)
        self.conv2 = nn.Conv2d(32, 64, kernel_size=3, padding=1)
        self.fc1 = nn.Linear(64 * 8 * 8, 128) # Assuming input image size leads to this flattened size
        self.fc2 = nn.Linear(128, num_classes)

    def forward(self, x):
        x = self.pool(self.relu(self.conv1(x)))
        x = self.pool(self.relu(self.conv2(x)))
        x = x.view(-1, 64 * 8 * 8) # Flatten
        x = self.relu(self.fc1(x))
        x = self.fc2(x)
        return x

# Initialize model, loss function, and optimizer
# model = SimpleCNN(num_classes=10)
# criterion = nn.CrossEntropyLoss()
# optimizer = optim.Adam(model.parameters(), lr=0.001)

# Example training loop structure (simplified):
# for epoch in range(num_epochs):
#     for inputs, labels in dataloader:
#         optimizer.zero_grad()
#         outputs = model(inputs)
#         loss = criterion(outputs, labels)
#         loss.backward()
#         optimizer.step()
```

--------------------------------

### Install NodeJS Requirements for Dolos

Source: https://github.com/openai/mle-bench/blob/main/extras/README.md

Installs the necessary Node.js dependencies for the Dolos plagiarism detector. This command should be run within the plagiarism detector directory.

```console
cd extras/plagiarism_detector
npm install
```

--------------------------------

### Download and Prepare Benchmark Tool

Source: https://github.com/openai/mle-bench/blob/main/mlebench/competitions/tensorflow-speech-recognition-challenge/description.md

This sequence of commands downloads the benchmark_model executable, makes it executable, and then runs it with specific parameters to evaluate a frozen TensorFlow graph. It's used to test model performance against the special prize requirements.

```bash
curl -O https://storage.googleapis.com/download.tensorflow.org/deps/pi/2017_10_07/benchmark_model
chmod +x benchmark_model
./benchmark_model --graph=conv_actions_frozen.pb --input_layer="decoded_sample_data:0,decoded_sample_data:1" --input_layer_shape="16000,1:" --input_layer_type="float,int32" --input_layer_values=":16000" --output_layer="labels_softmax:0" --show_run_order=false --show_time=false --show_memory=false --show_summary=true --show_flops=true
```

--------------------------------

### Install Dependencies for Familiarity Experiment

Source: https://github.com/openai/mle-bench/blob/main/experiments/familiarity/README.md

Installs the necessary Python packages and Playwright for web scraping. 'requirements.txt' should contain specific dependencies for older OpenAI API versions and additional packages not in the main library.

```bash
pip install -r requirements.txt
playwright install
```

--------------------------------

### R Beginner Tutorial

Source: https://github.com/openai/mle-bench/blob/main/mlebench/competitions/spooky-author-identification/kernels.txt

This snippet provides a basic introduction to using the R programming language. It covers fundamental concepts like data types, basic operations, and potentially data manipulation with packages like dplyr.

```r
# Assign a value to a variable
x <- 10
y <- 20

# Perform a calculation
result <- x + y

# Print the result
print(result)

# Create a vector
my_vector <- c(1, 2, 3, 4, 5)
print(my_vector)

# Create a data frame (example)
data <- data.frame(
  ID = 1:3,
  Name = c("Alice", "Bob", "Charlie"),
  Value = c(100, 150, 120)
)
print(data)

# Basic summary statistics
summary(data)

```

--------------------------------

### Simple CNN on PyTorch for Beginners

Source: https://github.com/openai/mle-bench/blob/main/mlebench/competitions/aerial-cactus-identification/kernels.txt

An introductory example of building a Convolutional Neural Network (CNN) using PyTorch. This snippet is designed for beginners and covers the fundamental steps of creating a CNN model.

```python
import torch
import torch.nn as nn

class SimpleCNN(nn.Module):
    def __init__(self, num_classes):
        super(SimpleCNN, self).__init__()
        self.conv1 = nn.Conv2d(3, 16, kernel_size=3, padding=1)
        self.relu1 = nn.ReLU()
        self.pool1 = nn.MaxPool2d(kernel_size=2, stride=2)
        self.conv2 = nn.Conv2d(16, 32, kernel_size=3, padding=1)
        self.relu2 = nn.ReLU()
        self.pool2 = nn.MaxPool2d(kernel_size=2, stride=2)
        self.flatten = nn.Flatten()
        self.fc1 = nn.Linear(32 * 8 * 8, 128) # Assuming input image size leads to this flattened size
        self.relu3 = nn.ReLU()
        self.fc2 = nn.Linear(128, num_classes)

    def forward(self, x):
        x = self.pool1(self.relu1(self.conv1(x)))
        x = self.pool2(self.relu2(self.conv2(x)))
        x = self.flatten(x)
        x = self.relu3(self.fc1(x))
        x = self.fc2(x)
        return x

model = SimpleCNN(num_classes=num_classes)
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=0.001)
```

--------------------------------

### Interact with Grading Server API

Source: https://context7.com/openai/mle-bench/llms.txt

Demonstrates how to interact with the grading server running inside a Docker container. It includes a bash example for health checks and submission validation using curl, and a Python example for programmatic validation within agent code.

```bash
# Health check endpoint
curl http://localhost:5000/health
# Response: {"status": "running"}

# Validate a submission file
curl -X POST http://localhost:5000/validate \
    -F "file=@/home/submission/submission.csv"
# Response: {"result": "Submission is valid."}
# Or on error: {"result": "Submission invalid! ...error details..."}
```

```python
# Python example for agents to use inside the container
import requests

def validate_my_submission(submission_path: str) -> tuple[bool, str]:
    """Validate submission format using the grading server."""
    with open(submission_path, 'rb') as f:
        response = requests.post(
            'http://localhost:5000/validate',
            files={'file': f}
        )

    if response.status_code == 200:
        result = response.json()['result']
        is_valid = "valid" in result.lower() and "invalid" not in result.lower()
        return is_valid, result
    else:
        return False, f"Server error: {response.json().get('error', 'Unknown')}"

# Usage
is_valid, message = validate_my_submission("/home/submission/submission.csv")
if is_valid:
    print("Submission is ready!")
else:
    print(f"Fix submission: {message}")
```

--------------------------------

### Prepare Datasets via CLI

Source: https://context7.com/openai/mle-bench/llms.txt

The 'mlebench prepare' command automates the downloading, splitting, and verification of Kaggle competition data. It supports full, lite, and custom-list preparation modes, with options for directory configuration and file retention.

```bash
# Prepare a single competition dataset
mlebench prepare -c spaceship-titanic

# Prepare all 75 competitions (takes ~2 days)
mlebench prepare --all

# Prepare the "lite" version with 22 low-complexity competitions (faster)
mlebench prepare --lite

# Prepare competitions from a custom list file
mlebench prepare -l experiments/splits/low.txt

# Keep raw downloaded files after preparation
mlebench prepare -c dog-breed-identification --keep-raw

# Specify custom data directory
mlebench prepare -c aerial-cactus-identification --data-dir /custom/data/path
```

--------------------------------

### GET /device/imu

Source: https://github.com/openai/mle-bench/blob/main/mlebench/competitions/smartphone-decimeter-2022/description.md

Retrieves IMU sensor readings including accelerometer, gyroscope, and magnetometer data.

```APIDOC
## GET /device/imu

### Description
Returns raw and bias-compensated readings from the device's IMU instruments.

### Method
GET

### Endpoint
/[train/test]/[drive_id]/[phone_name]/device_imu.csv

### Parameters
#### Path Parameters
- **drive_id** (string) - Required - The unique identifier for the drive session.
- **phone_name** (string) - Required - The device identifier.

### Response
#### Success Response (200)
- **MessageType** (string) - Instrument type (accelerometer, gyroscope, or magnetometer).
- **utcTimeMillis** (long) - Timestamp in UTC.
- **MeasurementX/Y/Z** (float) - Uncalibrated sensor readings.
- **BiasX/Y/ZMicroT** (float) - Estimated bias compensation.

### Response Example
{
  "MessageType": "accelerometer",
  "utcTimeMillis": 1625097600000,
  "MeasurementX": 0.05,
  "BiasXMicroT": 0.001
}
```