### Setup Inference Engine Environment

Source: https://github.com/facebookresearch/bigobench/blob/main/README.md

Install dependencies for the inference engine. Consult src/inference/README.md for more information.

```bash
cd src/inference
bash create_vllm_env.sh
```

--------------------------------

### Setup Evaluation Harness Environment

Source: https://github.com/facebookresearch/bigobench/blob/main/README.md

Install dependencies for the evaluation harness. Navigate to src/eval/README.md for usage details.

```bash
cd src/eval
bash create_eval_env.sh
```

--------------------------------

### Setup Complexity Framework Environment

Source: https://github.com/facebookresearch/bigobench/blob/main/README.md

Install dependencies specifically for the complexity framework. Refer to src/complexity/README.md for detailed usage.

```bash
cd src/complexity
bash create_complexity_env.sh
```

--------------------------------

### Install All Dependencies and Run Benchmark

Source: https://github.com/facebookresearch/bigobench/blob/main/README.md

Execute these commands to install all dependencies at once and prepare to run the BigOBench benchmark. Further instructions are in the src/README.md.

```bash
cd src
bash create_bigobench_env.sh
```

--------------------------------

### Install Complexity Framework Dependencies

Source: https://github.com/facebookresearch/bigobench/blob/main/src/complexity/README.md

Navigate to the complexity directory and execute the setup script. Activate the conda environment afterward.

```bash
cd src/complexity
bash create_complexity_env.sh
conda activate complexity
```

--------------------------------

### Install and Login to Hugging Face Hub

Source: https://github.com/facebookresearch/bigobench/blob/main/src/inference/README.md

Installs the huggingface_hub library and logs you into your Hugging Face account. This is necessary for downloading gated models.

```bash
pip install -U "huggingface_hub[cli]"
huggingface-cli login
huggingface-cli download meta-llama/Llama-3.1-70B-Instruct
```

--------------------------------

### Create and Activate VLLM Environment

Source: https://github.com/facebookresearch/bigobench/blob/main/src/inference/README.md

Use this script to install the necessary dependencies for VLLM inference and activate the created conda environment.

```bash
bash create_vllm_env.sh
conda activate vllm
```

--------------------------------

### Create BigO(Bench) Environment

Source: https://github.com/facebookresearch/bigobench/blob/main/src/README.md

Installs all project dependencies using a script and activates the conda environment. Ensure you are in the 'src/' directory before running.

```bash
cd src/
bash create_bigobench_env.sh
conda activate bigobench
```

--------------------------------

### Example Input String for Array Problem

Source: https://github.com/facebookresearch/bigobench/blob/main/src/complexity/README.md

This is an example of a concatenated input string for the 'A. Array' problem, demonstrating how multiple lines of input are provided as a single string in Input Format 1.

```text
'4\n-1 -2 -3 0\n'
```

--------------------------------

### Launch vLLM Server

Source: https://github.com/facebookresearch/bigobench/blob/main/src/inference/README.md

Starts a vLLM inference server with specified model, tensor parallelism, and maximum model length. Ensure model weights are downloaded and Hugging Face terms are accepted prior to execution.

```bash
vllm serve meta-llama/Llama-3.1-70B-Instruct --enforce-eager --max-model-len=32000 --tensor-parallel-size=8  --trust-remote-code
```

--------------------------------

### Create and Activate Evaluation Environment

Source: https://github.com/facebookresearch/bigobench/blob/main/src/eval/README.md

Use this script to install evaluation-specific dependencies and activate the conda environment. Ensure you are in the 'src/eval' directory.

```bash
cd src/eval
bash create_eval_env.sh
conda activate eval
```

--------------------------------

### Test Local VLLM Inference

Source: https://github.com/facebookresearch/bigobench/blob/main/src/inference/README.md

Use this script to test a local deployment of vLLM. Ensure the model path is correct for your setup.

```bash
python test_vllm.py --host 0.0.0.0 --model meta-llama/Llama-3.1-70B-Instruct
```

--------------------------------

### Configure VLLM Server Timeout

Source: https://github.com/facebookresearch/bigobench/blob/main/src/eval/README.md

Example of configuring timeout settings for an OpenAI client when using a VLLM engine. Adjust 'timeout' and 'connect' values as needed.

```python
OpenAI(
    base_url=url, 
    api_key="EMPTY",
    max_retries = 1,
    timeout = httpx.Timeout(timeout=5000.0, connect=5.0),
)
```

--------------------------------

### Run Evaluation with VLLM on SLURM

Source: https://github.com/facebookresearch/bigobench/blob/main/src/eval/README.md

Launch evaluation inference with a VLLM instance on a SLURM cluster. Ensure the `OPENAI_API_KEY` is set and specify the `--host` pointing to your SLURM node. This command is similar to the Llama-3.1 example but assumes a SLURM environment.

```bash
export OPENAI_API_KEY="your_openai_key_here"

python -u eval.py \
    --host "node-1" \
    --model "meta-llama/Llama-3.1-70B-Instruct" \
    --max_concurrent_requests 256 \
    --task.data_file_path "../../data/time_complexity_test_set.jsonl" \
    --task.tasks_str "complexity_prediction/time_at_10,complexity_generation/time_at_10" \
    --task.write_eval "True" \
    --task.batch_size 256 \
    --task.use_sampling "True" \
    --task.temperature 0.8 \
    --task.top_p 0.95 \
    --dump_dir "./results"
```

--------------------------------

### Standalone Input Handler Example

Source: https://github.com/facebookresearch/bigobench/blob/main/src/complexity/README.md

This Python code defines a class to process a list of integers, categorizing them into negative, positive, and zero. It demonstrates conditional logic for manipulating these lists before printing their lengths and contents. This snippet is used with the 'standalone' input handler.

```python
class Array_300_A:
    def solve(self, n, l):
        #A. Array
        a,b,c =[],[],[]
        for i in l:
            if i<0:
                a.append(i)
            elif i>0:
                b.append(i)
            else:
                c.append(i)

        if len(b)==0 and len(a)>2:
            b.append(a.pop())
            b.append(a.pop()) 
        if len(a)%2==0:
            c.append(a.pop())    
        print(len(a),*a)
        print(len(b),*b)
        print(len(c),*c)
```

--------------------------------

### Run Complexity Framework for Quick Testing with Logging

Source: https://github.com/facebookresearch/bigobench/blob/main/src/complexity/README.md

Execute the framework for quick testing purposes, logging all outputs. This is useful for basic verification and debugging.

```bash
python -u -m command_line \
    --path_to_jsonl_file="source_of_data.jsonl" \
    --code_end_index = 1 \
    --log_outputs
```

--------------------------------

### Python Array Manipulation Example

Source: https://github.com/facebookresearch/bigobench/blob/main/src/complexity/README.md

This Python code snippet demonstrates array manipulation for a problem involving sorting numbers into positive, negative, and zero lists. It's an example of code that uses Input Format 1.

```python
#A. Array
n = int(input())
a,b,c =[],[],[]
l = list(map(int,input().split()))
for i in l:
    if i<0:
        a.append(i)
    elif i>0:
        b.append(i)
    else:
        c.append(i)

if len(b)==0 and len(a)>2:
    b.append(a.pop())
    b.append(a.pop()) 
if len(a)%2==0:
    c.append(a.pop())    
print(len(a),*a)
print(len(b),*b)
print(len(c),*c)
```

--------------------------------

### Run Complexity Framework with Default Parameters

Source: https://github.com/facebookresearch/bigobench/blob/main/src/complexity/README.md

Execute the framework with default settings. Ensure to specify the path to your data file using `--path_to_jsonl_file`.

```bash
python -u -m command_line \
    --path_to_jsonl_file="source_of_data.jsonl"
```

--------------------------------

### Run Complexity Framework for Quick Testing with Output Logging

Source: https://github.com/facebookresearch/bigobench/blob/main/src/complexity/README.md

For basic testing, set a small `code_end_index` and enable `log_outputs` to True. This allows for rapid iteration and verification of framework behavior.

```python
from main import run_complexity_framework

run_complexity_framework(
    "source_of_data.jsonl",
    code_end_index = 1,
    log_outputs = True,
)
```

--------------------------------

### Running Complexity Framework with JSONL Input

Source: https://github.com/facebookresearch/bigobench/blob/main/src/complexity/README.md

This bash command demonstrates how to run the complexity framework using a JSONL file as input, specifically for Input Format 1 where `input_handler` is set to `with_dataclass`.

```bash
python -u -m command_line    --path_to_jsonl_file="../../data/example_input_format_1.jsonl"    --log_outputs
```

--------------------------------

### Get List of Head Nodes for Running Jobs

Source: https://github.com/facebookresearch/bigobench/blob/main/src/inference/README.md

Retrieves a comma-separated list of head nodes for active vLLM server instances, particularly useful when using SLURM arrays.

```bash
bash get_list_of_head_nodes.sh start_vllm_server_llama_70B.sh
```

--------------------------------

### Run Complexity Framework CLI

Source: https://github.com/facebookresearch/bigobench/blob/main/src/README.md

Use this command to post-process output files with the complexity framework via the command line. Ensure you are in the 'complexity' directory and specify the correct path to your JSONL file.

```bash
cd complexity
python -u -m command_line \
    --path_to_jsonl_file="../eval/results/eval_results/complexity_generation/time_at_10-time_complexity_test_set.jsonl"
```

--------------------------------

### Standalone Input Format Example

Source: https://github.com/facebookresearch/bigobench/blob/main/src/complexity/README.md

This JSON object represents the input for the 'standalone' input handler. It maps argument names ('n', 'l') to their string representations, which are then parsed by the associated Python code.

```json
{"n": "4", "l": "[-1, -2, -3, 0]"}
```

--------------------------------

### Command Line Execution with Standalone Input Handler

Source: https://github.com/facebookresearch/bigobench/blob/main/src/complexity/README.md

This bash command demonstrates how to run the Big-O Bench framework using the 'standalone' input handler. It specifies the path to the JSONL file containing the data samples and enables output logging.

```bash
python -u -m command_line \
    --path_to_jsonl_file="../../data/example_input_format_2.jsonl" \
    --input_handler="standalone" \
    --log_outputs
```

--------------------------------

### Run Complexity Framework with Default Settings

Source: https://github.com/facebookresearch/bigobench/blob/main/src/complexity/README.md

Call the framework directly from a Python script or Jupyter notebook. Ensure the path to your jsonl data file is correctly specified.

```python
from main import run_complexity_framework

run_complexity_framework(
    "source_of_data.jsonl"
)
```

--------------------------------

### ECharts Chart Options Configuration

Source: https://github.com/facebookresearch/bigobench/blob/main/docs/leaderboard.html

Defines the configuration options for an ECharts chart, including legend, grid, axes, tooltips, and series data. This setup is tailored for displaying model performance metrics.

```javascript
var chartOption = {
  legend: { data: ["all@1"], },
  grid: {
    left: "1%",
    right: "4%",
    bottom: "3%",
    containLabel: true,
  },
  xAxis: {
    name: "Act. Size",
    type: "category",
    boundaryGap: false,
    data: [],
    axisLabel: {
      formatter: function (value) {
        return value + "B";
      },
    },
  },
  yAxis: {
    name: "ALL@1 (greedy decoding)",
    type: "value",
    show: true,
    nameTextStyle: {
      align: "left",
    },
    splitLine: {
      show: true,
      lineStyle: {
        type: "dashed",
      },
    },
  },
  legend: {
    data: ["models"],
    itemStyle: {
      opacity: 1.0,
    },
  },
  tooltip: {
    trigger: "item",
    axisPointer: {
      type: "cross",
    },
  },
  series: [
    {
      name: "models",
      type: "scatter",
      data: [],
      itemStyle: {
        color: "#5470c6",
        opacity: 0.2,
      },
      emphasis: {
        focus: "series",
      },
      lineStyle: {
        width: 2,
      },
      markLine: {
        symbol: "none",
        emphasis: {
          label: {
            position: "middle",
            formatter: function (params) {
              return params.data.name;
            },
          },
        },
        data: [],
      },
    },
  ],
};
```

--------------------------------

### Launch vLLM Server with SLURM

Source: https://github.com/facebookresearch/bigobench/blob/main/src/inference/README.md

Launches a vLLM server using a SLURM job. Ensure the SLURM headers in the script are configured for your environment.

```bash
sbatch start_vllm_server_llama_70B.sh
```

--------------------------------

### Configure vLLM Server for Different Models

Source: https://github.com/facebookresearch/bigobench/blob/main/src/inference/README.md

Modify these parameters at the beginning of `start_vllm_server_llama_70B.sh` to use different models, adjust context length, or set GPU utilization.

```bash
# tune parameters below
MODEL=meta-llama/Llama-3.1-405B-Instruct
MAX_MODEL_LEN=32000
GPU_UTILIZATION=0.9
```

```bash
# tune parameters below
MODEL=deepseek-ai/DeepSeek-R1-Distill-Llama-70B
MAX_MODEL_LEN=32000
GPU_UTILIZATION=0.9
```

--------------------------------

### Run Complexity Framework with SLURM (Generation)

Source: https://github.com/facebookresearch/bigobench/blob/main/src/eval/README.md

Execute the complexity framework using a SLURM script for complexity generation tasks. Ensure the slurm.sh script is configured with appropriate parameters.

```bash
#SBATCH --nodes=1
#SBATCH --tasks-per-node 1
#SBATCH --cpus-per-task ***TODO***
#SBATCH --gpus-per-node=0
#SBATCH --output=logs/complexity-%x.%A.%a.%j.out
#***TODO*** set any addition parameters for SBATCH (account, partition, ...)
#SBATCH --time=1:00:00
#SBATCH --exclusive
#SBATCH --mem 0
#SBATCH --array=1-1%1

python -u -m command_line    --path_to_jsonl_file="../eval/results/eval_results/complexity_generation/time_at_10-time_complexity_test_set.jsonl"    --slurm_array_task_id=$SLURM_ARRAY_TASK_ID
    --slurm_array_task_max=$SLURM_ARRAY_TASK_MAX
    --slurm_array_task_min=$SLURM_ARRAY_TASK_MIN
    --slurm_array_job_id=$SLURM_ARRAY_JOB_ID
```

--------------------------------

### Run Complexity Framework with SLURM (Ranking)

Source: https://github.com/facebookresearch/bigobench/blob/main/src/eval/README.md

Execute the complexity framework using a SLURM script for complexity ranking tasks. The configuration is similar to generation, with a different input file path.

```bash
#SBATCH --nodes=1
#SBATCH --tasks-per-node 1
#SBATCH --cpus-per-task ***TODO***
#SBATCH --gpus-per-node=0
#SBATCH --output=logs/complexity-%x.%A.%a.%j.out
#***TODO*** set any addition parameters for SBATCH (account, partition, ...)
#SBATCH --time=1:00:00
#SBATCH --exclusive
#SBATCH --mem 0
#SBATCH --array=1-1%1

python -u -m command_line    --path_to_jsonl_file="../eval/results/eval_results/complexity_ranking/time_at_10-time_complexity_test_set.jsonl"    --slurm_array_task_id=$SLURM_ARRAY_TASK_ID
    --slurm_array_task_max=$SLURM_ARRAY_TASK_MAX
    --slurm_array_task_min=$SLURM_ARRAY_TASK_MIN
    --slurm_array_job_id=$SLURM_ARRAY_JOB_ID
```

--------------------------------

### SLURM Job Configuration Script

Source: https://github.com/facebookresearch/bigobench/blob/main/src/complexity/README.md

A sample slurm.sh script for configuring and running the complexity framework. Modify parameters like --cpus-per-task and SBATCH directives for specific resource allocation. The --array parameter can be adjusted to split work across multiple jobs.

```bash
#!/bin/bash

#SBATCH --nodes=1
#SBATCH --tasks-per-node 1
#SBATCH --cpus-per-task ***TODO***
#SBATCH --gpus-per-node=0
#SBATCH --output=logs/complexity-%x.%A.%a.%j.out
#***TODO*** set any addition parameters for SBATCH (account, partition, ...)
#SBATCH --time=1:00:00
#SBATCH --exclusive
#SBATCH --mem 0
#SBATCH --array=1-1%1

python -u -m command_line \
    --path_to_jsonl_file="source_of_data.jsonl" \
    --slurm_array_task_id=$SLURM_ARRAY_TASK_ID \
    --slurm_array_task_max=$SLURM_ARRAY_TASK_MAX \
    --slurm_array_task_min=$SLURM_ARRAY_TASK_MIN \
    --slurm_array_job_id=$SLURM_ARRAY_JOB_ID
```

--------------------------------

### Navigate to Complexity Framework Directory

Source: https://github.com/facebookresearch/bigobench/blob/main/src/eval/README.md

Change the current directory to the complexity framework's root directory before running SLURM scripts.

```bash
cd ../complexity
```

--------------------------------

### Initialize Global Variables for Complexity Demo

Source: https://github.com/facebookresearch/bigobench/blob/main/docs/demo.html

This JavaScript code initializes global variables used in a web-based demonstration for analyzing algorithm complexity. It sets up variables for problem details, complexity metrics, model performance, and UI elements.

```javascript
$(document).ready(function () {
    // FIRST, let's define the global variables!
    let searchParams = new URLSearchParams(window.location.search);
    let problemName = "";
    let currentProblemNumber = 1;
    let totalProblemCount = 5;
    let problemDescription = "";
    let complexity = "";
    let currentComplexityNumber = 1;
    let totalComplexityCount = 2;
    let humanSolution = "";
    let modelScore = "";
    let modelCorrectSolution = "";
    let modelCorrectSolutionCurve = "";
    let complexityNote = "";
    let correctNote = "";
    let incorrectNote = "";
    let explanationQuery = "";
    let modelIncorrectSolution = "";
    let incorrectSolutionDetails = "";
    let incorrectSolutionDetails2 = "";

    // Special variables from dropdown menus
    let currentModel = "Llama 3.1 8B";
    let complexityType = "time";
    let complexityTask = "prediction";
    let complexityOptions = "time-prediction";

    // SECOND, let's load the data!
    let tinyTimeTestSet;
    let tinySpaceTestSet;
    let modelResults;

    // Set up titles
    let problemDescriptionTitle = "Problem description";
    let humanSolutionTitle = "Reference human solution";
    let modelTitle = "Model";
    let modelScoreTitle = "Model Scores";
    let modelCorrectSolutionTitle = "Correct Model Solution";
    let modelIncorrectSolutionTitle = "Incorrect Model Solution";

    // THIRD, let's link the variables to the HTML elements!
    $("#problemName").text(problemName);
    $("#currentProblemNumber").text(currentProblemNumber);
    $("#totalProblemCount").text(totalProblemCount);
    $("#problemDescription").text(problemDescription);
    $("#complexity").text(complexity);
    $("#currentComplexityNumber").text(currentComplexityNumber);
    $("#totalComplexityCount").text(totalComplexityCount);
    $("#humanSolution").text(humanSolution);
    $("#modelScore").text(modelScore);
    $("#modelCorrectSolution").text(modelCorrectSolution);
    $("#modelCorrectSolutionCurve").text(modelCorrectSolutionCurve);
    $("#explanationQuery").text("");
    $("#complexityNote").text("");
    $("#correctNote").text("");
    $("#incorrectNote").text("");
    $("#modelIncorrectSolution").text(modelIncorrectSolution);
    $("#incorrectSolutionDetails").text(incorrectSolutionDetails);
    $("#incorrectSolutionDetails2").text(incorrectSolutionDetails2);
    $("#problemDescriptionTitle").text(problemDescriptionTitle);
    $("#humanSolutionTitle").text(humanSolutionTitle);
    $("#modelTitle").text(modelTitle);
    $("#modelScoreTitle").text(modelScoreTitle);
    $("#modelCorrectSolutionTitle").text(modelCorrectSolutionTitle);
    $("#modelIncorrectSolutionTitle").text(modelIncorrectSolutionTitle);

    // Now, specific for the histogram
    let humanCoeffList = [];
    let humanCoeffFiltered = [];
    let modelCoeff = null;
    let showHistogram = false;
    let frequencyMap = {};
    let ctx = document.createElement('canvas').getContext('2d');

    // let chart = new Chart(ctx, {
    //     type: 'bar',
    //     data: {
    //         labels: Object.keys(frequencyMap),
    //         datasets: [{
    //             label: 'Frequency',
    //             data: Object.values(frequencyMap),
    //             backgroundColor: 'rgba(255, 99, 132, 0.2)',
    //             borderColor: 'rgba(255, 99, 132, 1)',
    //             borderWidth: 1
    //         }]
    //     },
    //     options: {
    //         scales: {
    //             x: {
    //                 title: {
    //                     display: true,
    //                     text: 'Values'
    //                 }
    //             },
    //             y: {
    //                 beginAtZero: true,
    //                 title: {
    //                     display: true,
    //                     text: 'Frequency'
    //                 }
    //             }
    //         },
    //         animation: {
    //             duration: 1000
    //         }
    //     }
    // });
});
```

--------------------------------

### Download BigOBench Dataset using Hugging Face CLI

Source: https://github.com/facebookresearch/bigobench/blob/main/data/README.md

This snippet shows how to download the BigOBench dataset using the Hugging Face CLI. It navigates to the data directory, downloads the dataset into a temporary directory, moves the data files to the current directory, and then removes the temporary directory.

```bash
cd /data
huggingface-cli download facebook/BigOBench --repo-type dataset --local-dir ./temp_dir
mv ./temp_dir/data/* .
rm -r ./temp_dir
```

--------------------------------

### Initialize Problem and Complexity Navigation

Source: https://github.com/facebookresearch/bigobench/blob/main/docs/demo.html

Initializes the current problem number and sets up event listeners for navigation buttons and keyboard input. It also handles updates for complexity selection.

```javascript
$(document).ready(function () {
    currentProblemNumber = Math.floor(Math.random() * totalProblemCount) + 1;
    $("#currentProblemNumber").text(currentProblemNumber);
    updateData();
    updateButtonStatus();
});
$("#rightButtonProblem").click(function () {
    if (currentProblemNumber < totalProblemCount) {
        currentProblemNumber++;
        $("#currentProblemNumber").text(currentProblemNumber);
        updateData();
    }
    updateButtonStatus();
});
$("#leftButtonComplexity").click(function () {
    if (currentComplexityNumber > 1) {
        currentComplexityNumber--;
        $("#currentComplexityNumber").text(currentComplexityNumber);
        updateData();
    }
    updateButtonStatus();
});
$("#randButtonComplexity").click(function () {
    currentComplexityNumber = Math.floor(Math.random() * totalComplexityCount) + 1;
    $("#currentComplexityNumber").text(currentComplexityNumber);
    updateData();
    updateButtonStatus();
});
$("#rightButtonComplexity").click(function () {
    if (currentComplexityNumber < totalComplexityCount) {
        currentComplexityNumber++;
        $("#currentComplexityNumber").text(currentComplexityNumber);
        updateData();
    }
    updateButtonStatus();
});
$(document).keydown(function (e) {
    switch (e.which) {
        case 37: // left arrow key
            if (currentComplexityNumber > 1) {
                currentComplexityNumber--;
                $("#currentComplexityNumber").text(currentComplexityNumber);
                updateData();
            } else {
                if (currentProblemNumber > 1) {
                    currentComplexityNumber = 1000;
                    $("#currentComplexityNumber").text(currentComplexityNumber);
                    currentProblemNumber--;
                    $("#currentProblemNumber").text(currentProblemNumber);
                    updateData();
                }
            }
            updateButtonStatus();
            break;
        case 39: // right arrow key
            if (currentComplexityNumber < totalComplexityCount) {
                currentComplexityNumber++;
                $("#currentComplexityNumber").text(currentComplexityNumber);
                updateData();
            } else {
                if (currentProblemNumber < totalProblemCount) {
                    currentComplexityNumber = 1;
                    $("#currentComplexityNumber").text(currentComplexityNumber);
                    currentProblemNumber++;
                    $("#currentProblemNumber").text(currentProblemNumber);
                    updateData();
                }
            }
            updateButtonStatus();
            break;
        default:
            return; // exit this handler for other keys
    }
    e.preventDefault(); // prevent the default action (scroll / move caret)
});
$("#currentModel").change(function () {
    currentModel = $('#currentModel').val();
    updateData();
});
// $("#complexityOptions").change(function () {
// complexityOptions = $('#complexityOptions').val();
// updateData();
// });
$("input[name='complexityOptions']").change(function () {
    complexityOptions = $("input[name='complexityOptions']:checked").attr('id');
    let parts = complexityOptions.split("-");
    complexityType = parts[0];
    complexityTask = parts[1];
    updateData();
    updateButtonStatus();
});
// $("#complexityType").change(function () {
// complexityType = $('#complexityType').val();
// updateData();
// });
// $("#complexityTask").change(function () {
// complexityTask = $('#complexityTask').val();
// updateData();
// });
});
```

--------------------------------

### Calculate Complexity Ranking Metrics

Source: https://github.com/facebookresearch/bigobench/blob/main/src/README.md

This script calculates metrics for the complexity ranking task. It requires the output from the complexity framework and the test set file path. The `--results_folder_or_file_path` can be a .zip file or a folder containing .zip files.

```bash
cd eval
python -u metrics/postprocessing_complexity_ranking.py \
    --results_folder_or_file_path "../complexity/results/results_datetime_xxx_id_yyy.zip" \
    --dump_dir "./results" \
    --test_set_file_path "../../data/time_complexity_test_set.jsonl" \
    --time_or_space "time" \
    --generate_baseline False \
    --at_k 10 
```

--------------------------------

### Run Evaluation Inference Script

Source: https://github.com/facebookresearch/bigobench/blob/main/src/README.md

Execute the evaluation script with specified parameters for LLM benchmarking. Adjust task parameters and output directories as needed. Remove the `--host` argument if using the OpenAI API and ensure `OPENAI_API_KEY` is set.

```bash
cd eval
python -u eval.py \
    --host "host_address" \
    --model "model_name" \
    --max_concurrent_requests 256 \
    --task.data_file_path "../../data/time_complexity_test_set.jsonl" \
    --task.tasks_str "complexity_prediction/time_at_10,complexity_generation/time_at_10,complexity_ranking/time_at_10" \
    --task.write_eval "True" \
    --task.batch_size 256 \
    --task.use_sampling "True" \
    --task.temperature 0.8 \
    --task.top_p 0.95 \
    --dump_dir "./results"
```

--------------------------------

### Calculate Complexity Generation Metrics

Source: https://github.com/facebookresearch/bigobench/blob/main/src/README.md

Use this script to compute metrics for the complexity generation task, following the post-processing of the complexity framework. The results folder or file path should point to the output of the framework.

```bash
cd eval
python -u metrics/postprocessing_complexity_generation.py \
    --results_folder_or_file_path "../complexity/results/results_datetime_xxx_id_yyy.zip" \
    --dump_dir "./results" \
    --test_set_file_path "../../data/time_complexity_test_set.jsonl" \
    --time_or_space "time" \
    --generate_baseline False \
    --at_k 10 
```

--------------------------------

### Output Results Folder Structure

Source: https://github.com/facebookresearch/bigobench/blob/main/src/complexity/README.md

This markdown illustrates the typical directory structure for results saved by the Big-O Bench framework when using the command-line interface. It shows how results from multiple runs are organized into zipped folders, each containing various JSON files with analysis details.

```markdown
results_folder_name_root/
├── results_run_1.zip/
│   ├── complexity_labels_light.json
│   ├── complexity_labels_full.json
│   ├── additional_logs.json
│   └── arguments_of_complexity_framework.json
├── results_run_2.zip/
│   └── ...
└── ...
```

--------------------------------

### Run Evaluation with GPT-4o via OpenAI API

Source: https://github.com/facebookresearch/bigobench/blob/main/src/eval/README.md

Execute evaluation inference using GPT-4o via the OpenAI API. Set the `OPENAI_API_KEY` environment variable and specify the model as `gpt-4o`. The `--host` parameter is not required for API-based inference.

```bash
export OPENAI_API_KEY="your_openai_key_here"

python -u eval.py \
    --model "gpt-4o" \
    --max_concurrent_requests 256 \
    --task.data_file_path "../../data/time_complexity_test_set.jsonl" \
    --task.tasks_str "complexity_prediction/time_at_10,complexity_generation/time_at_10" \
    --task.write_eval "True" \
    --task.batch_size 256 \
    --task.use_sampling "True" \
    --task.temperature 0.8 \
    --task.top_p 0.95 \
    --dump_dir "./results"
```

--------------------------------

### Run Complexity Framework Disabling CPU Optimizations

Source: https://github.com/facebookresearch/bigobench/blob/main/src/complexity/README.md

Run the framework while disabling CPU optimizations. This is useful when running on machines with fewer than ~40 CPUs. All CPU-related parameters are set to 'None' or a step of 1.

```bash
python -u -m command_line \
    --path_to_jsonl_file="source_of_data.jsonl" \
    --main_process_cpu_id_list="None" \
    --forkserver_cpu_id_list="None" \
    --sandbox_cpu_id_step=1 \
    --sandbox_incremental_cpu_id_list="None" \
    --distinct_forkservers_incremental_cpu_id_list="None" \

```

--------------------------------

### Launch CLI Command with SLURM

Source: https://github.com/facebookresearch/bigobench/blob/main/src/eval/README.md

Use this command to launch the evaluation inference CLI command via SLURM, ensuring a dedicated node orchestrates the process. The `--host` and `--model` parameters must be configured appropriately for your inference engine.

```bash
sbatch slurm.sh
```

--------------------------------

### Clone BigOBench Repository

Source: https://github.com/facebookresearch/bigobench/blob/main/README.md

Use this command to clone the BigOBench repository locally. Navigate into the cloned directory afterwards.

```bash
git clone git@github.com:facebookresearch/BigOBench.git
cd BigOBench
```

--------------------------------

### Configure SLURM Array for Parallel Instances

Source: https://github.com/facebookresearch/bigobench/blob/main/src/inference/README.md

Tune SLURM array parameters to launch multiple vLLM instances concurrently. Set 'n' and 'k' to the desired number of parallel jobs.

```bash
#SBATCH --array=1-n%k
```