SRBench

https://github.com/stogiannidis/srbench
Admin
SRBench is a comprehensive benchmarking suite for evaluating spatial reasoning capabilities in...
Tokens:987
Snippets:20
Trust Score:7.9
License:MIT
Update:4 months ago
Show doc for...
Tokens:
Raw
### Install Project Dependencies

Source: https://github.com/stogiannidis/srbench/blob/main/README.md

Install all required Python packages listed in the requirements.txt file. Ensure your virtual environment is activated.

```bash
pip install -r requirements.txt
```

--------------------------------

### Clone SRBench Repository

Source: https://github.com/stogiannidis/srbench/blob/main/README.md

Clone the SRBench repository to your local machine. This is the first step to get started with the project.

```bash
git clone https://github.com/stogiannidis/srbench.git
cd srbench
```

--------------------------------

### Evaluate Model with One-Shot Examples

Source: https://github.com/stogiannidis/srbench/blob/main/README.md

Evaluate a model using one-shot examples provided in a JSON file. This helps the model understand the task format.

```bash
python src/eval.py --model <model_id> --dataset <dataset_id> --one-shot <path_to_example.json> --batch_size 16
```

--------------------------------

### Create and Activate Virtual Environment (venv)

Source: https://github.com/stogiannidis/srbench/blob/main/README.md

Create a Python virtual environment using venv and activate it. This isolates project dependencies.

```bash
python3 -m venv venv
source venv/bin/activate
```

--------------------------------

### Create and Activate Virtual Environment (conda)

Source: https://github.com/stogiannidis/srbench/blob/main/README.md

Create a Python virtual environment using conda and activate it. This isolates project dependencies.

```bash
conda create -n srbench python=3.12
conda activate srbench
```

--------------------------------

### Evaluate Model with Standard Prompting

Source: https://github.com/stogiannidis/srbench/blob/main/README.md

Run the main evaluation script for a specified model and dataset using standard prompting. Adjust batch size and seed as needed.

```bash
python src/eval.py --model <model_id> --dataset <dataset_id> --batch_size 16 --seed 42
```

--------------------------------

### Run Batch Evaluations

Source: https://github.com/stogiannidis/srbench/blob/main/README.md

Execute the main evaluation script configured in scripts/run.sh. This script evaluates multiple models on the SRBench dataset.

```bash
bash scripts/run.sh
```

--------------------------------

### Evaluate Model with Chain-of-Thought Prompting

Source: https://github.com/stogiannidis/srbench/blob/main/README.md

Evaluate a model using Chain-of-Thought prompting. This strategy encourages more detailed reasoning steps.

```bash
python src/eval.py --model <model_id> --dataset <dataset_id> --cot --batch_size 16
```

--------------------------------

### Compute Accuracy from Results

Source: https://github.com/stogiannidis/srbench/blob/main/README.md

Run the script to compute accuracy from evaluation results. This script processes CSV files in the output directory.

```bash
bash scripts/run_closed.sh
```
Tokens:
Raw