RelBench

https://relbench.stanford.edu/
Admin
RelBench is an open-source benchmark platform for machine learning over relational databases,...
Tokens:7,584
Snippets:130
Trust Score:10
Update:6 months ago
Show doc for...
Tokens:
Raw
### Install Example Dependencies

Source: https://relbench.stanford.edu/start

Install dependencies required to run scripts in the examples directory.

```bash
pip install relbench[example]
```

--------------------------------

### Install gradio_leaderboard

Source: https://relbench.stanford.edu/gradio_app/README

Use pip to install the package.

```bash
pip install gradio_leaderboard
```

--------------------------------

### Run Example Script

Source: https://relbench.stanford.edu/start

Clone the repository and execute a GNN example script.

```bash
git clone https://github.com/snap-stanford/relbench
cd relbench/examples
python gnn_entity.py --dataset rel-f1 --task driver-position
```

--------------------------------

### Install 4DBInfer Datasets

Source: https://relbench.stanford.edu/start

Install the package with support for 4DBInfer datasets.

```bash
pip install relbench[dbinfer]
```

--------------------------------

### Install Full Dependencies

Source: https://relbench.stanford.edu/start

Install RelBench along with PyTorch, PyTorch Geometric, and PyTorch Frame for modeling.

```bash
pip install relbench[full]
```

--------------------------------

### Install CTU Datasets

Source: https://relbench.stanford.edu/start

Install the package with support for CTU datasets.

```bash
pip install relbench[ctu]
```

--------------------------------

### Install RelBench Core

Source: https://relbench.stanford.edu/start

Install the core RelBench package to access data and task loading functionality.

```bash
pip install relbench
```

--------------------------------

### Customize the MIMIC-IV dataset

Source: https://relbench.stanford.edu/datasets/rel-mimic

Example of how to load the dataset with custom parameters such as table limits and column filtering.

```python
drop_columns_per_table = {
    "admissions": [...],
    "chartevents": [...],
    ...
}
tables_limit = ["patients", "admissions", "icustays", "chartevents", "procedureevents", "d_items"]


dataset = MimicDataset(
    patients_limit=20000,
    out_path='/data',
    cache_dir='/cache',
    tables_limit=tables_limit,
    db_params=db_params,
    drop_columns_per_table=drop_columns_per_table
)
db = dataset.make_db()
```

--------------------------------

### Load rel-salt Dataset

Source: https://relbench.stanford.edu/datasets/rel-salt

Use this code to load the rel-salt dataset into RelBench. Ensure RelBench is installed.

```python
from relbench.datasets import get_dataset
dataset = get_dataset("rel-salt")
```

--------------------------------

### Load rel-avito Dataset

Source: https://relbench.stanford.edu/datasets/rel-avito

Use this code to load the rel-avito dataset into RelBench. Ensure you have the relbench library installed.

```python
from relbench.datasets import get_dataset
dataset = get_dataset("rel-avito")
```

--------------------------------

### Load rel-f1 Dataset

Source: https://relbench.stanford.edu/datasets/rel-f1

Use this code to load the F1 dataset from RelBench. Ensure RelBench is installed.

```python
from relbench.datasets import get_dataset
dataset = get_dataset("rel-f1")
```

--------------------------------

### Load rel-stack Dataset

Source: https://relbench.stanford.edu/datasets/rel-stack

Use this code to load the rel-stack dataset using the get_dataset function from the relbench.datasets library. Ensure relbench is installed.

```python
from relbench.datasets import get_dataset
dataset = get_dataset("rel-stack")
```

--------------------------------

### Load rel-trial Dataset

Source: https://relbench.stanford.edu/datasets/rel-trial

Use this code to load the rel-trial clinical trial dataset into RelBench. Ensure RelBench is installed.

```python
from relbench.datasets import get_dataset
dataset = get_dataset("rel-trial")

```

--------------------------------

### Load rel-ratebeer Dataset

Source: https://relbench.stanford.edu/datasets/rel-ratebeer

Use this code to load the rel-ratebeer dataset into RelBench. Ensure you have the relbench library installed.

```python
from relbench.datasets import get_dataset
dataset = get_dataset("rel-ratebeer")
```

--------------------------------

### Initialize a basic Leaderboard

Source: https://relbench.stanford.edu/gradio_app/README

Integrate the Leaderboard component into a Gradio Blocks application using a pandas DataFrame.

```python
import gradio as gr
from gradio_leaderboard import Leaderboard
from pathlib import Path
import pandas as pd

abs_path = Path(__file__).parent

# Any pandas-compatible data
df = pd.read_json(str(abs_path / "leaderboard_data.json"))

with gr.Blocks() as demo:
    gr.Markdown("""
    # ðŸ¥‡ Leaderboard Component
    """)
    Leaderboard(
        value=df,
        select_columns=["T", "Model", "Average â¬†ï¸", "ARC",
            "HellaSwag", "MMLU", "TruthfulQA",
            "Winogrande", "GSM8K"],
        search_columns=["model_name_for_query", "Type"],
        hide_columns=["model_name_for_query", "Model Size"],
        filter_columns=["T", "Precision", "Model Size"],
    )

if __name__ == "__main__":
    demo.launch()
```

--------------------------------

### Load rel-arxiv dataset

Source: https://relbench.stanford.edu/datasets/rel-arxiv

Use the get_dataset function to initialize the rel-arxiv dataset within the RelBench framework.

```python
from relbench.datasets import get_dataset
dataset = get_dataset("rel-arxiv")
```

--------------------------------

### Dataframe Initialization Parameters

Source: https://relbench.stanford.edu/gradio_app/README

Configuration options for initializing the Dataframe component.

```APIDOC
## Initialization Parameters

### Description
Parameters used to configure the behavior and appearance of the Dataframe component.

### Parameters
- **value** (pd.DataFrame | None) - Optional - Default value to display in the DataFrame.
- **datatype** (str | list[str]) - Optional - Datatype of values in sheet (e.g., "str", "number", "bool", "date", "markdown").
- **search_columns** (list[str] | SearchColumns) - Optional - Configuration for searchable columns.
- **select_columns** (list[str] | SelectColumns) - Optional - Configuration for selectable columns.
- **filter_columns** (list[str | ColumnFilter] | None) - Optional - Configuration for filterable columns.
- **hide_columns** (list[str] | None) - Optional - List of columns to hide by default.
- **latex_delimiters** (list[dict[str, str | bool]] | None) - Optional - Delimiters for rendering LaTeX expressions in markdown columns.
- **label** (str | None) - Optional - The label for this component.
- **show_label** (bool | None) - Optional - If True, will display label.
- **every** (float | None) - Optional - If value is a callable, run the function every number of seconds.
- **height** (int) - Optional - The maximum height of the dataframe in pixels.
- **scale** (int | None) - Optional - Relative size compared to adjacent components.
- **min_width** (int) - Optional - Minimum pixel width.
- **interactive** (bool | None) - Optional - If True, allows users to edit the dataframe.
- **visible** (bool) - Optional - If False, component will be hidden.
- **elem_id** (str | None) - Optional - HTML DOM id.
- **elem_classes** (list[str] | str | None) - Optional - HTML DOM classes.
- **render** (bool) - Optional - If False, component will not render in the Blocks context.
- **wrap** (bool) - Optional - If True, text in table cells will wrap.
- **line_breaks** (bool) - Optional - If True, enables Github-flavored Markdown line breaks.
```

--------------------------------

### Configure Google Cloud authentication

Source: https://relbench.stanford.edu/datasets/rel-mimic

Commands to set up Application Default Credentials and project quotas for BigQuery access.

```bash
gcloud auth application-default login
```

```bash
gcloud auth application-default set-quota-project YOUR_PROJECT_ID
```

```bash
~/.config/gcloud/application_default_credentials.json
```

```bash
export GOOGLE_APPLICATION_CREDENTIALS="/path/to/application_default_credentials.json"
```

```bash
export PROJECT_ID="your_project_id"
```

--------------------------------

### Load a RelBench dataset

Source: https://relbench.stanford.edu/start

Retrieves a dataset by name, with options to download and cache data locally.

```python
dataset: Dataset = get_dataset("rel-amazon", download=True)
```

--------------------------------

### Load rel-event dataset in RelBench

Source: https://relbench.stanford.edu/datasets/rel-event

Use the get_dataset function to initialize the rel-event dataset for use in RelBench workflows.

```python
from relbench.datasets import get_dataset
dataset = get_dataset("rel-event")
```

--------------------------------

### Load H&M Dataset

Source: https://relbench.stanford.edu/datasets/rel-hm

Use the get_dataset function to initialize the H&M database within the RelBench framework.

```python
from relbench.datasets import get_dataset
dataset = get_dataset("rel-hm")
```

--------------------------------

### Load the MIMIC-IV dataset

Source: https://relbench.stanford.edu/datasets/rel-mimic

Use this snippet to load the pre-prepared subset of the MIMIC-IV dataset in RelBench.

```python
from relbench.datasets import get_dataset
dataset = get_dataset("rel-mimic")
```

--------------------------------

### Load the Amazon dataset

Source: https://relbench.stanford.edu/datasets/rel-amazon

Use the get_dataset function to initialize the rel-amazon dataset within the RelBench framework.

```python
from relbench.datasets import get_dataset
dataset = get_dataset("rel-amazon")
```

--------------------------------

### Configure search functionality

Source: https://relbench.stanford.edu/gradio_app/README

Use the SearchColumns class to define primary and secondary search columns and customize the search textbox.

```python
import pandas as pd
import gradio as gr
from gradio_leaderboard import Leaderboard, SearchColumns

with gr.Blocks() as demo:
    Leaderboard(
        value=pd.DataFrame({"name": ["Freddy", "Maria", "Mark"], "country": ["USA", "Mexico", "USA"]}),
        search_columns=SearchColumns(primary_column="name", secondary_columns="country",
                                     placeholder="Search by name or country. To search by country, type 'country:<query>'",
                                     label="Search"),
    )

demo.launch()
```

--------------------------------

### Import RelBench components

Source: https://relbench.stanford.edu/start

Essential imports for accessing base classes and utility functions in RelBench.

```python
from relbench.base import Table, Database, Dataset, EntityTask
from relbench.datasets import get_dataset
from relbench.tasks import get_task
```

--------------------------------

### Configuration: column_widths

Source: https://relbench.stanford.edu/gradio_app/README

Defines the structure and usage of the column_widths parameter for table components.

```APIDOC
## Configuration: column_widths

### Description
An optional parameter used to define the width of each column in a table. If provided, the browser attempts to fit the table within the page width based on these values.

### Parameters
#### Request Body
- **column_widths** (list[str | int] | None) - Optional - A list representing the width of each column. Elements can be strings like "100px" or "10%", or integers which are converted to pixel values. If omitted, widths are determined automatically.
```

--------------------------------

### Gradio Leaderboard with Custom Filters

Source: https://relbench.stanford.edu/gradio_app/README

Use this snippet to create an interactive leaderboard with customizable filters for columns like name, country, age, and score. Requires pandas and gradio_leaderboard libraries.

```python
import pandas as pd
import gradio as gr
from gradio_leaderboard import Leaderboard, ColumnFilter

with gr.Blocks() as demo:
    Leaderboard(
        value=pd.DataFrame({"name": ["Freddy", "Maria", "Mark"], "country": ["USA", "Mexico", "USA"],
                            "age": [25, 30, 35], "score": [100, 200, 300]}),
        filter_columns=[
            "name",
            ColumnFilter("country", type="dropdown", label="Select Country ðŸ‡ºðŸ‡¸ðŸ‡²ðŸ‡½"),
            ColumnFilter("age", type="slider", min=20, max=40, greater_than=True),
            ColumnFilter("score", type="slider", min=50, max=350, greater_than=True)],
    )

demo.launch()

```

--------------------------------

### Define and retrieve tasks

Source: https://relbench.stanford.edu/start

Loads a specific task for a dataset and retrieves its associated train, validation, or test tables.

```python
task: EntityTask = get_task("rel-amazon", "user-churn", download=True)
```

```python
train_table: Table = task.get_table("train")
val_table: Table = task.get_table("val")
test_table: Table = task.get_table("test")
```

```python
full_test_table: Table = task.get_table("test", mask_input_cols=False)
```

--------------------------------

### Cite 4DBInfer

Source: https://relbench.stanford.edu/start

BibTeX citation for the 4DBInfer benchmarking toolbox.

```bibtex
@article{dbinfer,
  title={4DBInfer: A 4D Benchmarking Toolbox for Graph-Centric Predictive Modeling on Relational DBs},
  author={Wang, Minjie and Gan, Quan and Wipf, David and Cai, Zhenkun and Li, Ning and Tang, Jianheng and Zhang, Yanlin and Zhang, Zizhao and Mao, Zunyao and Song, Yakun and Wang, Yanbo and Li, Jiahang and Zhang, Han and Yang, Guang and Qin, Xiao and Lei, Chuan and Zhang, Muhan and Zhang, Weinan and Faloutsos, Christos and Zhang, Zheng},
  journal={arXiv preprint arXiv:2404.18209},
  year={2024}
}
```

--------------------------------

### Access database from dataset

Source: https://relbench.stanford.edu/start

Retrieves the database object from a loaded dataset. Use the upto_test_timestamp parameter to control temporal leakage.

```python
db: Database = dataset.get_db()
```

```python
full_db: Database = dataset.get_db(upto_test_timestamp=False)
```

--------------------------------

### Configure column selection

Source: https://relbench.stanford.edu/gradio_app/README

Use the SelectColumns class to customize the checkboxgroup behavior and appearance for column selection.

```python
import pandas as pd
import gradio as gr
from gradio_leaderboard import Leaderboard, SelectColumns

with gr.Blocks() as demo:
    Leaderboard(
        value=pd.DataFrame({"a": [1, 2, 3], "b": [4, 5, 6], "c": [7, 8, 9]}),
        select_columns=SelectColumns(default_selection=["a", "b"],
                                    cant_deselect="a",
                                    label="Select The Columns",
                                    info="Helpful information")
    )

demo.launch()
```

--------------------------------

### Cite ReDeLEx

Source: https://relbench.stanford.edu/start

BibTeX citation for the ReDeLEx framework.

```bibtex
@misc{peleska2025redelex,
  title={REDELEX: A Framework for Relational Deep Learning Exploration},
  author={Jakub Peleška and Gustav Šír},
  year={2025},
  eprint={2506.22199},
  archivePrefix={arXiv},
  primaryClass={cs.LG},
  url={https://arxiv.org/abs/2506.22199},
}
```

--------------------------------

### Evaluate model predictions

Source: https://relbench.stanford.edu/start

Computes evaluation metrics for model predictions against task tables.

```python
task.evaluate(test_pred)
```

```python
task.evaluate(val_pred, val_table)
```

--------------------------------

### Cite RelBench papers

Source: https://relbench.stanford.edu/start

BibTeX citations for RelBench research papers.

```bibtex
@inproceedings{rdl,
  title={Position: Relational Deep Learning - Graph Representation Learning on Relational Databases},
  author={Fey, Matthias and Hu, Weihua and Huang, Kexin and Lenssen, Jan Eric and Ranjan, Rishabh and Robinson, Joshua and Ying, Rex and You, Jiaxuan and Leskovec, Jure},
  booktitle={Forty-first International Conference on Machine Learning}
}
```

```bibtex
@misc{relbench,
      title={RelBench: A Benchmark for Deep Learning on Relational Databases},
      author={Joshua Robinson and Rishabh Ranjan and Weihua Hu and Kexin Huang and Jiaqi Han and Alejandro Dobles and Matthias Fey and Jan E. Lenssen and Yiwen Yuan and Zecheng Zhang and Xinwei He and Jure Leskovec},
      year={2024},
      eprint={2407.20060},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2407.20060},
}
```

```bibtex
@misc{gu2026relbenchv2,
      title={{RelBench} v2: A Large-Scale Benchmark and Repository for Relational Data},
      author={Justin Gu and Rishabh Ranjan and Charilaos Kanatsoulis and Haiming Tang and Martin Jurkovic and Valter Hudovernik and Mark Znidar and Pranshu Chaturvedi and Parth Shroff and Fengyu Li and Jure Leskovec},
      year={2026},
      eprint={2602.12606},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2602.12606},
}
```
Tokens:
Raw