EEG-FM-Bench

https://github.com/xw1216/eeg-fm-bench
Admin
EEG-FM-Bench is a comprehensive benchmark platform designed to standardize the evaluation of EEG...
Tokens:7,189
Snippets:69
Trust Score:8.4
License:Apache-2.0
Update:3 months ago
Show doc for...
Tokens:
Raw
### Complete Configuration Example (YAML)

Source: https://github.com/xw1216/eeg-fm-bench/blob/main/README.md

A comprehensive example of a YAML configuration file for training experiments. It covers seed, data, model, training, and logging parameters.

```yaml
# Training pattern flags
seed: 42
master_port: 51001
multitask: true
model_type: 'eegpt'

# Data configuration
data:
  batch_size: 32
  num_workers: 1
  datasets:
    tuab: 'finetune'
    seed: 'finetune_sub_dependent'
    hmc: 'finetune'
    # ...


# EEGPT-specific model configuration
model:
  # Pretrained weights - each model will load from this checkpoint
  pretrained_path: null

  # Classifier head configuration
  classifier_head:
    head_type: 'avg_pool'  # Options: avg_pool, attention_pool, dual_stream_fusion, flatten_mlp
    
    # Common parameters
    hidden_dims: [128]
    dropout: 0.3

  # EEGPT architecture parameters
  patch_size: 64
  patch_stride: 32
  embed_num: 4
  embed_dim: 512
  depth: 8
  num_heads: 8
  mlp_ratio: 4.0

# Training configuration
training:
  max_epochs: 50
  weight_decay: 0.01
  max_grad_norm: 3.0

  # Optimizer settings
  max_lr: 5e-4
  encoder_lr_scale: 0.1    # Scale factor for encoder learning rate
  warmup_epochs: 5
  warmup_scale: 1e-2
  min_lr: 1e-6            # For CosineAnnealingLR

  # Training options
  freeze_encoder: false     # Whether to freeze encoder weights
  use_amp: true           # Use automatic mixed precision

  label_smoothing: 0.1    # Label smoothing factor

  # LoRA configuration
  lora:
    use_lora: false
    lora_r: 8

# Logging configuration
logging:
  experiment_name: "eegpt"
  run_dir: "assets/run"
  
  # Cloud logging configuration
  use_cloud: true
  
  project: 'eegpt'           # Project name (uses experiment_name if not specified)
  offline: false
  
  tags: ['eegpt', 'unified', 'full', "debug"]

  # Logging intervals
  log_step_interval: 1      # Log every N steps
  ckpt_interval: 5       # Evaluate every N epochs

```

--------------------------------

### Example Experiment Configuration (YAML)

Source: https://github.com/xw1216/eeg-fm-bench/blob/main/README.md

An example YAML file demonstrating how to configure an experiment, including model, data, training, and logging parameters. Values not set are filled by Pydantic.

```yaml
# Example: assets/conf/baseline/eegpt/eegpt_unified.yaml
model:
  pretrained_path: "/path/to/eegpt/weights"
data:
  batch_size: 128
  num_workers: 4
  datasets:
    tuab: 'finetune'
training:
  max_lr: 1e-4
  max_epochs: 50
log:
  run_dir: "/path/to/run"
```

--------------------------------

### Clone Repository and Install Dependencies

Source: https://github.com/xw1216/eeg-fm-bench/blob/main/README.md

Clone the EEG-FM-Bench repository and install the necessary Python packages, including PyTorch. Note potential version conflicts with braindecode and moabb.

```bash
git clone https://github.com/xw1216/EEG-FM-Bench.git
cd EEG-FM-Bench

# Please install torch by official command in https://pytorch.org/get-started/locally/
# torchaudio is not supported since torch 2.9
pip3 install torch torchaudio torchvision

# Install dependencies
pip install -r requirements.txt

# These packages are required in some scenarios but may cause conflicts with other packages
# braindecode : required by EEGNet and EEGConformer, but needs torch < 2.9
# moabb : required by BENDR
# captum : supports numpy > 2.0, install with --no-deps option
```

--------------------------------

### List Available Dataset Implementations

Source: https://github.com/xw1216/eeg-fm-bench/blob/main/README.md

Use this command to browse all available dataset implementations within the project. This helps in identifying the Python files corresponding to each dataset.

```bash
# Browse all available dataset implementations
ls data/dataset/
find data/dataset/ -name "*.py"
```

--------------------------------

### Execute Model Fine-tuning

Source: https://github.com/xw1216/eeg-fm-bench/blob/main/README.md

Initiate model fine-tuning using a specific configuration file and model type. Use 'list-models' to see supported model types.

```bash
# Fine-Tuning (examples for different models)
python baseline_main.py conf_file=baseline/eegpt/eegpt_unified.yaml model_type=eegpt

# List model types supported by the unified entrypoint
python baseline_main.py list-models
```

--------------------------------

### View Dataset Class Documentation

Source: https://github.com/xw1216/eeg-fm-bench/blob/main/README.md

Execute this Python command to view the documentation and description of a specific dataset class. Replace 'WorkloadConfig' and 'finetune' with the relevant dataset class and configuration name.

```python
python -c "
from data.dataset.workload import WorkloadConfig  # Example
conf = WorkloadConfig(name='finetune')
print(conf.description)
print(conf.citation)
"
```

--------------------------------

### Develop Custom Dataset Adapter

Source: https://github.com/xw1216/eeg-fm-bench/blob/main/README.md

Create a DatasetAdapter to handle runtime data conversion if your model requires a different input format. Implement _setup_adapter, _process_sample, and get_supported_channels.

```python
# baseline/your_model/your_model_adapter.py
# If your model uses different input data format, you should create an DatasetAdapter to do runtime data conversion
class YourModelDatasetAdapter(AbstractDatasetAdapter):
    def _setup_adapter(self):
      """Initialize specific adapter configurations."""
        self.model_name = 'your_model'
        super()._setup_adapter()

    def _process_sample(self, sample: Dict[str, Any]) -> Dict[str, Union[torch.Tensor, str, List[str], int]]:
      """Process a single sample according to model requirements."""
      pass

    def get_supported_channels(self) -> List[str]:
        """Return list of channels supported by your model"""
        return []

```

--------------------------------

### Execute Dataset Preprocessing

Source: https://github.com/xw1216/eeg-fm-bench/blob/main/README.md

Run the preprocessing pipeline using a specified configuration file. The config file can be an absolute path or relative to EEGFM_CONF_ROOT.

```bash
# First, download datasets from their original sources (see Dataset Guide)
# Then preprocess with standardized pipeline
# Config file can be identified by absolute path or relative path to CONF_ROOT
python preproc.py conf_file=preproc/preproc_example.yaml
```

--------------------------------

### Set Project Paths via Environment Variables

Source: https://github.com/xw1216/eeg-fm-bench/blob/main/README.md

Configure environment variables for project root, configuration root, and run root. This is essential for the project to locate its assets and output directories.

```bash
export EEGFM_PROJECT_ROOT=$PWD
export EEGFM_CONF_ROOT=$PWD/assets/conf
export EEGFM_RUN_ROOT=$PWD/assets/run
```

```powershell
$env:EEGFM_PROJECT_ROOT = (Get-Location).Path
$env:EEGFM_CONF_ROOT = "$env:EEGFM_PROJECT_ROOT/assets/conf"
$env:EEGFM_RUN_ROOT = "$env:EEGFM_PROJECT_ROOT/assets/run"
```

--------------------------------

### Gradient/Representation Analysis - Stage 2 Visualization

Source: https://github.com/xw1216/eeg-fm-bench/blob/main/README.md

Generate figures from saved tensors and metrics. The --data-dir can point to the run root or a specific seed directory.

```bash
# Stage-2: visualize a single run/seed
# Tip: --data-dir can point to the run root (auto-discovered) or a specific seed directory.
python analysis_vis.py \
  --data-dir ./analysis_results/scratch_vs_pretrained_YYYYMMDD_HHMMSS
```

--------------------------------

### Create Custom Model Trainer

Source: https://github.com/xw1216/eeg-fm-bench/blob/main/README.md

Implement a custom trainer by inheriting from AbstractTrainer. Override the build_model method to return your foundation model.

```python
# baseline/your_model/your_model_trainer.py
class YourModelTrainer(AbstractTrainer):
    """Main trainer class for your model."""
    
    def __init__(self, config: YourModelConfig):
        super().__init__(config)
        
    def build_model(self) -> nn.Module:
        """Build the foundation model (include classifier) architecture."""
        return YourFoundationModel(...)
    
    def load_checkpoint(self, checkpoint_path: str):
        """Load model checkpoint."""
        pass

```

--------------------------------

### Create Integrated Gradients Analysis

Source: https://github.com/xw1216/eeg-fm-bench/blob/main/README.md

Generate integrated gradients analysis plots using specified configuration files for the model and the plotting parameters.

```bash
# Create integrated gradients analysis
python plot_vis.py integrated_gradients assets/conf/baseline/csbrain/csbrain_unified.yaml plot/configs/example/integrated_gradients_config_csbrain.yaml
```

--------------------------------

### Implement Custom Foundation Model Architecture

Source: https://github.com/xw1216/eeg-fm-bench/blob/main/README.md

Define the architecture of your custom foundation model by inheriting from nn.Module. Implement the __init__ and forward methods.

```python
# baseline/your_model/model.py
class YourFoundationModel(nn.Module):
    """Your foundation model architecture."""
    
    def __init__(self, encoder, classifier, ):
        super().__init__()
        # Implement your architecture
        self.encoder = encoder
        self.classifier = classifier
    
    def forward(self, x: torch.Tensor) -> torch.Tensor:
        """Full forward pass."""
        pass

```

--------------------------------

### Custom Dataset Integration (Python)

Source: https://github.com/xw1216/eeg-fm-bench/blob/main/README.md

Python code defining a Pydantic configuration class and a dataset builder for integrating new datasets. Requires implementing specific interfaces.

```python
from dataclasses import dataclass, field
from typing import Optional, Union, Any

import datasets

from common.config import EEGConfig
from common.dataset import EEGDatasetBuilder, DatasetTaskType
from pandas import DataFrame


@dataclass
class TemplateConfig(EEGConfig):
    name: str = 'finetune'
    version: Optional[Union[datasets.utils.Version, str]] = datasets.utils.Version("0.0.0")
    description: Optional[str] = ""
    citation: Optional[str] = """
    bibtex Citation
    """

    filter_notch: float = 50.0
    is_notched: bool = False

    dataset_name: Optional[str] = 'template'
    task_type: DatasetTaskType = DatasetTaskType.UNKNOWN
    file_ext: str = 'edf'
    montage: dict[str, list[str]] = field(default_factory=lambda: {
        '10_20': [
            'Fp1',
            'Fp2',
        ]
    })

    valid_ratio: float = 0.10
    test_ratio: float = 0.10
    wnd_div_sec: int = 10
    suffix_path: str = 'template'
    scan_sub_dir: str = "data"

    category: list[str] = field(default_factory=lambda: ['class 1', 'class 2'])


class TemplateBuilder(EEGDatasetBuilder):
    BUILDER_CONFIG_CLASS = TemplateConfig
    BUILDER_CONFIGS = [
        BUILDER_CONFIG_CLASS(name='pretrain'),
        BUILDER_CONFIG_CLASS(name='finetune', is_finetune=True)
    ]

    def __init__(self, config_name='pretrain',**kwargs):
        super().__init__(config_name, **kwargs)
        self._load_meta_info()

    def _load_meta_info(self):
        pass

    def _resolve_file_name(self, file_path: str) -> dict[str, Any]:
        pass

    def _resolve_exp_meta_info(self, file_path: str) -> dict[str, Any]:
        pass

    def _resolve_exp_events(self, file_path: str, info: dict[str, Any]):
        return [('default', 0, -1)]

    def _divide_split(self, df: DataFrame) -> DataFrame:
        return self._divide_all_split_by_sub(df)

    def standardize_chs_names(self, montage: str):
        return self.config.montage[montage]

```

--------------------------------

### Submit SLURM Preprocessing Job

Source: https://github.com/xw1216/eeg-fm-bench/blob/main/README.md

Use this command to initiate large-scale preprocessing tasks on a SLURM cluster. Ensure datasets are downloaded prior to execution. Requires a preprocessing configuration file.

```bash
# Large-scale preprocessing (after downloading datasets)
sbatch scripts/slurm/preproc_submit.slurm conf_file=your_preproc_config.yaml
```

--------------------------------

### Update Preprocessing Configuration

Source: https://github.com/xw1216/eeg-fm-bench/blob/main/README.md

Edit the preprocessing configuration YAML file to include your downloaded datasets. This step is crucial for the preprocessing pipeline to recognize and utilize your data.

```bash
# Update preprocessing configuration with your data paths
vim assets/conf/preproc/preproc_example.yaml
# Edit the YAML file to add your downloaded datasets to preproc list
```

--------------------------------

### Gradient/Representation Analysis - Stage 1 Collection

Source: https://github.com/xw1216/eeg-fm-bench/blob/main/README.md

Collect gradients and features for analysis by running lightweight training loops and saving results to disk. This is the first stage of the two-stage workflow.

```bash
# Stage-1: collect gradients/features
python analysis_run.py \
  --config assets/conf/analysis/analysis_example.yaml \
  --trainer-config assets/conf/baseline/csbrain/csbrain_unified.yaml
```

--------------------------------

### Register Custom Model with Registry

Source: https://github.com/xw1216/eeg-fm-bench/blob/main/README.md

Register your custom model components (config, adapter factory, trainer) with the ModelRegistry. This makes your model available for use within the framework.

```python
# baseline/__init__.py
# Register your own classes to Registry
ModelRegistry.register_model(
    model_type='your_model',
    config_class=YourModelConfig,
    adapter_class=YourModelDataLoaderFactory, # or None if no conversion needed
    trainer_class=YourModelTrainer
)

```

--------------------------------

### Submit SLURM Distributed Model Training Job

Source: https://github.com/xw1216/eeg-fm-bench/blob/main/README.md

This command is used for distributed model training on a SLURM cluster. Specify the configuration file and the model type for training. Supports various models.

```bash
# Distributed model training (examples for different models)
sbatch scripts/slurm/baseline_submit.slurm conf_file=your_model_config.yaml model_type=eegpt
```

--------------------------------

### Integrate DatasetAdapter with DataLoaderFactory

Source: https://github.com/xw1216/eeg-fm-bench/blob/main/README.md

Extend AbstractDataLoaderFactory to create your custom DatasetAdapter. This allows the framework to use your adapter for data loading.

```python
# Integrate DatasetAdapter into DataLaderFactory class
class YourModelDataLoaderFactory(AbstractDataLoaderFactory):
    def create_adapter(
        self,
        dataset: HFDataset,
        dataset_names: List[str],
        dataset_configs: List[str]
    ) -> YourModelDatasetAdapter:
        return YourModelDatasetAdapter(dataset, dataset_names, dataset_configs)

```

--------------------------------

### Generate t-SNE Embeddings

Source: https://github.com/xw1216/eeg-fm-bench/blob/main/README.md

Create t-SNE embeddings for visualization using a specified configuration file and plot configuration.

```bash
# Generate t-SNE embeddings
python plot_vis.py t_sne assets/conf/baseline/csbrain/csbrain_unified.yaml plot/configs/example/tsne_config_csbrain.yaml
```

--------------------------------

### Define Custom Model Configuration

Source: https://github.com/xw1216/eeg-fm-bench/blob/main/README.md

Extend BaseModelArgs to define specific parameters for your custom model. Ensure all model-specific parameters have type annotations.

```python
# baseline/your_model/your_model_config.py
class YourModelConfig(BaseModelArgs):
    """Model-specific configuration extending BaseModelArgs."""
    hidden_dim: int = 256
    num_layers: int = 4
    dropout: float = 0.1
    # Add your model-specific parameters with type annotations

```

--------------------------------

### EEG Foundation Model Citations

Source: https://github.com/xw1216/eeg-fm-bench/blob/main/README.md

Citations for specific EEG foundation models. Use these when referencing individual models within the EEG-FM-Bench framework.

```bibtex
@article{kostas2021bendr,
  title={BENDR: Using transformers and a contrastive self-supervised learning task to learn from massive amounts of EEG data},
  author={Kostas, Demetres and Aroca-Ouellette, Stephane and Rudzicz, Frank},
  journal={Frontiers in Human Neuroscience},
  volume={15},
  pages={653659},
  year={2021},
  publisher={Frontiers Media SA}
}
```

```bibtex
@article{yang2023biot,
  title={Biot: Biosignal transformer for cross-data learning in the wild},
  author={Yang, Chaoqi and Westover, M and Sun, Jimeng},
  journal={Advances in Neural Information Processing Systems},
  volume={36},
  pages={78240--78260},
  year={2023}
}
```

```bibtex
@article{wang2024cbramod,
  title={Cbramod: A criss-cross brain foundation model for eeg decoding},
  author={Wang, Jiquan and Zhao, Sha and Luo, Zhiling and Zhou, Yangxuan and Jiang, Haiteng and Li, Shijian and Li, Tao and Pan, Gang},
  journal={arXiv preprint arXiv:2412.07236},
  year={2024}
}
```

```bibtex
@article{wang2024eegpt,
  title={Eegpt: Pretrained transformer for universal and reliable representation of eeg signals},
  author={Wang, Guangyu and Liu, Wenchao and He, Yuhong and Xu, Cong and Ma, Lin and Li, Haifeng},
  journal={Advances in Neural Information Processing Systems},
  volume={37},
  pages={39249--39280},
  year={2024}
}
```

```bibtex
@article{jiang2024labram,
  title={Large brain model for learning generic representations with tremendous EEG data in BCI},
  author={Jiang, Wei-Bang and Zhao, Li-Ming and Lu, Bao-Liang},
  journal={arXiv preprint arXiv:2405.18765},
  year={2024}
}
```

```bibtex
@article{Ouahidi2025reve,
  title={REVE: A Foundation Model for EEG - Adapting to Any Setup with Large-Scale Pretraining on 25,000 Subjects},
  author={Yassine El Ouahidi and Jonathan Lys and Philipp Th{"o}lke and Nicolas Farrugia and Bastien Pasdeloup and Vincent Gripon and Karim Jerbi and Giulia Lioi},
  journal={ArXiv},
  year={2025},
  volume={abs/2510.21585},
}
```

```bibtex
@article{zhou2025csbrain,
  title={CSBrain: A Cross-scale Spatiotemporal Brain Foundation Model for EEG Decoding},
  author={Zhou, Yuchen and Wu, Jiamin and Ren, Zichen and Yao, Zhouheng and Lu, Weiheng and Peng, Kunyu and Zheng, Qihao and Song, Chunfeng and Ouyang, Wanli and Gou, Chao},
  journal={arXiv preprint arXiv:2506.23075},
  year={2025}
}
```

--------------------------------

### General Time-Series Foundation Model Citations

Source: https://github.com/xw1216/eeg-fm-bench/blob/main/README.md

Citations for general time-series foundation models. Include these if the research utilizes broader time-series foundation models beyond EEG-specific ones.

```bibtex
@article{feofanov2025mantis,
  title={Mantis: Lightweight calibrated foundation model for user-friendly time series classification},
  author={Feofanov, Vasilii and Wen, Songkang and Alonso, Marius and Ilbert, Romain and Guo, Hongbo and Tiomoko, Malik and Pan, Lujia and Zhang, Jianfeng and Redko, Ievgen},
  journal={arXiv preprint arXiv:2502.15637},
  year={2025}
}
```

```bibtex
@article{goswami2024moment,
  title={Moment: A family of open time-series foundation models},
  author={Goswami, Mononito and Szafer, Konrad and Choudhry, Arjun and Cai, Yifu and Li, Shuo and Dubrawski, Artur},
  journal={arXiv preprint arXiv:2402.03885},
  year={2024}
}
```

--------------------------------

### Classical Baseline Model Citations

Source: https://github.com/xw1216/eeg-fm-bench/blob/main/README.md

Citations for classical baseline models. These are important for comparative studies and understanding the evolution of EEG analysis techniques.

```bibtex
@article{lawhern2018eegnet,
  title={EEGNet: a compact convolutional neural network for EEG-based brain--computer interfaces},
  author={Lawhern, Vernon J and Solon, Amelia J and Waytowich, Nicholas R and Gordon, Stephen M and Hung, Chou P and Lance, Brent J},
  journal={Journal of neural engineering},
  volume={15},
  number={5},
  pages={056013},
  year={2018},
  publisher={iOP Publishing}
}
```

```bibtex
@article{song2022eeg-conformer,
  title={EEG conformer: Convolutional transformer for EEG decoding and visualization},
  author={Song, Yonghao and Zheng, Qingqing and Liu, Bingchuan and Gao, Xiaorong},
  journal={IEEE Transactions on Neural Systems and Rehabilitation Engineering},
  volume={31},
  pages={710--719},
  year={2022},
  publisher={IEEE}
}
```

--------------------------------

### EEG-FM-Bench Paper Citation

Source: https://github.com/xw1216/eeg-fm-bench/blob/main/README.md

Cite this BibTeX entry when using the EEG-FM-Bench system in your research.

```bibtex
@misc{xiong2025eegfmbenchcomprehensivebenchmarksystematic,
      title={EEG-FM-Bench: A Comprehensive Benchmark for the Systematic Evaluation of EEG Foundation Models},
      author={Wei Xiong and Jiangtong Li and Jie Li and Kun Zhu},
      year={2025},
      eprint={2508.17742},
      archivePrefix={arXiv},
      primaryClass={eess.SP},
      url={https://arxiv.org/abs/2508.17742},
}
```
Tokens:
Raw