{"id":862,"date":"2026-05-07T16:37:32","date_gmt":"2026-05-07T08:37:32","guid":{"rendered":"https:\/\/connectword.dpdns.org\/?p=862"},"modified":"2026-05-07T16:37:32","modified_gmt":"2026-05-07T08:37:32","slug":"meta-ai-releases-neuralbench-a-unified-open-source-framework-to-benchmark-neuroai-models-across-36-eeg-tasks-and-94-datasets","status":"publish","type":"post","link":"https:\/\/connectword.dpdns.org\/?p=862","title":{"rendered":"Meta AI Releases NeuralBench: A Unified Open-Source Framework to Benchmark NeuroAI Models Across 36 EEG Tasks and 94 Datasets"},"content":{"rendered":"
Evaluating AI models trained on brain signals has long been a messy, inconsistent topic. Different research groups use different preprocessing pipelines, train models on different datasets, and report results on a narrow set of tasks \u2014 making it nearly impossible to know which model actually works best, or for what. A new framework from Meta AI team is designed to fix that.<\/p>\n
Meta Researchers have released NeuralBench<\/strong>, a unified, open-source framework for benchmarking AI models of brain activity. Its first release, NeuralBench-EEG v1.0<\/strong>, is the largest open benchmark of its kind: 36 downstream tasks, 94 datasets, 9,478 subjects, 13,603 hours of electroencephalography (EEG) data, and 14 deep learning architectures evaluated under a single standardized interface.<\/p>\n
The broader field of NeuroAI where deep learning meets neuroscience has exploded in recent years. Self-supervised learning techniques originally developed for language, speech and images are now being adapted to build brain foundation models<\/em>: large models pretrained on unlabeled brain recordings and fine-tuned for downstream tasks ranging from clinical seizure detection to decoding what a person is seeing or hearing.<\/p>\n
But the evaluation landscape has been badly fragmented. Existing benchmarks like MOABB cover up to 148 brain-computer interfacing (BCI) datasets but limit evaluation to just 5 downstream tasks. Other efforts \u2014 EEG-Bench, EEG-FM-Bench, AdaBrain-Bench \u2014 are each constrained in their own ways. For modalities like magnetoencephalography (MEG) and functional magnetic resonance imaging (fMRI), there is no systematic benchmark at all.<\/p>\n
The result \u2014 claims about foundation models being \u201cgeneralizable\u201d or \u201cfoundational\u201d often rest on cherry-picked tasks with no common reference point.<\/p>\n
What is NeuralBench?<\/strong><\/h3>\n
NeuralBench is built on three core Python packages that form a modular pipeline.<\/strong><\/p>\n
NeuralFetch<\/strong> handles dataset acquisition, pulling curated data from public repositories including OpenNeuro, DANDI, and NEMAR. NeuralSet<\/strong> prepares data as PyTorch-ready dataloaders, wrapping existing neuroscience tools like MNE-Python and nilearn for preprocessing, and HuggingFace for extracting stimulus embeddings (for tasks involving images, speech, or text). NeuralTrain<\/strong> provides modular training code built on PyTorch-Lightning, Pydantic, and the exca<\/code> execution and caching library.<\/p>\n
Once installed via pip install neuralbench<\/code>, the framework is controlled via a command-line interface (CLI). Running a task is as simple as three commands: download the data, prepare the cache, and execute. Every task is configured through a lightweight YAML file that specifies the data source, train\/validation\/test splits, preprocessing steps, target processing, training hyperparameters, and evaluation metrics.<\/p>\n
What NeuralBench-EEG v1.0 Covers<\/strong><\/h3>\n
The first release focuses on EEG and spans eight task categories: cognitive decoding<\/strong> (image, sentence, speech, typing, video, and word decoding), brain-computer interfacing (BCI)<\/strong>, evoked responses<\/strong>, clinical tasks<\/strong>, internal state<\/strong>, sleep<\/strong>, phenotyping<\/strong>, and miscellaneous<\/strong>.<\/p>\n
Three classes of models are compared:<\/strong><\/p>\n
\n
Task-specific architectures<\/strong> (~1.5K\u20134.2M parameters, trained from scratch): ShallowFBCSPNet, Deep4Net, EEGNet, BDTCN, ATCNet, EEGConformer, SimpleConvTimeAgg, and CTNet.<\/li>\n
EEG foundation models<\/strong> (~3.2M\u2013157.1M parameters, pretrained and fine-tuned): BENDR, LaBraM, BIOT, CBraMod, LUNA, and REVE.<\/li>\n
Handcrafted feature baselines<\/strong>: sklearn-style pipelines using symmetric positive definite (SPD) matrix representations fed into logistic or Ridge regression.<\/li>\n<\/ul>\n
All foundation models are fine-tuned end-to-end using a shared training recipe \u2014 AdamW optimizer, learning rate of 10\u207b\u2074, weight decay of 0.05, cosine-annealing with 10% warmup, up to 50 epochs with early stopping (patience=10). The sole exception is BENDR, for which the learning rate is lowered to 10\u207b\u2075 and gradient clipping is applied at 0.5 to obtain stable learning curves. This intentional standardization otherwise removes model-specific optimization tricks \u2014 such as layer-wise learning rate decay, two-stage probing, or LoRA \u2014 so that architecture and pretraining methodology are what actually gets evaluated.<\/p>\n
Data splitting is handled differently per task type to reflect real-world generalization constraints: predefined splits where provided by dataset research team, leave-concept-out<\/em> for cognitive decoding tasks (all subjects seen in training, but a held-out set of stimuli used for testing), cross-subject splits for most clinical and BCI tasks, and within-subject splits for datasets with very few participants. Each model is trained three times per task using three different random seeds.<\/p>\n
Evaluation metrics are standardized by task type: balanced accuracy for binary and multiclass classification, macro F1-score for multilabel classification, Pearson correlation for regression, and top-5 accuracy for retrieval tasks. All results are additionally reported as normalized scores (s\u0303), where 0 corresponds to dummy-level performance and 1 corresponds to perfect performance, enabling fair cross-task comparisons regardless of metric scale.<\/p>\n
One important methodological note: some EEG foundation models were pretrained on datasets that overlap with NeuralBench\u2019s downstream evaluation sets. Rather than discarding these results, the benchmark flags them with hashed bars in result figures so readers can identify potential pretraining data leakage \u2014 no strong trend suggesting leakage inflates performance was observed, but the transparency is preserved.<\/p>\n
The benchmark offers two variants: NeuralBench-EEG-Core v1.0<\/strong>, which uses a single representative dataset per task for broad coverage, and NeuralBench-EEG-Full v1.0<\/strong>, which expands to up to 24 datasets per task to study within-task variability across recording hardware, labs, and subject populations. A Kendall\u2019s \u03c4 of 0.926 (p < 0.001) between Core and Full rankings confirms that the Core variant is a reliable proxy \u2014 though a few model positions do shift, including CTNet overtaking LUNA when more datasets are included.<\/p>\n
Finding 1: Foundation models only marginally outperform task-specific models.<\/strong> The top-ranked models overall are REVE (69.2M parameters, mean normalized rank 0.20), LaBraM (5.8M, rank 0.21), and LUNA (40.4M, rank 0.30). But several task-specific models trained from scratch \u2014 CTNet (150K parameters, rank 0.32), SimpleConvTimeAgg (4.2M, rank 0.35), and Deep4Net (146K, rank 0.43) \u2014 trail closely behind. CTNet actually overtakes the LUNA foundation model to rank third in the Full variant, despite having roughly 270\u00d7 fewer parameters. This shows the gap between task-specific and foundation models is narrow enough that expanding dataset coverage alone is sufficient to change global rankings.<\/p>\n
Finding 2: Many tasks remain genuinely hard.<\/strong> Cognitive decoding tasks \u2014 recovering dense representations of images, speech, sentences, video, or words from brain activity \u2014 are particularly challenging, with even the best models scoring well below ceiling. Tasks like mental imagery, sleep arousal, psychopathology decoding, and cross-subject motor imagery and P300 classification frequently yield performance close to dummy level. These tasks represent the best benchmarks for stress-testing the next generation of EEG foundation models.<\/p>\n
Tasks approaching saturation include SSVEP classification, pathology detection, seizure detection, sleep stage classification, and phenotyping tasks like age regression and sex classification.<\/p>\n
Beyond EEG: MEG and fMRI<\/strong><\/h3>\n
Even in this initial EEG-focused release, NeuralBench already supports MEG and fMRI tasks as proof of concept. Notably, the REVE model \u2014 pretrained exclusively on EEG data \u2014 achieves the best performance among all tested models on the typing decoding task in MEG. This is a striking early signal that EEG-pretrained representations may transfer meaningfully across brain recording modalities, a hypothesis the framework is positioned to rigorously test in future releases.<\/p>\n
The infrastructure is explicitly designed for expansion to intracranial EEG (iEEG), functional near-infrared spectroscopy (fNIRS), and electromyography (EMG).<\/p>\n
How to Get Started<\/h2>\n
Installation takes a single command: pip install neuralbench<\/code>. From there, running the audiovisual stimulus classification task on EEG looks like this:<\/p>\n
![\"\"](\"https:\/\/www.marktechpost.com\/wp-content\/uploads\/2026\/05\/Screenshot-2026-05-07-at-1.34.54-AM-1.png\")
![\"\"](\"https:\/\/www.marktechpost.com\/wp-content\/uploads\/2026\/05\/Screenshot-2026-05-07-at-1.35.55-AM-1.png\")
![\"\"](\"https:\/\/www.marktechpost.com\/wp-content\/uploads\/2026\/05\/Screenshot-2026-05-07-at-1.36.48-AM-1.png\")