Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

https://github.com/willxxy/awesome-mmps

Corpus of resources for multimodal machine learning with physiological signals
https://github.com/willxxy/awesome-mmps

List: awesome-mmps

deep-learning machine-learning multimodal multimodal-data multimodal-deep-learning multimodal-learning physiological-signals signal-processing

Last synced: 2 months ago
JSON representation

Corpus of resources for multimodal machine learning with physiological signals

Lists

README 

        
# awesome-mmps

Corpus of resources for multimodal machine learning with physiological signals.



Any additions, corrections, or concerns please submit an issue. For additions to the list, please provide the relevant information. Thank you! :)



***



## Table of Contents



- [Publications and Preprints](#publications-and-preprints)



- [Datasets](#datasets)



- [Laboratories](#laboratories)



## Publications-and-Preprints



- [Comparing Recognition Performance and Robustness of Multimodal Deep Learning Models for Multimodal Emotion Recognition](https://ieeexplore.ieee.org/abstract/document/9395500), IEEE TCDS 2021; [EEG, Eye Movement, Peripheral Physiological Signals, ECG]

- [Multi-view Emotion Recognition Using Deep Canonical Correlation Analysis](https://bcmi.sjtu.edu.cn/~blu/papers/2018/Qiu2018Multi-viewEmotion.pdf), ICONIP 2018; [EEG, Eye Movement]

- [Correlated Attention Networks for Multimodal Emotion Recognition](https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=8621129), IEEE BIBM 2018; [EEG, Eye Movement]

- [Investigating Sex Differences in Classification of Five Emotions from EEG and Eye Movement Signals](https://drive.google.com/file/d/1pCln0DI3fiIPHHiRiCRcXJWb4LCWskXm/view), IEEE EMBC 2019; [EEG, Eye Movement]

- [Transfer Knowledge from Natural Language to Electrocardiography: Can We Detect Cardiovascular Disease Through Language Models?](https://arxiv.org/pdf/2301.09017.pdf), EACL Findings 2023; [ECG, Language]

- [Can Brain Signals Reveal Inner Alignment with Human Languages?](https://arxiv.org/pdf/2208.06348.pdf), EMNLP Findings 2023; [EEG, Language]

- [Open Vocabulary Electroencephalography-to-Text Decoding and Zero-Shot Sentiment Classification](https://arxiv.org/abs/2112.02690), AAAI 2022; [EEG, Language]

- [Decoding EEG Brain Activity for Multi-Modal Natural Language Processing](https://www.frontiersin.org/articles/10.3389/fnhum.2021.659410/full?&utm_source=Email_to_authors_&utm_medium=Email&utm_content=T1_11.5e1_author&utm_campaign=Email_publication&field=&journalName=Frontiers_in_Human_Neuroscience&id=659410), Frontiers in Human Neuroscience 2021; [EEG, Eye Movement, Language]

- [Advancing NLP with Cognitive Language Processing Signals](https://arxiv.org/pdf/1904.02682.pdf), arxiv 2019; [EEG, Language, Gaze]

- [Converting ECG Signals to Images for Efficient Image-text Retrieval via Encoding](https://arxiv.org/pdf/2304.06286), Machine Learning for Health 2023; [ECG, Image, Language]

- [Multimodal Representation Learning of Cardiovascular Magnetic Resonance Imaging](https://arxiv.org/pdf/2304.07675.pdf), ICML Workshop on Machine Learning for Multimodal Healthcare Data 2023; [CMR, Language]

- [Recognition of emotions using multimodal physiological signals and an ensemble deep learning model](https://www.sciencedirect.com/science/article/pii/S0169260716305090), Computer Methods and Programs in Biomedicine 2017; [EEG, Peripheral Physiological Signals]

- [Multi-modal emotion analysis from facial expressions and electroencephalogram](https://www.sciencedirect.com/science/article/pii/S1077314215002106), Computer Vision and Image Understanding 2016; (EEG, Facial Expressions)

- [An Ensemble Learning Method for Emotion Charting Using Multimodal Physiological Signals](https://www.mdpi.com/1424-8220/22/23/9480), Sensors 2022; [EEG, ECG, GSR]

- [Biometric Recognition Using Multimodal Physiological Signals](https://ieeexplore.ieee.org/document/8740847), IEEE 2019; [Heart Rate, Breathing Rate, Palm Electrodermal Activity, Perinasal Perspitation]

- [Machine-Learning-Based Detection of Craving for Gaming Using Multimodal Physiological Signals: Validation of Test-Retest Reliability for Practical Use](https://www.mdpi.com/1424-8220/19/16/3475), Sensors 2019; [PPG, GSR, EOG]

- [Automated detection of panic disorder based on multimodal physiological signals using machine learning](https://onlinelibrary.wiley.com/doi/full/10.4218/etrij.2021-0299), ETRI Journal 2022; [ECG, EDA, Respiration, Peripheral Temperature]

- [Video-based multimodal spontaneous emotion recognition using facial expressions and physiological signals](https://openaccess.thecvf.com/content/CVPR2022W/ABAW/papers/Ouzar_Video-Based_Multimodal_Spontaneous_Emotion_Recognition_Using_Facial_Expressions_and_Physiological_CVPRW_2022_paper.pdf), CVPR Workshop 2022; [Image, iPPG, Heart Rate]

- [Emotion Recognition from Multimodal Physiological Signals for Emotion Aware Healthcare Systems](https://link.springer.com/article/10.1007/s40846-019-00505-7), Journal of Medical and Biological Engineering 2020; [Respiratory Belt, Photoplethysmography, Fingertip Temperature]

- [A Multimodal Music Recommendation System with Listeners' Personality and Physiological Signals](https://dl.acm.org/doi/abs/10.1145/3383583.3398623), ACM/IEEE JCDL 2020; [Heart Rate, EDA, IBI, Skin Temperature, BVP]

- [Emotion recognition based on multi-modal physiological signals and transfer learning](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9493208/), Frontiers in Neuroscience 2022; [EEG, RSP, GSR, Photo-Plethysmograph]

- [Multimodal Affective States Recognition Based on Multiscale CNNs and Biologically Inspired Decision Fusion Model](https://arxiv.org/abs/1911.12918), IEEE Transactions on Affective Computing 2023; [EEG, Peripheral Physiological Signals]

- [Stress Detection with Deep Learning Approaches Using Physiological Signals](https://link.springer.com/chapter/10.1007/978-3-030-69963-5_7), IoT Technologies for Healthcare 2020; [EDA, BVP]

- [A Multimodal Approach to Estimating Vigilance Using EEG and Forehead EOG](https://iopscience.iop.org/article/10.1088/1741-2552/aa5a98/pdf), Journal of Neural Engineering 2017; [EEG, EOG]

- [Multimodal Emotion Recognition Using Deep Neural Networks](https://bcmi.sjtu.edu.cn/~liuwei/Wei%20Liu's%20HomePage_files/th_paper_2017.pdf), ICONIP 2017; [EEG, Eye Movement]

- [ECGBERT: Understanding Hidden Language of ECGs with Self-Supervised Representation Learning](https://arxiv.org/abs/2306.06340), arxiv 2023; [ECG, Language]

- [ECG Language Processing (ELP): A new technique to analyze ECG signals](https://www.sciencedirect.com/science/article/pii/S0169260721000341), Computer Methods and Programs in Biomedicine; [ECG, Language]

- [Robust Patient Information Embedding and Retrieval Mechanism for ECG Signals](https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=9201451), IEEE Access 2020; [ECG, Language]

- [EmotionMeter: A Multimodal Framework for Recognizing Human Emotions](https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=8283814), IEEE Transactions on Cybernetics 2019; [EEG, Eye Movement]

- [A Deep Learning Architecture for Temporal Sleep Stage Classification Using Multivariate and Multimodal Time Series](https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=8307462), IEEE Transactions on Neural Systems and Rehabilitation Engineering 2018; [EEG, EOG, EMG]

- [Multimodal Physiological Signals and Machine Learning for Stress Detection by Wearable Devices](https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=9856558), IEEE MEMEA 2022; [EDA, ECG, PPG]

- [Multimodal Adaptive Emotion Transformer with Flexible Modality Inputs on A Novel Dataset with Continuous Labels](https://www.semanticscholar.org/paper/Multimodal-Adaptive-Emotion-Transformer-with-Inputs-Jiang-Liu/a08b89ea7269be93bbe0d7f5f8493471b4d1c39c), ACM 2023; [EEG, Eye Movement]

- [Emotion Transformer Fusion: Complementary Representation Properties of EEG and Eye Movements on Recognizing Anger and Surprise](https://bcmi.sjtu.edu.cn/~lubaoliang/papers/2021/2021-14.pdf), IEEE BIBM 2023; [EEG, Eye Movement]

- [Multimodal Physiological Signals Fusion for Online Emotion Recognition](https://dl.acm.org/doi/pdf/10.1145/3581783.3612555), ACM Multimedia 2023; [EEG, ECG, GSR]

- [Graph to Grid: Learning Deep Representations for Multimodal Emotion Recognition](https://dl.acm.org/doi/pdf/10.1145/3581783.3612074), ACM Multimedia 2023; [EEG, Eye Movement, GSR, Respiration, ECG]

- [Automated detection of panic disorder based on multimodal physiological signals using machine learning](https://onlinelibrary.wiley.com/doi/epdf/10.4218/etrij.2021-0299?src=getftr), ETRI Journal 2023; [ECG, EDA, RESP, PT]

- [Stress Classification by Multimodal Physiological Signals Using Variational Mode Decomposition and Machine Learning](https://www.semanticscholar.org/reader/f23f2f2a8bd3f80b44042fd5b58c57c76fcdf281), Hindawi Journal of Healthcare Engineering 2021; [ECG, EEG]

- [Transformer-Based Physiological Feature Learning for Multimodal Analysis of Self-Reported Sentiment](https://dl.acm.org/doi/abs/10.1145/3536221.3556576), ICMI 2022; [EDA, BVP, HR, TEMP]

- [Language Mapping using tEEG and EEG Data with Convolutional Neural Networks](https://www.semanticscholar.org/paper/Language-Mapping-using-tEEG-and-EEG-Data-with-Adhikari-Pham/c960eee117d06c20bb5c9a54aedb707a8c277289), IEEE 2022; [EEG, tEEG]

- [Understanding language-elicited EEG data by predicting it from a fine-tuned language model](https://www.semanticscholar.org/paper/Understanding-language-elicited-EEG-data-by-it-from-Schwartz-Mitchell/dacd64e55bea910486289ceac2a5f94217433b79), NAACL 2019; [EEG, ERP, Language]

- [Validation and Interpretation of a Multimodal Drowsiness Detection System Using Explainable Machine Learning](https://www.sciencedirect.com/science/article/pii/S0169260723005916), Computer Methods and Programs in Biomedicine 2023; [EEG, ECG, EOG]

- [A Novel Algorithm for Movement Artifact Removal in ECG Signals Acquired from Wearable Systems Applied to Horses](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4618928/), PLoS One 2015; [ECG, Accelerometer]

- [Every word counts: A multilingual analysis of individual human alignment with model attention](https://arxiv.org/abs/2210.04963), AACl 2022; [Eye-Tracking, Language]

- [Early Life Stress Detection Using Physiological Signals and Machine Learning Pipelines](https://www.mdpi.com/2079-7737/12/1/91), Biology 2023; [Heart Rate, GSR]

- [Automatic stress detection in car drivers based on non-invasive physiological signals using machine learning techniques](https://link.springer.com/article/10.1007/s00521-023-08428-w), Neural Computing and Applications 2023; [ECG, EMG, GSR, Respiration Rate]

- [Hierarchical extreme puzzle learning machine-based emotion recognition using multimodal physiological signals](https://www.sciencedirect.com/science/article/pii/S1746809423000575), Biomedical Signal Processing and Control 2023; [EDA, ECG, TEMP, RESP, EMR]

- [Machine Learning–Enabled Multimodal Fusion of Intra-Atrial and Body Surface Signals in Prediction of Atrial Fibrillation Ablation Outcomes](https://www.ahajournals.org/doi/full/10.1161/CIRCEP.122.010850), Journal of American Heart Association 2022; [ECG, iECG]

- [A Hybrid Deep Learning Emotion Classification System Using Multimodal Data](https://www.mdpi.com/1424-8220/23/23/9333), Sensors 2023; [EDA, IBI, TEMP, Text, Audio]

- [Exploring the Potential of Multimodal Emotion Recognition for Hearing-Impaired Children Using Physiological Signals and Facial Expressions](https://dl.acm.org/doi/pdf/10.1145/3610661.3616240), ICMI Companion 2023; [ST, EDA, BVP, ACC, HR, Vision]

- [A Classification Model for Sensing Human Trust in Machines Using EEG and GSR](https://dl.acm.org/doi/pdf/10.1145/3132743), ACM Transactions on Interactive Intelligent Systems 2018; [EEG, GSR]

- [Performance Exploration of RNN Variants for Recognizing Daily Life Stress Levels by Using Multimodal Physiological Signals](https://dl.acm.org/doi/pdf/10.1145/3577190.3614159), ICMI 2023; [EDA, ST, ACC, HR]

- [Evaluating the Stressful Commutes Using Physiological Signals and Machine Learning Techniques](https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=9990849), IEEE 3ICT 2022; [HR, BP, EEG]

- [Review of Machine and Deep Learning Techniques in Epileptic Seizure Detection using Physiological Signals and Sentiment Analysis](https://dl.acm.org/doi/pdf/10.1145/3552512), ACM 2022; [iEEG, EEG]

- [Comparative Analysis of Emotion Classification Based on Facial Expression and Physiological Signals Using Deep Learning](https://www.mdpi.com/2076-3417/12/3/1286), Applied Sciences 2022; [Facial Expressions, HRV]

- [Towards Autonomous Physiological Signal Extraction From Thermal Videos Using Deep Learning](https://dl.acm.org/doi/10.1145/3577190.3614123), ICMI 2023; [HR, Respiration Rate, Temperature]

- [Transformer-based Self-supervised Multimodal Representation Learning for Wearable Emotion Recognition](https://arxiv.org/abs/2303.17611), IEEE Transactions On Affective Computing 2023; [EDA, BVP, TEMP]

- [ECG and EEG based detection and multilevel classification of stress using machine learning for specified genders: A preliminary study](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10473514/), PLoS One 2023; [EEG, ECG]

- [Modeling Multivariate Biosignals With Graph Neural Networks and Structured State Space Models](https://proceedings.mlr.press/v209/tang23a.html), CHIL 2023; [EEG, ECG, PSG]

- [Cognitive Load Prediction from Multimodal Physiological Signals using Multiview Learning.](https://www.semanticscholar.org/paper/Cognitive-Load-Prediction-from-Multimodal-Signals-Liu-Yu/5103ff5e420ff93bd37ff414e9789fda99e46424), IEEE journal of biomedical and health informatics 2023; [EEG, EDA, ECG, EOG, & Eye Movements]

- [An Efficient Multimodal Emotion Identification Using FOX Optimized Double Deep Q-Learning](https://dl.acm.org/doi/10.1007/s11277-023-10685-w), Wireless Personal Communications 2023; [GSR, ECG, and EEG]

- [A Framework for Cognitive Load Recognition Based on Machine Learning and Multimodal Physiological Signals by Wearable Sensors](https://ieeexplore.ieee.org/abstract/document/10348206), IEEE PRML 2023; [ECG, EDA, and PPG]

- [Multimodal 12-lead ECG data classification using multi-label DenseNet for heart disease detection](https://ieeexplore.ieee.org/document/9943957), IEEE DSIT 2022; [HRV, ECG, EHR]

- [Stress appraisal in the workplace and its associations with productivity and mood: Insights from a multimodal machine learning analysis](https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0296468), PLoS One 2024; [EDA, ST, BVP, Accelerometer]

- [Naturalistic Emotion Recognition Using EEG and Eye Movements](https://bcmi.sjtu.edu.cn/home/blu/papers/2023/2023-8.pdf), Springer Nature Singapore 2023; [EEG, Eye Movement]

- [SIM-CNN: Self-Supervised Individualized Multimodal Learning for Stress Prediction on Nurses Using Biosignals](https://www.medrxiv.org/content/medrxiv/early/2023/08/28/2023.08.25.23294640.full.pdf), medRxiv 2023; [EDA, HR, ST, Interbeat Interval, BVP, Accelerometer]

- [Time Synchronization of Multimodal Physiological Signals through Alignment of Common Signal Types and Its Technical Considerations in Digital Health](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9145353/), J Imaging 2022; [ECG]

- [Single-modal and multi-modal false arrhythmia alarm reduction using attention-based convolutional and recurrent neural networks](https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0226990), PLoS One 2024; [PPG, ABP, ECG]

- [Effects of Physiological Signals in Different Types of Multimodal Sentiment Estimation](https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=9726810&tag=1), IEEE AC 2023; [Language, EDA, Audio, Vision]

- [EmotionKD: A Cross-Modal Knowledge Distillation Framework for Emotion Recognition Based on Physiological Signals](https://dl.acm.org/doi/pdf/10.1145/3581783.3612277), ACM 2023; [EEG, GSR]

- [Interpretation of Intracardiac Electrograms Through Textual Representations](https://arxiv.org/abs/2402.01115), arxiv 2024; [EGM, Language]

- [Negative emotion recognition using multimodal physiological signals for advanced driver assistance systems](https://link.springer.com/article/10.1007/s10015-023-00858-y), Artificial Life and Robotics 2023; [ECG, EDA, EEG, and fNIRS]

- [Multimodal Risk Prediction with Physiological Signals, Medical Images and Clinical Notes](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10246140/), medRxiv 2023; [Vision, Language, EHR]

- [Physiological Fusion Net: Quantifying Individual VR Sickness with Content Stimulus and Physiological Response](https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=8802983), IEEE International Conference on Image Processing (ICIP) 2019; [Vision, EEG, ECG, GSR]

- [Text-to-ECG: 12-Lead Electrocardiogram Synthesis conditioned on Clinical Text Reports](https://arxiv.org/pdf/2303.09395.pdf), ICASSP 2023; [ECG, Language]

- [DeWave: Discrete EEG Waves Encoding for Brain Dynamics to Text Translation](https://arxiv.org/abs/2309.14030), Neurips 2023; [EEG, Language]

- [DreamDiffusion: Generating High-Quality Images from Brain EEG Signals](https://arxiv.org/pdf/2306.16934.pdf), arxiv 2023; [EEG, Vision]

- [Multi-Signal Reconstruction Using Masked Autoencoder From EEG During Polysomnography](https://arxiv.org/abs/2311.07868), IEEE BCI 2023; [EEG, EOG, Chin EMG, Event Makers]

- [Multimodal Physiological Signal Emotion Recognition Based on Convolutional Recurrent Neural Network](https://iopscience.iop.org/article/10.1088/1757-899X/782/3/032005), IOP 2020; [EOG, EMG, GSR, RSP, BVP, TMP, EEG]

- [Comparative analysis of physiological signals and electroencephalogram (EEG) for multimodal emotion recognition using generative models](https://ieeexplore.ieee.org/abstract/document/7010181), IEEE STSIVA 2014; [EEG, EOG, EMG, GSR, HR]

- [An Exploratory Study of Multimodal Physiological Data in Jazz Improvisation Using Basic Machine Learning Techniques](https://arxiv.org/ftp/arxiv/papers/2401/2401.12266.pdf), arxiv 2024; [EDA, EEG]

- [Integrating multimodal information in machine learning for classifying acute myocardial infarction](https://pubmed.ncbi.nlm.nih.gov/36963114/), Physiological Measurement 2023; [ECG, Text]

- [Large Language Models for Time Series: A Survey](https://arxiv.org/pdf/2402.01801.pdf), arxiv 2024; [EEG, ECG, Text, Vision]

- [Position Paper: What Can Large Language Models Tell Us about Time Series Analysis](https://arxiv.org/pdf/2402.02713.pdf), arxiv 2024, [ECG, Text]

- [Large Language Model-informed ECG Dual Attention Network for Heart Failure Risk Prediction](https://arxiv.org/abs/2403.10581), arxiv 2024; [ECG, Text]

- [Frozen Language Model Helps ECG Zero-Shot Learning](https://arxiv.org/pdf/2303.12311.pdf), MIDL 2023; [ECG, Text]

- [Zero-Shot ECG Diagnosis with Large Language Models and Retrieval-Augmented Generation](https://proceedings.mlr.press/v225/yu23b/yu23b.pdf), ML4H 2023; [ECG, Text]

- [Do Self-Supervised Speech and Language Models Extract Similar Representations as Human Brain?](https://ieeexplore.ieee.org/abstract/document/10446334), ICASSP 2024; [ECoG, Speech, Text]

- [Multimodal Multi-View Spectral-Spatial-Temporal Masked Autoencoder for Self-Supervised Emotion Recognition](https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=10447194), ICASSP 2024; [EEG, Eye Movement]

- [Functional Emotion Transformer for EEG-Assisted Cross-Modal Emotion Recognition](https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=10446937), ICASSP 2024; [EEG, Eye Movement]

- [Emotion Recognition From Multimodal Physiological Signals via Discriminative Correlation Fusion With a Temporal Alignment Mechanism](https://pubmed.ncbi.nlm.nih.gov/37862275/), IEEE Transactions on Cybernetics 2023; [CNS, ANS]

- [EEG-GPT: Exploring Capabilities of Large Language Models for EEG Classification and Interpretation](https://arxiv.org/abs/2401.18006), arxiv 2024; [EEG, Language]

- [Large Brain Model for Learning Generic Representations with Tremendous EEG Data in BCI](https://openreview.net/forum?id=QzTpTRVtrP), ICLR 2024; [EEG, Language]

- [A Household Multimodal Physiological Signals Monitoring System](https://ieeexplore.ieee.org/document/10082658), IEEE 2023; [ECG, lLEMG, rLEMG, pBCG]

- [Temporal-Spatial Prediction: Pre-Training on Diverse Datasets for EEG Classification](https://ieeexplore.ieee.org/abstract/document/10447845), ICASSP 2024; [EEG]

- [ECG-QA: A Comprehensive Question Answering Dataset Combined With Electrocardiogram](https://proceedings.neurips.cc/paper_files/paper/2023/hash/d0b67349dd16b83b2cf6167fb4e2be50-Abstract-Datasets_and_Benchmarks.html), NeurIPS 2023; [ECG, Text]

- [Joint Contrastive Learning with Feature Alignment for Cross-Corpus EEG-based Emotion Recognition](https://arxiv.org/abs/2404.09559), arxiv 2024; [EEG]

- [EEGFormer: Towards Transferable and Interpretable Large-Scale EEG Foundation Model](https://arxiv.org/abs/2401.10278), arxiv 2024; [EEG]

- [AC-CfC: An attention-based convolutional closed-form continuous-time neural network for raw multi-channel EEG-based emotion recognition](https://www.sciencedirect.com/science/article/pii/S1746809424003070), Biomedical Signal Processing and Control 2024; [EEG]



## Datasets

- [SEED](https://bcmi.sjtu.edu.cn/home/seed/seed.html); [EEG, Eye Movement, Video]

- [SEED-IV](https://bcmi.sjtu.edu.cn/home/seed/seed-iv.html); [EEG, Eye Movement, Video]

- [SEED-VIG](https://bcmi.sjtu.edu.cn/home/seed/seed-vig.html); [EEG, EOG]

- [SEED-V](https://bcmi.sjtu.edu.cn/home/seed/seed-v.html); [EEG, Eye Movement, Video]

- [SEED-GER](https://bcmi.sjtu.edu.cn/home/seed/seed-GER.html); [EEG, Eye Movement, Video]

- [SEED-FRA](https://bcmi.sjtu.edu.cn/home/seed/seed-FRA.html); [EEG, Eye Movement, Video]

- [ZuCo 2.0](https://osf.io/2urht/); [EEG, Language, Eye Movement]

- [DEAP](https://www.eecs.qmul.ac.uk/mmv/datasets/deap/); [EEG, Peripheral Physiological Signals, Video]

- [K-EmoCon](https://zenodo.org/record/3762962); [EEG, Peripheral Physiological Signals, Video, Audio]

- [AMIGOS](http://www.eecs.qmul.ac.uk/mmv/datasets/amigos/index.html); [EEG, ECG, GSR, Video]

- [Multimodal Adaptive Emotion Transformer with Flexible Modality Inputs on A Novel Dataset with Continuous Labels](https://www.semanticscholar.org/paper/Multimodal-Adaptive-Emotion-Transformer-with-Inputs-Jiang-Liu/a08b89ea7269be93bbe0d7f5f8493471b4d1c39c); [EEG, Eye Movement]

- [Dataset of Speech Production in intracranial Electroencephalography](https://www.nature.com/articles/s41597-022-01542-9#code-availability); [Intracranial EEG, Language]

- [Temple University Hospital EEG Seizure Corpus](https://isip.piconepress.com/projects/tuh_eeg/html/downloads.shtml); [EEG, Artifacts]

- [ECG & EEG Stress Features](https://www.kaggle.com/datasets/apithm/ecg-and-eeg-stress-features/); [EEG, ECG]

- [A multi-modal open dataset for mental-disorder analysis](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9018722/); [EEG, Language]

- [Sleep-EDF](https://www.physionet.org/content/sleep-edfx/1.0.0/); [EEG, EOG, Chin EMG, Event Makers]

- [Neurosity EEG Dataset](https://github.com/JeremyNixon/neurosity/tree/main); [EEG]

- [ECG-QA](https://github.com/Jwoo5/ecg-qa); [ECG, Text]



## Laboratories

- [Neural Interfacing Lab @ Maastricht University](https://neuralinterfacinglab.github.io/)

- [BCMI Lab @ Shanghai Jiao Tong University](https://bcmi.sjtu.edu.cn/)

- [Centre for Language Technology @ University of Copenhagen](https://cst.ku.dk/english/)

- [Safe AI Lab @ Carnegie Mellon University](https://safeai-lab.github.io/)

- [NeuroMechatronics Lab @ Carnegie Mellon University](https://www.meche.engineering.cmu.edu/faculty/neuromechatronics-lab.html)

- [Interactive Computing Lab @ KAIST](https://ic.kaist.ac.kr/)

- [Laboratory for the Neural Mechanisms of Attention @ UC Davis](https://mangunlab.ucdavis.edu/)

- [HUman Bio-behavioral Signals Lab @ Texas A&M University](https://hubbs.engr.tamu.edu/)

- [Computational Intelligence & Neural Engineering Lab @ Hanyang University](http://cone.hanyang.ac.kr/index_e.html)

- [Bioelectronics Laboratory @ Incheon National University](https://sites.google.com/view/bioelectronics-lab)

- [Healthy ML @ MIT](https://healthyml.org/people/)

- [Auton Lab @ Carnegie Mellon University](https://autonlab.org/)

- [Biomedical Functional Imaging and Neuroengineering Laboratory @ Carnegie Mellon University](https://www.cmu.edu/bme/helab/)

- [Laboratoire des Systèmes Perceptifs @ École Normale Supérieure](https://lsp.dec.ens.fr/fr)

- [Brain and AI Team @ Meta](https://ai.meta.com/blog/studying-the-brain-to-build-ai-that-processes-language-as-people-do/)

- [Computational Arrhythmia Research Laboratory @ Stanford University](http://web.stanford.edu/group/narayanlab/cgi-bin/wordpress/)

- [Wu Tsai Institute @ Yale University](https://wti.yale.edu/)

- [Shah Lab @ Stanford University](https://shahlab.stanford.edu/)

- [Edward Yoonjae Choi's Lab @ KAIST](https://mp2893.com/index.html)



## Citation

If you found this repository helpful in your research, please cite the following:



```

@software{Han_A_corpus_of_2024,

author = {Han, William},

month = apr,

title = {{A corpus of resources for Multimodal Learning with Physiological Signals}},

url = {https://github.com/willxxy/awesome-mmps},

version = {2.0.4},

year = {2024}

}

```