{"id":38610238,"url":"https://github.com/gretatuckute/auditory_brain_dnn","last_synced_at":"2026-01-17T08:43:17.795Z","repository":{"id":195012285,"uuid":"592514950","full_name":"gretatuckute/auditory_brain_dnn","owner":"gretatuckute","description":"Comparison of auditory DNNs and human brain acitivity.","archived":false,"fork":false,"pushed_at":"2025-04-22T21:08:04.000Z","size":7145,"stargazers_count":13,"open_issues_count":1,"forks_count":3,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-04-22T22:22:35.891Z","etag":null,"topics":["audio","brain","deep-learning"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/gretatuckute.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2023-01-23T22:18:10.000Z","updated_at":"2025-04-22T21:08:07.000Z","dependencies_parsed_at":"2025-04-22T22:32:25.241Z","dependency_job_id":null,"html_url":"https://github.com/gretatuckute/auditory_brain_dnn","commit_stats":null,"previous_names":["gretatuckute/auditory_brain_dnn"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/gretatuckute/auditory_brain_dnn","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gretatuckute%2Fauditory_brain_dnn","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gretatuckute%2Fauditory_brain_dnn/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gretatuckute%2Fauditory_brain_dnn/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gretatuckute%2Fauditory_brain_dnn/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/gretatuckute","download_url":"https://codeload.github.com/gretatuckute/auditory_brain_dnn/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gretatuckute%2Fauditory_brain_dnn/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28504415,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-17T06:57:29.758Z","status":"ssl_error","status_checked_at":"2026-01-17T06:56:03.931Z","response_time":85,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["audio","brain","deep-learning"],"created_at":"2026-01-17T08:43:17.285Z","updated_at":"2026-01-17T08:43:17.790Z","avatar_url":"https://github.com/gretatuckute.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"## Many but not all deep neural network audio models capture brain responses and exhibit correspondence between model stages and brain regions\n\nThis repository contains code and data accompanying: \nGreta Tuckute*, Jenelle Feather*, Dana Boebinger, Josh H. McDermott (2023): _Many but not all deep neural network audio models capture brain responses and exhibit correspondence between model stages and brain regions_. PLoS Biol 21(12): e3002366. https://doi.org/10.1371/journal.pbio.3002366\n\n[![DOI](https://zenodo.org/badge/592514950.svg)](https://zenodo.org/badge/latestdoi/592514950)\n\n## Environment\nThe environment does not require any sophisticated packages and would run in most Python 3.6 environments with [pandas](https://pandas.pydata.org/), [scikit-learn](https://scikit-learn.org/stable/), [statsmodels](https://www.statsmodels.org/stable/index.html), [seaborn](https://seaborn.pydata.org/) and [matplotlib](https://matplotlib.org/). However, to use the exact Python 3.6.10 environment used in the paper, install it as:\n\n```\nconda env create -f env_auditory_brain_dnn.yml\n```\n\n## Downloading data\nTo download data used in the paper, run the [download_files.py](https://github.com/gretatuckute/auditory_brain_dnn/blob/main/setup_utils/download_files.py) script. You can specify what you want to download according to the four flags: 1) _get_data = True_ will download the two neural datasets (NH2015, B2021) and the component dataset (NH2015comp). 2) _get_model_actv = True_ will download the activations of the models used in the paper. 3) _get_results = True_ will download the outputs of the analyses in the paper. 4) _get_fsavg_surf = True_ will download the surface maps in the paper.\n\n## Obtaining predictivity scores for DNNs\nWe used to common model-brain evaluation metrics, namely regression and representational similarity analysis (RSA), as demonstrated in the figure below.\n\n\u003cimg src=\"./illustrations/fig1.png\" width=\"600\"/\u003e\n\n\n### Regression\nTo perform regression from DNN activations (regressors) to brain/component responses, run [/aud_dnn/AUD_main.py](https://github.com/gretatuckute/auditory_brain_dnn/blob/main/aud_dnn/AUD_main.py). This script 1. Loads a DNN unit activations from a given model (*source_model*) and layer (*source_layer*), 2. Loads the target (*target*) of interest (either neural data: *NH2015* (Norman-Haignere et al., 2015; 7,694 voxels across 8 participants) or *B2021* (Boebinger et al., 2021; 26,792 voxels across 20 participants), or component data *NH2015comp* (Norman-Haignere et al., 2015; 6 components), 3. Runs a ridge-regression across 10 splits of the data (165 sounds; 83 sounds in train and 82 sounds in test) and stores the outputs in /results/ in subfolders with an identifier corresponding to the DNN name.\n\n### Representational Similarity Analysis (RSA)\nTo reproduce all RSA analyses in the paper, run [/aud_dnn/analyze/rsa_matrix_calculation_all_models.py](https://github.com/gretatuckute/auditory_brain_dnn/blob/main/aud_dnn/analyze/rsa_matrix_calculation_all_models.py). A call to this script from the command line will run all RSA analyses and exports results into the /results/rsa_anlaysis directory. If the specified analysis is already present in the directory, the results will be loaded to make the plots. \n\n#### Note on how DNN unit activations are organized\nIn the study we used either in-house models (trained by us, in lab) or external models (publicly available models). Both types of model activations are loaded using the function [get_source_features](https://github.com/gretatuckute/auditory_brain_dnn/blob/main/aud_dnn/utils.py#L211) which for in-house models will load the activations stored in a single h5 file (see e.g., /model-actv/Kell2018word/natsound_activations.h5). For external models, the model activations are stored as a single pickle file per sound using each sound's identifier (see e.g., /model-actv/S2T/stim5_alarm_clock_activations.pkl) which contains the activations across layers for the sound \"stim5_alarm_clock\" according to the sound nomenclature (see /data/stimuli/ -- the corresponding sound file would be named \"stim5_alarm_clock.wav\"). The [PytorchWrapper class](https://github.com/gretatuckute/auditory_brain_dnn/blob/main/aud_dnn/utils.py#L104) loads and concatenates these individual pickle files according to the specified sound order.\nFor both in-house and external models, we denote model activations obtained using a permuted network architecture with \"_randnetw\", e.g. /model-actv/Kell2018word/natsound_activations_randnetw.h5 or /model-actv/S2T/stim5_alarm_clock_activations_randnetw.pkl).\n\n\n## Generating plots\nThe figures in the paper can be reproduced via the notebooks in the [analyze/generate_figures/](https://github.com/gretatuckute/auditory_brain_dnn/tree/main/aud_dnn/analyze/generate_figures) directory, e.g., [generate_Figure2.ipynb](https://github.com/gretatuckute/auditory_brain_dnn/blob/main/aud_dnn/analyze/generate_figures/generate_Figure2.ipynb) and so forth.\n\n## DNN activations\n\n### In-house models\nThe following repository contains checkpoints for loading the in-house trained CochCNN9 and CochResNet50 models:\n\n[CochDNN](https://github.com/jenellefeather/cochdnn)\u003cbr\u003e\n\n### External models (publicly available)\nThe following repositories were used to extract DNN representations from the external DNN models:\n\n[AST](https://github.com/gretatuckute/ast)\u003cbr\u003e\n[DCASE2020](https://github.com/gretatuckute/dcase-2020-baseline)\u003cbr\u003e\n[DeepSpeech2](https://github.com/gretatuckute/deepspeech.pytorch)\u003cbr\u003e\n[MetricGAN](https://github.com/gretatuckute/speechbrain_activation_extraction)\u003cbr\u003e\n[SepFormer](https://github.com/gretatuckute/speechbrain_activation_extraction)\u003cbr\u003e\n[S2T](https://github.com/gretatuckute/asr)\u003cbr\u003e\n[VGGish](https://github.com/gretatuckute/torchvggish)\u003cbr\u003e\n[wav2vec](https://github.com/gretatuckute/asr)\u003cbr\u003e\n[VQ-VAE (ZeroSpeech)](https://github.com/gretatuckute/ZeroSpeech)\u003cbr\u003e\n\nWe thank the authors and developers for making these weights publicly available!\nThe permuted weight indices that were used to extract permuted model activations can be [downloaded here](https://drive.google.com/drive/folders/1yySqaHAC-Zda23IXaRtNOtg-pgf54dWa?usp=sharing) for each model. \n\n## Citation\n```\n@article{10.1371/journal.pbio.3002366,\n    doi = {10.1371/journal.pbio.3002366},\n    author = {Tuckute, Greta AND Feather, Jenelle AND Boebinger, Dana AND McDermott, Josh H.},\n    journal = {PLOS Biology},\n    publisher = {Public Library of Science},\n    title = {Many but not all deep neural network audio models capture brain responses and exhibit correspondence between model stages and brain regions},\n    year = {2023},\n    month = {12},\n    volume = {21},\n    url = {https://doi.org/10.1371/journal.pbio.3002366},\n    pages = {1-70},\n    abstract = {Models that predict brain responses to stimuli provide one measure of understanding of a sensory system and have many potential applications in science and engineering. Deep artificial neural networks have emerged as the leading such predictive models of the visual system but are less explored in audition. Prior work provided examples of audio-trained neural networks that produced good predictions of auditory cortical fMRI responses and exhibited correspondence between model stages and brain regions, but left it unclear whether these results generalize to other neural network models and, thus, how to further improve models in this domain. We evaluated model-brain correspondence for publicly available audio neural network models along with in-house models trained on 4 different tasks. Most tested models outpredicted standard spectromporal filter-bank models of auditory cortex and exhibited systematic model-brain correspondence: Middle stages best predicted primary auditory cortex, while deep stages best predicted non-primary cortex. However, some state-of-the-art models produced substantially worse brain predictions. Models trained to recognize speech in background noise produced better brain predictions than models trained to recognize speech in quiet, potentially because hearing in noise imposes constraints on biological auditory representations. The training task influenced the prediction quality for specific cortical tuning properties, with best overall predictions resulting from models trained on multiple tasks. The results generally support the promise of deep neural networks as models of audition, though they also indicate that current models do not explain auditory cortical responses in their entirety.},\n    number = {12},\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgretatuckute%2Fauditory_brain_dnn","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgretatuckute%2Fauditory_brain_dnn","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgretatuckute%2Fauditory_brain_dnn/lists"}