{"id":19214788,"url":"https://github.com/garethjns/msimodels","last_synced_at":"2025-05-12T23:12:23.661Z","repository":{"id":37624545,"uuid":"103420104","full_name":"garethjns/MSIModels","owner":"garethjns","description":"Exploring multi-sensory integration and decision making in biologically inspired deep neural networks.","archived":false,"fork":false,"pushed_at":"2022-11-21T21:20:47.000Z","size":48197,"stargazers_count":3,"open_issues_count":5,"forks_count":2,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-04-20T19:37:12.960Z","etag":null,"topics":["audiodag","brain","decision-making","deep-neural-networks","event-detection","keras-neural-networks","lstm","matlab","multi-modal","multisensory-processing","psychophysics"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/garethjns.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-09-13T15:53:37.000Z","updated_at":"2022-11-07T11:13:54.000Z","dependencies_parsed_at":"2022-09-06T08:10:42.561Z","dependency_job_id":null,"html_url":"https://github.com/garethjns/MSIModels","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/garethjns%2FMSIModels","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/garethjns%2FMSIModels/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/garethjns%2FMSIModels/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/garethjns%2FMSIModels/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/garethjns","download_url":"https://codeload.github.com/garethjns/MSIModels/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253837460,"owners_count":21971984,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["audiodag","brain","decision-making","deep-neural-networks","event-detection","keras-neural-networks","lstm","matlab","multi-modal","multisensory-processing","psychophysics"],"created_at":"2024-11-09T14:11:20.828Z","updated_at":"2025-05-12T23:12:23.644Z","avatar_url":"https://github.com/garethjns.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Deep multi-sensory integration models\n![Tests](https://github.com/garethjns/MSIModels/workflows/Tests/badge.svg) [![Quality Gate Status](https://sonarcloud.io/api/project_badges/measure?project=garethjns_MSIModels\u0026metric=alert_status)](https://sonarcloud.io/dashboard?id=garethjns_MSIModels)\n\nTrying out early and late integration in multi-input neural networks, for event detection and rate discrimination.\n\nContents:  \n1) [Setup](#setup)  \n2) [Background](#background)  \n3) [Main experiment results](#main_experiment_results)  \n4) [Custom experiments](#custom_experiments)  \n\n# Setup \u003ca name=\"setup\"\u003e\u003c/a\u003e\n```bash\ngit clone https://github.com/garethjns/MSIModels.git\ncd MSIModels\npip install -r requirements.txt\n```\n\n## Run experiment\n\n### Python\nRun main experiment:\n```bash\npython3 run.py\n```\n\nOr, Run smaller example experiment:\n```bash\npython3 -m scripts.run_full_experiment\n```\n\n#### View results\n1) Run MLflow server. Use a Linux VM or WSL if running project in Windows.\n    ````bash\n    mlflow ui\n    ````\n2) View at http://localhost:5000\n3) And \"contained_experiment/_[run_number]\"\n\n\n### Docker container\nBuild the docker container:\n```bash\ndocker build . -t msimodels\n```\nAnd run to access results folder at /tmp/msi and mlflow server at http://localhost:5000 \n```\nmkdir /tmp/msi/\ndocker run -p 5000:5000 --mount type=bind,source=/tmp/msi,target=/contained_experiment msimodels\n```\n\n# Background \u003ca name=\"background\"\u003e\u003c/a\u003e\nThe human brain integrates incoming information over time, and combines evidence between different sensory modalities. Humans are able to use sensory information in a statistically-optimal fashion; reliable information is weighted as more important than unreliable or noisy information. However, exactly how and where the brain combines modalities is unclear, with multisensory processing occurring early in the cortex in cortices traditionally believed to be unisensory.\n\nThis project sets up analogies of two models of evidence integration - early and late - in ANNs and has two main aims:\n - To see how performance varies with \"where\" the integration occurs.\n - Compare the results to those observed in mammals, particularly with regard to \n    - The effect of temporal asynchrony between sensory modalities.\n    - Which parts of the stimuli are most import for decision making.\n\n### Early vs late integration\nIn early integration models, multisensory processing first occurs in sensory cortices (not necessarily exclusively). In late integration models, modalities are processed separately by the appropriate sensory cortices and then combined in later, decision making cortices (such as parietal and frontal) [[1](http://www.sciencedirect.com/science/article/pii/S0959438816300678)].  \n\nBehavioural evidence implies late integration occurs [[2](http://www.jneurosci.org/content/32/11/3726.short)], anatomical evidence tends to imply integration occurs early and late.\n\n## The task\nGiven noisy \"sensory\" input, the aim is to categorise the stimuli as \"fast\" or \"slow\", depending on the number of events in the input.\n\nIn the psychosocial version of this task, the appropriate fast/slow threshold is learned by the subject, and events are, for example, light flashes or tone pips that are embedded in a noisy background. \n\nFor the neural networks, the events are the same, and the task is binary classification with a decision threshold set according to the mean of the known stim rates. In order to reach the final decision, each each individual component of the network performance sub-tasks such as event detection, rate estimation, etc.  These losses are also monitored.\n\n## Stimuli\nThe stimuli used are based on those used in a number of psychophysical tasks (eg. [[2](http://www.jneurosci.org/content/32/11/3726.short), [3](http://www.eposters.net/poster/exploring-the-role-of-synchrony-in-auditory-visual-integration-in-ferrets-and-humans), [4](http://www.eposters.net/poster/exploring-the-role-of-synchrony-in-spatial-and-temporal-auditory-visual-integration-1)]). The stimuli are either \"unisensory\" (single channel input) or \"multisensory\" (two-channel input).   \n\nEach channel consists a number of events embedded in a noisy background. These events are separated by gaps of two discrete lengths. For comparison, in a auditory and visual psycphysics task, these inputs are analagous to the raw voltage traces used to drive speakers (AC) and LEDs (rectified). In this case, the\nevent might be a short flash (~20 ms) for the visual channel, and a a short tone pip (~20 ms) for the auditory channel.\n\nIn the multisensory case, the two channels can be:\n - Matched: The same number of events on each channel\n   - Asynchronous: Matched, but events between channel temporally aligned between channels\n   - Synchronous: Matched AND the events align temporally between channels\n - Unmatched: \n   - Agreeing: Differing number of events on each channel, but indicating the same decision (ie. both \"fast\" or both \"slow\")\n   - Conflicting: Differing number of events on each channel, and indicating contradictory decisions (ie. one \"fast, one \"slow\")\n\n![Example stim](images/stimExample.png) \n\n## Models\nThe models represent experimental subjects that use different strategies to make their decision. They have two inputs and multiple outputs, and the total number of parameters in each model is roughly matched at ~160,000.\n\n### Inputs\n2 Channels, nominally \"left\" and \"right\" or \"auditory\" or \"visual\" to reuse terminology from the psychophysics task. Each channel takes input similar to the voltage trace stimuli for the auditory stimuli plotted above, rather than the square wave signal designed for use with LEDs. \n\nEach channel connects to an branch containing 1D convolutional layers. Where this branch merges defines the model type (see below).\n\n### Output\nThere are two main outputs for the network, a binary \"fast\" or \"slow\" decision and a linear \"rate\" estimation. By default the loss weights are 50/50 for these. The binary decision is used to evaluate model performance.\n\nThere are additional outputs at the layers before the input branches merged. By default the loss weights for these are set to 0, and these outputs have no effect on training. Modifying the weights of the outputs allows creation of models that either focus on the final decision, or the intermediate event detection stages. See also [model debug](#model_debug)\n\n### Architectures\n#### Early\n![Early integration model architecture](images/mod_early.png) \n#### Intermediate\n![Intermediate integration model architecture](images/mod_intermediate.png) \n#### Late\n![Late integration model architecture](images/mod_late.png) \n\n# Main experiment results \u003ca name=\"main_experiment_results\"\u003e\u003c/a\u003e  \nThe main experiment is defined in run.py, which can either be run using Python, or built into a Docker container which will also run the results server.\n\n```bash\npython3 run.py\n```\n\n## Experimental setup\n - 3 Model types - early, intermediate, late integration (see above)\n - 4 datasets (50k rows each) - easy, medium, hard, very hard (see below) and containing:\n     - 5 Types of stimuli \n        1) Left channel only\n        2) Right channel only\n        3) Multisensory matched, sync\n        4) Multisensory matched, async\n        5) Multisensory unmatched, async\n  - 3 Repeats of each model on each dataset\n\n## Experimental stimuli\n### Types\nExamples from the \"easy\" set.  \n#### Type 1: Left channel only\n![example_type_1](images/contained_experiment/easy/example_type_1.png)  \n#### Type 2: Right channel only\n![example_type_2](images/contained_experiment/easy/example_type_2.png)\n#### Type 3: Multisensory matched, sync \n![example_type_3](images/contained_experiment/easy/example_type_3.png)  \n#### Type 4: Multisensory matched, async \n![example_type_4](images/contained_experiment/easy/example_type_4.png)\n#### Type 5: Multisensory unmatched, async \n![example_type_5](images/contained_experiment/easy/example_type_5.png)  \n\n### Difficulty\n4 levels ranging from easy to hard. Examples of type 3 at each difficulty:\n#### Easy\n![example_easy](images/contained_experiment/easy/example_type_3.png)\n#### Medium  \n![example_medium](images/contained_experiment/medium/example_type_3.png)\n#### Hard  \n![example_hard](images/contained_experiment/hard/example_type_3.png)\n#### Very hard  \n![example_very_hard](images/contained_experiment/very_hard/example_type_3.png)  \n\n### Overall model accuracy\n![Model accuracy](images/contained_experiment/graphs/dt_test.png)  \nAveraging over all stimuli types, early model is most accurate on all datasets. Increasing difficulty (stim noise) decreases performance and variability. Intermediate and late models perform similarly.\n\n| Model | Dataset: Easy | Medium | Hard | Very hard |  \n|---|---|---|---|---|\n| Early | 0.985 ± 0.002 | 0.945 ± 0.005 | 0.894 ± 0.004 | 0.672 ± 0.032 |\n| Intermediate | 0.951 ± 0.004 | 0.888 ± 0.005 | 0.832 ± 0.017 | 0.652 ± 0.020 |\n| Late | 0.946 ± 0.003 | 0.885 ± 0.007 | 0.825 ± 0.008 | 0.636 ± 0.015 |\n\n### Discrimination thresholds\nDiscrimination thresholds are calculated from the psychometric curve fits, see [PsychometricCurveFitting](https://github.com/garethjns/PsychometricCurveFitting) for more info. Intuitively, the lower this threshold, the fewer events are required to flip a decision from fast to slow.  \n\n#### Unisensory: Types 1 and 2\n![Discrimination threshold for type1](images/contained_experiment/graphs/dt_test_type1.png) ![Discrimination threshold for type2](images/contained_experiment/graphs/dt_test_type2.png)  \nPerformance for all model types is consistent across types - this is expected. Stimuli on each channel are generated in the same way, and models are symmetrical, so inconsistency in performance would indicate a problem. Similar to decision accuracy, performance decrease (discrimination threshold increases) as difficulty increases. Variability also increases. The early integration model generally performs best.\n\n#### Multisensory: Types 3\n![Discrimination threshold for type3](images/contained_experiment/graphs/dt_test_type3.png)    \nAll 3 models perform between on type 3 stimuli than type 2. They are able to extract more useful information from the stimuli when it is presented on two channels synchronously (note, events are synchronous across channels, but background noise is independent). The early integration model performances best across all difficulties.\n\n#### Multisensory: Type 4\n![Discrimination threshold for type4](images/contained_experiment/graphs/dt_test_type4.png)  \nAll 3 models perform between on type 4 stimuli than type 2. They are able to extract more useful information from the stimuli when it is presented on two channels asynchronously (note, there are the same number of events in each channel, but they are presented asynchronously between channels, and background noise is independent). Again, the early integration model performances best across all difficulties.  Performance between type 3 and 4 is similar. \n\n### Psychometric curves\n#### Early integration model\n![Early integration on easy data](images/contained_experiment/easy/early_integration/aggregated_results.png)\n![Early integration on medium data](images/contained_experiment/medium/early_integration/aggregated_results.png)  \n![Early integration on hard data](images/contained_experiment/hard/early_integration/aggregated_results.png)  \n![Early integration on very hard data](images/contained_experiment/very_hard/early_integration/aggregated_results.png)  \n\n#### Intermediate integration model\n![Intermediate integration on easy data](images/contained_experiment/easy/intermediate_integration/aggregated_results.png)  \n![Intermediate integration on medium data](images/contained_experiment/medium/intermediate_integration/aggregated_results.png)  \n![Intermediate integration on hard data](images/contained_experiment/hard/intermediate_integration/aggregated_results.png)  \n![Intermediate integration on very hard data](images/contained_experiment/very_hard/intermediate_integration/aggregated_results.png)  \n\n#### Late integration model\n![Late integration on easy data](images/contained_experiment/easy/late_integration/aggregated_results.png)  \n![Late integration on medium data](images/contained_experiment/medium/late_integration/aggregated_results.png)  \n![Late integration on hard data](images/contained_experiment/hard/late_integration/aggregated_results.png)  \n![Late integration on very hard data](images/contained_experiment/very_hard/late_integration/aggregated_results.png)  \n\n#### Psychometric curves summary\nDiscrimination threshold averaged over types and repetitions.  \n\n| Model | Dataset: Easy | Medium | Hard | Very hard |  \n|---|---|---|---|---|\n| Early integration | 0.234 ± 0.013 | 0.42 ± 0.077 | 0.492 ± 0.036 | 3.025 ± 0.536|\n| Intermediate integration | 0.427 ± 0.027 | 0.852 ± 0.116 | 1.159 ± 0.051 | 3.72 ± 0.424 |\n| Late integration | 0.455 ± 0.007 | 0.677 ± 0.08 | 1.02 ± 0.052 | 4.104 ± 0.189 |\n\n# Custom experiments \u003ca name=\"custom_experiments\"\u003e\u003c/a\u003e\nThe main experimental entrypoint is the Experiment object and is the best place to start. It can be used to fairly-flexibly define experiments and handles a lot of house keeping. The lower level objects are also public and usable independently for more customised experiments. \n\nRoughly the structure of an Experiment is:\n\n - Experiment (class)\n   - ExperimentalResults\n     - Methods fo plotting and evaluation (comparison across model and data types)\n   - ExperimentalRun\n     - ExperimentalModel (x n repeats)\n       - MultisensoryModel of type early, intermediate, or late integration\n        - Single Model plotting and evaluation methods \n     - ExperimentalDataset (x 1)\n       - Data summary and plotting methods\n       - MultiChannel\n         - Methods stimuli loading, splitting between train, test, etc.\n   - ExperimentalRunResults\n     - Methods fo plotting and evaluation (aggregated over n runs or \"subjects\")\n\nSee scripts/ for example usage of each object, and below (although this may be slightly out of date)\n\n## Experiment class\nThe Experiment class is constructed from ExperimentalModels and ExperimentalDatasets. For each, it generates ExperientalRuns to handle running multiple repeats.  \n\nResults are persisted to mlflow into two experiments:\n - *[name]_summary*\n   - Contains summary results, with a run for each model.\n - *[name]*\n   - Contains a run for every stimulus type, for every subject.\n\nAnd into the folder [experiment_name]/_[run number]/\n\nSmall example (scripts/run_full_experiment.py):\n```python\nfrom msi_models.experiment.experiment import Experiment\nfrom msi_models.experiment.experimental_model import ExperimentalModel\nfrom msi_models.models.conv.multisensory_classifier import MultisensoryClassifier\nfrom msi_models.experiment.experimental_dataset import ExperimentalDataset\n\nN_REPS = 5\nN_EPOCHS = 2000\nN_DATA_ROWS = 5000\n\n# Prepare experiment\nexp = Experiment(name='scripts_example_experiment', n_epochs=N_EPOCHS, n_reps=N_REPS)\n\n# Prepare and add data (data doesn't need to have been pre-generated)\nfor exp_data in [ExperimentalDataset(\"scripts_example_easy\",\n                                     n=N_DATA_ROWS, difficulty=12).build(\"data/scripts_example_mix_easy.hdf5\"),\n                 ExperimentalDataset(\"scripts_example_hard\",\n                                     n=N_DATA_ROWS, difficulty=12).build(\"data/scripts_example_mix_hard.hdf5\")]:\n    exp.add_data(exp_data)\n\n# Prepare and add models\ncommon_model_kwargs = {'opt': 'adam', 'batch_size': int(min(N_DATA_ROWS / 10, 15000)), 'lr': 0.01}\nfor int_type in ['early_integration', 'intermediate_integration', 'late_integration']:\n    mod = ExperimentalModel(MultisensoryClassifier(integration_type=int_type, **common_model_kwargs), name=int_type)\n    exp.add_model(mod)\n\n# Run experiment\nexp.run()\n```\n\n## Data preparation\nIndividual stimuli are constructed with [AudioDAG](https://github.com/garethjns/AudioDAG) and combined into datasets using msi_models.stimset.channel and msi_models.stimset.multi_channel objects.\n\nTemplates are available for different stimulus components, types and combinations of stimuli. The typically specify compatible defaults for the params, which can be modified.\n\nMultiTwoGapStim.generate creates a .hdf5 file with the following structure:\n```\n    |--left/\n            |--x (n, duration_pts, 1)\n            |--x_indicators (n, duration_pts, 1)\n            |--y_rate (n,)\n            |--y_dec (n,)\n            |--configs \n    |--right/\n            |--x (n, duration_pts, 1)\n            |--x_indicators (n, duration_pts, 1)\n            |--y_rate (n,)\n            |--y_dec (n,)\n            |--configs \n    /--agg/\n           |--y_rate\n           |--y_dec\n           |--summary (df)  \n           |--summary_train (df)  \n           |--summary_train (df)\n```\n\nGenerating multisensory (two channel) using templates:\n\n```python\nfrom msi_models.stim.multi_two_gap.multi_two_gap_stim import MultiTwoGapStim\nfrom msi_models.stim.multi_two_gap.multi_two_gap_template import MultiTwoGapTemplate\n\nMultiTwoGapStim.generate(templates=MultiTwoGapTemplate['matched_async'], n_jobs=-1,\n                         fs=500, n=200, batch_size=10, fn='data/sample_multisensory_data_sync.hdf5',\n                         template_kwargs={\"duration\": 1300,\n                                          \"background_mag\": 0.09,\n                                          \"duration_tol\": 0.5})\n```\n\nThis data is read using Channel definitions, and the MultiChannel object for combining channels:\n\n````python\nfrom msi_models.stimset.channel_config import ChannelConfig\nfrom msi_models.stimset.multi_channel import MultiChannel\nfrom msi_models.stimset.multi_channel_config import MultiChannelConfig\n\n# Create channel configs to load for /left/ and /right/ keys\npath = 'data/sample_multisensory_data_sync.hdf5'\ncommon_kwargs = {\"path\": path, \"train_prop\": 0.8, \"seed\": 100,\n                 \"x_keys\": [\"x\", \"x_mask\"], \"y_keys\": [\"y_rate\", \"y_dec\"]}\nleft_config = ChannelConfig(key='left', **common_kwargs)\nright_config = ChannelConfig(key='right', **common_kwargs)\n\n# Combine the channels and define aggregate (/agg/) key to use as y(s)\nmulti_config = MultiChannelConfig(path=path, key='agg', y_keys=[\"y_rate\", \"y_dec\"],\n                                  channels=[left_config, right_config])\nmc = MultiChannel(multi_config)\n\n# View some examples\n````\n\nAdditional examples:  \n - scripts/generate_single_stim_from_template_examples - example stims  \n - scripts/generate_single_type_multisensory_data - generate data to files  \n - scripts/generate_multi_type_multisensory_data - generate multiple datatypes to one file to create training and test sets  \n - scripts/generate_unisensory_data.py - generate unisensory data  \n\n### ExperimentalDataset\nThe ExperimentalDataset class specifies a dataset for use in experiments. It will automatically generate the data file if it doesn't exist and the appropriate Multichannel object to read and partition the data.\n\neg. \n```python\nfrom msi_models.experiment.experimental_dataset import ExperimentalDataset\n\nexp_data = ExperimentalDataset(\"example_mix\", n=500, difficulty=70).build(\"data/example_mix.hdf5\")\n\nexp_data.mc.plot_summary()\nexp_data.mc.plot_example()\nexp_data.mc.plot_example()\n```\n\n## Model preparation\n\n### Single model fitting\nModel templates are available in msi_models.models.conv.multisensory_templates, and define combinations of loss weights to create a binary classifier, event detector, etc. MultisensoryClassifier creates a model aims to correctly predict rate and the \"fast\"/\"slow\" decision.\n\nThe MultisensoryClassifier uses the agg/y_rate and agg/y_dec as output, targets, but also calculates the loss for the against the unisensory targets, eg left/y_rate and left/y_dec so the performance of the individual outputs can also be monitored.\n\nThe channels are first defined, then combined into a MultiChannel object to feed the model with pre-generated data (stored in hdf5), see generations scripts above.\n\nExamples of usage are shown in:\nAdditional examples:  \n - scripts/train_unisensory_model.py   \n - scripts/train_multisensory_model.py    \n\n## Model debug \u003ca name=\"custom_experiments\"\u003e\u003c/a\u003e\n![model debug plot](images/intermediate_mod_debug.png)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgarethjns%2Fmsimodels","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgarethjns%2Fmsimodels","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgarethjns%2Fmsimodels/lists"}