{"id":15902669,"url":"https://github.com/lromul/sensorium","last_synced_at":"2026-03-06T06:03:44.055Z","repository":{"id":207945683,"uuid":"656909310","full_name":"lRomul/sensorium","owner":"lRomul","description":"NeurIPS | 1st place solution for Sensorium 2023 Competition","archived":false,"fork":false,"pushed_at":"2023-11-22T09:25:26.000Z","size":1935,"stargazers_count":28,"open_issues_count":0,"forks_count":5,"subscribers_count":1,"default_branch":"master","last_synced_at":"2026-02-07T09:49:45.567Z","etag":null,"topics":["deep-learning","neural-prediction","neuroscience","pytorch","video","visual-cortex"],"latest_commit_sha":null,"homepage":"https://www.sensorium-competition.net","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lRomul.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2023-06-21T22:42:34.000Z","updated_at":"2025-07-15T09:48:41.000Z","dependencies_parsed_at":"2023-11-18T17:20:20.798Z","dependency_job_id":"ae3b86b0-bb35-426f-9e0a-6ac09b14de4e","html_url":"https://github.com/lRomul/sensorium","commit_stats":null,"previous_names":["lromul/sensorium"],"tags_count":290,"template":false,"template_full_name":null,"purl":"pkg:github/lRomul/sensorium","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lRomul%2Fsensorium","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lRomul%2Fsensorium/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lRomul%2Fsensorium/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lRomul%2Fsensorium/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lRomul","download_url":"https://codeload.github.com/lRomul/sensorium/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lRomul%2Fsensorium/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30164532,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-06T04:43:31.446Z","status":"ssl_error","status_checked_at":"2026-03-06T04:40:30.133Z","response_time":250,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deep-learning","neural-prediction","neuroscience","pytorch","video","visual-cortex"],"created_at":"2024-10-06T11:42:20.258Z","updated_at":"2026-03-06T06:03:43.088Z","avatar_url":"https://github.com/lRomul.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n\n# Solution for Sensorium 2023 Competition\n\n[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)\n[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.10155151.svg)](https://doi.org/10.5281/zenodo.10155151)\n\n\u003c/div\u003e\n\nThis repository contains the winning solution for the Sensorium 2023, part of the NeurIPS 2023 competition track.\nThe competition aimed to find the best model that can predict the activity of neurons in the primary visual cortex of mice in response to videos.\nThe competition introduced a temporal component using dynamic stimuli (videos) instead of static stimuli (images) used in Sensorium 2022, making the task more challenging.\n\nThe primary metric of the competition was a single trial correlation.\nYou can read about the metric, the data, and the task in the competition paper [^1].\nIt is important to note that additional data for five mice was introduced during the competition, which doubled the dataset's size ([old](https://gin.g-node.org/pollytur/Sensorium2023Data) and [new](https://gin.g-node.org/pollytur/sensorium_2023_dataset) data).\n\n## Solution\n\nKey points:\n* [DwiseNeuro](src/models/dwiseneuro.py) - novel model architecture for prediction neural activity in the mouse primary visual cortex.\n* Solid cross-validation strategy with splitting folds by perceptual video hash.\n* Training on all mice with an option to fill unlabeled samples via distillation.\n\n## Model Architecture\n\nDuring the competition, I dedicated most of my time to designing the model architecture since it significantly impacted the solution's outcome compared to other components.\nI iteratively tested various computer vision and deep learning techniques, integrating them into the architecture as the correlation metric improved.\n\nThe diagram below illustrates the final architecture, which I named DwiseNeuro:\n\n![architecture](data/readme_images/architecture.png)\n\nDwiseNeuro consists of three main parts: core, cortex, and readouts.\nThe core consumes sequences of video frames and mouse behavior activity in separate channels, processing temporal and spatial features.\nProduced features pass through global pooling over spatial dimensions to aggregate them.\nThe cortex processes the pooled features independently for each timestep, significantly increasing the channels.\nFinally, each readout predicts the activation of neurons for the corresponding mouse.\n\nIn the following sections, we will delve deeper into each part of the architecture.\n\n### Core\n\nThe first layer of the module is the stem. It's a point-wise 3D convolution for increasing the number of channels, followed by batch normalization.\nThe rest of the core consists of inverted residual blocks [^2][^3] with a `narrow -\u003e wide -\u003e narrow` channel structure. \n\n#### Techniques\n\nSeveral methods were added to the inverted residual block rewritten in 3D layers:\n* **Absolute Position Encoding** [^4] - summing the encoding to the input of each block allows convolutions to accumulate position information. It's quite important because of the subsequent spatial pooling after the core.\n* **Factorized (2+1)D convolution** [^5] - 3D depth-wise convolution was replaced with a spatial 2D depth-wise convolution followed by a temporal 1D depth-wise convolution. There are spatial convolutions with stride two in some blocks to compress output size.\n* **Shortcut Connections** - completely parameter-free residual shortcuts with three operations: \n    * Identity mapping if input and output dimensions are equal. It's the same as the connection proposed in ResNet [^6].\n    * Nearest interpolation in case of different spatial sizes. \n    * Cycling repeating of channels if they don't match.\n* **Squeeze-and-Excitation** [^7] - dynamic channel-wise feature recalibration.\n* **DropPath (Stochastic Depth)** [^8][^9] - regularization that randomly drops the block's main path for each sample in batch.\n\nBatch normalization is applied after each layer, including the shortcut. \nSiLU activation is used after expansion and depth-wise convolutions.\n\n#### Hyperparameters\n\nI found that the number of core blocks and their parameters dramatically affect the outcome.\nIt's possible to tune channels, strides, expansion ratio, and spatial/temporal kernel sizes.\nObviously, it is almost impossible to start experiments with optimal values.\nThe problem is mentioned in the EfficientNet [^3] paper, which concluded that it is essential to carefully balance model width, depth, and resolution.\n\nAfter conducting a lot of experiments, I chose the following parameters:\n* Four blocks with 64 output channels, three with 128, and two with 256.\n* Three blocks have stride two. They are the first in each subgroup from the point above.\n* Expansion ratio of the inverted residual block is six.\n* Kernel of spatial depth-wise convolution is (1, 3, 3).\n* Kernel of temporal depth-wise convolution is (5, 1, 1).\n\n### Cortex\n\nCompared with related works [^10][^11], the model architecture includes a new part called the cortex.\nIt is also common for all mice as the core.\nThe cortex receives features with channels and temporal dimensions only.\nSpatial information was accumulated through position encoding applied earlier in the core and compressed by average pooling after the core.\nThe primary purpose of the cortex is to smoothly increase the number of channels, which the readouts will further use.\n\nThe building element of the module is a grouped 1D convolution followed by the channel shuffle operation [^12].\nBatch normalization, SiLU activation, and shortcut connections with stochastic depth were applied similarly to the core.\n\n#### Hyperparameters\n\nHyperparameters of the cortex were also important:\n* Convolution with two groups and kernel size one (bigger kernel size over temporal dimension has not led to better results).\n* Three layers with 1024, 2048, and 4096 channels.\n\nAs you can see, the number of channels is quite large.\nGroups help optimize computation and memory efficiency.\nChannel shuffle operation allows the sharing of information between groups of different layers.\n\n### Readouts\n\nThe readout is a single 1D convolution with two groups and kernel size one, followed by Softplus activation.\nEach of the ten mice has its readout with the number of output channels equal to the number of neurons (7863, 7908, 8202, 7939, 8122, 7440, 7928, 8285, 7671, 7495, respectively).\n\n#### Softplus\n\nKeeping the response positive by using Softplus [^10] was essential in my pipeline.\nIt works much better than `ELU + 1` [^11], especially with tuning the Softplus beta parameter.\nIn my case, the optimal beta value was about 0.07, which resulted in a 0.018 increase in the correlation metric.\n\nYou can see a comparison of `ELU + 1` and Softplus in the plot below:\n\n![softplus](data/readme_images/softplus.png)\n\n#### Learnable Softplus\n\nI also conducted an experiment where the beta parameter was trainable.\nInterestingly, the trained value converged approximately to the optimal, which I found by grid search.\nI omitted the learnable Softplus from the solution because it resulted in a slightly worse score.\nBut this may be an excellent way to quickly and automatically find a good beta.\n\nHere's a numerical stable implementation of learnable Softplus in PyTorch:\n\n```Python\nimport torch\nfrom torch import nn\n\n\nclass LearnableSoftplus(nn.Module):\n    def __init__(self, beta: float):\n        super().__init__()\n        self.beta = nn.Parameter(torch.tensor(float(beta)))\n\n    def forward(self, x):\n        xb = x * self.beta\n        return (torch.clamp(xb, 0) + torch.minimum(xb, -xb).exp().log1p()) / self.beta\n```\n\n## Training\n\n### Validation strategy\n\nAt the end of the competition, I employed 7-fold cross-validation to check hypotheses and tune hyperparameters more precisely.\nI used all available labeled data to make folds.\nRandom splitting gave an overly optimistic metric estimation because some videos were duplicated (e.g., in the original validation split or between old and new datasets).\nTo solve this issue, I created group folds with non-overlapping videos.\nSimilar videos were found using [perceptual hashes](src/phash.py) of several frames fetched deterministically.\n\n### Basic training ([config](configs/true_batch_001.py))\n\nThe training was performed in two stages. The first stage is basic training with the following pipeline parameters:\n* Learning rate warmup for the first three epochs from 0 to 2.4e-03, cosine annealing last 18 epochs to 2.4e-05\n* Batch size 32, one training epoch comprises 72000 samples\n* Optimizer AdamW [^13] with weight decay 0.05\n* Poisson loss\n* Model EMA with decay 0.999\n* CutMix [^14] with alpha 1.0 and usage probability 0.5\n* The sampling of different mice in the batch is random by uniform distribution\n\nEach dataset sample consists of a grayscale video, behavior activity measurements (pupil center, pupil dilation, and running speed), and the neuron responses of a single mouse.\nAll data is presented at 30 FPS. During training, the model consumes 16 frames, skipping every second (equivalent to 16 neighboring frames at 15 FPS).\nThe video frames were zero-padded to 64x64 pixels. The behavior activities were added as separate channels.\nThe entire tensor channel was filled with the value for each behavior measurement.\nNo normalization is applied to the target and input tensors during training.\n\nThe ensemble of models from all folds gets 0.2905 single-trial correlation on the main track and 0.2207 on the bonus track in the final phase of the competition.\nThis result would be enough to take first place in the main and bonus (out-of-distribution) competition tracks.\n\n### Knowledge Distillation ([config](configs/distillation_001.py))\n\nFor an individual sample in the batch, the loss was calculated for the responses of only one mouse.\nBecause the input tensor is associated with a single mouse trial, and there are no neural activity data for other mice.\nHowever, the model can predict responses for all mice from the input tensor. In the second stage of training, I used a method similar to knowledge distillation [^15].\nI created a pipeline where models from the first stage predict unlabeled responses during training.\nAs a result, the second-stage models trained all their readouts via each batch sample.\nThe loss value on distilled predictions was weighed to be 0.36% of the overall loss.\n\nThe hyperparameters were identical, except for the expansion ratio in inverted residual blocks: seven in the first stage and six in the second.\n\nIn the second stage, the ensemble of models achieves nearly the same single-trial correlation as the ensemble from the first stage.\nHowever, what is fascinating is that each fold model performs better by an average score of 0.007 than the corresponding model from the first stage.\nThus, the distilled model works like an ensemble of undistilled models.\nAccording to the work [^16], the individual model is forced to learn the ensemble's performance during knowledge distillation, and an ensemble of distilled models offers no more performance boost.\nI can observe the same behavior in my solution.\n\nDistillation can be a great practice if you need one good model.\nBut in ensembles, this leads to minor changes in performance.\n\n## Prediction\n\nThe ensembles were produced by taking the arithmetic mean of predictions from multiple steps:\n* Overlapping a sliding window over each possible sequence of frames.\n* Models from cross-validations of one training stage.\n* Training stages (optional).\n\nI used the same model weights for both competition tracks.\nThe competition only evaluated the responses of five mice from the new dataset, so I only predicted those.\n\n## Competition progress\n\nYou can see the progress of solution development during the competition in [spreadsheets](https://docs.google.com/spreadsheets/d/1xJTB6lZvtjNSQbYQiB_hgKIrL9hUB0uL3xhUAK1Pmxw/edit#gid=0) (the document consists of multiple sheets).\nUnfortunately, the document contains less than half of the conducted experiments because sometimes I was too lazy to fill it :)\nHowever, if you need a complete chronology of valuable changes, you can see it in git history.\n\nTo summarize, an early model with depth-wise 3D convolution blocks achieved a score of around 0.19 on the main track during the live phase of the competition. \nSubsequently, implementing techniques from the core section, tuning hyperparameters, and training on all available data boosted the score to 0.25.\nApplying non-standard normalization, expected by evaluation servers, on postprocessing improved the score to 0.27.\nThe cortex and CutMix increased the score to 0.276. \nThen, the beta value of Softplus was tuned, resulting in a score of 0.294.\nLastly, adjusting drop rate and batch size parameters helped to achieve a score of 0.3 on the main track during the live phase.\n\nThe ensemble of the basic and distillation training stages achieved a single-trial correlation of 0.2913 on the main track and 0.2215 on the bonus track in the final phase (0.3005 and 0.2173 in the live phase, respectively).\nThis result is just a bit better than the basic training result alone, but I should provide it because it was the best submission in the competition.\nIn addition, it was interesting to research the relation between ensembling and distillation, which I wrote above.\n\nThanks to the Sensorium organizers and participants for the excellent competition. Thanks to my family and friends who supported me during the competition!\n\n\u003c!-- ## References --\u003e\n\n[^1]: Turishcheva, Polina, et al. (2023). The Dynamic Sensorium competition for predicting large-scale mouse visual cortex activity from videos. https://arxiv.org/abs/2305.19654  \n[^2]: Sandler, Mark, et al. (2018). Mobilenetv2: Inverted residuals and linear bottlenecks. https://arxiv.org/abs/1801.04381  \n[^3]: Tan, Mingxing, and Quoc Le. (2019). Efficientnet: Rethinking model scaling for convolutional neural networks. https://arxiv.org/abs/1905.11946  \n[^4]: Vaswani, Ashish, et al. (2017). Attention is all you need. https://arxiv.org/abs/1706.03762  \n[^5]: Tran, Du, et al. (2018). A closer look at spatiotemporal convolutions for action recognition. https://arxiv.org/abs/1711.11248  \n[^6]: He, Kaiming, et al. (2016). Deep residual learning for image recognition. https://arxiv.org/abs/1512.03385  \n[^7]: Hu, Jie, Li Shen, and Gang Sun. (2018). Squeeze-and-excitation networks. https://arxiv.org/abs/1709.01507  \n[^8]: Larsson, Gustav, Michael Maire, and Gregory Shakhnarovich. (2016). Fractalnet: Ultra-deep neural networks without residuals. https://arxiv.org/abs/1605.07648  \n[^9]: Huang, Gao, et al. (2016). Deep networks with stochastic depth. https://arxiv.org/abs/1603.09382  \n[^10]: Höfling, Larissa, et al. (2022). A chromatic feature detector in the retina signals visual context changes. https://www.biorxiv.org/content/10.1101/2022.11.30.518492  \n[^11]: Lurz, Konstantin-Klemens, et al. (2020). Generalization in data-driven models of primary visual cortex. https://www.biorxiv.org/content/10.1101/2020.10.05.326256  \n[^12]: Zhang, Xiangyu, et al. (2018). Shufflenet: An extremely efficient convolutional neural network for mobile devices. https://arxiv.org/abs/1707.01083  \n[^13]: Loshchilov, Ilya, and Frank Hutter. (2017). Decoupled weight decay regularization. https://arxiv.org/abs/1711.05101  \n[^14]: Yun, Sangdoo, et al. (2019). Cutmix: Regularization strategy to train strong classifiers with localizable features. https://arxiv.org/abs/1905.04899  \n[^15]: Hinton, Geoffrey, Oriol Vinyals, and Jeff Dean. (2015). Distilling the knowledge in a neural network. https://arxiv.org/abs/1503.02531  \n[^16]: Allen-Zhu, Zeyuan, and Yuanzhi Li. (2020). Towards understanding ensemble, knowledge distillation and self-distillation in deep learning. https://arxiv.org/abs/2012.09816  \n\n## Quick setup and start\n\n### Requirements\n\n* Linux (tested on Ubuntu 20.04 and 22.04)\n* NVIDIA GPU (models trained on RTX A6000)\n* NVIDIA Drivers \u003e= 535, CUDA \u003e= 12.2\n* [Docker](https://docs.docker.com/engine/install/)\n* [NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html)\n\nPipeline tuned for training on a single RTX A6000 with 48 GB.\nIn the case of GPU with less memory, you can use gradient accumulation by increasing the `iter_size` parameter in training configs.\nIt will worsen the result (by a 0.002 score for `\"iter_size\": 2`), but it has less than the effect of reducing the batch size.\n\n### Run\n\nClone the repo and enter the folder.\n\n```bash\ngit clone git@github.com:lRomul/sensorium.git\ncd sensorium\n```\n\nBuild a Docker image and run a container.\n\n\u003cdetails\u003e\u003csummary\u003eHere is a small guide on how to use the provided Makefile\u003c/summary\u003e\n\n```bash\nmake  # stop, build, run\n\n# do the same\nmake stop\nmake build\nmake run\n\nmake  # by default all GPUs passed\nmake GPUS=all  # do the same\nmake GPUS=none  # without GPUs\n\nmake run GPUS=2  # pass the first two GPUs\nmake run GPUS='\\\"device=1,2\\\"'  # pass GPUs numbered 1 and 2\n\nmake logs\nmake exec  # run a new command in a running container\nmake exec COMMAND=\"bash\"  # do the same\nmake stop\n```\n\n\u003c/details\u003e\n\n```bash\nmake\n```\n\nFrom now on, you should run all commands inside the docker container.\n\nIf you already have the Sensorium 2023 dataset (148 GB), copy it to the folder `./data/sensorium_all_2023/`.\nOtherwise, use the script for downloading:\n\n```bash\npython scripts/download_data.py\n```\n\nYou can now reproduce the final results of the solution using the following commands:\n```bash\n# Train\n# The training time is 3.5 days (12 hours per fold) for each experiment on a single A6000\n# You can speed up the process by using the --folds argument to train folds in parallel\n# Or just download trained weights in the section below\npython scripts/train.py -e true_batch_001\npython scripts/train.py -e distillation_001\n\n# Predict\n# Any GPU with more than 6 GB memory will be enough\npython scripts/predict.py -e true_batch_001 -s live_test_main\npython scripts/predict.py -e true_batch_001 -s live_test_bonus\npython scripts/predict.py -e true_batch_001 -s final_test_main\npython scripts/predict.py -e true_batch_001 -s final_test_bonus\npython scripts/predict.py -e distillation_001 -s live_test_main\npython scripts/predict.py -e distillation_001 -s live_test_bonus\npython scripts/predict.py -e distillation_001 -s final_test_main\npython scripts/predict.py -e distillation_001 -s final_test_bonus\n\n# Ensemble predictions of two experiments\npython scripts/ensemble.py -e distillation_001,true_batch_001 -s live_test_main\npython scripts/ensemble.py -e distillation_001,true_batch_001 -s live_test_bonus\npython scripts/ensemble.py -e distillation_001,true_batch_001 -s final_test_main\npython scripts/ensemble.py -e distillation_001,true_batch_001 -s final_test_bonus\n\n# Final predictions will be there\ncd data/predictions/distillation_001,true_batch_001\n```\n\n### Trained model weights\n\nYou can skip the training step by downloading model weights (9.5 GB) using [torrent file](data/experiments.torrent).  \n\nPlace the files in the data directory so that the folder structure is as follows:\n\n```\ndata\n├── experiments\n│   ├── distillation_001\n│   └── true_batch_001\n└── sensorium_all_2023\n    ├── dynamic29156-11-10-Video-8744edeac3b4d1ce16b680916b5267ce\n    ├── dynamic29228-2-10-Video-8744edeac3b4d1ce16b680916b5267ce\n    ├── dynamic29234-6-9-Video-8744edeac3b4d1ce16b680916b5267ce\n    ├── dynamic29513-3-5-Video-8744edeac3b4d1ce16b680916b5267ce\n    ├── dynamic29514-2-9-Video-8744edeac3b4d1ce16b680916b5267ce\n    ├── dynamic29515-10-12-Video-9b4f6a1a067fe51e15306b9628efea20\n    ├── dynamic29623-4-9-Video-9b4f6a1a067fe51e15306b9628efea20\n    ├── dynamic29647-19-8-Video-9b4f6a1a067fe51e15306b9628efea20\n    ├── dynamic29712-5-9-Video-9b4f6a1a067fe51e15306b9628efea20\n    └── dynamic29755-2-8-Video-9b4f6a1a067fe51e15306b9628efea20\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flromul%2Fsensorium","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flromul%2Fsensorium","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flromul%2Fsensorium/lists"}