{"id":15662965,"url":"https://github.com/bryanlimy/v1t","last_synced_at":"2025-10-17T11:27:01.080Z","repository":{"id":92146626,"uuid":"505467900","full_name":"bryanlimy/V1T","owner":"bryanlimy","description":"Code for \"V1T: Large-scale mouse V1 response prediction using a Vision Transformer\"","archived":false,"fork":false,"pushed_at":"2024-07-05T08:20:00.000Z","size":22461,"stargazers_count":20,"open_issues_count":0,"forks_count":6,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-09-05T15:47:22.800Z","etag":null,"topics":["deep-learning","neural-response","pytorch","vision-transformer","vit"],"latest_commit_sha":null,"homepage":"https://openreview.net/forum?id=qHZs2p4ZD4","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bryanlimy.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":"CITATION.bib","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-06-20T14:08:20.000Z","updated_at":"2025-08-22T10:08:27.000Z","dependencies_parsed_at":null,"dependency_job_id":"604ed2f1-13bb-4808-bcee-8d808ef50b2a","html_url":"https://github.com/bryanlimy/V1T","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/bryanlimy/V1T","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bryanlimy%2FV1T","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bryanlimy%2FV1T/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bryanlimy%2FV1T/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bryanlimy%2FV1T/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bryanlimy","download_url":"https://codeload.github.com/bryanlimy/V1T/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bryanlimy%2FV1T/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279334053,"owners_count":26150530,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-17T02:00:07.504Z","response_time":56,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deep-learning","neural-response","pytorch","vision-transformer","vit"],"created_at":"2024-10-03T13:35:04.621Z","updated_at":"2025-10-17T11:27:01.074Z","avatar_url":"https://github.com/bryanlimy.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# V\u003cfont size='5'\u003e1\u003c/font\u003eT: Large-scale mouse V1 response prediction using a Vision Transformer\n\nCode for TMLR2023 paper \"[V\u003cfont size='1'\u003e1\u003c/font\u003eT: Large-scale mouse V1 response prediction using a Vision Transformer](https://openreview.net/forum?id=qHZs2p4ZD4)\".\n\n![](misc/images/v1t.png)\n\nAuthors: [Bryan M. Li](https://twitter.com/bryanlimy), [Isabel M. Cornacchia](https://twitter.com/isabelmaria_c), [Nathalie L. Rochefort](https://twitter.com/RochefortLab), [Arno Onken](https://homepages.inf.ed.ac.uk/aonken/)\n\n```bibtex\n@article{\n  li2023vt,\n  title={V1T: large-scale mouse V1 response prediction using a Vision Transformer},\n  author={Bryan M. Li and Isabel Maria Cornacchia and Nathalie Rochefort and Arno Onken},\n  journal={Transactions on Machine Learning Research},\n  issn={2835-8856},\n  year={2023},\n  url={https://openreview.net/forum?id=qHZs2p4ZD4},\n  note={}\n}\n```\n\n## Acknowledgement\n\nWe sincerely thank [Willeke et al.](https://arxiv.org/abs/2206.08666) for organizing the [Sensorium](https://sensorium2022.net/home) challenge and, along with [Franke et al.](https://www.nature.com/articles/s41586-022-05270-3), for making their high-quality large-scale mouse V1 recordings publicly available. This codebase is inspired by [sinzlab/sensorium](https://github.com/sinzlab/sensorium), [sinzlab/neuralpredictors](https://github.com/sinzlab/neuralpredictors) and [sinzlab/nnfabrik](https://github.com/sinzlab/nnfabrik). \n\n## File structure\nThe codebase repository has the following structure. Check [.gitignore](.gitignore) for the ignored files.\n```\nsensorium2022/\n  data/\n    sensorium/\n      static21067-10-18-GrayImageNet-94c6ff995dac583098847cfecd43e7b6.zip\n      ...\n    franke2022/\n      static25311-4-6-ColorImageNet-104e446ed0128d89c639eef0abe4655b.zip\n      ...\n    README.md\n  misc/\n  src/\n    v1t/\n      ...\n  .gitignore\n  README.md\n  setup.sh\n  demo.ipynb\n  submission.py\n  sweep.py\n  train.py\n  ...\n```\n- [`demo.ipynb`](demo.ipynb) demonstrates how to load the best V1T model and inference the Sensorium+ test set, as well as extracting the attention rollout maps.\n- [`sweep.py`](sweep.py) performs hyperparameter tuning using [Weights \u0026 Biases](https://wandb.ai/site).\n- [`train.py`](train.py) contains the model training procedure.\n- [data](data/) store the datasets, please check [data/README.md](data/README.md) for more information.\n- [misc](misc/) contains scripts and notebooks to generate various plots and figures used in the paper.\n- [src/v1t](src/v1t/) contains the code for the main Python package.\n\n## Installation\n- Create a new [conda](https://docs.conda.io/en/latest/miniconda.html) environment in Python 3.10.\n  ```bash\n  conda create -n v1t python=3.10\n  ```\n- Activate `v1t` virtual environment\n  ```bash\n  conda activate v1t\n  ```\n- We have created a [`setup.sh`](setup.sh) script to install the relevant `conda` and `pip` packages for macOS and Ubuntu devices.\n  ```bash\n  sh setup.sh\n  ```\n- Alternatively, you can install PyTorch 2.0 and all the relevant packages with:\n  ```bash\n  # install PyTorch\n  conda install -c pytorch pytorch=2.0 torchvision torchaudio -y\n  # install V1T package\n  pip install -e .\n  ```\n\n## Train model\n- An example command to train a V1T core and Gaussian readout on the Sensorium+ dataset\n  ```bash\n  python train.py --dataset data/sensorium --output_dir runs/v1t_model --core vit --readout gaussian2d --behavior_mode 3 --batch_size 16\n  ```\n- use the `--help` flag to see all available options\n\n## Trained model weight(s)\n- The model weights trained on the Sensorium 2022 dataset and Franke et al. 2022 dataset are available at [huggingface.co/bryanlimy/V1T](https://huggingface.co/bryanlimy/V1T).\n  \n## Visualize training performance\n- The training code `train.py` uses both [TensorBoard](https://www.tensorflow.org/tensorboard) and [Weights \u0026 Biases](https://wandb.ai/site) to log training information.\n  - TensorBoard\n    - Use the following command to monitor training performance with TensorBoard\n      ```bash\n      tensorboard --logdir runs/v1t_model --port 6006\n      ```\n    - Visit `localhost:6006` on your browser\n  - Weights \u0026 Biases\n    - use `--use_wandb` and (optional) `--wandb_group \u003cgroup name\u003e` to enable `wandb` logging.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbryanlimy%2Fv1t","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbryanlimy%2Fv1t","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbryanlimy%2Fv1t/lists"}