{"id":15908290,"url":"https://github.com/jegp/nrf","last_synced_at":"2025-11-17T13:58:37.244Z","repository":{"id":237368251,"uuid":"794364670","full_name":"Jegp/nrf","owner":"Jegp","description":"Neuromorphic spatio-temporal receptive fields","archived":false,"fork":false,"pushed_at":"2025-11-05T12:17:07.000Z","size":807,"stargazers_count":5,"open_issues_count":0,"forks_count":2,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-11-05T14:21:05.890Z","etag":null,"topics":["computer-vision","event-based-camera","neural-network","neuromorphic"],"latest_commit_sha":null,"homepage":"https://jepedersen.dk/posts/202510_nrf/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Jegp.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2024-05-01T01:38:59.000Z","updated_at":"2025-11-05T12:17:11.000Z","dependencies_parsed_at":"2024-10-28T11:53:09.074Z","dependency_job_id":"96a716fd-7a4a-44a4-ab0c-cdf5f1d6bd84","html_url":"https://github.com/Jegp/nrf","commit_stats":{"total_commits":24,"total_committers":3,"mean_commits":8.0,"dds":0.125,"last_synced_commit":"71f2e56e533597c9377744b27f3bb5f0adf7a385"},"previous_names":["jegp/nrf"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/Jegp/nrf","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Jegp%2Fnrf","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Jegp%2Fnrf/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Jegp%2Fnrf/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Jegp%2Fnrf/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Jegp","download_url":"https://codeload.github.com/Jegp/nrf/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Jegp%2Fnrf/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":284893575,"owners_count":27080531,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-11-17T02:00:06.431Z","response_time":55,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["computer-vision","event-based-camera","neural-network","neuromorphic"],"created_at":"2024-10-06T14:20:35.230Z","updated_at":"2025-11-17T13:58:37.238Z","avatar_url":"https://github.com/Jegp.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Covariant spatio-temporal receptive fields for spiking neural networks\n\n[![Nature Communications Paper](https://zenodo.org/badge/DOI/10.1038/s41467-025-63493-0.svg)](https://doi.org/10.1038/s41467-025-63493-0)\n![GitHub Repo stars](https://img.shields.io/github/stars/jegp/nrf)\n\nThis repository contains the code for the paper \"Covariant spatio-temporal receptive fields for spiking neural networks\".\n\n## Video summary (watch on YouTube)\n[![Paper summary video](https://img.youtube.com/vi/qtHkrx4tYfI/0.jpg)](https://www.youtube.com/watch?v=qtHkrx4tYfI)\n\n\n## Introduction\n[Neuromorphic computing](https://en.wikipedia.org/wiki/Neuromorphic) exploits the laws of physics to perform computations, similar to the human brain.\nIf we can \"lower\" the computation into physics, we achieve **extreme energy gains**, [up to 27-35 orders of magnitude](https://ieeexplore.ieee.org/document/10363573).\nSo, why aren't we doing that? Presently, we *lack theories to guide efficient implementations*.\nWe can build the circuits, but we don't know how to combine them to achieve what we want.\n[Current neuromorphic models cannot compete with deep learning](https://www.nature.com/articles/s43588-021-00184-y).\n\n\nHere, **we provide a principled computational model for neuromorphic systems** based on tried-and-tested spatio-temporal covariance properties.\nWe demonstrate superior performance for simple neuromorphic primitives in an [event-based vision](https://en.wikipedia.org/wiki/Event_camera) task compared to naïve artificial neural networks.\nThe direction is exciting to us because\n\n1. We define mathematically coherent covariance properties, which are required to correctly handle signals in space and time,\n2. Use neuromorphic primitives (leaky integrator and leaky integrate-and-fire models) to outcompete a non-neuromorphic neural network, and\n3. Our results have immediate relevance in signal processing and event-based vision, with the possibility to extend to other tasks over space and time, such as memory and control.\n\n## Abstract\nBiological nervous systems constitute important sources of inspiration towards computers that are faster, cheaper, and more energy efficient.\nNeuromorphic disciplines view the brain as a coevolved system, simultaneously optimizing the hardware and the algorithms running on it.\nThere are clear efficiency gains when bringing the computations into a physical substrate, but we presently lack theories to guide efficient implementations.\nHere, we present a principled computational model for neuromorphic systems in terms of spatio-temporal receptive fields, based on affine Gaussian kernels over space and leaky-integrator and leaky integrate-and-fire models over time.\nOur theory is provably covariant to spatial affine and temporal scaling transformations, and with close similarities to the visual processing in mammalian brains.\nWe use these spatio-temporal receptive fields as a prior in an event-based vision task, and show that this improves the training of spiking networks, which otherwise is known as problematic for event-based vision.\nThis work combines efforts within scale-space theory and computational neuroscience to identify theoretically well-founded ways to process spatio-temporal signals in neuromorphic systems.\nOur contributions are immediately relevant for signal processing and event-based vision, and can be extended to other processing tasks over space and time, such as memory and control.\n\n**Read more at [jepedersen.dk](https://jepedersen.dk/posts/202510_nrf/)**\n\n## Dataset\nThe data is generated using the [Geometric Event-Response Data Generation](https://github.com/ncskth/gerd) repository (seen in the movie above, panel **d**).\nThe exact data used in this publication can be recreated using the script `data_generation.sh` or retrieved by reaching out to the author of the paper below.\nWe use the PyTorch dataset class in `dataset.py` to load the data.\nAdditional information about the data generation method is available in the [GERD repository](https://github.com/ncskth/gerd) and [the preprint](https://arxiv.org/abs/2412.03259).\n\n## Usage\n\nBefore our code can be run, the following dependencies must be installed:\n```\ntorch\u003e=2.2\nnorse\u003e=1.1.0\npytorch-lightning==1.9.4\ntensorboard==2.17.1\n```\n\nThe main entry-point is the `learn_shapes.py` file, which takes two main arguments (path to the training data and path to the log directory) and a host of hyperparameters.\n\n```\npython3 learn_shapes.py \u003cpath-to-data\u003e \u003cpath-to-log\u003e\n```\n\nA description of the parameters can be found using `python3 learn_shapes.py --help`.\n\n### Reproducing the results in the paper\n\nThe paper uses specific sets of parameters that are available in the `config_ann.txt` and `config_snn.txt` files. The data in the paper was procured from the [Swedish Alvis cluster](https://www.c3se.chalmers.se/about/Alvis/).\nThe cluster-specific code is available upon request, but generally, the code can be run with the following script:\n\n```bash\n# Set basic flags common for all models\nDATADIR=\u003cpath-to-dataset\u003e\nLOGDIR=\u003cpath-to-logs\u003e\nFLAGS=\"${DATADIR} ${LOGDIR} --n_spatial_scales=4 --n_temporal_scales=4 --max_epochs=50 --devices=1 --accelerator=gpu --strategy=dp\"\n\n# Use the flags in the config files to set the following variables:\nMODEL_FLAGS=\"$FLAGS --net=$net --coordinate=$coordinate --stack_frames=$stack_frames --init_scheme_spatial=$init_scheme_spatial \\\n --init_scheme_temporal=$init_scheme_temporal --batch_size=$batch_size --weight_sharing=$weight_sharing --dropout=$dropout \\\n --n_angles=$n_angles --n_ratios=$n_ratios --batch_normalization=$batch_normalization\"\n\n# Run the model\npython3 learn_shapes.py $MODEL_FLAGS\n```\n\n## Acknowledgements\nAll simulations, neuron models, and the spatio-temporal receptive fields rely on [the Norse library](https://github.com/norse/norse).\nThe implementation of affine directional derivatives is based on the [affscsp module in the pyscsp package](https://github.com/tonylindeberg/pyscsp) and parts of the temporal smoothing operations are based on the [pytempscsp package](https://github.com/tonylindeberg/pytempscsp).\n\nThe authors gratefully acknowledge support from the EC Horizon 2020 Framework\nProgramme under Grant Agreements 785907 and 945539 (HBP), the Swedish Research Council under contracts 2022-02969 and 2022-06725, and the Danish National Research Foundation grant number P1.\n\n## Citation\nOur work is freely available at [https://www.nature.com/articles/s41467-025-63493-0](https://www.nature.com/articles/s41467-025-63493-0) and can be cited as follows\n```bibtex\n@misc{pedersen2025covariant,\n    title={Covariant spatio-temporal receptive fields for spiking neural networks}, \n    author={Pedersen, J. E. and Conradt, J. and Lindeberg, T.},\n    year={2025},\n    pages={8231},\n    volume={16},\n    DOI={10.1038/s41467-025-63493-0},\n    number={1},\n    journal={Nature Communications},\n}\n```\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjegp%2Fnrf","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjegp%2Fnrf","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjegp%2Fnrf/lists"}