{"id":29115604,"url":"https://github.com/tub-rip/event_based_optical_flow","last_synced_at":"2025-06-29T11:13:21.783Z","repository":{"id":47817275,"uuid":"512650640","full_name":"tub-rip/event_based_optical_flow","owner":"tub-rip","description":"The official implementation of \"Secrets of Event-based Optical Flow\" (ECCV2022 Oral and IEEE T-PAMI 2024)","archived":false,"fork":false,"pushed_at":"2025-03-12T22:28:19.000Z","size":10104,"stargazers_count":162,"open_issues_count":4,"forks_count":15,"subscribers_count":9,"default_branch":"main","last_synced_at":"2025-03-12T23:26:29.873Z","etag":null,"topics":["depth-estimation","eccv2022","event-based-vision","event-camera","event-vision","optical-flow","pytorch"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tub-rip.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-07-11T07:04:50.000Z","updated_at":"2025-03-12T22:28:22.000Z","dependencies_parsed_at":"2024-05-06T11:27:48.218Z","dependency_job_id":"24aaf733-96e1-4559-aa36-65caa98db20b","html_url":"https://github.com/tub-rip/event_based_optical_flow","commit_stats":{"total_commits":23,"total_committers":4,"mean_commits":5.75,"dds":"0.30434782608695654","last_synced_commit":"756dfd0125ca0b9092d207e7aed982b763be1d32"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/tub-rip/event_based_optical_flow","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tub-rip%2Fevent_based_optical_flow","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tub-rip%2Fevent_based_optical_flow/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tub-rip%2Fevent_based_optical_flow/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tub-rip%2Fevent_based_optical_flow/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tub-rip","download_url":"https://codeload.github.com/tub-rip/event_based_optical_flow/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tub-rip%2Fevent_based_optical_flow/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":262581512,"owners_count":23331925,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["depth-estimation","eccv2022","event-based-vision","event-camera","event-vision","optical-flow","pytorch"],"created_at":"2025-06-29T11:13:10.342Z","updated_at":"2025-06-29T11:13:21.755Z","avatar_url":"https://github.com/tub-rip.png","language":"Python","funding_links":[],"categories":["Optical Flow Estimation"],"sub_categories":[],"readme":"👀 **The extension paper has been accepted to IEEE T-PAMI! ([Paper](https://doi.org/10.1109/TPAMI.2024.3396116))**\n\n👀 **We are now working to make this method more generic, easy-to-use functions (`flow = useful_function(events)`). Stay tuned!**\n\n# Secrets of Event-Based Optical Flow (T-PAMI 2024, ECCV 2022)\n\nThis is the official repository for [**Secrets of Event-Based Optical Flow**](https://arxiv.org/abs/2207.10022), **ECCV 2022 Oral** by  \n[Shintaro Shiba](http://shibashintaro.com/), [Yoshimitsu Aoki](https://aoki-medialab.jp/aokiyoshimitsu-en/) and [Guillermo Callego](http://www.guillermogallego.es).\n\nWe have extended this paper to a journal version: [**Secrets of Event-based Optical Flow, Depth and Ego-motion Estimation by Contrast Maximization**](https://doi.org/10.1109/TPAMI.2024.3396116), **IEEE T-PAMI 2024**.\n\n \u003c!-- - [Paper]() \n[[Video](https://youtu.be/nUb2ZRPdbWk)] [[PDF](https://link.springer.com/chapter/10.1007/978-3-031-19797-0_36)]\n [[arXiv](https://arxiv.org/pdf/2207.10022)]\n --\u003e\n\n\u003ch2 align=\"left\"\u003e\n  \n[Paper (IEEE T-PAMI 2024)](https://hal.science/hal-04655247v1/document) | [Paper (ECCV 2022)](https://arxiv.org/pdf/2207.10022) | [Video](https://youtu.be/nUb2ZRPdbWk) | [Poster](docs/img/2024_TPAMI_SecretsOfEVFlow_poster.pdf)\n\u003c/h2\u003e\n\n[![Secrets of Event-Based Optical Flow](docs/img/secretsevflow_eccv22.jpg)](https://youtu.be/nUb2ZRPdbWk)\n\n\nIf you use this work in your research, please cite it (see also [here](#citation)):\n\n```bibtex\n@Article{Shiba24pami,\n  author        = {Shintaro Shiba and Yannick Klose and Yoshimitsu Aoki and Guillermo Gallego},\n  title         = {Secrets of Event-based Optical Flow, Depth, and Ego-Motion by Contrast Maximization},\n  journal       = {IEEE Trans. Pattern Anal. Mach. Intell. (T-PAMI)},\n  year          = 2024,\n  pages         = {1--18},\n  doi           = {10.1109/TPAMI.2024.3396116}\n}\n\n@InProceedings{Shiba22eccv,\n  author        = {Shintaro Shiba and Yoshimitsu Aoki and Guillermo Gallego},\n  title         = {Secrets of Event-based Optical Flow},\n  booktitle     = {European Conference on Computer Vision (ECCV)},\n  pages         = {628--645},\n  doi           = {10.1007/978-3-031-19797-0_36},\n  year          = 2022\n}\n```\n\n## **List of datasets that the flow estimation is tested on**\n\nAlthough this codebase releases just MVSEC examples,\nI have tested the flow estimation is roughly good in the below datasets.\nThe list is being updated, and if you test new datasets please let us know.\n\n- [MVSEC](https://daniilidis-group.github.io/mvsec/)\n- [DSEC](https://dsec.ifi.uzh.ch/dsec-datasets/download/)\n- [ECD, both simulation and real data](http://rpg.ifi.uzh.ch/davis_data.html)\n- [TUM VIE](https://cvg.cit.tum.de/data/datasets/visual-inertial-event-dataset)\n- [UZH-FPV Drone Racing Dataset](https://fpv.ifi.uzh.ch/)\n- [EDS](https://rpg.ifi.uzh.ch/eds.html#dataset)\n- [M3ED](https://m3ed.io/)\n\nThe above is all public datasets, and in our paper (T-PAMI 2024) we also used some non-public dataset from previous works.\n\n-------\n# Setup\n\n## Requirements\n\nAlthough not all versions are strictly tested, the followings should work.\n\n- python: 3.8.x, 3.9.x, 3.10.x\n\nGPU is entirely optional.\nIf `torch.cuda.is_available()` then it automatically switches to use GPU.\nI'd recomment to use GPU for time-aware solutions, but CPU is ok for no-timeaware method as long as I tested.\n\n### Tested environments\n\n- Mac OS Monterey (both M1 and non-M1)\n- Ubuntu (CUDA 11.1, 11.3, 11.8)\n- PyTorch 1.9-1.12.1, or PyTorch 2.0 (1.13 raises an error during Burgers).\n\n## Installation\n\nI strongly recommend to use venv: `python3 -m venv \u003cnew_venv_path\u003e`\nAlso, you can use [poetry]().\n\n- Install pytorch **\u003c 1.13** or **\u003e= 2.0** and torchvision for your environment. Make sure you install the correct CUDA version if you want to use it.\n\n- If you use poetry, `poetry install`. If you use only venv, check dependecy libraries and install it from [here](./pyproject.toml).\n\n- If you are having trouble to install pytorch with cuda using poetry refer to this [link](https://github.com/python-poetry/poetry/issues/6409). \n\n## Download dataset\n\nDownload each dataset under `./datasets` directory.\nOptionally you can specify other root directory:\nplease check the [dataset readme](./datasets/README.md) for the details.\n\n# Execution\n\n```shell\npython3 main.py --config_file ./configs/mvsec_indoor_no_timeaware.yaml\n```\n\nIf you use poetry, simply add `poetry run` at the beginning.\nPlease run with `-h` option to know more about the other options.\n\n## Config file\n\nThe config (.yaml) file specifies various experimental settings.\nPlease check and change parameters as you like.\n\n### Optional tasks (for me)\n\n**The code here is already runnable, and explains the ideas of the paper enough.** (Please report bugs if any.)\n\nRather than releasing all of my (sometimes too experimental) codes,\nI published just a minimal set of the codebase to reproduce.\nSo the following tasks are more optional for me.\nBut if it helps you, I can publish other parts as well. For example:\n\n - Other data loader\n\n - Some other cost functions\n\n - Pretrained model checkpoint file ✔️ [released for MVSEC](https://drive.google.com/file/d/13m-waAt5X0C7f0JLBwb6KAApYxgXoA2J/view?usp=sharing)\n\n - Other solver (especially DNN)\n\n - The implementation of [the Sensors paper]((https://www.mdpi.com/1424-8220/22/14/5190))\n\nYour feedback is helpful to prioritize the tasks, so please contact me or raise issues.\nThe code is modularized well, so if you want to contribute, it should be easy too.\n\n# Citation\n\nIf you use this work in your research, please cite it **as stated above**, below the video.\n\nThis code also includes some implementation of the [following paper about event collapse in details](https://www.mdpi.com/1424-8220/22/14/5190).\nPlease check it :)\n\n```bibtex\n@Article{Shiba22sensors,\n  author        = {Shintaro Shiba and Yoshimitsu Aoki and Guillermo Gallego},\n  title         = {Event Collapse in Contrast Maximization Frameworks},\n  journal       = {Sensors},\n  year          = 2022,\n  volume        = 22,\n  number        = 14,\n  pages         = {1--20},\n  article-number= 5190,\n  doi           = {10.3390/s22145190}\n}\n```\n\n# Author\n\nShintaro Shiba [@shiba24](https://github.com/shiba24)\n\n## LICENSE\n\nPlease check [License](./LICENSE).\n\n## Acknowledgement\n\nI appreciate the following repositories for the inspiration:\n\n- [autograd-minimize](https://github.com/brunorigal/autograd-minimize)\n- [EVFlowNet-pytorch](https://github.com/CyrilSterling/EVFlowNet-pytorch)\n\n-------\n# Additional Resources\n\n* [Motion-prior Contrast Maximization (ECCV 2024)](https://github.com/tub-rip/MotionPriorCMax)\n* [EVILIP: Event-based Image Reconstruction as a Linear Inverse Problem (TPAMI 2022)](https://github.com/tub-rip/event_based_image_rec_inverse_problem)\n* [Event Collapse in Contrast Maximization Frameworks](https://github.com/tub-rip/event_collapse)\n* [CMax-SLAM (TRO 2024)](https://github.com/tub-rip/cmax_slam)\n* [EBOS: Event-based Background-Oriented Schlieren (TPAMI 2023)](https://github.com/tub-rip/event_based_bos)\n* [EPBA: Event-based Photometric Bundle Adjustment](https://github.com/tub-rip/epba)\n* [ES-PTAM: Event-based Stereo Parallel Tracking and Mapping](https://github.com/tub-rip/ES-PTAM)\n* [Research page (TU Berlin, RIP lab)](https://sites.google.com/view/guillermogallego/research/event-based-vision)\n* [Research page (Keio University, Aoki Media Lab)](https://aoki-medialab.jp/home-en/)\n* [Course at TU Berlin](https://sites.google.com/view/guillermogallego/teaching/event-based-robot-vision)\n* [Survey paper](http://rpg.ifi.uzh.ch/docs/EventVisionSurvey.pdf)\n* [List of Resources](https://github.com/uzh-rpg/event-based_vision_resources)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftub-rip%2Fevent_based_optical_flow","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftub-rip%2Fevent_based_optical_flow","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftub-rip%2Fevent_based_optical_flow/lists"}