{"id":29115626,"url":"https://github.com/tub-rip/ETAP","last_synced_at":"2025-06-29T11:13:24.937Z","repository":{"id":285559642,"uuid":"939886908","full_name":"tub-rip/ETAP","owner":"tub-rip","description":"The official implementation of \"ETAP: Event-based Tracking of Any Point\" (CVPR'25 Highlight) ","archived":false,"fork":false,"pushed_at":"2025-06-02T09:33:27.000Z","size":37002,"stargazers_count":49,"open_issues_count":1,"forks_count":3,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-06-02T20:23:46.258Z","etag":null,"topics":["cvpr2025","event-cameras","point-tracking"],"latest_commit_sha":null,"homepage":"https://arxiv.org/pdf/2412.00133","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tub-rip.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-02-27T09:16:39.000Z","updated_at":"2025-06-02T09:33:30.000Z","dependencies_parsed_at":"2025-06-02T10:43:41.348Z","dependency_job_id":null,"html_url":"https://github.com/tub-rip/ETAP","commit_stats":null,"previous_names":["tub-rip/etap"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/tub-rip/ETAP","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tub-rip%2FETAP","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tub-rip%2FETAP/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tub-rip%2FETAP/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tub-rip%2FETAP/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tub-rip","download_url":"https://codeload.github.com/tub-rip/ETAP/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tub-rip%2FETAP/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":262581512,"owners_count":23331925,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cvpr2025","event-cameras","point-tracking"],"created_at":"2025-06-29T11:13:10.704Z","updated_at":"2025-06-29T11:13:24.881Z","avatar_url":"https://github.com/tub-rip.png","language":"Python","funding_links":[],"categories":["Feature / Point Tracking"],"sub_categories":[],"readme":"# ETAP: Event-based Tracking of Any Point (CVPR'25 Highlight)\n\n[![Paper](https://img.shields.io/badge/arXiv-2409.03358-b31b1b.svg)](https://arxiv.org/pdf/2412.00133)\n[![Poster](https://img.shields.io/badge/Poster-PDF-FF6B6B.svg)](docs/etap_cvpr25_poster.pdf)\n[![Data](https://img.shields.io/badge/Dataset-GoogleDrive-4285F4.svg)](https://drive.google.com/drive/folders/1Mprj-vOiTP5IgXE9iuu4-4bazcZUswpp?usp=drive_link)\n[![Video](https://img.shields.io/badge/Video-YouTube-FF0000.svg?style=flat\u0026labelColor=444444\u0026color=FF0000)](https://youtu.be/LaeA9WJ7ptc)\n[![License](https://img.shields.io/badge/License-CC_BY--NC_4.0-lightgrey.svg)](https://creativecommons.org/licenses/by-nc/4.0/)\n\n## Introduction\n\nThis is the official repository for [**ETAP: Event-based Tracking of Any Point**](https://arxiv.org/pdf/2412.00133), by [Friedhelm Hamann](https://friedhelmhamann.github.io/), [Daniel Gehrig](https://danielgehrig18.github.io/), [Filbert Febryanto](https://github.com/filbert14), [Kostas Daniilidis](https://www.cis.upenn.edu/~kostas/) and [Guillermo Gallego](http://www.guillermogallego.es/).\n\n\u003cp align=\"center\"\u003e\n \u003ca href=\"https://youtu.be/LaeA9WJ7ptc\"\u003e\n  \u003cimg src=\"./docs/thumbnail.png\" alt=\"ETAP_thumbnail\" width=\"600\"/\u003e\n \u003c/a\u003e\n\u003c/p\u003e\n\n### Key Features\n\n- The first event-only point tracking method with strong cross-dataset generalization.\n- Robust tracking in challenging conditions (high-speed motion, lighting changes).\n- Evaluation benchmark based on six event camera datasets: EDS, EC, EVIMO2, EventKubric, E2D2 and PennAviary\n\n### Example Predictions\n\n\u003cdiv align=\"center\"\u003e\n  \u003ctable\u003e\n    \u003ctr\u003e\n      \u003ctd align=\"center\"\u003e\n        \u003cimg src=\"docs/pred_event_kubric.gif\" alt=\"Example 1\" width=\"200\"/\u003e\n        \u003cbr\u003eExample 1: Synthetic dataset EventKubric\n      \u003c/td\u003e\n      \u003ctd align=\"center\"\u003e\n        \u003cimg src=\"docs/pred_eds_peanuts_running.gif\" alt=\"Example 2\" width=\"265\"/\u003e\n        \u003cbr\u003eExample 2: Feature Tracking on EDS\n      \u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd align=\"center\"\u003e\n        \u003cimg src=\"docs/pred_e2d2_fidget.gif\" alt=\"Example 3\" width=\"200\"/\u003e\n        \u003cbr\u003eExample 3: E2D2\n      \u003c/td\u003e\n      \u003ctd align=\"center\"\u003e\n        \u003cimg src=\"docs/pred_evimo2.gif\" alt=\"Example 4\" width=\"265\"/\u003e\n        \u003cbr\u003eExample 4: EVIMO2\n      \u003c/td\u003e\n    \u003c/tr\u003e\n  \u003c/table\u003e\n\u003c/div\u003e\n\n## Table of Contents\n\n- [Quickstart Demo](#quickstart-demo)\n- [Installation](#installation)\n- [Model Selection](#model-selection)\n- [Evaluation Tasks](#evaluation-tasks)\n  - [Feature Tracking (EDS, EC)](#feature-tracking-eds-ec)\n  - [EVIMO2](#evaluation-evimo2)\n  - [EventKubric](#evaluation-eventkubric)\n  - [E2D2](#evaluation-e2d2)\n  - [PennAviary](#evaluation-pennaviary-qualitative)\n- [Synthetic Data Generation](#synthetic-data-generation-eventkubric)\n- [Acknowledgements](#acknowledgements)\n- [Citation](#citation)\n- [Additional Resources](#additional-resources)\n- [License](#license)\n\n## Quickstart Demo\n\nThe quickest way to try ETAP is using our demo:\n\n1. Clone the repository:\n   ```bash\n   git clone https://github.com/tub-rip/ETAP.git\n   cd ETAP\n   ```\n\n2. Download [the model weights](https://drive.google.com/file/d/18mnwu8CsrVJvDXeRtvU0Wgp6shdxn0HD/view?usp=drive_link) and save to `weights/ETAP_v1_cvpr25.pth`\n3. Download [the demo example](https://drive.google.com/drive/folders/1d5Yi1q6ZFom3Q_VrELzXjrxE5aC2ezOk?usp=drive_link) (30MB) and extract to `data/demo_example`\n4. Run the demo:\n   ```bash\n   python scripts/demo.py\n   ```\n\nThis demo requires only basic dependencies: `torch`, `numpy`, `tqdm`, `matplotlib`, `imageio`, and `pillow`. No dataset preprocessing needed!\n\n## Installation\n\n1. Clone the repository:\n   ```bash\n   git clone git@github.com:tub-rip/ETAP.git\n   cd ETAP\n   ```\n\n2. Set up the environment:\n   ```bash\n   conda create --name ETAP python=3.10\n   conda activate ETAP\n   ```\n\n3. Install PyTorch (choose a command compatible with your CUDA version from the [PyTorch website](https://pytorch.org/get-started/locally/)), e.g.:\n   ```bash\n   conda install pytorch torchvision pytorch-cuda=12.1 -c pytorch -c nvidia\n   ```\n\n4. Install other dependencies:\n   ```bash\n   pip install -r requirements.txt\n   ```\n\n## Model Selection\n\nDownload the [pre-trained model](https://drive.google.com/drive/folders/17YNqf603b3dEdETmNht-ih7cwosKyACN?usp=drive_link) and move it into the folder `\u003crepo-root\u003e/weights/`. \n\nTo reproduce the paper results, use the model `ETAP_v1_cvpr25.pth`.\n\n## Evaluation Tasks and Datasets\n\n### Evaluation: Feature Tracking (EDS, EC)\n\n#### Preparations\n\n##### Download EDS (Prophesee Gen3 640 x 480 px)\n\nThe four evaluation sequences of the \"Event-aided Direct Sparse Odometry Dataset\" (EDS) can be downloaded in two ways:\n\n**Option 1**\n\nDownload the four evaluation sequences of the \"Event-aided Direct Sparse Odometry Dataset\" (EDS) from the [official web page](https://rpg.ifi.uzh.ch/eds.html):\n\n* `01_peanuts_light`\n* `02_rocket_earth_light`\n* `08_peanuts_running`\n* `14_ziggy_in_the_arena`\n\nChoose the archive file option, which contains the events as an hdf5 file. Place all sequences in a common folder.\n\n**Option 2:** Use our download script:\n```bash\nbash scripts/download_eds.sh\n```\n\nWe also use the calibration data provided by EDS. No action is required, as it is included in this repository at `config/misc/eds/calib.yaml`. This is the same file as in the `00_calib` results from the official source.\n\nThe evaluation was introduced in [DDFT](https://github.com/uzh-rpg/deep_ev_tracker). As with the calibration data, we have hardcoded the ground truth tracks at `/config/misc/eds/gt_tracks`, so no additional steps are necessary. If you are interested in how the tracks are created, please refer to the DDFT repository.\n\nCreate a symbolic link to your data root into `\u003crepository-root\u003e/data`, or alternatively you can change the paths in the config files. The setup should look something like this:\n\n```\ndata/eds/\n   ├── 01_peanuts_light\n   │   └── events.h5\n   ├── 02_rocket_earth_light\n   │   └── events.h5\n   ├── 08_peanuts_running\n   │   └── events.h5\n   └── 14_ziggy_in_the_arena\n      └── events.h5\n```\n\n##### Download EC (DAVIS240C 240 x 180 px)\n\nDownload the five evaluation sequences of the \"Event Camera Dataset\" (EC) from the [official source](https://rpg.ifi.uzh.ch/davis_data.html). Download the option `Text (zip)`. Unzip the sequences into a folder structure like this:\n\n```\ndata/ec/\n   ├── boxes_rotation\n   │   ├── calib.txt\n   │   ├── events.txt\n   │   ├── groundtruth.txt\n   │   ├── images\n   │   ├── images.txt\n   │   └── imu.txt\n   ├── boxes_translation\n   │   ├── events.txt\n   │   ├── ...\n   ├── shapes_6dof\n   │   ├── events.txt\n   │   ├── ...\n   ├── shapes_rotation\n   │   ├── events.txt\n   │   ├── ...\n   └── shapes_translation\n       ├── events.txt\n       ├── ...\n```\n\nAs with EDS, the ground truth tracks are from the evaluation introduced in DDFT but we have included them at `config/misc/ec/gt` for convenience.\n\n##### Preprocessing\n\nPreprocess the data by transforming the raw events into event stacks with the following commands:\n\n```bash\n# For EDS dataset\npython scripts/prepare_event_representations.py --dataset eds --config config/exe/prepare_event_representations/eds.yaml\n\n# For EC dataset\npython scripts/prepare_event_representations.py --dataset ec --config config/exe/prepare_event_representations/ec.yaml\n```\n\n#### Inference\n\nRun the tracking inference with:\n\n```bash\npython scripts/inference_online.py --config config/exe/inference_online/feature_tracking.yaml\n```\n\n#### Evaluation\n\nRun the benchmarking script to evaluate the tracking results:\n\n```bash\npython scripts/benchmark_feature_tracking.py feature_tracking_eds_ec\n```\n\n### Evaluation: EVIMO2 (Samsung DVS Gen3 640 x 480 px)\n\n#### Preparations\n\n1. Download the required EVIMO2 sequences from the [official source](https://better-flow.github.io/evimo/download_evimo_2.html#imo). You only need **Motion Segmentation / Object Recognition** sequences for the **samsung_mono** camera in **.npz** format (2.4GB). Unzip them and move them into the data directory.\n\n2. Download the precomputed tracks [here](https://drive.google.com/drive/folders/1d5Yi1q6ZFom3Q_VrELzXjrxE5aC2ezOk?usp=drive_link) and merge them into the data directory.\n\nThe result should look like this:\n\n```\ndata/evimo/\n└── samsung_mono\n    └── imo\n        └── eval\n            ├── scene13_dyn_test_00_000000\n            │   ├── dataset_classical.npz\n            │   ├── dataset_depth.npz\n            │   ├── dataset_events_p.npy\n            │   ├── dataset_events_t.npy\n            │   ├── dataset_events_xy.npy\n            │   ├── dataset_info.npz\n            │   ├── dataset_mask.npz\n            │   └── dataset_tracks.h5\n            ├── scene13_dyn_test_05_000000\n            │   ├── dataset_classical.npz\n            ... ...\n```\n\n3. Precompute the event stacks:\n\n```bash\npython scripts/prepare_event_representations.py --dataset evimo2 --config config/exe/prepare_event_representations/evimo2.yaml\n```\n\n#### Inference \u0026 Evaluation\n\nRun inference and evaluation with a single command:\n\n```bash\npython scripts/inference_offline.py --config config/exe/inference_offline/evimo2.yaml\n```\n\n#### Ground Truth Track Generation (Optional)\n\nIf you want to generate the point tracks yourself instead of using the precomputed ones:\n\n```bash\npython scripts/create_evimo2_track_gt.py --config config/misc/evimo2/val_samples.csv --data_root data/evimo2\n```\n\n### Evaluation: EventKubric (synthetic 512 x 512 px)\n\n#### Preparations\n\n1. Download [the event_kubric test set](https://drive.google.com/drive/folders/1d5Yi1q6ZFom3Q_VrELzXjrxE5aC2ezOk?usp=drive_link) and move it to the data directory:\n\n```\ndata/event_kubric\n└── test\n    ├── sample_000042\n    │   ├── annotations.npy\n    │   ├── data.hdf5\n    │   ├── data_ranges.json\n    │   ├── events\n    │   │   ├── 0000000000.npz\n    │   │   ├── 0000000001.npz\n    │   │   ...\n    │   ├── events.json\n    │   └── metadata.json\n    ├── sample_000576\n    │   ├── annotations.npy\n    │   ...\n    ...\n```\n\n2. Prepare the event stacks:\n\n```bash\npython scripts/prepare_event_representations.py --dataset event_kubric --config config/exe/prepare_event_representations/event_kubric.yaml\n```\n\n#### Inference \u0026 Evaluation\n\nRun inference and evaluation with a single command:\n\n```bash\npython scripts/inference_offline.py --config config/exe/inference_offline/event_kubric.yaml\n```\n\n### Evaluation: E2D2 (SilkyEvCam 640 x 480 px)\n\n#### Preparations\n\n1. Download [the E2D2 fidget spinner sequence](https://drive.google.com/drive/folders/1d5Yi1q6ZFom3Q_VrELzXjrxE5aC2ezOk?usp=drive_link) and move it to the data directory:\n\n```\ndata/e2d2/\n└── 231025_110210_fidget5_high_exposure\n    ├── gt_positions.npy\n    ├── gt_timestamps.npy\n    ├── queries.npy\n    └── seq.h5\n```\n\n2. Prepare the event stacks:\n\n```bash\npython scripts/prepare_event_representations.py --dataset e2d2 --config config/exe/prepare_event_representations/e2d2.yaml\n```\n\n#### Inference\n\n```bash\npython scripts/inference_online.py --config config/exe/inference_online/e2d2.yaml\n```\n\n#### Evaluation\n\n```bash\npython scripts/benchmark_tap.py --gt_dir data/e2d2/231025_110210_fidget5_high_exposure --pred_dir output/inference/tap_e2d2\n```\n\n#### Ground Truth Generation (Optional)\n\nThe ground truth is calculated from the turning speed of the fidget spinner and is provided for download. To calculate the ground truth tracks yourself, run:\n\n```bash\npython scripts/create_e2d2_fidget_spinner_gt.py\n```\n\n### Evaluation: PennAviary (SilkyEvCam 640 x 480 px, Qualitative)\n\n#### Preparations\n\nDownload [the penn_aviary sequence](https://drive.google.com/drive/folders/1d5Yi1q6ZFom3Q_VrELzXjrxE5aC2ezOk?usp=drive_link) and move it to the data directory:\n\n```\ndata/penn_aviary/\n└── 231018_174107_view2\n    ├── mask00082.png\n    └── seq.h5\n```\n\n#### Inference\n\nRun the inference with:\n\n```bash\npython scripts/inference_online.py --config config/exe/inference_online/penn_aviary.yaml\n```\n\n## Synthetic Data Generation (EventKubric)\n\nWe provide a [10 sample test set of EventKubric](https://drive.google.com/drive/folders/1d5Yi1q6ZFom3Q_VrELzXjrxE5aC2ezOk?usp=drive_link) for quick evaluation. The complete dataset consists of approximately 10,000 samples.\n\nTo generate your own synthetic event data, please refer to the [Data Pipeline Instructions](data_pipeline/README.md).\n\n## Acknowledgements\n\nWe gratefully appreciate the following repositories and thank the authors for their excellent work:\n\n- [CoTracker](https://github.com/facebookresearch/co-tracker)\n- [TapVid](https://github.com/google-deepmind/tapnet/tree/main/tapnet/tapvid)\n- [DDFT](https://github.com/uzh-rpg/deep_ev_tracker)\n\n## Citation\n\nIf you find this work useful in your research, please consider citing:\n\n```bibtex\n@InProceedings{Hamann25cvpr,\n  author={Friedhelm Hamann and Daniel Gehrig and Filbert Febryanto and Kostas Daniilidis and Guillermo Gallego},\n  title={{ETAP}: Event-based Tracking of Any Point},\n  booktitle={{IEEE/CVF} Conf. Computer Vision and Pattern Recognition ({CVPR})},\n  year=2025,\n}\n```\n\n## Additional Resources\n\n* [Research page (TU Berlin, RIP lab)](https://sites.google.com/view/guillermogallego/research/event-based-vision)\n* [Course at TU Berlin](https://sites.google.com/view/guillermogallego/teaching/event-based-robot-vision)\n* [Survey paper](http://rpg.ifi.uzh.ch/docs/EventVisionSurvey.pdf)\n* [List of Event-based Vision Resources](https://github.com/uzh-rpg/event-based_vision_resources)\n\n## License\n\nThis project is licensed under the Creative Commons Attribution-NonCommercial 4.0 International License - see the [LICENSE](LICENSE) file for details. This means you are free to share and adapt the material for non-commercial purposes, provided you give appropriate credit and indicate if changes were made.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftub-rip%2FETAP","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftub-rip%2FETAP","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftub-rip%2FETAP/lists"}