{"id":33053197,"url":"https://github.com/wangxiao5791509/VisEvent_SOT_Benchmark","last_synced_at":"2025-11-23T15:00:40.155Z","repository":{"id":45570728,"uuid":"319866929","full_name":"wangxiao5791509/VisEvent_SOT_Benchmark","owner":"wangxiao5791509","description":"[IEEE TCYB 2023] The first large-scale tracking dataset by fusing RGB and Event cameras. ","archived":false,"fork":false,"pushed_at":"2025-05-14T23:18:54.000Z","size":20949,"stargazers_count":142,"open_issues_count":6,"forks_count":9,"subscribers_count":5,"default_branch":"main","last_synced_at":"2025-05-15T00:29:25.155Z","etag":null,"topics":["dvs","dynamic-vision-sensors","event","frame-event-tracking","multi-modal-fusion","neuromorphic","neuromorphic-vision","pengchenglab","single-object-tracking","spike","visevent"],"latest_commit_sha":null,"homepage":"https://sites.google.com/view/viseventtrack/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/wangxiao5791509.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"License","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-12-09T06:43:39.000Z","updated_at":"2025-05-14T23:18:58.000Z","dependencies_parsed_at":"2023-01-29T18:16:24.976Z","dependency_job_id":"eb0d576b-c0d4-414f-97f3-0f30c538ac22","html_url":"https://github.com/wangxiao5791509/VisEvent_SOT_Benchmark","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/wangxiao5791509/VisEvent_SOT_Benchmark","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wangxiao5791509%2FVisEvent_SOT_Benchmark","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wangxiao5791509%2FVisEvent_SOT_Benchmark/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wangxiao5791509%2FVisEvent_SOT_Benchmark/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wangxiao5791509%2FVisEvent_SOT_Benchmark/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/wangxiao5791509","download_url":"https://codeload.github.com/wangxiao5791509/VisEvent_SOT_Benchmark/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wangxiao5791509%2FVisEvent_SOT_Benchmark/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":285968540,"owners_count":27262718,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-11-23T02:00:06.149Z","response_time":135,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dvs","dynamic-vision-sensors","event","frame-event-tracking","multi-modal-fusion","neuromorphic","neuromorphic-vision","pengchenglab","single-object-tracking","spike","visevent"],"created_at":"2025-11-14T03:00:36.947Z","updated_at":"2025-11-23T15:00:40.149Z","avatar_url":"https://github.com/wangxiao5791509.png","language":"Python","funding_links":[],"categories":["⚓ Awesome Repositories","Python","RGBE Tracking"],"sub_categories":["RGB-Multi Tracking","Papers"],"readme":"\u003cdiv align=\"center\"\u003e\n\n\u003cimg src=\"https://github.com/wangxiao5791509/VisEvent_SOT_Benchmark/blob/main/figures/visevent_art.jpg\" width=\"600\"\u003e\n  \n**The First Large-scale Benchmark Dataset for Reliable Object Tracking by fusing RGB and Event Cameras**\n\n------\n\n\u003cp align=\"center\"\u003e\n  • \u003ca href=\"https://sites.google.com/view/viseventtrack/\"\u003eProject\u003c/a\u003e •\n  \u003ca href=\"https://arxiv.org/abs/2108.05015\"\u003earXiv\u003c/a\u003e • \n  \u003ca href=\"https://github.com/wangxiao5791509/RGB-DVS-SOT-Baselines\"\u003eBaselines\u003c/a\u003e •\n  \u003ca href=\"https://youtu.be/U4uUjci9Gjc\"\u003eDemoVideo\u003c/a\u003e • \n  \u003ca href=\"https://youtu.be/vGwHI2d2AX0\"\u003eTutorial\u003c/a\u003e •\n\u003c/p\u003e\n\n\u003c/div\u003e\n\n\u003e **VisEvent: Reliable Object Tracking via Collaboration of Frame and Event Flows[J]** arXiv preprint arXiv:2108.05015, 2021, Xiao Wang, Jianing Li, Lin Zhu, Zhipeng Zhang, Zhe Chen, Xin Li, Yaowei Wang, Yonghong Tian, Feng Wu \n\n\n\n\n## News: \n\n* :fire: [2024.03.12] A New Long-term RGB-Event based Visual Object Tracking Benchmark Dataset (termed **FELT**) is available at\n  [[Paper](https://arxiv.org/pdf/2403.05839.pdf)] \n  [[Code](https://github.com/Event-AHU/FELT_SOT_Benchmark)] \n  [[DemoVideo](https://youtu.be/6zxiBHTqOhE?si=6ARRGFdBLSxyp3G8)]  \n\n* :fire: [2023.09.27] A High Definition (HD) Event based Visual Object Tracking Benchmark Dataset (termed **EventVOT**) is available at\n[[arXiv](https://arxiv.org/abs/2309.14611)] [[Github](https://github.com/Event-AHU/EventVOT_Benchmark)] \n\n* :fire: [2023.09.20] VisEvent is accepted by IEEE Transactions on Cybernetics [[IEEE](https://ieeexplore.ieee.org/abstract/document/10284004)]\n\n* :fire: [2022.11.27] Due to some aedat4 files are missing, you can use part of this dataset according to this list: [[aedat4HARDVS_list](https://github.com/wangxiao5791509/VisEvent_SOT_Benchmark/blob/main/aedat4HARDVS_list.txt)]  \n\n* :fire: [2022.11.23] A new Color Frame + Event Stream based Tracking dataset **COESOT** is available at \n[[arXiv](https://arxiv.org/abs/2211.11010)] [[GitHub](https://github.com/Event-AHU/COESOT)]\n\n* :fire: [2022.10.19] Event camera (DVS, Spike) based Papers Published on Top International Conference [[\nEvent_Camera_in_Top_Conference](https://github.com/Event-AHU/Event_Camera_in_Top_Conference)]\n\n* [2022.07.14] Update the VOT2019-RGB-Event dataset used in our paper [BaiduYun]. \n\n* [2022.02.10] Update paper list for event camera based tracking [[Event_Tracking_Paper_List](https://github.com/wangxiao5791509/VisEvent_SOT_Benchmark/blob/main/Event_Tracking_Paper_List.md)]. \n\n* [2021.10.13] Update the links for onedrive. \n\n\n\n## Introduction \nDifferent from visible cameras which record intensity images frame by frame, the biologically inspired event camera produces a stream of asynchronous and sparse events with much lower latency. In practice, the visible cameras can better perceive texture details and slow motion, while event cameras can be free from motion blurs and have a larger dynamic range which enables them to work well under fast motion and low illumination. Therefore, the two sensors can cooperate with each other to achieve more reliable object tracking. In this work, we propose a large-scale Visible-Event benchmark (termed VisEvent) due to the lack of a realistic and scaled dataset for this task. Our dataset consists of 820 video pairs captured under low illumination, high speed, and background clutter scenarios, and it is divided into a training and a testing subset, each of which contains 500 and 320 videos, respectively. Based on VisEvent, we transform the event flows into event images and construct more than 30 baseline methods by extending current single-modality trackers into dual-modality versions. More importantly, we further build a simple but effective tracking algorithm by proposing a cross-modality transformer, to achieve more effective feature fusion between visible and event data. Extensive experiments on the proposed VisEvent dataset, and two simulated datasets (i.e., OTB-DVS and VOT-DVS), validated the effectiveness of our model. \n\n\n## Demo Video\nA demo video for the VisEvent can be found by cliking the image below: \n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://youtu.be/U4uUjci9Gjc\"\u003e\n    \u003cimg src=\"https://github.com/wangxiao5791509/VisEvent_SOT_Benchmark/blob/main/figures/visevent_demoIMG.jpg\" alt=\"VisEvent_DemoVideo\" width=\"700\"/\u003e\n  \u003c/a\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://youtu.be/U4uUjci9Gjc\"\u003e\n    \u003cimg src=\"https://github.com/wangxiao5791509/RGB_Event_Tracking_Benchmark/blob/main/videosamples.png\" alt=\"VisEvent_examples\" width=\"700\"/\u003e\n  \u003c/a\u003e\n\u003c/p\u003e\n\n\n\n\n## Video Tutorial\nThe tutorial for this paper can be found by cliking the image below: \n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://youtu.be/vGwHI2d2AX0\"\u003e\n    \u003cimg src=\"https://github.com/wangxiao5791509/VisEvent_SOT_Benchmark/blob/main/figures/visevent_tutorial.jpg\" alt=\"VisEvent_DemoVideo\" width=\"700\"/\u003e\n  \u003c/a\u003e\n\u003c/p\u003e\n\n\n\n\n## How to Download VisEvent Dataset? \n\n* **BaiduYun (about 216 GB):** \n```\nLink：https://pan.baidu.com/s/1VhdORXT4OvG8TUESfDZHfw\nPassword：AHUE \n```\n\n* **DropBox**:\n```\nhttps://www.dropbox.com/scl/fo/r406wsgll56fy0hhhwu62/AFo3cjXjSI4Dzjn5nlnXNW0?rlkey=ecgyd26j1ycfl1jbm4pwc3vbn\u0026st=rzf95buf\u0026dl=0\n```\n\n\n## Links for **VOT2019-RGB-Event dataset** (36.3 GB) used in our paper \n\n* **BaiduYun:**\n```\nLink：https://pan.baidu.com/s/1cS79d1dJFD8mF0AwuGG5Og   Password: AHUT \n```\n\n* **Googledrive:**  Click [[here](https://drive.google.com/drive/folders/19kSDi1VcDWJ9Tpr1fw34LK50Nm4zR2u8?usp=sharing)]\n\n\n\n## Baseline Methods\nThe source code of baseline trackers by fusing dual-modalities can be found at: [[RGB-DVS-SOT-Baselines](https://github.com/wangxiao5791509/RGB-DVS-SOT-Baselines)].\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://github.com/wangxiao5791509/RGB-DVS-SOT-Baselines\"\u003e\n    \u003cimg src=\"https://github.com/wangxiao5791509/VisEvent_SOT_Benchmark/blob/main/figures/visevent_baselines.png\" alt=\"VisEvent_Baselines\" width=\"700\"/\u003e\n  \u003c/a\u003e\n\u003c/p\u003e\n\n\n## How to load the aedat4 file? \nWe provide a python script ([**read_aedat4.py**](https://github.com/wangxiao5791509/VisEvent_SOT_Benchmark/blob/main/scripts/read_aedat4.py)) to load the aedat4 file. You can download one aedat4 file to feel the data style: [[dvSave-2021_12_21_16_32_19.aedat4](https://drive.google.com/file/d/1cMCVocWA3bfnIX1WlKqMNACWiYvFZ1rC/view?usp=share_link)]\n\n\nHere is an example: \n\n1). Install the required toolkit [[**dv-gui**](https://inivation.gitlab.io/dv/dv-docs/docs/getting-started.html)]. Please use different scripts for various Ubuntu OS: \n\n**Ubuntu 20.04**:\n\n    sudo add-apt-repository ppa:inivation-ppa/inivation\n    sudo apt-get update\n    sudo apt-get install dv-gui\n\n**Ubuntu 18.04**: \n\n    sudo add-apt-repository ppa:ubuntu-toolchain-r/test\n    sudo add-apt-repository ppa:inivation-ppa/inivation-bionic\n    sudo apt-get update\n    sudo apt-get install dv-gui\n\n**Ubuntu 16.04**: \n\n    sudo add-apt-repository ppa:ubuntu-toolchain-r/test \n    sudo add-apt-repository ppa:lkoppel/opencv \n    sudo add-apt-repository ppa:janisozaur/cmake-update \n    sudo add-apt-repository ppa:inivation-ppa/inivation-xenial \n    sudo apt-get update \n    sudo apt-get install dv-gui\n\nOther softwares:\n\n    pip install dv\n    pip install opencv-python numpy pillow -i https://pypi.tuna.tsinghua.edu.cn/simple \n\n[ref] [https://gitlab.com/inivation/dv/dv-python](https://gitlab.com/inivation/dv/dv-python)\n\n2). Open your terminal and run the script: \n\n    python read_aedat4.py\n\n\n\n\n\n\n\n\n\n\n## Evaluation ToolKit \nOnly matlab version is available. \n\n**1. Download this github:**\n    \n    git clone https://github.com/wangxiao5791509/VisEvent_SOT_Benchmark\n\n**2. Download the tracking results of our benchmark:**\n[[**GoogleDrive (185MB)**](https://drive.google.com/file/d/1fILCNMrwt2PiITPWIQFZpk1PJvg_JAjX/view?usp=sharing)]\n\n    unzip tracking_results_VisEvent_SOT_benchmark.zip, and put it into the folder \"tracking_results\". \n\n    unzip the \"annos.zip\" in the folder \"annos\"\n\n**3. Open your matlab, and run the script \"Evaluate_VisEvent_SOT_benchmark.m\". Wait and check the final evaluated figures**\n\n\u003cimg src=\"res_fig/VisEvent_benchmark_results.png\" width=\"650\"\u003e \n\n\n\n\n\n\n## More Related Materials \n* [**Github-1**] https://github.com/wangxiao5791509/SNN_CV_Applications_Resources \n* [**Github-2**] https://github.com/uzh-rpg/event-based_vision_resources \n* [**Github-3**] https://github.com/wangxiao5791509/Single_Object_Tracking_Paper_List\n* [**Survey**] **神经形态视觉传感器的研究进展及应用综述**，计算机学报，李家宁, 田永鸿 [[Paper](https://drive.google.com/file/d/1d7igUbIrEWxmUI7xq75P6h_I4H7uI3FA/view?usp=sharing)] \n* [**Survey**] **Event-based Vision: A Survey**, Guillermo Gallego, et al., IEEE T-PAMI 2020, [[Paper](https://arxiv.org/abs/1904.08405)]\n* [**Survey**] **Deep Learning for Event-based Vision: A Comprehensive Survey and Benchmarks**, Xu Zheng, Yexin Liu, Yunfan Lu, Tongyan Hua, Tianbo Pan, Weiming Zhang, Dacheng Tao, Lin Wang, [[Paper](https://arxiv.org/abs/2302.08890)] \n\n* [**FE108 dataset**] **Object Tracking by Jointly Exploiting Frame and Event Domain**, Jiqing Zhang, et al., ICCV 2021, \n[[Project](https://zhangjiqing.com/dataset/)] [[added 33 videos, FE240 dataset, Baidu Cloud: password 68x3](https://pan.baidu.com/s/1gpAdfQ5Eb_GhhCDJlK3j2w)]\n[[DemoVideo](https://www.youtube.com/watch?v=EeMRO8XVv04\u0026ab_channel=JiqingZhang)] \n[[Github](https://github.com/Jee-King/ICCV2021_Event_Frame_Tracking)] \n[[Dataset](https://zhangjiqing.com/dataset/contact.html)] \n[[Paper](https://arxiv.org/pdf/2109.09052.pdf)]\n[[Baiduyun](链接：https://pan.baidu.com/s/1GFfCULGbSiv7FWCKgkb8_g 提取码：AHUT)]\n* [**SpikingJelly**] (SpikingJelly is an open-source deep learning framework for Spiking Neural Network (SNN) based on PyTorch) [[OpenI from PCL](https://git.openi.org.cn/OpenI/spikingjelly)] [[GitHub](https://github.com/fangwei123456/spikingjelly)] [[Documents](https://spikingjelly.readthedocs.io/zh_CN/latest/)]\n* [**Event-Toolkit**] https://github.com/TimoStoff/event_utils (Various representations can be obtained with (a) the raw events, (b) the voxel grid, (c) the event image, (d) the timestamp image.)\n* [**aedat 2.0.1**] AEDAT is a fast AEDAT 4 python reader, with a Rust underlying implementation. Run **pip install aedat** to install it. \n[[pypi.org](https://pypi.org/project/aedat/)] [[Github](https://github.com/neuromorphicsystems/aedat)]\n\n\n\n\u003cimg src=\"res_fig/event_representations.png\" width=\"650\"\u003e \n\n\n\n\n\n## License\nThis project is under the MIT license. See [[license](https://github.com/wangxiao5791509/VisEvent_SOT_Benchmark/blob/main/License)] for details. \n\n\n## :page_with_curl: BibTex: \nIf you find this work useful for your research, please cite the following papers: \n\n```bibtex\n@article{wang2021viseventbenchmark,\n  title={VisEvent: Reliable Object Tracking via Collaboration of Frame and Event Flows},\n  author={Xiao Wang, Jianing Li, Lin Zhu, Zhipeng Zhang, Zhe Chen, Xin Li, Yaowei Wang, Yonghong Tian, Feng Wu},\n  journal={arXiv:2108.05015},\n  year={2021}\n}\n```\n\nIf you have any questions about this work, please submit an issue or contact me via **Email**: wangxiaocvpr@foxmail.com, xiaowang@ahu.edu.cn, or **Wechat**: wangxiao5791509. Thanks for your attention! \n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwangxiao5791509%2FVisEvent_SOT_Benchmark","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwangxiao5791509%2FVisEvent_SOT_Benchmark","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwangxiao5791509%2FVisEvent_SOT_Benchmark/lists"}