{"id":18385977,"url":"https://github.com/uzh-rpg/event_representation_study","last_synced_at":"2025-04-07T00:32:43.169Z","repository":{"id":197507553,"uuid":"551409707","full_name":"uzh-rpg/event_representation_study","owner":"uzh-rpg","description":"Official PyTorch implementation of the ICCV 2023 paper: From Chaos Comes Order: Ordering Event Representations for Object Recognition and Detection.","archived":false,"fork":false,"pushed_at":"2024-04-03T13:12:46.000Z","size":82390,"stargazers_count":58,"open_issues_count":4,"forks_count":7,"subscribers_count":6,"default_branch":"master","last_synced_at":"2025-03-22T09:43:30.542Z","etag":null,"topics":["event-based-camera","event-based-vision","event-camera","event-cameras","events","gen1","gen4","gromov-wasserstein","gromov-wasserstein-distance","gryffin","imagenet","machine-learning","machine-learning-algorithms","neural-network","object-detection","optimal-transport","optimization","python","yolo","yolov6"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/uzh-rpg.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2022-10-14T10:50:37.000Z","updated_at":"2025-03-07T12:10:14.000Z","dependencies_parsed_at":null,"dependency_job_id":"a4770699-9f4f-49f7-93ef-3185912c0341","html_url":"https://github.com/uzh-rpg/event_representation_study","commit_stats":null,"previous_names":["uzh-rpg/event_representation_study"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/uzh-rpg%2Fevent_representation_study","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/uzh-rpg%2Fevent_representation_study/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/uzh-rpg%2Fevent_representation_study/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/uzh-rpg%2Fevent_representation_study/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/uzh-rpg","download_url":"https://codeload.github.com/uzh-rpg/event_representation_study/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247574088,"owners_count":20960495,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["event-based-camera","event-based-vision","event-camera","event-cameras","events","gen1","gen4","gromov-wasserstein","gromov-wasserstein-distance","gryffin","imagenet","machine-learning","machine-learning-algorithms","neural-network","object-detection","optimal-transport","optimization","python","yolo","yolov6"],"created_at":"2024-11-06T01:19:46.919Z","updated_at":"2025-04-07T00:32:38.155Z","avatar_url":"https://github.com/uzh-rpg.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# From Chaos Comes Order: Ordering Event Representations for Object Recognition and Detection\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://rpg.ifi.uzh.ch/img/papers/iccv23_zubic.png\"\u003e\n\u003c/p\u003e\n\nOfficial PyTorch implementation of the ICCV 2023 paper: [From Chaos Comes Order: Ordering Event Representations for Object Recognition and Detection](https://arxiv.org/abs/2304.13455).\n\n## 🖼️ Check Out Our Poster! 🖼️ [here](https://download.ifi.uzh.ch/rpg/event_representation_study/ICCV23_Zubic.pdf)\n\n## Citation\nIf you find this work useful, please consider citing:\n```bibtex\n@InProceedings{Zubic_2023_ICCV,\n    author    = {Zubi\\'c, Nikola and Gehrig, Daniel and Gehrig, Mathias and Scaramuzza, Davide},\n    title     = {From Chaos Comes Order: Ordering Event Representations for Object Recognition and Detection},\n    booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},\n    month     = {October},\n    year      = {2023},\n    pages     = {12846-12856}\n}\n```\n\n## Conda Installation\nWe highly recommend using [Mambaforge](https://github.com/conda-forge/miniforge#mambaforge) to reduce the installation time.\n```Bash\nconda create -y -n event_representation python=3.8\nconda activate event_representation\nconda install -y pytorch==1.13.1 torchvision==0.14.1 torchaudio==0.13.1 pytorch-cuda=11.7 -c pytorch -c nvidia\npip install matplotlib tonic tqdm numba POT scikit-learn wandb pyyaml opencv-python bbox-visualizer pycocotools h5py hdf5plugin timm tensorboard addict\nconda install -y pyg -c pyg\nconda install -y pytorch-scatter -c pyg\ncd ev-licious\npip install .\ncd ..\ncd gryffin\npip install .\n```\n\n## Required Data\n* To evaluate or train the model, you will need to download the required preprocessed datasets:\n  \u003ctable\u003e\u003ctbody\u003e\n  \u003cth valign=\"bottom\"\u003e\u003c/th\u003e\n  \u003cth valign=\"bottom\"\u003eTrain\u003c/th\u003e\n  \u003cth valign=\"bottom\"\u003eValidation\u003c/th\u003e\n  \u003cth valign=\"bottom\"\u003eTest\u003c/th\u003e\n  \u003ctr\u003e\u003ctd align=\"left\"\u003eGen1\u003c/td\u003e\n  \u003ctd align=\"center\"\u003e\u003ca href=\"https://download.ifi.uzh.ch/rpg/event_representation_study/gen1/training.h5\"\u003edownload\u003c/a\u003e\u003c/td\u003e\n  \u003ctd align=\"center\"\u003e\u003ca href=\"https://download.ifi.uzh.ch/rpg/event_representation_study/gen1/validation.h5\"\u003edownload\u003c/a\u003e\u003c/td\u003e\n  \u003ctd align=\"center\"\u003e\u003ca href=\"https://download.ifi.uzh.ch/rpg/event_representation_study/gen1/testing.h5\"\u003edownload\u003c/a\u003e\u003c/td\u003e\n  \u003c/tbody\u003e\u003c/table\u003e\n\n* 1 Mpx dataset needs to be downloaded from the following [repository](https://github.com/wds320/AAAI_Event_based_detection) and then processed using [precompute_reps.py](https://github.com/uzh-rpg/event_representation_study/blob/master/ev-YOLOv6/yolov6/data/gen4/precompute_reps.py) file.\n\n* Annotations for GEN1 and 1 Mpx datasets can be downloaded from [here](https://download.ifi.uzh.ch/rpg/event_representation_study/annotations.zip).\n\n## Pre-trained Checkpoints\n### [Gen1](https://download.ifi.uzh.ch/rpg/event_representation_study/GEN1.zip)\n### [1 Mpx](https://download.ifi.uzh.ch/rpg/event_representation_study/GEN4.zip)\nContains folders of all trained models (in the end full YOLOv6 backbone that is now on-par with Swin-V2 but faster training and less memory). Each folder has weights folder, and we use `best_ckpt.pt` as the checkpoint.\u003cbr\u003e\nCurrently, contains two optimized representations we found (small variations), by default the second one is used - aim for `gen1_optimized_2` and `gen1_optimized_augment_2` weights when evaluating.\u003cbr\u003e\nIf you want to use the first one, uncomment it at lines 16-66 ([optimized_representation.py](https://github.com/uzh-rpg/event_representation_study/blob/master/representations/optimized_representation.py)) and comment out the second one (lines 86-134).\u003cbr\u003e\n`gen1_optimized_augment_2` should produce the following results (50.6% mAP):\n```\n Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.506\n Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.775\n Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.539\n Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.420\n Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.580\n Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.528\n Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.319\n Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.635\n Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.666\n Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.622\n Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.712\n Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.666\n```\n\n\n## Evaluation\n- Set `DATASET_PATH` as the path to either the 1 Mpx or Gen1 dataset directory\n- Set `OUTPUT_DIR` to the path where you want to save evaluation outputs.\n- Set `conf-file`'s (e.g. ev-YOLOv6/configs/gen1_optimized_augment2.py by default; configs are same for both gen1 and gen4) pre-trained parameter to the path of the model (best_ckpt.pt)\n- Evaluation scripts also start from `train.py` file, but use `testing` parameter\n\nFor simplicity, we are only showing the validation script for Gen1. For 1 Mpx it should be similar.\n### Gen1 (no augment)\n```Bash\npython ev-YOLOv6/tools/train.py --wandb_name test_gen1_optimized_augment --file $DATASET_PATH \\\n--data-path ev-YOLOv6/data/gen1.yaml --conf-file ev-YOLOv6/configs/gen1_optimized_augment2.py \\\n--img-size 640 --batch-size 32 --epochs 100 --device 0 --output-dir $OUTPUT_DIR \\\n--name test_gen1_optimized_augment --representation OptimizedRepresentation --dataset gen1 --testing\n```\n### Gen1 (augment)\n```Bash\npython ev-YOLOv6/tools/train.py --wandb_name test_gen1_optimized_augment --file $DATASET_PATH \\\n--data-path ev-YOLOv6/data/gen1.yaml --conf-file ev-YOLOv6/configs/gen1_optimized_augment2.py \\\n--img-size 640 --batch-size 32 --epochs 100 --device 0 --output-dir $OUTPUT_DIR \\\n--name test_gen1_optimized_augment --representation OptimizedRepresentation --dataset gen1 --testing --augment\n```\n\n## Training\nPretrained Swin-V2 weights (`swinv2_yolov6l6.pt`) can be downloaded from [here](https://download.ifi.uzh.ch/rpg/event_representation_study/swinv2_yolov6l6.pt). \u003cbr\u003e\nYou can set the [pretrained variable](https://github.com/uzh-rpg/event_representation_study/blob/master/ev-YOLOv6/configs/swinv2_yolov6l6_finetune.py#L4) to your path to `swinv2_yolov6l6.pt` file.\n\n### Gen1\n- Set `OUTPUT_DIR` to the directory where you want to store training outputs\n\nTraining without augmentation:\n```Bash\npython ev-YOLOv6/tools/train.py --wandb_name gen1_optimized --file /shares/rpg.ifi.uzh/dgehrig/gen1 \\\n--data-path ev-YOLOv6/data/gen1.yaml --conf-file ev-YOLOv6/configs/swinv2_yolov6l6_finetune.py \\\n--img-size 640 --batch-size 32 --epochs 100 --device 0 --output-dir $OUTPUT_DIR \\\n--name gen1_optimized --representation OptimizedRepresentation --dataset gen1\n```\nTraining with augmentation:\n```Bash\npython ev-YOLOv6/tools/train.py --wandb_name gen1_optimized_augment --file /shares/rpg.ifi.uzh/dgehrig/gen1 \\\n--data-path ev-YOLOv6/data/gen1.yaml --conf-file ev-YOLOv6/configs/swinv2_yolov6l6_finetune.py \\\n--img-size 640 --batch-size 32 --epochs 100 --device 0 --output-dir $OUTPUT_DIR \\\n--name gen1_optimized_augment --representation OptimizedRepresentation --dataset gen1 --augment\n```\n\n### 1 Mpx\n- Set `OUTPUT_DIR` to the directory where you want to store training outputs\n\nTraining without augmentation:\n```Bash\npython ev-YOLOv6/tools/train.py --wandb_name gen4_optimized \\\n--file /shares/rpg.ifi.uzh/nzubic/datasets/gen4/OptimizedRepresentation \\\n--data-path ev-YOLOv6/data/gen4.yaml --conf-file ev-YOLOv6/configs/swinv2_yolov6l6_finetune.py \\\n--img-size 640 --batch-size 32 --epochs 100 --device 0 --output-dir $OUTPUT_DIR \\\n--name gen4_optimized --representation OptimizedRepresentation --dataset gen4\n```\nTraining with augmentation:\n```Bash\npython ev-YOLOv6/tools/train.py --wandb_name gen4_optimized_augment \\\n--file /shares/rpg.ifi.uzh/nzubic/datasets/gen4/OptimizedRepresentation \\\n--data-path ev-YOLOv6/data/gen4.yaml --conf-file ev-YOLOv6/configs/swinv2_yolov6l6_finetune.py \\\n--img-size 640 --batch-size 32 --epochs 100 --device 0 --output-dir $OUTPUT_DIR \\\n--name gen4_optimized_augment --representation OptimizedRepresentation --dataset gen4 --augment\n```\n\n## Mini N-ImageNet experiments\n* All details regarding the execution of Mini N-ImageNet experiments can be seen in [n_imagenet/scripts](https://github.com/uzh-rpg/event_representation_study/tree/master/n_imagenet/scripts) folder.\n* Details on how to download the Mini N-ImageNet dataset and prepare data can be seen at their official repo [here](https://github.com/82magnolia/n_imagenet).\n\n## Running GWD computation\n* Computation can be run with the following command:\n```Bash\nID=0\nREP_NAME=VoxelGrid\n\nCUDA_VISIBLE_DEVICES=$ID python representations/representation_search/gen1_compute.py \\\n--event_representation_name $REP_NAME\n```\nwhere `ID` represents ID of the device, and `REP_NAME` represents the representation name.\n\n## Running Gryffin optimization\n`python representations/representation_search/optimization.py` \u003cbr\u003e\nChange file [Path](https://github.com/uzh-rpg/event_representation_study/blob/master/representations/representation_search/optimization.py#L294) to the directory of GEN1 folder where `training.h5`, `validation.h5` and `testing.h5` files are. \u003cbr\u003e\nChange [`SAVE_PATH`](https://github.com/uzh-rpg/event_representation_study/blob/master/representations/representation_search/optimization.py#L272) of run_optimization function to the path where you want to save the results.\n\u003cbr\u003e\u003cbr\u003e\nObtained optimal representation (ERGO-12):\u003cbr\u003e\n![ERGO-12](https://github.com/uzh-rpg/event_representation_study/blob/master/viz/ergo12_visualization.png)\n\n## Code Acknowledgments\nThis project has used code from the following projects:\n- [Swin-Transformer](https://github.com/microsoft/Swin-Transformer) for the Swin Transformer version 2 implementation in PyTorch\n- [YOLOv6](https://github.com/meituan/YOLOv6) for the object detection pipeline\n- [n_imagenet](https://github.com/82magnolia/n_imagenet) for Mini N-ImageNet experiments\n- [AAAI_Event_based_detection](https://github.com/wds320/AAAI_Event_based_detection) for processed/filtered 1 Mpx dataset\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fuzh-rpg%2Fevent_representation_study","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fuzh-rpg%2Fevent_representation_study","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fuzh-rpg%2Fevent_representation_study/lists"}