{"id":29115699,"url":"https://github.com/tub-rip/event_penguins","last_synced_at":"2025-06-29T11:13:27.498Z","repository":{"id":238878144,"uuid":"788999721","full_name":"tub-rip/event_penguins","owner":"tub-rip","description":"The official implementation of \"Low-power, Continuous Remote Behavioral Localization with Event Cameras\" (CVPR 2024) ","archived":false,"fork":false,"pushed_at":"2024-07-15T15:39:52.000Z","size":1204,"stargazers_count":9,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2024-07-15T19:05:59.670Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tub-rip.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-04-19T14:02:39.000Z","updated_at":"2024-07-15T15:39:54.000Z","dependencies_parsed_at":"2024-05-08T17:16:16.207Z","dependency_job_id":"64f5ce70-3e87-4bd7-9dfa-149b43d2c60f","html_url":"https://github.com/tub-rip/event_penguins","commit_stats":null,"previous_names":["tub-rip/event_penguins"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/tub-rip/event_penguins","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tub-rip%2Fevent_penguins","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tub-rip%2Fevent_penguins/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tub-rip%2Fevent_penguins/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tub-rip%2Fevent_penguins/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tub-rip","download_url":"https://codeload.github.com/tub-rip/event_penguins/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tub-rip%2Fevent_penguins/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":262581513,"owners_count":23331925,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-06-29T11:13:11.610Z","updated_at":"2025-06-29T11:13:27.415Z","avatar_url":"https://github.com/tub-rip.png","language":"Python","funding_links":[],"categories":["Segmentation"],"sub_categories":["Temporal Action Detection"],"readme":"# Event Penguins (CVPR 2024)\n\nThis is the official repository for **Low-power, Continuous Remote Behavioral Localization with Event Cameras** accepted at **CVPR 2024** by [Friedhelm Hamann](https://friedhelmhamann.github.io/), [Suman Ghosh](https://scholar.google.com/citations?user=4QgBeWAAAAAJ\u0026hl=de\u0026oi=sra), [Ignacio Juarez Martinez](https://www.biology.ox.ac.uk/people/ignacio-nacho-juarez-martinez), [Tom Hart](https://scholar.google.co.uk/citations?user=HxUEZy0AAAAJ\u0026hl=en), [Alex Kacelnik](https://users.ox.ac.uk/~kgroup/people/alexkacelnik.shtml), [Guillermo Gallego](https://sites.google.com/view/guillermogallego)\n\n\u003ch2 align=\"left\"\u003e\n\n [Project Page](https://tub-rip.github.io/eventpenguins/) | [Paper](https://arxiv.org/pdf/2312.03799.pdf) | [Video](https://www.youtube.com/watch?v=o79wbZh0gU4\u0026feature=youtu.be) | [Data](https://drive.google.com/drive/folders/1VoKEg6CSITmPH27R19tGzyzbUIrmhRDV?usp=drive_link)\n\n\u003c/h2\u003e\n\n[![Low-power, Continuous Remote Behavioral Localization with Event Cameras](docs/eventpenguins_yt_thumbnail.png)](https://youtu.be/o79wbZh0gU4)\n\n## Table of Contents\n\n1. [Citation](#citation)\n2. [Quickstart](#quickstart)\n   - [Setup](#setup)\n   - [Preprocessing the data](#preprocessing-the-data)\n   - [Inference](#inference)\n3. [Details](#details)\n   - [Original Data](#original-data)\n   - [Pre-processed Data](#pre-processed-data)\n   - [Annotations](#annotations)\n4. [Acknowledgements](#acknowledgements)\n5. [In the Press](#in-the-press)\n6. [Additional Resources](#additional-resources)\n\n## Citation\n\nIf you use this work in your research, please consider citing:\n\n```bibtex\n@InProceedings{Hamann24cvpr,\n    author    = {Hamann, Friedhelm and Ghosh, Suman and Martinez, Ignacio Juarez and Hart, Tom and Kacelnik, Alex and Gallego, Guillermo},\n    title     = {Low-power Continuous Remote Behavioral Localization with Event Cameras},\n    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},\n    month     = {June},\n    year      = {2024},\n    pages     = {18612-18621}\n}\n```\n\n## Quickstart\n\n### Setup\n\nYou can use [Miniconda](https://docs.conda.io/en/latest/miniconda.html) to set up an environment:\n\n```bash\nconda create --name eventpenguins python=3.8\nconda activate eventpenguins\n```\n\nInstall PyTorch by choosing a command that matches your CUDA version. You can find the compatible commands on the [PyTorch official website](https://pytorch.org/get-started/locally/) (tested with PyTorch 2.2.2), e.g.:\n\n```bash\nconda install pytorch torchvision pytorch-cuda=12.1 -c pytorch -c nvidia\n```\n\nInstall other required packages:\n\n```bash\npip install -r requirements.txt\n```\n\n### Preprocessing the data\n\n1. Create a folder for the data:\n\n```bash\ncd \u003cproject-root\u003e\nmkdir data\n```\n\n2. [Download the data](https://drive.google.com/drive/folders/1VoKEg6CSITmPH27R19tGzyzbUIrmhRDV?usp=drive_link) and save it in `\u003cproject-root\u003e/data`.\n\n3. Create the pre-processed dataset with the following command:\n\n```bash\npython scripts/preprocess.py --data_root data/EventPenguins --output_dir data --recording_info_path config/annotations/recording_info.csv\n```\n\nThis crops the events according to the pre-annotated nests and stores the recordings according to the split specified in [the paper](https://arxiv.org/pdf/2312.03799.pdf).\n\n### Inference\n\n1. Create a folder for models:\n\n```bash\nmkdir models\n```\n\n2. Download the pre-trained model weights from [here](https://drive.google.com/drive/folders/1A4JB97u4879oQ88FsgXa_LAU42ffpW1K?usp=sharing) and save them in the `models` folder.\n\n3. Run inference with the following command:\n\n```bash\npython scripts/inference.py --config config/exp/inference.yaml --verbose\n```\n\n## Details\n\n### Original Data\n\nThe [EventPenguins dataset](https://drive.google.com/drive/folders/1VoKEg6CSITmPH27R19tGzyzbUIrmhRDV?usp=drive_link) contains 24 ten-minute recordings, with 16 annotated nests.\nAn overview of the data can be found in `config/annotations/recording_info.csv`.\nEach recording has a `roi_group_id`, which links to the location of the 16 pre-annotated regions of interest, which can be found in `config/annotations/rois` (new set of ROIs when the camera was moved).\nThe dataset is structured as follows:\n\n```\nEventPenguins/\n├── \u003cyy-mm-dd\u003e_\u003chh-mm-ss\u003e/  # (these folders are referred to as \"recordings\")\n│   ├── frames/\n│   │   ├── 000000000000.png\n│   │   ├── 000000000001.png\n│   │   └── ...\n│   ├── events.h5\n│   ├── frame_timestamps.txt  # [us]\n│   └── metadata.yaml       \n└── ...\n```\n\nPlease note that we do not use the grayscale frames in our method but provide them for completeness.\n\n### Pre-processed Data\n\n#### Structure\n\nThe processed data is stored in a single HDF5 file named `preprocessed.h5`. The file structure is organized as follows:\n\n- Each ten-minute recording is stored in a group labeled by its timestamp (e.g., `22-01-12_17-26-00`).\n- Each group (timestamp) contains multiple subgroups, each corresponding to a specific ROI (nest) identified by an ID (e.g., `N01`).\n- Each ROI subgroup contains:\n  - An `events` dataset, where each event is represented as a row `[x, y, t, p]` indicating the event's x-position, y-position, timestamp (us), and polarity, respectively.\n  - Attributes `height` and `width` indicating the dimensions of the ROI.\n\n#### Attributes\n\nEach subgroup (ROI) has the following attributes:\n- `height`: The height of the ROI in pixels.\n- `width`: The width of the ROI in pixels.\n\nEach main group (recording timestamp) has the following attribute:\n- `split`: Indicates the data split (e.g., `train`, `test`, `validate`) that the recording belongs to.\n\n### Annotations\n\nThe annotations are in `config/annotations/annotations.json`.\nThe structure is very similar to [ActivityNet](https://github.com/activitynet/ActivityNet), with an additional layer to consider different nests.\n\n```json\n{\n  \"version\": \"VERSION 0.0\",\n  \"database\": {\n    \"\u003cyy-mm-dd\u003e_\u003chh-mm-ss\u003e\": {\n      \"annotations\": {\n        \"\u003croi_id\u003e\": [\n          {\n            \"label\": \u003clabel\u003e,\n            \"segment\": [\n              \u003ct_start\u003e,\n              \u003ct_end\u003e\n            ]\n          },\n          ...\n        ]\n      }\n    }\n  }\n}\n```\n* `\u003cyy-mm-dd\u003e_\u003chh-mm-ss\u003e` is the identifier for a ten-minute recording\n* `roi_id` is an integer number encoding the nest\n* `t_start` and `t_end` are the start and end times of an action in seconds \n* The `label` is one of `[\"ed\", \"adult_flap\", \"chick_flap\"]`.\n\n`\"adult_flap\"` and `\"chick_flap\"` are other types of wing flapping easily confused with the ecstatic display (`ed`).\nWe provide these labels for completeness, but they are not considered in our method.\n\n## Acknowledgements\n\nThe evaluation for activity detection is largely inspired by [ActivityNet](https://github.com/activitynet/ActivityNet). We thank the authors for their excellent work.\n\n## In the Press\n\n* [TU Berlin Press Release](https://www.tu.berlin/ueber-die-tu-berlin/profil/pressemitteilungen-nachrichten/pinguine-in-ekstase)\n* [Tagesspiegel Article (German)](https://www.tagesspiegel.de/wissen/tierbeobachtung-in-der-antarktis-das-ratsel-der-ekstatischen-pinguine-11967704.html)\n\n## Additional Resources\n\n* [Recording Software (CoCapture)](https://github.com/tub-rip/CoCapture)\n* [Stereo Co-capture System for Recording and Tracking Fish with Frame-and Event Cameras](https://arxiv.org/pdf/2207.07332.pdf)\n* [Homepage (TU Berlin, RIP lab)](https://sites.google.com/view/guillermogallego/research/event-based-vision)\n* [Homepage (Science Of Intelligence)](https://www.scienceofintelligence.de/)\n* [Class at TU Berlin](https://sites.google.com/view/guillermogallego/teaching/event-based-robot-vision)\n* [Survey paper](http://rpg.ifi.uzh.ch/docs/EventVisionSurvey.pdf)\n* [List of Resources](https://github.com/uzh-rpg/event-based_vision_resources)","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftub-rip%2Fevent_penguins","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftub-rip%2Fevent_penguins","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftub-rip%2Fevent_penguins/lists"}