{"id":22067892,"url":"https://github.com/tub-rip/MouseSIS","last_synced_at":"2025-07-24T05:31:05.524Z","repository":{"id":265753665,"uuid":"842405148","full_name":"tub-rip/MouseSIS","owner":"tub-rip","description":"The official implementation of \"MouseSIS: Space-Time Instance Segmentation of Mice\" (ECCVW 2024) ","archived":false,"fork":false,"pushed_at":"2025-06-11T22:29:12.000Z","size":9103,"stargazers_count":17,"open_issues_count":0,"forks_count":4,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-06-11T23:35:03.699Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tub-rip.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-08-14T09:31:53.000Z","updated_at":"2025-06-11T22:29:15.000Z","dependencies_parsed_at":null,"dependency_job_id":"491eabc1-fada-4ab6-853e-8d963e51636f","html_url":"https://github.com/tub-rip/MouseSIS","commit_stats":null,"previous_names":["tub-rip/mousesis"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/tub-rip/MouseSIS","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tub-rip%2FMouseSIS","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tub-rip%2FMouseSIS/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tub-rip%2FMouseSIS/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tub-rip%2FMouseSIS/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tub-rip","download_url":"https://codeload.github.com/tub-rip/MouseSIS/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tub-rip%2FMouseSIS/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":266796832,"owners_count":23985483,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-24T02:00:09.469Z","response_time":99,"last_error":null,"robots_txt_status":null,"robots_txt_updated_at":null,"robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-30T20:03:35.892Z","updated_at":"2025-07-24T05:31:05.513Z","avatar_url":"https://github.com/tub-rip.png","language":"Python","funding_links":[],"categories":["Paper List","Segmentation"],"sub_categories":["Follow-up Papers","Instance Segmentation"],"readme":"# MouseSIS: Space-Time Instance Segmentation of Mice\n\n[![Paper](https://img.shields.io/badge/arXiv-2409.03358-b31b1b.svg)](https://arxiv.org/abs/2409.03358)\n[![Poster](https://img.shields.io/badge/Poster-PDF-FF6B6B.svg)](docs/2024_ECCVW_MouseSIS_poster.pdf)\n[![Dataset](https://img.shields.io/badge/Dataset-GoogleDrive-4285F4.svg)](https://drive.google.com/drive/folders/1TQns9-WZw-n26FaUE3gqdAhGgrlRUzCp?usp=sharing)\n[![License](https://img.shields.io/badge/License-MIT-green.svg)](https://opensource.org/licenses/MIT)\n\nThis is the official repository for [**MouseSIS: A Frames-and-Events Dataset for Space-Time Instance Segmentation of Mice**](https://arxiv.org/pdf/2409.03358), accepted at the **Workshop on Neuromorphic Vision** in conjunction with **ECCV 2024** by [Friedhelm Hamann](https://friedhelmhamann.github.io/), [Hanxiong Li](https://www.linkedin.com/in/hanxiong-li-1201011bb), [Paul Mieske](https://scholar.google.de/citations?user=wQPmm6kAAAAJ\u0026hl=de), [Lars Lewejohann](https://www.vetmed.fu-berlin.de/einrichtungen/vph/we11/mitarbeitende/lewejohann_lars3/index.html) and [Guillermo Gallego](http://www.guillermogallego.es).\n\n👀 **This dataset the base for the [SIS Challenge](https://www.codabench.org/competitions/5600/) hosted in conjunction with the [CVPR 2025 Workshop on Event-based Vision](https://tub-rip.github.io/eventvision2025/).**\n\n\u003e 🏆 **[View the complete challenge results and winning teams here!](docs/challenge_results.md)**\n\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"./image/visualization_seq12_0003.jpg\" alt=\"MouseSIS Visualization\" width=\"600\"/\u003e\n\u003c/p\u003e\n\n## Key Features\n- Space-time instance segmentation dataset focused on mice tracking\n- Combined frames and event data from neuromorphic vision sensor\n- 33 sequences (~20 seconds each, ~600 frames per sequence)\n- YouTubeVIS-style annotations\n- Baseline implementation and evaluation metrics included\n\n## Timeline\n\n- **SIS challenge results** (June 2025): We released [the results](docs/challenge_results.md) of the [SIS challenge](https://www.codabench.org/competitions/5600) and presented it at the [CVPR'25 Workshop on Event-based Vision](https://tub-rip.github.io/eventvision2025/).\n\n- **v1.0.0** (Current, February 2024): Major refactoring and updates, including improved documentation.\n- **v0.1.0** (September 2023): Initial release with basic functionality and dataset.\n\n## Table of Contents\n- [Quickstart](#quickstart)\n- [Installation](#installation)\n- [Data Preparation](#data-preparation)\n    - [Data](#data)\n    - [Pretrained Weights](#pretrained-weights)\n    - [Preprocess Events](#preprocess-events)\n- [Evaluation](#evaluation)\n    - [Quickstart Evaluation](#quickstart-evaluation)\n    - [Evaluation on Full Validation Set](#evaluation-on-full-validation-set)\n    - [Evaluation on Test Set](#evaluation-on-test-set)\n- [Acknowledgements](#acknowledgements)\n- [Citation](#citation)\n- [Additional Resources](#additional-resources)\n- [License](#license)\n\n## Quickstart\n\nIf you want to work with the dataset the quickest way to access the data and get an idea of it's structure is [downloading one sequence](https://drive.google.com/drive/folders/1amY4kuaZFWdpgHg4RfTrw9Qb-tKrM-8h?usp=drive_link) and the annotations of the according split and visualizing the data, e.g. `seq12.h5`:\n\n```bash\npython scripts/visualize_events_frames_and_masks.py --h5_path data/MouseSIS/top/val/seq12.h5 --annotation_path data/MouseSIS/val_annotations.json\n```\n\nThis requires `h5py, numpy, Pillow, tqdm`. The full dataset structure is explained [here](docs/DATASET.md).\n\n## Installation\n\n1. Clone the repository:\n   ```bash\n   git clone git@github.com:tub-rip/MouseSIS.git\n   cd MouseSIS\n   ```\n\n2. Set up the environment:\n   ```bash\n   conda create --name MouseSIS python=3.8\n   conda activate MouseSIS\n   ```\n\n3. Install PyTorch (choose a command compatible with your CUDA version from the [PyTorch website](https://pytorch.org/get-started/locally/)), e.g.:\n   ```bash\n   conda install pytorch torchvision pytorch-cuda=12.1 -c pytorch -c nvidia\n   ```\n\n4. Install other dependencies:\n   ```bash\n   pip install -r requirements.txt\n   ```\n\n## Data Preparation\n\n### Data\n\n1. Create a folder for the original data\n\n    ```bash\n    cd \u003cproject-root\u003e\n    mkdir -p data/MouseSIS\n    ```\n\n2. [Download the data and annotation](https://drive.google.com/drive/folders/1amY4kuaZFWdpgHg4RfTrw9Qb-tKrM-8h) and save it in `\u003cproject-root\u003e/data/MouseSIS`.\n**You do not necessarily need to download the whole dataset, e.g. you can only download the sequences needed for the sequences you want to evaluate on**.\nThe `data/MouseSIS` folder should be organized as follows:\n\n    ```txt\n    data/MouseSIS\n    │\n    ├── top/\n    │   ├── train\n    │   │   ├── seq_02.hdf5\n    │   │   ├── seq_05.hdf5\n    │   │   ├── ...\n    │   │   └── seq_33.hdf5\n    |   ├── val\n    │   │   ├── seq_03.hdf5\n    │   │   ├── seq_04.hdf5\n    │   │   ├── ...\n    │   │   └── seq_25.hdf5\n    │   └── test\n    │       ├── seq_01.hdf5\n    │       ├── seq_07.hdf5\n    │       ├── ...\n    │       └── seq_32.hdf5\n    ├── dataset_info.csv\n    ├── val_annotations.json\n    └── train_annotations.json\n    ```\n\n    - **`top/`**: This directory contains the frame and event data for the Mouse dataset captured from top view, stored as 33 individual `.hdf5` files, each containing approximately 20 seconds of data (around 600 frames), along with temporally aligned events.\n    - **`dataset_info.csv`**: This CSV file contains metadata for each sequence, such as recording dates, providing additional context and details about the dataset.\n    - **`\u003csplit\u003e_annotations.json`**: The annotation file of top view for the respective splits follows a structure similar to MSCOCO's format in JSON, with some modifications. Note that the test annotations are not publicly available. The definition of json files is:\n\n    ```txt\n    {\n        \"info\": {\n            \"description\": \"string\",     // Dataset description\n            \"version\": \"string\",         // Version identifier\n            \"date_created\": \"string\"     // Creation timestamp\n        },\n        \"videos\": [\n            {\n                \"id\": \"string\",          // Video identifier (range: \"01\" to \"33\")\n                \"width\": integer,        // Frame width in pixels (1280)\n                \"height\": integer,       // Frame height in pixels (720)\n                \"length\": integer        // Total number of frames\n            }\n        ],\n        \"annotations\": [\n            {\n                \"id\": integer,           // Unique instance identifier\n                \"video_id\": \"string\",    // Reference to parent video\n                \"category_id\": integer,  // Object category (1 = mouse)\n                \"segmentations\": [\n                    {\n                        \"size\": [height: integer, width: integer],  // Mask dimensions\n                        \"counts\": \"string\"                          // RLE-encoded segmentation mask\n                    }\n                ],\n                \"areas\": [float],        // Object area in pixels\n                \"bboxes\": [              // Bounding box coordinates\n                    [x_min: float, y_min: float, width: float, height: float]\n                ],\n                \"iscrowd\": integer      // Crowd annotation flag (0 or 1)\n            }\n        ],\n        \"categories\": [\n            {\n                \"id\": integer,          // Category identifier\n                \"name\": \"string\",       // Category name\n                \"supercategory\": \"string\" // Parent category\n            }\n        ]\n    }\n    ```\n\n### Pretrained Weights\n\nDownload the [model weights](https://drive.google.com/drive/folders/1-P1HN4FZEy3ETn5rrQiMoDQx3378HLQW?usp=drive_link):\n   ```bash\n   cd \u003cproject-root\u003e\n   mkdir models\n   # Download yolo_e2vid.pt, yolo_frame.pt, and XMem.pth from the provided link\n   # and place them in the models directory\n   ```\n\nAfterwards, the `models` folder should be organized as follows:\n\n```txt\nmodels\n├── XMem.pth\n├── yolo_e2vid.pt\n└── yolo_frame.pt\n```\n\n### Preprocess Events\n\nThis preprocessing step is required only when evaluating the ModelMixSort method from the paper. It relies on e2vid images reconstructed at the grayscale image timesteps.\n\n```bash\npython scripts/preprocess_events_to_e2vid_images.py --data_root data/MouseSIS\n```\n\n## Evaluation\n\nAfter downloading the data and model weights, proceed with evaluation. First run inference, e.g. our provided inference script like:\n\n```bash\npython3 scripts/inference.py --config \u003cpath-to-config-yaml\u003e\n```\n\nThis saves a file `output/\u003ctracker-name\u003e/final_results.json`. The file contains the predictions in this structure:\n\n```txt\n[\n  {\n    \"video_id\": int,\n    \"score\": float,\n    \"instance_id\": int,\n    \"category_id\": int,\n    \"segmentations\": [\n      null | {\n        \"size\": [int, int],\n        \"counts\": \"RLE encoded string\"\n      },\n      ...\n    ],\n  },\n  ...\n]\n```\n\nThen run the evaluation script like this:\n\n```bash\npython scripts/eval.py --TRACKERS_TO_EVAL \u003ctracker-name\u003e --SPLIT_TO_EVAL \u003csplit-name\u003e\n```\n\nBelow are specific options listed.\n\n### Quickstart Evaluation\n\nThis section describes how to run a minimal evaluation workflow on one sequence of the validation set. Only download the sequence `seq_25.hdf5` from the validation set and the according annotations `val_annotations.json`. The resulting folder should look as follows:\n\n```txt\ndata/MouseSIS\n│\n├── top/\n|   ├── val\n│   │   └── seq_25.hdf5\n└── val_annotations.json\n```\n\nNow you can run inference as\n\n```bash\npython3 scripts/inference.py --config configs/predict/quickstart.yaml\n```\n\nand then evaluation as\n\n```bash\npython scripts/eval.py --TRACKERS_TO_EVAL quickstart --SPLIT_TO_EVAL val\n```\n\nThis should return the following results\n\n| Sequence |   HOTA   |   MOTA   |   IDF1    |\n|----------|----------|----------|-----------|\n| 25       | 30.15    | 39.125   | 35.315    |\n| **Avg.** | 30.15    | 39.125   | 35.315    |\n\n### Evaluation on Full Validation Set\n\nSimilar as for quickstart but download all sequences of the validation set (sequences 3, 4, 12, 25).\n\n```bash\npython3 scripts/inference.py --config configs/predict/combined_on_validation.yaml\npython scripts/eval.py --TRACKERS_TO_EVAL combined_on_validation --SPLIT_TO_EVAL val\n```\n\nHere you should get the following results\n\n| Sequence |   HOTA   |   MOTA   |   IDF1    |\n|----------|----------|----------|-----------|\n| 3        | 54.679   | 72.432   | 60.212    |\n| 4        | 51.717   | 64.942   | 58.36     |\n| 12       | 39.497   | 66.049   | 45.431    |\n| 25       | 30.15    | 39.125   | 35.315    |\n| **Avg.** | 45.256   | 62.097   | 50.459    |\n\n### Evaluation on Test Set Without Sequences 1 \u0026 7 (SIS Challenge)\n\nIn this case, download all test sequences and run\n\n```bash\npython3 scripts/inference.py --config configs/predict/sis_challenge_baseline.yaml\n```\n\nFor evaluation you can upload the `final_results.json` to the challenge/benchmark page, which results in the following combined metrics:\n\n| Sequence |   HOTA   |   MOTA   |   IDF1    |\n|----------|----------|----------|-----------|\n| **Avg.** |  0.43    |  0.45    |  0.5      |\n\nPlease note that results vary slightly from the ones reported in the paper after updates for the challenge. Please refer to version v0.1.0 to reproduce the exact paper results.\n\n## Acknowledgements\n\nWe greatfully appreciate the following repositories and thank the authors for their excellent work:\n\n- [XMem](https://github.com/hkchengrex/XMem)\n- [TrackEval](https://github.com/JonathonLuiten/TrackEval)\n\n## Citation\n\nIf you find this work useful in your research, please consider citing:\n\n```bibtex\n@inproceedings{hamann2024mousesis,\n  title={{MouseSIS}: A Frames-and-Events Dataset for Space-Time Instance Segmentation of Mice},\n  author={Friedhelm Hamann and Hanxiong Li and Paul Mieske and Lars Lewejohann and Guillermo Gallego},\n  booktitle={European Conference on Computer Vision Workshops (ECCVW)},\n  year={2024}\n}\n```\n\n## Additional Resources\n\n* [Recording Software (CoCapture)](https://github.com/tub-rip/CoCapture)\n* [Secrets of Event-Based Optical Flow (TPAMI 2024)](https://github.com/tub-rip/event_based_optical_flow)\n* [EVILIP: Event-based Image Reconstruction as a Linear Inverse Problem (TPAMI 2022)](https://github.com/tub-rip/event_based_image_rec_inverse_problem)\n* [Research page (TU Berlin, RIP lab)](https://sites.google.com/view/guillermogallego/research/event-based-vision)\n* [Science Of Intelligence Homepage](https://www.scienceofintelligence.de/)\n* [Course at TU Berlin](https://sites.google.com/view/guillermogallego/teaching/event-based-robot-vision)\n* [Survey paper](http://rpg.ifi.uzh.ch/docs/EventVisionSurvey.pdf)\n* [List of Event-based Vision Resources](https://github.com/uzh-rpg/event-based_vision_resources)\n\n## License\n\nThis project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftub-rip%2FMouseSIS","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftub-rip%2FMouseSIS","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftub-rip%2FMouseSIS/lists"}