{"id":18792432,"url":"https://github.com/prbonn/mask4d","last_synced_at":"2025-08-20T07:30:52.137Z","repository":{"id":198712763,"uuid":"694552036","full_name":"PRBonn/Mask4D","owner":"PRBonn","description":"Mask4D: End-to-End Mask-Based 4D Panoptic Segmentation for LiDAR Sequences, RA-L, 2023 ","archived":false,"fork":false,"pushed_at":"2024-07-25T10:40:54.000Z","size":321,"stargazers_count":57,"open_issues_count":0,"forks_count":3,"subscribers_count":3,"default_branch":"master","last_synced_at":"2024-12-10T10:02:26.890Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/PRBonn.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-09-21T08:20:13.000Z","updated_at":"2024-12-04T03:06:02.000Z","dependencies_parsed_at":null,"dependency_job_id":"36243b89-2b89-48de-baa6-180f4e77efa1","html_url":"https://github.com/PRBonn/Mask4D","commit_stats":null,"previous_names":["prbonn/mask4d"],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PRBonn%2FMask4D","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PRBonn%2FMask4D/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PRBonn%2FMask4D/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PRBonn%2FMask4D/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/PRBonn","download_url":"https://codeload.github.com/PRBonn/Mask4D/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":230400616,"owners_count":18219831,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-07T21:19:52.199Z","updated_at":"2024-12-19T08:09:17.929Z","avatar_url":"https://github.com/PRBonn.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Mask4D: End-to-End Mask-Based 4D Panoptic Segmentation for LiDAR Sequences\n\nThis is the official implementation of [Mask4D](https://www.ipb.uni-bonn.de/wp-content/papercite-data/pdf/marcuzzi2023ral-meem.pdf).\n\n## Overview\n* Mask4D is a method for 4D panoptic segmentation using masks. It builds on top of [MaskPLS](https://github.com/PRBonn/MaskPLS) using [SphereFormer](https://github.com/dvlab-research/SphereFormer/tree/master) as feature extractor.\n* We reuse the output queries of previous steps to decode and track the same instance over time.\n* It is an end-to-end approach without post-processing step like clustering.\n* We propose Position-aware mask attention to provide prior positional information to the cross-attention and improve the segmentation.\n\n![](pics/overview.jpg)\n\n## Get started\n\nInstall this package by running in the root directory of this repo:\n```\npip3 install -U -e .\n```\n\nInstall pdependencies (we test on python=3.8.10, pytorch==1.12.0, cuda==11.3)\n\n```\npip3 install torch==1.12.0+cu113 torchvision==0.13.0+cu113 --extra-index-url https://download.pytorch.org/whl/cu113\npip3 install -r requirements.txt\n```\n\nInstall [SparseTransformer](https://github.com/dvlab-research/SparseTransformer).\n\n## Data preparation: SemanticKITTI\nDownload the [SemanticKITTI](http://www.semantic-kitti.org/dataset.html#overview) dataset inside the directory `data/kitti/`. The directory structure should look like this:\n```\n./\n└── data/\n    └── kitti\n        └── sequences\n            ├── 00/           \n            │   ├── velodyne/\t\n            |   |\t├── 000000.bin\n            |   |\t├── 000001.bin\n            |   |\t└── ...\n            │   └── labels/ \n            |       ├── 000000.label\n            |       ├── 000001.label\n            |       └── ...\n            ├── 08/ # for validation\n            ├── 11/ # 11-21 for testing\n            └── 21/\n                └── ...\n```\n\n## Pretrained models\n\n* [Mask4D](https://www.ipb.uni-bonn.de/html/projects/mask_4d/mask4d.ckpt)\n* [MaskPLS](https://www.ipb.uni-bonn.de/html/projects/mask_4d/maskpls.ckpt)\n\n## Reproducing results\n```\npython3 scripts/evaluate_model.py --w [path_to_4D_model]\n```\n\n## Training\n\nWe leverage the weights of the 3D model MaskPLS with SphereFormer as backbone.\n\n```\npython3 scripts/train_model.py --w [path_to_3D_model]\n\n```\n\n## Citation\n```bibtex\n@article{marcuzzi2023ral-meem,\n  author = {R. Marcuzzi and L. Nunes and L. Wiesmann and E. Marks and J. Behley and C. Stachniss},\n  title = {{Mask4D: End-to-End Mask-Based 4D Panoptic Segmentation for LiDAR Sequences}},\n  journal = ral,\n  year = 2023,\n  volume = {8},\n  number = {11},\n  pages = {7487-7494},\n  doi = {10.1109/LRA.2023.3320020},\n  url = {https://www.ipb.uni-bonn.de/wp-content/papercite-data/pdf/marcuzzi2023ral-meem.pdf},\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fprbonn%2Fmask4d","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fprbonn%2Fmask4d","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fprbonn%2Fmask4d/lists"}