{"id":13444361,"url":"https://github.com/Angzz/awesome-panoptic-segmentation","last_synced_at":"2025-03-20T18:32:29.082Z","repository":{"id":37359357,"uuid":"164067097","full_name":"Angzz/awesome-panoptic-segmentation","owner":"Angzz","description":"Panoptic Segmentation Resources List","archived":false,"fork":false,"pushed_at":"2020-08-25T10:04:54.000Z","size":1498,"stargazers_count":544,"open_issues_count":2,"forks_count":99,"subscribers_count":39,"default_branch":"master","last_synced_at":"2024-05-23T04:02:01.869Z","etag":null,"topics":["computer-vision","deep-learning","panoptic-segmentation"],"latest_commit_sha":null,"homepage":"","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Angzz.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-01-04T06:23:24.000Z","updated_at":"2024-05-10T08:42:27.000Z","dependencies_parsed_at":"2022-09-12T23:51:44.463Z","dependency_job_id":null,"html_url":"https://github.com/Angzz/awesome-panoptic-segmentation","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Angzz%2Fawesome-panoptic-segmentation","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Angzz%2Fawesome-panoptic-segmentation/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Angzz%2Fawesome-panoptic-segmentation/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Angzz%2Fawesome-panoptic-segmentation/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Angzz","download_url":"https://codeload.github.com/Angzz/awesome-panoptic-segmentation/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244670553,"owners_count":20491008,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["computer-vision","deep-learning","panoptic-segmentation"],"created_at":"2024-07-31T04:00:20.792Z","updated_at":"2025-03-20T18:32:29.076Z","avatar_url":"https://github.com/Angzz.png","language":null,"funding_links":[],"categories":["Uncategorized","SemanticSeg","🌐 Panoptic Segmentation","CV","Other Lists"],"sub_categories":["Uncategorized","📚 Key Implementations","TeX Lists"],"readme":"# Awesome-Panoptic-Segmentation [![Awesome](https://awesome.re/badge.svg)](https://awesome.re)\nThis repo is a collection of the challenging panoptic segmentation, including papers, codes, and benchmark results, etc.\n\n##  Outline\n* [Panoptic Segmentation](#panoptic-segmentation)\n* [Datasets](#datasets)\n* [Evaluation](#evaluation)\n* [Benchmark Results](#benchmark-results)\n* [Papers](#papers)\n* [Tutorials](#tutorials)\n* [Blogs](#blogs)\n\n\n## Panoptic Segmentation\n\u003cdiv align=\"center\"\u003e\u003cimg src=\"img/panoptic_segmentation_overview2.png\" width=\"900\" height=\"180\"\u003e\u003c/div\u003e\nSummarize in one sentence : Panoptic Segmentation proposes to solve the semantic segmentation(*Stuff*) and instance segmentation(*Thing*) in a unified and general manner.\n\n\n### Structure Overview\n\u003cdiv align=\"center\"\u003e\u003cimg src=\"img/panoptic_structure.png\" width=\"800\" height=\"240\"\u003e\u003c/div\u003e\n\nfrom [UPSNet](https://arxiv.org/pdf/1901.03784.pdf).\n\n\n## Datasets\nGenerally, the datasets which contains both semantic and instance annotations can be used to solve the challenging *panoptic* task.\n\n* [COCO-Panoptic](http://cocodataset.org/)\n* [Cityscapes](https://www.cityscapes-dataset.com/)\n* [Mapillary Vistas](https://blog.mapillary.com/product/2017/05/03/mapillary-vistas-dataset.html)\n* [ADE20K](http://groups.csail.mit.edu/vision/datasets/ADE20K/)\n* [IDD20K](http://idd.insaan.iiit.ac.in/)\n\n\n## Evaluation\n### Metrics\n* ``PQ`` are the standard metrics described in [Panoptic Segmentation](https://arxiv.org/pdf/1801.00868.pdf).\n\u003cdiv align=\"center\" width=\"10\" height=\"5\"\u003e\u003cimg src=\"img/pq_metric.png\" width=\"600\" height=\"150\"\u003e\u003c/div\u003e\n\n* ``PC`` are the standard metrics described in [DeeperLab](https://arxiv.org/pdf/1902.05093).\n\u003cdiv align=\"center\" width=\"10\" height=\"5\"\u003e\u003cimg src=\"img/pc_metric.png\" width=\"600\" height=\"207\"\u003e\u003c/div\u003e\n\n### Evaluation Code\n* [cocodataset/panopticapi](https://github.com/cocodataset/panopticapi)\n* [mcordts/cityscapesScripts](https://github.com/mcordts/cityscapesScripts)\n\n### Competition\n* [AutoNUE 2019 Panoptic Segmentation Challenge (ICCV 2019 Workshop, Closed)](https://cvit.iiit.ac.in/autonue2019/challenge/)\n* [COCO 2019 Panoptic Segmentation Task (ICCV 2019 Workshop, Closed)](http://cocodataset.org/#panoptic-2019)\n* [Mapillary 2019 Panoptic Segmentation Task (ICCV 2019 Workshop, Closed)](https://research.mapillary.com/eccv18/#panoptic)\n* [Cityscapes Panoptic Semantic Labeling Task (Open)](https://www.cityscapes-dataset.com/benchmarks/#panoptic-scene-labeling-task)\n* [COCO 2018 Panoptic Segmentation Task (ECCV 2018 Workshop, Closed)](http://cocodataset.org/index.htm#panoptic-2018)\n* [Mapillary Vistas 2018 Panoptic Segmentation Task (ECCV 2018 Workshop, Closed)](https://research.mapillary.com/eccv18/#panoptic)\n\n\n## Benchmark Results\n### COCO `val` Benchmark\n| Method | Backbone | PQ | PQ-Thing | PQ-Stuff | SQ | RQ | mIoU | AP-Mask | PC |  e2e | \n| :----------: | :-----------: | :-----------: | :-----------: |:-----------: |:-----------: |:-----------: |:-----------: |:-----------: | :-----------: | :-----------: |\n| SOGNet | ResNet-50 | 43.7 | 50.6 | 33.2 | 78.7 | 53.5 | 54.56 | 34.2 | - | :white_check_mark: |\n| UPSNet | ResNet-50 | 42.5 | 48.6 | 33.4 | - | - | 54.3 | 34.3 | - | :white_check_mark: |\n| OANet | ResNet-101 | 41.3 | 50.4 | 27.7 | - | - | - | - | - | :white_check_mark: |\n| OCFusion | ResNet-50 | 41.0 | 49.0 | 29.0 | 77.1 | 50.6 | - | - | - | :white_check_mark: |\n| Panoptic FPN | ResNet-101 | 40.9 | 48.3 | 29.7 | - | - | - | - | - | :white_check_mark: |\n| AUNet | ResNet-50 | 39.6 | 49.1 | 25.2 | - | - | 45.1 | 34.7 | - | :white_check_mark: |\n| AdaptIS | ResNet-101  | 37.0 | 41.8 | 29.9 | - | - | - | - | - | :white_check_mark: |\n| DeeperLab | Xception-71 | 34.3 | 37.5 | 29.6 | 77.1 | 43.1 | - | - | 56.8 | :white_check_mark: |\n\n### Cityscapes `val`Benchmark\n| Method | Backbone | PQ | PQ-Thing | PQ-Stuff | SQ | RQ | mIoU | AP-Mask | PC |  e2e | \n| :----------: | :-----------: | :-----------: | :-----------: |:-----------: |:-----------: |:-----------: |:-----------: |:-----------: | :-----------: | :-----------: |\n| Panoptic(Merge) | - | 61.2 | 66.4 | 54.0 | 80.9 | 74.4 | - | - | - | :negative_squared_cross_mark: |\n| AdaptIS | ResNet-101  | 60.6 | 58.7 | 64.4 | - | - | 79.2 | 36.3 | - | :white_check_mark: |\n| SOGNet | ResNet-50  | 60.0 | 56.7 | 62.5 | - | - | - | - | - | :white_check_mark: |\n| Seamless | ResNet-50  | 59.8 | 53.4 | 64.5 | - | - | 75.4 | 31.9 | - | :white_check_mark: |\n| UPSNet | ResNet-50 | 59.3 | 54.6 | 62.7 | 79.7 | 73.0 | 75.2 | 33.3 | - | :white_check_mark: |\n| TASCNet | ResNet-101 | 59.2 | 56 | 61.5 | - | - | 77.8 | 37.6 | - |  :white_check_mark: |\n| AUNet | ResNet-101 | 59.0 | 54.8 | 62.1 | - | - | 75.6 | 34.4 | - | :white_check_mark: |\n| Panoptic FPN | ResNet-101 | 58.1 | 52.0 | 62.5 | - | - | 75.7 |33.0 | - | :white_check_mark: |\n| DeeperLab | Xception-71 | 56.5 | - | - | - | - | - | - | 75.6 |  :white_check_mark: |\n\n### Mapillary `val` Benchmark\n| Method | Backbone | PQ | PQ-Thing | PQ-Stuff | SQ | RQ | mIoU | AP-Mask | PC |  e2e | \n| :----------: | :-----------: | :-----------: | :-----------: |:-----------: |:-----------: |:-----------: |:-----------: |:-----------: | :-----------: | :-----------: |\n| Panoptic(Merge) | -  | 38.3 | 41.8 | 35.7 | 73.6 | 47.7 | - | - | - | :negative_squared_cross_mark: |\n| Seamless | ResNet-50  | 37.2 | 33.2 | 42.5 | - | - | 50.2 | 16.3 | - | :white_check_mark: |\n| AdaptIS | ResNet-101  | 33.4 | 28.3 | 40.3 | - | - | - | - | - | :white_check_mark: |\n| TASCNet | ResNet-101 | 32.6 | 31.3 | 34.4 | - | - | 35.0 | 18.5 | - | :white_check_mark: |\n| DeeperLab | Xception-71 | 32.0 | - | - | - | - | - | - | 55.3 |  :white_check_mark: |\n\n\n## Papers \n### AAAI2020\n* **SOGNet:** Yibo Yang, Hongyang Li, Xia Li, Qijie Zhao, Jianlong Wu, Zhouchen Lin.\u003cbr /\u003e\"SOGNet: Scene Overlap Graph Network for Panoptic Segmentation.\" AAAI (2020). [[paper](https://arxiv.org/pdf/1911.07527.pdf)]\n\n### ICCV2019\n* **AdaptIS:** Konstantin Sofiiuk, Olga Barinova, Anton Konushin.\u003cbr /\u003e\"AdaptIS: Adaptive Instance Selection Network.\" ICCV (2019). [[paper](https://arxiv.org/pdf/1909.07829.pdf)]\n\n* Cheng-Yang Fu, Tamara L. Berg, Alexander C. Berg.\u003cbr /\u003e\"IMP: Instance Mask Projection for High Accuracy Semantic Segmentation of Things.\" ICCV (2019). [[paper](https://arxiv.org/pdf/1906.06597.pdf)]\n\n* Bowen Cheng, Maxwell D. Collins, Yukun Zhu, Ting Liu, Thomas S. Huang, Hartwig Adam, Liang-Chieh Chen.\u003cbr /\u003e\"Panoptic-DeepLab: A Simple, Strong, and Fast Baseline for Bottom-Up Panoptic Segmentation Bowen.\" ICCVW (2019). [[paper](https://arxiv.org/pdf/1911.10194.pdf)]\n\n### CVPR2019\n* **Panoptic Segmentation:** Alexander Kirillov, Kaiming He, Ross Girshick, Carsten Rother, Piotr Dollár.\u003cbr /\u003e\"Panoptic Segmentation.\" CVPR (2019). [[paper](https://arxiv.org/pdf/1801.00868.pdf)]\n\n* **Panoptic FPN:** Alexander Kirillov, Ross Girshick, Kaiming He, Piotr Dollár.\u003cbr /\u003e\"Panoptic Feature Pyramid Networks.\" CVPR (2019 **oral**). [[paper](https://arxiv.org/pdf/1901.02446.pdf)] [[unofficial code](https://github.com/Angzz/panoptic-fpn-gluon)][[detectron2]](https://github.com/facebookresearch/detectron2)\n\n* **AUNet:** Yanwei Li, Xinze Chen, Zheng Zhu, Lingxi Xie, Guan Huang, Dalong Du, Xingang Wang.\u003cbr /\u003e\"Attention-guided Unified Network for Panoptic Segmentation.\" CVPR (2019). [[paper](https://arxiv.org/pdf/1812.03904.pdf)]\n\n* **UPSNet:** Yuwen Xiong, Renjie Liao, Hengshuang Zhao, Rui Hu, Min Bai, Ersin Yumer, Raquel Urtasun.\u003cbr /\u003e\"UPSNet: A Unified Panoptic Segmentation Network.\" CVPR (2019 **oral**). [[paper](https://arxiv.org/pdf/1901.03784.pdf)] [[code](https://github.com/uber-research/UPSNet)]\n\n* **DeeperLab:** Tien-Ju Yang, Maxwell D. Collins, Yukun Zhu, Jyh-Jing Hwang, Ting Liu, Xiao Zhang, Vivienne Sze, George Papandreou, Liang-Chieh Chen.\u003cbr /\u003e\"DeeperLab: Single-Shot Image Parser.\" CVPR (2019 **oral**). [[paper](https://arxiv.org/pdf/1902.05093)] [[project](http://deeperlab.mit.edu)] [[code](https://github.com/tensorflow/models/tree/master/research/deeplab/evaluation)]\n\n* **OANet:** Huanyu Liu, Chao Peng, Changqian Yu, Jingbo Wang, Xu Liu, Gang Yu, Wei Jiang.\u003cbr /\u003e\"An End-to-End Network for Panoptic Segmentation.\" CVPR (2019). [[paper](https://arxiv.org/pdf/1903.05027.pdf)]\n\n* Eirikur Agustsson, Jasper R. R. Uijlings, Vittorio Ferrari\n.\u003cbr /\u003e\"Interactive Full Image Segmentation by Considering All Regions Jointly.\" CVPR (2019). [[paper](https://arxiv.org/pdf/1812.01888.pdf)]\n\n* **Seamless:** Lorenzo Porzi, Samuel Rota Bulo, Aleksander Colovic, Peter Kontschieder.\u003cbr /\u003e\"Seamless Scene Segmentation.\" CVPR (2019) (Extended Version). [[paper](https://arxiv.org/pdf/1905.01220.pdf)][[code](https://github.com/mapillary/seamseg)]\n\n### ECCV2018\n* Qizhu Li, Anurag Arnab, Philip H.S. Torr.\u003cbr /\u003e\"Weakly- and Semi-Supervised Panoptic Segmentation.\" ECCV (2018). [[paper](https://arxiv.org/pdf/1812.01192.pdf)] [[code](https://github.com/qizhuli/Weakly-Supervised-Panoptic-Segmentation)]\n\n### ArXiv\n* Rohit Mohan, Abhinav Valada.\u003cbr /\u003e\n\"EfficientPS: Efficient Panoptic Segmentation.\" arXiv (2020). [[paper]](https://arxiv.org/abs/2004.02307)\n\n* Rui Hou, Jie Li, Arjun Bhargava, Allan Raventos, Vitor Guizilini, Chao Fang, Jerome Lynch, Adrien Gaidon.\u003cbr /\u003e\n\"Real-Time Panoptic Segmentation from Dense Detections.\" arXiv (2019). [[paper]](https://arxiv.org/pdf/1912.01202.pdf)\n\n* Mark Weber, Jonathon Luiten, Bastian Leibe.\u003cbr /\u003e\"Single-Shot Panoptic Segmentation.\" arXiv (2019). [[paper](https://arxiv.org/pdf/1911.00764.pdf)]\n\n* Qiang Chen, Anda Cheng, Xiangyu He, Peisong Wang, Jian Cheng.\u003cbr /\u003e\"SpatialFlow: Bridging All Tasks for Panoptic Segmentation.\" arXiv (2019). [[paper](https://arxiv.org/pdf/1910.08787.pdf)]\n\n* Sagi Eppel, Alan Aspuru-Guzik.\u003cbr /\u003e\"Generator evaluator-selector net: a modular approach for panoptic segmentation.\" arXiv (2019). [[paper](https://arxiv.org/pdf/1908.09108.pdf)]\n\n* Jasper R. R. Uijlings, Mykhaylo Andriluka, Vittorio Ferrari.\u003cbr /\u003e\"Panoptic Image Annotation with a Collaborative Assistant.\" arXiv (2019). [[paper](https://arxiv.org/pdf/1906.06798.pdf)]\n\n* **OCFusion:** Justin Lazarow, Kwonjoon Lee, Zhuowen Tu.\u003cbr /\u003e\"Learning Instance Occlusion for Panoptic Segmentation.\" arXiv (2019). [[paper](https://arxiv.org/pdf/1906.05896.pdf)]\n\n* **PEN:** Yuan Hu, Yingtian Zou, Jiashi Feng.\u003cbr /\u003e\"Panoptic Edge Detection.\" arXiv (2019). [[paper](https://arxiv.org/pdf/1906.00590.pdf)]\n\n* **TASCNet:** Jie Li, Allan Raventos, Arjun Bhargava, Takaaki Tagawa, Adrien Gaidon.\u003cbr /\u003e\"Learning to Fuse Things and Stuff.\" arXiv (2018). [[paper](https://arxiv.org/pdf/1812.01192.pdf)]\n\n* Daan de Geus, Panagiotis Meletis, Gijs Dubbelman.\u003cbr /\u003e\"Panoptic Segmentation with a Joint Semantic and Instance Segmentation Network.\" arXiv (2018). [[paper](https://arxiv.org/pdf/1809.02110.pdf)]\n\n* Daan de Geus, Panagiotis Meletis, Gijs Dubbelman.\u003cbr /\u003e\"Single Network Panoptic Segmentation for Street Scene Understanding.\" arXiv (2019). [[paper](https://arxiv.org/pdf/1902.02678.pdf)]\n\n* David Owen, Ping-Lin Chang.\u003cbr /\u003e\"Detecting Reflections by Combining Semantic and Instance Segmentation.\" arXiv (2019). [[paper](https://arxiv.org/pdf/1904.13273.pdf)]\n\n* Gaku Narita, Takashi Seno, Tomoya Ishikawa, Yohsuke Kaji.\u003cbr /\u003e\"PanopticFusion: Online Volumetric Semantic Mapping at the Level of Stuff and Things.\" arXiv (2019, IROS). [[paper](https://arxiv.org/pdf/1903.01177.pdf)]\n\n\n## Tutorials\n* CVPR 2019 Tutorial on Visual Recognition and Beyond. [[slides](https://www.dropbox.com/s/t6tg87t78pdq6v3/cvpr19_tutorial_alexander_kirillov.pdf?dl=0)] [[homepage](http://feichtenhofer.github.io/cvpr2019-recognition-tutorial/?nsukey=sJf%2BXalFUZ1SdTHfNF9ApK0yHb3RiOUTjCdoDI0FTj2gtGZgjyITEf3MIwlgv1CWJywF4qeEOFiUd14dVkeQjn61Yh4mOoqDVb%2Ff4BDiWtBZCNZzozDG5ryVLAM4y8kHxz2NXKdlyjgF2BwgPUMBLs4RrvMNRpgQl8PZ9KPBmhwEXq71r6E4dCCvEFCdio1Lj3aQc%2FoyG%2FdLIN3tBmSjPQ%3D%3D)]\n* COCO 2017 Workshop. [[slides](http://presentations.cocodataset.org/COCO17-Stuff-FAIR.pdf)]\n\n\n## Blogs\n* Megvii(Face++) Detection Team. [[zhihu]](https://zhuanlan.zhihu.com/p/59141570)\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FAngzz%2Fawesome-panoptic-segmentation","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FAngzz%2Fawesome-panoptic-segmentation","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FAngzz%2Fawesome-panoptic-segmentation/lists"}