{"id":25509200,"url":"https://github.com/tonyduan/maskformer","last_synced_at":"2025-09-12T21:20:16.391Z","repository":{"id":161433710,"uuid":"636131105","full_name":"tonyduan/maskformer","owner":"tonyduan","description":"Maskformer (Per-Pixel Classification is Not All You Need) in PyTorch.","archived":false,"fork":false,"pushed_at":"2023-05-04T07:51:11.000Z","size":14,"stargazers_count":3,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-05-20T07:32:31.047Z","etag":null,"topics":["computer-vision","instances-segmentation","semantic-segmentation"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tonyduan.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2023-05-04T07:33:02.000Z","updated_at":"2025-02-20T15:37:21.000Z","dependencies_parsed_at":"2023-07-13T11:18:11.926Z","dependency_job_id":null,"html_url":"https://github.com/tonyduan/maskformer","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/tonyduan/maskformer","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tonyduan%2Fmaskformer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tonyduan%2Fmaskformer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tonyduan%2Fmaskformer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tonyduan%2Fmaskformer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tonyduan","download_url":"https://codeload.github.com/tonyduan/maskformer/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tonyduan%2Fmaskformer/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":274879250,"owners_count":25367095,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-12T02:00:09.324Z","response_time":60,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["computer-vision","instances-segmentation","semantic-segmentation"],"created_at":"2025-02-19T08:54:14.574Z","updated_at":"2025-09-12T21:20:16.378Z","avatar_url":"https://github.com/tonyduan.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"### Maskformer\n\nLast update: May 2023.\n\n---\n\nImplementation of Maskformer (Cheng et al. 2021) for instance segmentation.\n\nThe setup is very similar to set prediction for object detection (Carion et al. 2020). We query a Transformer with an overdetermined set of queries, each of which represents an instance-level mask. At training time, Hungarian matching allows each query to learn to snap to one particular mode rather than splitting across modes.\n\nIn the figure below, the groundtruth is on the left. Queries use positional embeddings to separate instances.\n\n![](examples/ex_squares.png)\n\nOne noteworthy innovation of the Maskformer paper is representing each query's instance segmentation output as the dot product of a high resolution feature with a query-specific feature. This saves a significant amount of time compared to up-sampling an instance mask per query.\n\nEach mask has a loss consisting of classification and mask terms:\n```math\n\\mathcal{L}(\\mathbf{y}, \\hat{\\mathbf{y}}) = \\lambda_\\mathrm{cls}\\mathcal{L}_\\mathrm{cls}(y_\\mathrm{cls}, \\hat{y}_\\mathrm{cls}) + \\lambda_\\mathrm{mask}\\mathcal{L}_\\mathrm{cls}(\\mathbf{y}_\\mathrm{mask}, \\hat{\\mathbf{y}}_\\mathrm{mask})\n```\n\nWe implement a couple loss functions for the mask, given per-pixel predictions $p \\in [0, 1]$ and groundtruth $y \\in \\{0, 1\\}$.\n1. Binary Cross-Entropy. $-y\\log p - (1-y)\\log (1-p)$.\n2. Brier Score. $(p - y)^2$.\n3. Dice Loss.\n\nThe forward pass is extremely simple:\n\n```python\n    def forward(self, x):\n        low_res_feat, high_res_feat = self.nn_module(x)\n        low_res_feat = rearrange(low_res_feat, \"b c h w -\u003e b (h w) c\")\n\n        # Transformer blocks attend to the low resolution feature\n        queries = repeat(self.query_embed, \"t c -\u003e b t c\", b=len(x))\n        for block in self.decoder_blocks:\n            queries = block(queries, memory=low_res_feat)\n        queries = self.ln(queries)\n\n        # Predicted masks formed via per-pixel dot product w high resolution feature\n        mask_embed = self.mask_embed_convs(queries)\n        masks = torch.einsum(\"b t c, b c h w -\u003e b t h w\", mask_embed, high_res_feat)\n\n        # Predicted classes via straightforward MLP\n        classes = self.class_convs(queries)\n        return masks, classes\n```\n\nTodo: find an implementation of Hungarian matching i.e. `linear_sum_assignment` on GPU.\n\n#### References\n\n[1] Cheng, B., Schwing, A. \u0026 Kirillov, A. Per-Pixel Classification is Not All You Need for Semantic Segmentation. in Advances in Neural Information Processing Systems vol. 34 17864–17875 (Curran Associates, Inc., 2021).\n\n[2] Carion, N. et al. End-to-End Object Detection with Transformers. in Computer Vision – ECCV 2020 (eds. Vedaldi, A., Bischof, H., Brox, T. \u0026 Frahm, J.-M.) 213–229 (Springer International Publishing, 2020). doi:10.1007/978-3-030-58452-8_13.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftonyduan%2Fmaskformer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftonyduan%2Fmaskformer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftonyduan%2Fmaskformer/lists"}