{"id":13637732,"url":"https://github.com/xmed-lab/AllSpark","last_synced_at":"2025-04-19T12:32:24.033Z","repository":{"id":226424669,"uuid":"768132186","full_name":"xmed-lab/AllSpark","owner":"xmed-lab","description":"CVPR 2024: AllSpark: Reborn Labeled Features from Unlabeled in Transformer for Semi-Supervised Semantic Segmentation","archived":false,"fork":false,"pushed_at":"2024-04-09T14:59:37.000Z","size":14756,"stargazers_count":49,"open_issues_count":7,"forks_count":7,"subscribers_count":4,"default_branch":"main","last_synced_at":"2024-08-03T01:11:48.741Z","etag":null,"topics":["attention","cvpr2024","semantic-segmentation","semi-supervised-segmentation","transformer"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/xmed-lab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2024-03-06T14:27:17.000Z","updated_at":"2024-07-24T10:44:53.000Z","dependencies_parsed_at":"2024-04-09T16:02:49.883Z","dependency_job_id":null,"html_url":"https://github.com/xmed-lab/AllSpark","commit_stats":null,"previous_names":["xmed-lab/allspark"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xmed-lab%2FAllSpark","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xmed-lab%2FAllSpark/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xmed-lab%2FAllSpark/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xmed-lab%2FAllSpark/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/xmed-lab","download_url":"https://codeload.github.com/xmed-lab/AllSpark/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":223799384,"owners_count":17204774,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["attention","cvpr2024","semantic-segmentation","semi-supervised-segmentation","transformer"],"created_at":"2024-08-02T01:00:27.375Z","updated_at":"2024-11-09T07:31:18.254Z","avatar_url":"https://github.com/xmed-lab.png","language":"Python","funding_links":[],"categories":["2024"],"sub_categories":[],"readme":"# [CVPR-2024] _AllSpark_: Reborn Labeled Features from Unlabeled in Transformer for Semi-Supervised Semantic Segmentation\n\n\n[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/allspark-reborn-labeled-features-from/semi-supervised-semantic-segmentation-on-21)](https://paperswithcode.com/sota/semi-supervised-semantic-segmentation-on-21?p=allspark-reborn-labeled-features-from)\u003cbr\u003e\n[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/allspark-reborn-labeled-features-from/semi-supervised-semantic-segmentation-on-4)](https://paperswithcode.com/sota/semi-supervised-semantic-segmentation-on-4?p=allspark-reborn-labeled-features-from)\u003cbr\u003e\n[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/allspark-reborn-labeled-features-from/semi-supervised-semantic-segmentation-on-9)](https://paperswithcode.com/sota/semi-supervised-semantic-segmentation-on-9?p=allspark-reborn-labeled-features-from)\u003cbr\u003e\n[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/allspark-reborn-labeled-features-from/semi-supervised-semantic-segmentation-on-44)](https://paperswithcode.com/sota/semi-supervised-semantic-segmentation-on-44?p=allspark-reborn-labeled-features-from)\u003cbr\u003e\n[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/allspark-reborn-labeled-features-from/semi-supervised-semantic-segmentation-on-27)](https://paperswithcode.com/sota/semi-supervised-semantic-segmentation-on-27?p=allspark-reborn-labeled-features-from)\u003cbr\u003e\n[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/allspark-reborn-labeled-features-from/semi-supervised-semantic-segmentation-on-28)](https://paperswithcode.com/sota/semi-supervised-semantic-segmentation-on-28?p=allspark-reborn-labeled-features-from)\u003cbr\u003e\n[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/allspark-reborn-labeled-features-from/semi-supervised-semantic-segmentation-on-29)](https://paperswithcode.com/sota/semi-supervised-semantic-segmentation-on-29?p=allspark-reborn-labeled-features-from)\u003cbr\u003e\n[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/allspark-reborn-labeled-features-from/semi-supervised-semantic-segmentation-on-30)](https://paperswithcode.com/sota/semi-supervised-semantic-segmentation-on-30?p=allspark-reborn-labeled-features-from)\u003cbr\u003e\n[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/allspark-reborn-labeled-features-from/semi-supervised-semantic-segmentation-on-coco)](https://paperswithcode.com/sota/semi-supervised-semantic-segmentation-on-coco?p=allspark-reborn-labeled-features-from)\u003cbr\u003e\n[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/allspark-reborn-labeled-features-from/semi-supervised-semantic-segmentation-on-coco-1)](https://paperswithcode.com/sota/semi-supervised-semantic-segmentation-on-coco-1?p=allspark-reborn-labeled-features-from)\u003cbr\u003e\n[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/allspark-reborn-labeled-features-from/semi-supervised-semantic-segmentation-on-coco-2)](https://paperswithcode.com/sota/semi-supervised-semantic-segmentation-on-coco-2?p=allspark-reborn-labeled-features-from)\u003cbr\u003e\n[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/allspark-reborn-labeled-features-from/semi-supervised-semantic-segmentation-on-coco-3)](https://paperswithcode.com/sota/semi-supervised-semantic-segmentation-on-coco-3?p=allspark-reborn-labeled-features-from)\n\n\nThis repo is the official implementation of [_AllSpark_: Reborn Labeled Features from Unlabeled in Transformer for Semi-Supervised Semantic Segmentation](https://arxiv.org/abs/2403.01818) which is accepted at CVPR-2024.\n\n\n\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"./docs/allspark.jpg\" width=39% height=65% class=\"center\"\u003e\n\u003cimg src=\"./docs/framework.png\" width=60% height=65% class=\"center\"\u003e\n\u003c/p\u003e\n\nThe _**AllSpark**_ is a powerful Cybertronian artifact in the film series of _Transformers_. It was used to reborn Optimus Prime in _Transformers: Revenge of the Fallen_, which aligns well with our core idea.\n\n-------\n\n## 💥 Motivation\nIn this work, we discovered that simply converting existing semi-segmentation methods into a pure-transformer framework is ineffective. \n\u003cp align=\"center\"\u003e\n\u003cimg src=\"./docs/backbone.png\" width=50% height=80% class=\"center\"\u003e\n\u003cimg src=\"./docs/issue.jpg\" width=35% height=65% class=\"center\"\u003e\n\u003c/p\u003e\n\n- The first reason is that transformers inherently possess weaker inductive bias compared to CNNs, so transformers heavily rely on a large volume of training data to perform well. \n\n- The more critical issue lies in the existing semi-supervised segmentation frameworks. These frameworks separate the training flows for labeled and unlabeled data, which aggravates the overfitting issue of transformers on the limited labeled data.\n\nThus, we propose to _intervene and diversify_ the labeled data flow with unlabeled data in the feature domain, leading to improvements in generalizability.\n\n-------\n\n## 🛠️ Usage\n\n**‼️ IMPORTANT: This version is not the final version. We made some mistakes when re-organizing the code. We will release the correct version soon. Sorry for any inconvenience this may cause.**\n\n### 1. Environment\n\nFirst, clone this repo:\n\n```shell\ngit clone https://github.com/xmed-lab/AllSpark.git\ncd AllSpark/\n```\n\nThen, create a new environment and install the requirements:\n```shell\nconda create -n allspark python=3.7\nconda activate allspark\npip install torch==1.12.0+cu116 torchvision==0.13.0+cu116 torchaudio==0.12.0 --extra-index-url https://download.pytorch.org/whl/cu116\npip install tensorboard\npip install six\npip install pyyaml\npip install -U openmim\nmim install mmcv==1.6.2\npip install einops\npip install timm\n```\n\n### 2. Data Preparation \u0026 Pre-trained Weights\n\n#### 2.1 Pascal VOC 2012 Dataset\nDownload the dataset with wget:\n```shell\nwget https://hkustconnect-my.sharepoint.com/:u:/g/personal/hwanggr_connect_ust_hk/EcgD_nffqThPvSVXQz6-8T0B3K9BeUiJLkY_J-NvGscBVA\\?e\\=2b0MdI\\\u0026download\\=1 -O pascal.zip\nunzip pascal.zip\n```\n\n#### 2.2 Cityscapes Dataset\nDownload the dataset with wget:\n```shell\nwget https://hkustconnect-my.sharepoint.com/:u:/g/personal/hwanggr_connect_ust_hk/EWoa_9YSu6RHlDpRw_eZiPUBjcY0ZU6ZpRCEG0Xp03WFxg\\?e\\=LtHLyB\\\u0026download\\=1 -O cityscapes.zip\nunzip cityscapes.zip\n```\n\n#### 2.3 COCO Dataset\nDownload the dataset with wget:\n```shell\nwget https://hkustconnect-my.sharepoint.com/:u:/g/personal/hwanggr_connect_ust_hk/EXCErskA_WFLgGTqOMgHcAABiwH_ncy7IBg7jMYn963BpA\\?e\\=SQTCWg\\\u0026download\\=1 -O coco.zip\nunzip coco.zip\n```\n\n\nThen your file structure will be like:\n\n```\n├── VOC2012\n    ├── JPEGImages\n    └── SegmentationClass\n    \n├── cityscapes\n    ├── leftImg8bit\n    └── gtFine\n    \n├── coco\n    ├── train2017\n    ├── val2017\n    └── masks\n```\n\nNext, download the following [pretrained weights](https://hkustconnect-my.sharepoint.com/:f:/g/personal/hwanggr_connect_ust_hk/Eobv9tk6a6RJqGXEDm2D_TcB2mEn4r2-BLDkotZHkd2l6w?e=fJBy7v).\n```\n├── ./pretrained_weights\n    ├── mit_b2.pth\n    ├── mit_b3.pth\n    ├── mit_b4.pth\n    └── mit_b5.pth\n```\n\nFor example, mit-B5:\n```shell\nmkdir pretrained_weights\nwget https://hkustconnect-my.sharepoint.com/:u:/g/personal/hwanggr_connect_ust_hk/ET0iubvDmcBGnE43-nPQopMBw9oVLsrynjISyFeGwqXQpw?e=9wXgso\\\u0026download\\=1 -O ./pretrained_weights/mit_b5.pth\n```\n\n\n### 3. Training \u0026 Evaluating\n\n```bash\n# use torch.distributed.launch\nsh scripts/train.sh \u003cnum_gpu\u003e \u003cport\u003e\n# to fully reproduce our results, the \u003cnum_gpu\u003e should be set as 4 on all three datasets\n# otherwise, you need to adjust the learning rate accordingly\n\n# or use slurm\n# sh scripts/slurm_train.sh \u003cnum_gpu\u003e \u003cport\u003e \u003cpartition\u003e\n```\n\nTo train on other datasets or splits, please modify\n``dataset`` and ``split`` in [train.sh](https://github.com/xmed-lab/AllSpark/blob/main/scripts/train.sh).\n\n\n### 4. Results\n\nModel weights and training logs will be released soon.\n\n#### 4.1 PASCAL VOC 2012 _original_\n\u003cp align=\"left\"\u003e\n\u003cimg src=\"./docs/pascal_org.png\" width=60% class=\"center\"\u003e\n\u003c/p\u003e\n\n\n| Splits | 1/16 | 1/8  | 1/4 | 1/2 | Full |\n| :- | - | - | - | - | - |\n| Weights of _**AllSpark**_ | 76.07 | 78.41 | 79.77 | 80.75 | 82.12 |\n| Reproduced | [76.06](https://hkustconnect-my.sharepoint.com/:u:/g/personal/hwanggr_connect_ust_hk/EZY5whWCIuNOnaAYpPk9cyoBmspntpHoa75z2DAnD1Tg2g?e=AkHect\\\u0026download\\=1) \u0026#124; [log](https://hkustconnect-my.sharepoint.com/:u:/g/personal/hwanggr_connect_ust_hk/ERqwnOSTFyZBlMmLfqA1xegBJDJEW5pqM-WHTG3JMWdwKg?e=2tL8Fs) | [78.41](https://hkustconnect-my.sharepoint.com/:u:/g/personal/hwanggr_connect_ust_hk/ESsfJbP0ipxAmhzzdESOIdgBKv3OLceKhpJscDaxTo9Grg?e=UDxRmb\\\u0026download\\=1) | [79.93](https://hkustconnect-my.sharepoint.com/:u:/g/personal/hwanggr_connect_ust_hk/ETHrHSQotHVOvKPjZKl3DUEByst-7FHBZFm7lSLt0L71HA?e=qb9sv5\\\u0026download\\=1) \u0026#124; [log](https://hkustconnect-my.sharepoint.com/:u:/g/personal/hwanggr_connect_ust_hk/EeBcskpwXhpJr1839qVe-f4Bjq68paMI0GSxaWJQjnrlQQ?e=V9VPkx) | [80.70](https://hkustconnect-my.sharepoint.com/:u:/g/personal/hwanggr_connect_ust_hk/EasoxicajudMmMmBzDCwNjIBnzg5kXXVga4lsYFIrq587A?e=njs6W7\\\u0026download\\=1) \u0026#124; [log](https://hkustconnect-my.sharepoint.com/:u:/g/personal/hwanggr_connect_ust_hk/EXI_9n636B9Hsql6JXSzT6oBzkweKDqxtLbPf5zrV0NFyA?e=iZTKWy) | [82.56](https://hkustconnect-my.sharepoint.com/:u:/g/personal/hwanggr_connect_ust_hk/Ec-K5rthA2xLheg0blwnPbkBI2zvYs7xcPdZ7Hr2RBHTag?e=kaiBEE\\\u0026download\\=1) \u0026#124; [log](https://hkustconnect-my.sharepoint.com/:u:/g/personal/hwanggr_connect_ust_hk/EcIWs3l1faBKh1HOM29QgmgBPjALel1IG9FPLJaA9xBPZg?e=of3NlB) |\n\n\n#### 4.2 PASCAL VOC 2012 _augmented_\n\n\u003cp align=\"left\"\u003e\n\u003cimg src=\"./docs/pascal_aug.png\" width=60% class=\"center\"\u003e\n\u003c/p\u003e\n\n| Splits | 1/16 | 1/8  | 1/4 | 1/2 |\n| :- | - | - | - | - |\n| Weights of _**AllSpark**_ | 78.32 | 79.98 | 80.42 | 81.14 |\n\n\n\n\n#### 4.3 Cityscapes\n\n\u003cp align=\"left\"\u003e\n\u003cimg src=\"./docs/cityscapes.png\" width=60% class=\"center\"\u003e\n\u003c/p\u003e\n\n\n| Splits | 1/16 | 1/8  | 1/4 | 1/2 |\n| :- | - | - | - | - |\n| Weights of _**AllSpark**_ | 78.33 | 79.24 | 80.56 | 81.39 |\n\n\n\n#### 4.4 COCO\n\n\u003cp align=\"left\"\u003e\n\u003cimg src=\"./docs/coco.png\" width=60% class=\"center\"\u003e\n\u003c/p\u003e\n\n\n| Splits | 1/512 | 1/256  | 1/128 | 1/64 |\n| :- | - | - | - | - |\n| Weights of _**AllSpark**_ | [34.10](https://hkustconnect-my.sharepoint.com/:u:/g/personal/hwanggr_connect_ust_hk/EaabBYut1PNEtPeQRCIlMtEBxpmkvbZ_ERmBGwTObS0H_g?e=69ToFl\\\u0026download\\=1) \u0026#124; [log](https://hkustconnect-my.sharepoint.com/:u:/g/personal/hwanggr_connect_ust_hk/EbOcn45vPcJGvERaP2tnJdQBLY-xF6GExOP2BsTFQta1xg?e=lSodC0) | [41.65](https://hkustconnect-my.sharepoint.com/:u:/g/personal/hwanggr_connect_ust_hk/EfIyzut1SwBMha25yKpeIWIBwPfhc3NzdGLjdlyuKdr0ig?e=H58uKd\\\u0026download\\=1) \u0026#124; [log](https://hkustconnect-my.sharepoint.com/:u:/g/personal/hwanggr_connect_ust_hk/EU9w0R77DpxOqE8pW62BSh0BC7OZ9Kqml2eluvGftx-fJw?e=9Ugtwc) | [45.48](https://hkustconnect-my.sharepoint.com/:u:/g/personal/hwanggr_connect_ust_hk/EUHmlDEXNPZPuq5qRfhTChgBs9GZ2n9qVRYdPWHGwgkYBQ?e=yRNTcg\\\u0026download\\=1) \u0026#124; [log](https://hkustconnect-my.sharepoint.com/:u:/g/personal/hwanggr_connect_ust_hk/EYIjEixClLVKtMaYJz8KZssB56s5nduIR31dAxUSALWF7A?e=IGRUSs) | [49.56](https://hkustconnect-my.sharepoint.com/:u:/g/personal/hwanggr_connect_ust_hk/ETeZ7agRCkRIjJeONaL8BYEBKIe4rDI3ZgRkEDdBcVPPOA?e=56diA2\\\u0026download\\=1) \u0026#124; [log](https://hkustconnect-my.sharepoint.com/:u:/g/personal/hwanggr_connect_ust_hk/EckkrL4B2HFHtNHY0IBwizYBsU7iSdh5zO36hzPhDUIlDg?e=tA2Eov) |\n\n\n\n\n## Citation\n\nIf you find this project useful, please consider citing:\n\n```bibtex\n@inproceedings{allspark,\n  title={AllSpark: Reborn Labeled Features from Unlabeled in Transformer for Semi-Supervised Semantic Segmentation},\n  author={Wang, Haonan and Zhang, Qixiang and Li, Yi and Li, Xiaomeng},\n  booktitle={CVPR},\n  year={2024}\n}\n```\n\n\n## Acknowlegement\n_**AllSpark**_ is built upon [UniMatch](https://github.com/LiheYoung/UniMatch) and [SegFormer](https://github.com/NVlabs/SegFormer). We thank their authors for making the source code publicly available.\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fxmed-lab%2FAllSpark","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fxmed-lab%2FAllSpark","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fxmed-lab%2FAllSpark/lists"}