{"id":20065496,"url":"https://github.com/cvi-szu/clims","last_synced_at":"2025-05-05T18:31:45.175Z","repository":{"id":45204904,"uuid":"466430359","full_name":"CVI-SZU/CLIMS","owner":"CVI-SZU","description":"[CVPR 2022] CLIMS: Cross Language Image Matching for Weakly Supervised Semantic Segmentation","archived":false,"fork":false,"pushed_at":"2024-06-07T01:20:19.000Z","size":43542,"stargazers_count":129,"open_issues_count":8,"forks_count":12,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-04-09T02:23:52.889Z","etag":null,"topics":["semantic-segmentation","weakly-supervised-learning","weakly-supervised-segmentation"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/CVI-SZU.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-03-05T11:12:33.000Z","updated_at":"2025-04-07T11:12:36.000Z","dependencies_parsed_at":"2024-11-13T14:02:16.719Z","dependency_job_id":null,"html_url":"https://github.com/CVI-SZU/CLIMS","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CVI-SZU%2FCLIMS","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CVI-SZU%2FCLIMS/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CVI-SZU%2FCLIMS/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CVI-SZU%2FCLIMS/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/CVI-SZU","download_url":"https://codeload.github.com/CVI-SZU/CLIMS/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252553035,"owners_count":21766816,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["semantic-segmentation","weakly-supervised-learning","weakly-supervised-segmentation"],"created_at":"2024-11-13T13:50:57.308Z","updated_at":"2025-05-05T18:31:40.166Z","avatar_url":"https://github.com/CVI-SZU.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# CLIMS\n\nCode repository for our paper \"[CLIMS: Cross Language Image Matching for Weakly Supervised Semantic Segmentation](https://openaccess.thecvf.com/content/CVPR2022/html/Xie_CLIMS_Cross_Language_Image_Matching_for_Weakly_Supervised_Semantic_Segmentation_CVPR_2022_paper.html)\" in **CVPR 2022**.\n\n:heart_eyes: Code for our\npaper \"[CCAM: Contrastive learning of Class-agnostic Activation Map for Weakly Supervised Object Localization and Semantic Segmentation](https://openaccess.thecvf.com/content/CVPR2022/html/Xie_C2AM_Contrastive_Learning_of_Class-Agnostic_Activation_Map_for_Weakly_Supervised_CVPR_2022_paper.html)\"\nin **CVPR 2022** is also available [here](https://github.com/CVI-SZU/CCAM).\n\n![](clims.png)\n\n**Please to NOTE that this repository is an **improved version** of our camera-ready version (you can refer to the directory of `previous_version/`). We recommend to use our improved version of CLIMS instead of camera-ready version.**\n\n\n## Dataset\n### PASCAL VOC2012\nYou will need to download the images (JPEG format) in PASCAL VOC2012 dataset from [here](http://host.robots.ox.ac.uk/pascal/VOC/voc2012/) and train_aug ground-truth can be found [here](http://home.bharathh.info/pubs/codes/SBD/download.html). Make sure your `data/VOC2012 folder` is structured as follows:\n```\n├── VOC2012/\n|   ├── Annotations\n|   ├── ImageSets\n|   ├── SegmentationClass\n|   ├── SegmentationClassAug\n|   └── SegmentationObject\n```\n### MS-COCO 2014\nYou will need to download the images (JPEG format) in MSCOCO 2014 dataset [here](https://cocodataset.org/#download) and ground-truth mask can be found [here](https://drive.google.com/drive/folders/18l3aAs64Ld_uvAJm57O3EiHuhEXkdwUy?usp=share_link). Make sure your `data/COCO folder` is structured as follows:\n```\n├── COCO/\n|   ├── train2014\n|   ├── val2014\n|   ├── annotations\n|   |   ├── instances_train2014.json\n|   |   ├── instances_val2014.json\n|   ├── mask\n|   |   ├── train2014\n|   |   ├── val2014\n```\n\n## Training on PASCAL VOC2012\n1. Install CLIP.\n```\n$ pip install ftfy regex tqdm\n$ pip install git+https://github.com/openai/CLIP.git\n```\n2. Download pre-trained baseline CAM ('res50_cam.pth') [here](https://drive.google.com/drive/folders/1CCYduc2L_V_s7MtXEuA_LzIscdlFFJag?usp=sharing) and put it at the directory of `cam-baseline-voc12/`.\n3. Train CLIMS on PASCAL V0C2012 dataset to generate initial CAMs.\n```\nCUDA_VISIBLE_DEVICES=0 python run_sample.py --voc12_root /data1/xjheng/dataset/VOC2012/ --hyper 10,24,1,0.2 --clims_num_epoches 15 --cam_eval_thres 0.15 --work_space clims_voc12 --cam_network net.resnet50_clims --train_clims_pass True --make_clims_pass True --eval_cam_pass True\n```\n3. Train IRNet and generate pseudo semantic masks.\n```\nCUDA_VISIBLE_DEVICES=0 python run_sample.py --voc12_root /data1/xjheng/dataset/VOC2012/ --cam_eval_thres 0.15 --work_space clims_voc12 --cam_network net.resnet50_clims --cam_to_ir_label_pass True --train_irn_pass True --make_sem_seg_pass True --eval_sem_seg_pass True\n```\n4. Train DeepLabv2 using pseudo semantic masks.\n```shell\ncd segmentation/\n```\n\n## Evaluation Results\n### The quality of initial CAMs and pseudo masks on PASCAL VOC2012.\n\n| Method    | backbone | CAMs | + RW | + IRNet |\n|:---------:|:--------:|:----:|:----:|:----:|\n| **CLIMS(camera-ready)** | R50      | 56.6 | 70.5 | - |\n| **CLIMS(this repo)**    | R50      | 58.6 | ~73 | 74.1 |\n\n### Evaluation results on PASCAL VOC2012 val and test sets.\n**Please cite the results of camera-ready version**\n\n| Method    | Supervision | Network  | Pretrained  | val  | test |\n|:---------:|:-----------:|:----:|:----:|:----:|:----:|\n| AdvCAM    | I           | DeepLabV2 |  ImageNet | 68.1 | 68.0 |\n| EDAM      | I+S         | DeepLabV2 |  COCO     | 70.9 | 70.6 |\n| **CLIMS(camera-ready)** | I     | DeepLabV2 |  ImageNet | 69.3 | 68.7 |\n| **CLIMS(camera-ready)** | I     | DeepLabV2 |  COCO     | 70.4 | 70.0 |\n| **CLIMS(this repo)** | I     | DeepLabV2 |ImageNet | 70.3 | 70.6 |\n| **CLIMS(this repo)** | I     | DeepLabV2 | COCO     | 71.4 | 71.2 |\n| **CLIMS(this repo)** | I     | DeepLabV1-R38 | ImageNet    | 73.3 | 73.4 |\n\n(**Please cite the results of camera-ready version**. Initial CAMs, pseudo semantic masks, and pre-trained models of camera-ready version can be found at [Google Drive](https://drive.google.com/drive/folders/1njCaolWacqSmw7HVNecwvCAMm7NsCFPq?usp=sharing))\n\n## Training on MSCOCO 2014\n1. Download pre-trained baseline CAM ('res50_cam.pth') [here](https://drive.google.com/drive/folders/18l3aAs64Ld_uvAJm57O3EiHuhEXkdwUy?usp=share_link) and put it at the directory of `cam-baseline-coco/`.\n2. Train CLIMS on MSCOCO 2014 dataset to generate initial CAMs.\n```\nCUDA_VISIBLE_DEVICES=6,7 python -m torch.distributed.launch --nproc_per_node=2 run_sample_coco.py --work_space clims_coco --clims_network net.resnet50_clims --train_clims_pass True --make_clims_pass True --eval_cam_pass True --clims_num_epoches 8 --cam_eval_thres 0.15 --hyper 2,14,1.25,0.2 --cam_batch_size 16 --clims_learning_rate 0.0005 --use_distributed_train True --cbs_loss_thresh 0.285\n```\n\nIf you are using our code, please consider citing our paper.\n\n```\n@InProceedings{Xie_2022_CVPR,\n    author    = {Xie, Jinheng and Hou, Xianxu and Ye, Kai and Shen, Linlin},\n    title     = {CLIMS: Cross Language Image Matching for Weakly Supervised Semantic Segmentation},\n    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},\n    month     = {June},\n    year      = {2022},\n    pages     = {4483-4492}\n}\n@article{xie2022cross,\n  title={Cross Language Image Matching for Weakly Supervised Semantic Segmentation},\n  author={Xie, Jinheng and Hou, Xianxu and Ye, Kai and Shen, Linlin},\n  journal={arXiv preprint arXiv:2203.02668},\n  year={2022}\n}\n```\nThis repository was highly based on [IRNet](https://github.com/jiwoon-ahn/irn), thanks for Jiwoon Ahn's great code.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcvi-szu%2Fclims","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcvi-szu%2Fclims","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcvi-szu%2Fclims/lists"}