{"id":20305509,"url":"https://github.com/vitae-transformer/lemevit","last_synced_at":"2025-04-11T14:51:24.720Z","repository":{"id":240152529,"uuid":"792951970","full_name":"ViTAE-Transformer/LeMeViT","owner":"ViTAE-Transformer","description":"The official repo for [IJCAI'24] \"LeMeViT: Efficient Vision Transformer with Learnable Meta Tokens for Remote Sensing Image Interpretation\"","archived":false,"fork":false,"pushed_at":"2024-11-11T02:58:22.000Z","size":24585,"stargazers_count":48,"open_issues_count":1,"forks_count":4,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-03-25T11:01:42.173Z","etag":null,"topics":["attention","deep-learning","object-detection","remote-sensing","scene-classification","semantic-segmentation","vision-transformer"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ViTAE-Transformer.png","metadata":{"files":{"readme":"README.md","changelog":"change_detection/eval.py","contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-04-28T02:31:39.000Z","updated_at":"2025-03-17T09:36:23.000Z","dependencies_parsed_at":"2024-05-22T13:30:30.700Z","dependency_job_id":"5331a1aa-dfd1-43b9-8ef6-171560f0cd04","html_url":"https://github.com/ViTAE-Transformer/LeMeViT","commit_stats":null,"previous_names":["vitae-transformer/lemevit"],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ViTAE-Transformer%2FLeMeViT","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ViTAE-Transformer%2FLeMeViT/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ViTAE-Transformer%2FLeMeViT/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ViTAE-Transformer%2FLeMeViT/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ViTAE-Transformer","download_url":"https://codeload.github.com/ViTAE-Transformer/LeMeViT/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248424791,"owners_count":21101232,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["attention","deep-learning","object-detection","remote-sensing","scene-classification","semantic-segmentation","vision-transformer"],"created_at":"2024-11-14T17:08:47.324Z","updated_at":"2025-04-11T14:51:24.681Z","avatar_url":"https://github.com/ViTAE-Transformer.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003ch1 align=\"center\"\u003e LeMeViT: Efficient Vision Transformer with Learnable Meta Tokens for Remote\nSensing Image Interpretation  \u003ca href=\"https://doi.org/10.24963/ijcai.2024/103\"\u003e\u003cimg src=\"https://img.shields.io/badge/IJCAI-2024-blue\"\u003e\u003c/a\u003e \u003ca href=\"https://arxiv.org/abs/2405.09789\"\u003e\u003cimg src=\"https://img.shields.io/badge/arXiv-Paper-\u003ccolor\u003e\"\u003e\u003c/a\u003e \u003c/h1\u003e\n\u003cp align=\"center\"\u003e\n\u003ch4 align=\"center\"\u003eThe official repo for [IJCAI'24] \u003ca href=\"https://doi.org/10.24963/ijcai.2024/103\"\u003eLeMeViT: Efficient Vision Transformer with Learnable Meta Tokens for Remote\nSensing Image Interpretation\u003c/a\u003e.\u003c/h4\u003e\n\u003ch5 align=\"center\"\u003e\u003cem\u003eWentao Jiang, Jing Zhang, Di Wang, Qiming Zhang, Zengmao Wang and Bo Du\u003c/em\u003e\u003c/h5\u003e\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"#news\"\u003eNews\u003c/a\u003e |\n  \u003ca href=\"#abstract\"\u003eAbstract\u003c/a\u003e |\n  \u003ca href=\"#method\"\u003eMethod\u003c/a\u003e |\n  \u003ca href=\"#results\"\u003eResults\u003c/a\u003e |\n  \u003ca href=\"#usage\"\u003eUsage\u003c/a\u003e |\n  \u003ca href=\"#statement\"\u003eStatement\u003c/a\u003e\n\u003c/p\u003e\n\n# News\n\n***17/05/2024***  \n\n- :rocket: The paper is post on arxiv now! The code is released.\n\n***17/04/2024***\n\n- :tada: :tada: LeMeViT is accepted by IJCAI 2024! :tada: :tada:\n\n***\n\nRelevant Project: \n\n\u003e :sparkles: **RSP: An Empirical Study of Remote Sensing Pretraining** | [Paper](https://arxiv.org/abs/2204.02825) | [Code](https://github.com/ViTAE-Transformer/RSP)\n\u003e\n\u003e :sparkles: **RVSA: Advancing Plain Vision Transformer Towards Remote Sensing Foundation Model** | [Paper](https://arxiv.org/abs/2208.03987) | [Code](https://github.com/ViTAE-Transformer/ViTAE-Transformer-Remote-Sensing)\n\n\n# Comparison on ImageNet-1K\n\n\u003cfigure align=\"center\"\u003e\n\u003cimg src=\"figs/infer.png\" width=\"70%\"\u003e\n\u003c/figure\u003e\n\nHere's the comparison of LemeViT with other state-of-the-arts on ImageNet-1K, \u003ci\u003e i.e. \u003c/i\u003e \n[**PVTv2**](https://arxiv.org/abs/2106.13797), \n[**Swin Transformer**](https://arxiv.org/abs/2103.14030), \n[**BiFormer**](https://arxiv.org/abs/2303.08810) and \n[**PacaViT**](https://arxiv.org/abs/2203.11987).\n\n# Abstract\n\n\u003cp align=\"left\"\u003e Due to spatial redundancy in remote sensing images, sparse tokens containing rich information are usually involved in self-attention (SA) to reduce the overall token numbers within the calculation, avoiding the high computational cost issue in Vision Transformers. However, such methods usually obtain sparse tokens by hand-crafted or parallel-unfriendly designs, posing a challenge to reach a better balance between efficiency and performance. Different from them, this paper proposes to use learnable meta tokens to formulate sparse tokens, which effectively learn key information meanwhile improving the inference speed. Technically, the meta tokens are first initialized from image tokens via cross-attention. Then, we propose Dual Cross-Attention (DCA) to promote information exchange between image tokens and meta tokens, where they serve as query and key (value) tokens alternativelyin a dual-branch structure, significantly reducingthe computational complexity compared to self-attention. By employing DCA in the early stages with dense visual tokens, we obtain the hierarchical architecture LeMeViT with various sizes. Experimental results in classification and dense prediction tasks show that LeMeViT has a significant 1.7× speedup, fewer parameters, and competitive performance compared to the baseline models, and achieves a better trade-off between efficiency and performance.\n\n# Motivation\n\n\u003cfigure align=\"center\"\u003e\n\u003cimg src=\"figs/motivation1.png\" width=\"70%\"\u003e\n\u003cfigcaption align = \"left\"\u003e \u003cb\u003eFigure 1: \u003c/b\u003e Due to the high correlation between neighboring pixels and image patches, as well as the repetitive nature of textures and patterns in remote sensing images, there is a significant amount of spatial redundancy. This results in redundant computation in self-attention mechanism. \u003c/figcaption\u003e\n\u003c/figure\u003e\n\n\u003cfigure align=\"center\"\u003e\n\u003cimg src=\"figs/motivation2.png\" width=\"70%\"\u003e\n\u003cfigcaption align = \"left\"\u003e \u003cb\u003eFigure 2: Existing methods \u003c/b\u003e commonly use downsampling or clustering to reduce the number of image tokens within the current block which relies on strong priors or is parallel-computation unfriendly. \u003cb\u003e Our method \u003c/b\u003e learns meta tokens to sparsely represent dense image tokens. Meta tokens exchange information with image tokens via the computationally efficient Dual Cross-Attention Block in an end-to-end way, promoting information flow stage-by-stage. \u003c/figcaption\u003e\n\u003c/figure\u003e\n\n# Method\n\n\u003cfigure\u003e\n\u003cimg src=\"figs/framework.png\"\u003e\n\u003cfigcaption align = \"left\"\u003e \u003cb\u003eFigure 3: The Overall Architecture of LeMeViT.\u003c/b\u003e LeMeViT consists of three different attention blocks, arranged from left to right as Cross Attention Block, Dual Cross-Attention Block, and Standard Attention Block. The meta tokens are first initialized from image tokens via cross-attention. Then, DCA is employed in the early stages to promote information exchange between image tokens and meta tokens, where they serve as query and key (value) tokens alternatively in a dual-branch structure. In the later stages, standard attention blocks based on self-attention are used. \u003c/figcaption\u003e\n\u003c/figure\u003e\n\n# Visualization\n\n\u003cfigure\u003e\n\u003cimg src=\"figs/vis2.png\"\u003e\n\u003cfigcaption align = \"left\"\u003e \u003cb\u003eFigure 4:\u003c/b\u003e Visualization of the attention maps between three meta tokens in the last layer and image tokens. (a) illustrates the attention maps on natural images, and (b) illustrates attention maps on remote sensing images. \u003c/figcaption\u003e\n\u003c/figure\u003e\n\n# Results\n\n\u003e :alarm_clock: All the weights will be released soon.\n\n### Classfication\n\n|    Model     |  Resolution  | Throughput | Params |  MACs | Acc@1 | Weight |\n|:-------------|:------------:|:----------:|:------:|:-----:|:-----:|:------:|\n|  LeMeViT-Tiny |   224 × 224  |   5316.58  |  8.64  |  1.68 | 79.07 |    -   |\n| LeMeViT-Small |   224 × 224  |   3608.12  |  16.40 |  3.74 | 81.88 |    -   |\n|  LeMeViT-Base |   224 × 224  |   1482.70  |  53.10 | 11.06 | 84.35 |    -   |\n\n### Scene Recognition\n\n|    Model     |  Resolution  | Throughput | Params |  MACs | Acc@1 | Weight |\n|:-------------|:------------:|:----------:|:------:|:-----:|:-----:|:------:|\n|  LeMeViT-Tiny |   224 × 224  |   5327.47  |  8.33  |  1.78 | 98.80 |    -   |\n| LeMeViT-Small |   224 × 224  |   3612.68  |  16.04 |  3.74 | 99.00 |    -   |\n|  LeMeViT-Base |   224 × 224  |   1484.09  |  52.61 | 11.06 | 99.17 |    -   |\n\n### Object Detection\n\n|   Backbone   | Method         | Params |  MACs  |  mAP  | Weight |\n|:-------------|----------------|:------:|:------:|:-----:|:------:|\n|  LeMeViT-Tiny | Oriented R-CNN |  25.37 | 154.12 | 76.63 |    -   |\n| LeMeViT-Small | Oriented R-CNN |  33.15 | 193.91 | 77.58 |    -   |\n|  LeMeViT-Base | Oriented R-CNN |  69.76 | 335.53 | 78.00 |    -   |\n\n### Semantic Segmentation\n\n|   Backbone   | Method  | Params |  MACs  |   OA  | mF1   | Weight |\n|:-------------|---------|:------:|:------:|:-----:|:-----:|:------:|\n|  LeMeViT-Tiny | UperNet |  37.05 | 217.88 | 91.03 | 90.55 |    -   |\n| LeMeViT-Small | UperNet |  45.59 | 228.16 | 91.23 | 90.62 |    -   |\n|  LeMeViT-Base | UperNet |  83.19 | 263.75 | 91.35 | 90.85 |    -   |\n\n### Change Detection\n\n|   Backbone   | Method | Params |  MACs | F1 score | Weight |\n|:-------------|:------:|:------:|:-----:|:--------:|:------:|\n|  LeMeViT-Tiny |   BIT  |  8.56  |  5.75 |   95.56  |    -   |\n| LeMeViT-Small |   BIT  |  16.75 | 10.71 |   96.64  |    -   |\n|  LeMeViT-Base |   BIT  |  53.34 | 28.47 |   97.32  |    -   |\n\n# Usage\n\nThe script examples can be found in the directory [scripts](scripts), which can be modified and run easily.\n\nSuch as:\n\n```\n# For classification \u0026 scene recognition training\nbash scripts/train.sh\n\n# For semantic segmentation evaluation\nbash scripts/test_ss.sh \n```\n## Preparation\n\u003cdetails\u003e\n\n\u003csummary\u003eDatasets\u003c/summary\u003e\n\n- Classification: [ImageNet-1k](https://image-net.org/)\n- Scene Recognition: [MillionAid](https://captain-whu.github.io/DiRS/)\n- Object Detection: [DOTA-v1.0](https://captain-whu.github.io/DOTA/index.html)\n- Semantic Segmentation: [Potsdam](https://www.isprs.org/education/benchmarks/UrbanSemLab/2d-sem-label-potsdam.aspx)\n- Change Detection: [CDD](https://drive.google.com/file/d/1GX656JqqOyBi_Ef0w65kDGVto-nHrNs9/edit)\n  \n\u003c/details\u003e\n\n\n\n## Classification \u0026 Scene Recognition\n\n### Installation\n\n**Environment:** Python=3.8, Pytorch=2.1.0, CUDA=12.2\n\nFirst, install the necessary libraries.\n\n```\npip install timm=0.9.16\npip install einops\n```\n\nThen, optionally install the following libraries for further acceleration.\n\n```\n# Speed: flash_attn \u003e xformers \u003e torch\n\npip install flash_attn\npip install xformers\n```\n\n\n### Evaluation\n\nFor ImageNet-1K classification, the evaluation script can be:\n```\n  python \\\n    validate.py \\\n    --data-dir \u003cdata_path\u003e \\\n    --dataset imagenet \\\n    --model lemevit_small \\\n    --num-classes 1000 \\\n    --checkpoint \u003cckpt_path\u003e \\\n    --img-size 224 \\\n    --batch-size \u003cbatch_size\u003e \\\n    --amp \\\n```\n\nFor MillionAid scene recognition, the evaluation script can be:\n```\n  python \\\n    validate.py \\\n    --data-dir \u003cdata_path\u003e \\\n    --dataset millionaid \\\n    --model lemevit_small \\\n    --num-classes 51 \\\n    --checkpoint \u003cckpt_path\u003e \\\n    --img-size 224 \\\n    --batch-size \u003cbatch_size\u003e \\\n    --amp \\\n```\n\nAlso, you can follow script [test.sh](scripts/test.sh) for recommended usage.\n\n### Training\n\nThe training script for both classication and scene recognition can be:\n\n```\nCUDA_VISIBLE_DEVICES=\u003cdevices_ids\u003e \\\n  torchrun \\\n    --rdzv_backend c10d \\\n    --rdzv-endpoint=localhost:0 \\\n    --nnodes 1 \\\n    --nproc_per_node \u003cdevices_num\u003e \\\n    main.py \\\n    --config \u003cconfig_file_path\u003e \\\n    --output \u003coutput_path\u003e \\\n    --experiment \u003cexp_name\u003e \\\n    --model \u003cmodel_name\u003e \\\n    --img-size 224 \\\n    --batch-size \u003cbatch_size\u003e \\\n```\n\nAlso, you can follow script [train.sh](scripts/train.sh) for recommended usage.\n\n\n## Object Detection \u0026 Semantic Segmentation\n\n### Installation\n\nSince we use [OBBDetection](https://github.com/jbwang1997/OBBDetection) and [MMSegmenation](https://github.com/open-mmlab/mmsegmentation) to implement corresponding detection or segmentation models, we only provide necessary config and backbone files. \n\nThe installation and dataset preparation can separately refer [OBBDetection-installation](https://github.com/jbwang1997/OBBDetection/blob/master/docs/install.md) and [MMSegmentation-installation](https://github.com/open-mmlab/mmsegmentation/blob/master/docs/en/get_started.md#installation)\n\nThen put these files into corresponding folders.\n\n### Evalation \u0026 Training\n\n```\n# For object detection evaluation\nbash scripts/test_od.sh\n\n# For object detection training\nbash scripts/train_od.sh\n\n# For semantic segmentation evaluation\nbash scripts/test_ss.sh\n\n# For semantic segmentation training\nbash scripts/train_ss.sh\n```\n\n## Change Detection\n\n### Installation\n\n\nPlease refer to [BIT-Readme.md](https://github.com/justchenhao/BIT_CD/blob/master/README.md) for installing.\n\n### Evalation \u0026 Training\n\n```\n# For evaluation\nbash scripts/test_cd.sh\n\n# For training\nbash scripts/train_cd.sh\n```\n\n\n# Statement\n\n## Acknowledgement\nThis repository is built using the [timm](https://github.com/rwightman/pytorch-image-models), [OBBDetection](https://github.com/jbwang1997/OBBDetection) and [mmsegmenation](https://github.com/open-mmlab/mmsegmentation) library.\n\n\n## License\nThis project is for research purpose only. \n\n## Citation\n\nIf you find LeMeViT helpful, please consider giving this repo a star :star: and citing:\n\n```\n@inproceedings{jiang_lemevit_2024,\n  title={{LeMeViT}: Efficient Vision Transformer with Learnable Meta Tokens for Remote Sensing Image Interpretation},\n  author={Jiang, Wentao and Zhang, Jing and Wang, Di and Zhang, Qiming and  Wang, Zengmao and Du, Bo},\n  booktitle={International Joint Conference on Artificial Intelligence},\n  year={2024}\n}\n```\n\nFor any other questions please contact [jiang_wentao@whu.edu.cn](mailto:jiang_wentao@whu.edu.cn).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvitae-transformer%2Flemevit","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvitae-transformer%2Flemevit","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvitae-transformer%2Flemevit/lists"}