{"id":27595952,"url":"https://github.com/chagmgang/dinov2-remote-sensing","last_synced_at":"2025-04-22T12:37:39.892Z","repository":{"id":269389993,"uuid":"907264885","full_name":"chagmgang/dinov2-remote-sensing","owner":"chagmgang","description":"Implementation dino v2 for remote sensing with huggingface transformers","archived":false,"fork":false,"pushed_at":"2025-03-12T01:49:26.000Z","size":25077,"stargazers_count":4,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-03-12T02:21:58.647Z","etag":null,"topics":["computer-vision","deep-learning","dino-v1","dino-v2","foundation-models","remote-sensing","transformers"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/chagmgang.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-12-23T07:43:06.000Z","updated_at":"2025-03-12T01:49:29.000Z","dependencies_parsed_at":null,"dependency_job_id":"d95b07b1-1538-481d-bd74-72b3e00b5c66","html_url":"https://github.com/chagmgang/dinov2-remote-sensing","commit_stats":null,"previous_names":["chagmgang/dinov2-remote-sensing"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chagmgang%2Fdinov2-remote-sensing","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chagmgang%2Fdinov2-remote-sensing/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chagmgang%2Fdinov2-remote-sensing/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chagmgang%2Fdinov2-remote-sensing/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/chagmgang","download_url":"https://codeload.github.com/chagmgang/dinov2-remote-sensing/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250242292,"owners_count":21398092,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["computer-vision","deep-learning","dino-v1","dino-v2","foundation-models","remote-sensing","transformers"],"created_at":"2025-04-22T12:37:35.717Z","updated_at":"2025-04-22T12:37:39.885Z","avatar_url":"https://github.com/chagmgang.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Reimplementation Self-Supervised Vision Transformers for DINO v2 with Huggingface 🤗\n---\n* Pytorch implementation and pretrained models for DINO v2 in remote sensing.\n* See Official Paper and Github for information in detail.\n[[`arXiv #1`]](https://arxiv.org/abs/2304.07193)\n[[`arXiv #2`]](https://arxiv.org/abs/2309.16588)\n[[`Github`]](https://github.com/facebookresearch/dinov2)\n\n---\n## Training\n\nThis project use the deepspeed interface for multi gpu training\n```\ndeepspeed --include localhost:0,1,2,3... vit_train.py\n```\n```\ndeepspeed --include localhost:0,1,2,3... convvit_train.py\n```\n\n---\n## Training Dataset for Remote Sensing\n\n| Dataset name | # of corpus | Dataset Paper |\n| :-: | :-: | :-: |\n| Million-AID | 990,666 | [Link](https://arxiv.org/abs/2006.12485) |\n| SkyScript | 5,181,068 | [Link](https://arxiv.org/abs/2312.12856) |\n| Total | 6,171,734 | |\n\n---\n## Pretrained Model on Huggingface\n| Model | Epoch | Total Params | Student Backbone Params | Student DINO Head Params | Student iBOT Head Params | Weight \u0026 Config | Logs |\n| :-: | :-: | :-: | :-: | :-: | :-: | :-: | :-: |\n| ViT-S/16-e25 | 25 | 132M | 21M | 22M | 22M | [Link](https://huggingface.co/KevinCha/dinov2-vit-small-remote-sensing) | [logs](https://huggingface.co/KevinCha/dinov2-vit-small-remote-sensing/tensorboard) |\n| ViT-S/16-e100 | 25 | 132M | 21M | 22M | 22M | [Link](https://huggingface.co/KevinCha/dinov2-vit-small-remote-sensing-100ep) | [logs](https://huggingface.co/KevinCha/dinov2-vit-small-remote-sensing-100ep/tensorboard) |\n| ViT-B/16-e25 | 25 | 264M | 88M | 21M | 21M | [Link](https://huggingface.co/KevinCha/dinov2-vit-base-remote-sensing) | [logs](https://huggingface.co/KevinCha/dinov2-vit-base-remote-sensing/tensorboard) |\n| ViT-L/14-e25 | 25 | 837M | 303M | 57M | 57M | [Link](https://huggingface.co/KevinCha/dinov2-vit-large-remote-sensing) | [logs](https://huggingface.co/KevinCha/dinov2-vit-large-remote-sensing/tensorboard) |\n| ViT-L/14-e50 | 50 | 837M | 303M | 57M | 57M | [Link](https://huggingface.co/KevinCha/dinov2-vit-large-remote-sensing-50ep) | [logs](https://huggingface.co/KevinCha/dinov2-vit-large-remote-sensing-50ep/tensorboard) |\n| ConvViT-S-e25(DINOv1) | 25 | 88.5M | 22.2M | 22M | x | [Link](https://huggingface.co/KevinCha/dinov2-conv-vit-small-remote-sensing) | [logs](https://huggingface.co/KevinCha/dinov2-conv-vit-small-remote-sensing/tensorboard) |\n\n---\n\n## Evaluation\n\nThe evaluation methods for DINOv2 are k-nn clustering and linear probing. 90% of the data is randomly selected as the training set while the 10% is selected as test set. The `k=20` is selected for evaluation with K-NN. The evaluation datasets are including below table. The splited data is stored in [linprob_data_lists](/linprob_data_lists).\n\n| Dataset Name | Dataset Paper |\n| :-: | :-: |\n| `RESISC` | [Remote Sensing Image Scene Classification: Benchmark and State of the Art](https://arxiv.org/abs/1703.00121) |\n|`Optimal 31` | [Scene Classification With Recurrent Attention of VHR Remote Sensing Images](https://ieeexplore.ieee.org/document/8454883) |\n| `MLRSNet`| [MLRSNet: A Multi-label High Spatial Resolution Remote Sensing Dataset for Semantic Scene Understanding](https://arxiv.org/abs/2010.00243) |\n| `WHU-RS19` |  |\n| `EuroSAT` | [EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification](https://arxiv.org/abs/1709.00029) |\n| `UC Merced` | [Bag-of-visual-words and spatial extensions for land-use classification](https://dl.acm.org/doi/10.1145/1869790.1869829) |\n| `Cv-BrCT` | [AiRound and CV-BrCT: Novel Multi-View Datasets for Scene Classification](https://arxiv.org/abs/2008.01133) |\n| `AiRound`| [AiRound and CV-BrCT: Novel Multi-View Datasets for Scene Classification](https://arxiv.org/abs/2008.01133) |\n|`RSI-CB128` | [RSI-CB: A Large Scale Remote Sensing Image Classification Benchmark via Crowdsource Data](https://arxiv.org/abs/1705.10450) |\n\n### Linear Probing Evaluation\n\n```\n# train_textfile = linprob_data_lists/RESISC/train.txt\n# test_textfile = linprob_data_lists/RESISC/test.txt\n\npython3 evaluation/linprob.py --model-path {model_registry} \\\n                              --data-root {data_root} \\\n                              --train-text {train_textfile} \\\n                              --test-text {test_textfile}\n```\n\n| Model | RESISC | Optimal 31 | MLRSNet | WHU-RS19 | EuroSAT | UC Merced | Cv-BrCT | AiRound | RSI-CB128 |\n| :-: | :-: | :-: | :-: | :-: | :-: | :-: | :-: | :-: | :-: |\n| ViT-S/16-e25 | 94.381 | 96.237 | 96.642 | 99.811 | 98.037 | 99.048 | 77.613 | 78.644 | 99.593 |\n| ViT-S/16-e100 | 94.381 | 95.161 | 96.349 | 100.00 | 97.704 | 99.048 | 76.910 | 79.407 | 99.539 |\n| ViT-B/16-e25 | 95.460 | 98.925 | 97.301 | 100.00 | 97.889 | 98.571 | 79.058 | 80.339 | 99.675 |\n| ViT-L/14-e25 | 96.603 | 96.774 | 98.161 | 100.000 | 98.704 | 99.048 | 80.132 | 82.627 | 99.729 |\n| ViT-L/14-e50 | 96.762 | 96.774 | 97.511 | 100.00 | 98.407 | 98.571 | 80.463 | 85.508 | 99.702 |\n| ConvViT-S-e25(DINOv1) | 94.476 | 93.548 | 95.919 | 99.065 | 96.778 | 98.095 | 77.695 | 81.949 | 99.295 |\n\n### KNN Evaluation\n\n```\n# train_textfile = linprob_data_lists/RESISC/train.txt\n# test_textfile = linprob_data_lists/RESISC/test.txt\n\npython3 evaluation/knn.py --model-path {model_registry} \\\n                              --data-root {data_root} \\\n                              --train-text {train_textfile} \\\n                              --test-text {test_textfile}\n```\n\n| Model | RESISC | Optimal 31 | MLRSNet | WHU-RS19 | EuroSAT | UC Merced | Cv-BrCT | AiRound | RSI-CB128 |\n| :-: | :-: | :-: | :-: | :-: | :-: | :-: | :-: | :-: | :-: |\n| Official ViT-S/14 | 87.778 | 85.484 | 91.820 | 99.065 | 92.074 | 91.429 | 73.936 | 74.068 | 96.504 |\n| Official ViT-B/14 | 90.571 | 89.247 | 91.948 | 96.262 | 90.667 | 92.857 | 74.721 | 75.847 | 96.585 |\n| ViT-S/16-e25 | 93.365 | 89.785 | 96.981 | 97.196 | 95.741 | 87.143 | 76.208 | 77.881 | 98.943 |\n| ViT-S/16-e100 | 93.746 | 94.624 | 97.081 | 97.196 | 96.222 | 86.667 | 75.960 | 76.695 | 98.808 |\n| ViT-B/16-e25 | 94.286 | 90.323 | 97.328 | 100.00 | 95.704 | 87.143 | 76.456 | 77.373 | 99.106 |\n| ViT-L/14-e25 | 93.778 | 91.398 | 97.392 | 99.065 | 96.963 | 88.095 | 79.430 | 80.085 | 99.133 |\n| ViT-L/14-e50 | 94.063 | 92.473 | 97.511 | 100.00 | 96.704 | 87.619 | 79.843 | 82.203 | 99.079 |\n| ConvViT-S-e25(DINOv1) | 92.508 | 91.935 | 95.947 | 98.131 | 94.074 | 90.000 | 75.630 | 76.271 | 98.374 |\n\n---\n\n## Property Analysis\n\n* Feature Mapping - [feature_mapping.ipynb](/notebook/feature_mapping.ipynb)\n![feature mapping1](/assets/feature_vis_1.png)\n![feature mapping2](/assets/feature_vis_2.png)\n* Sparse Feature Matching - [vit-feature-matching.ipynb](/notebook/vit-feature-matching.ipynb)\n![sparse matching](/assets/sparse_matching.png)\n* Image Retrieval - [index_search.ipynb](/notebook/index_search.ipynb)\n![index search1](/assets/1.png)\n![index search2](/assets/2.png)\n![index search3](/assets/3.png)\n![index search4](/assets/4.png)\n![index search5](/assets/5.png)\n![index search6](/assets/6.png)\n![index search7](/assets/7.png)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchagmgang%2Fdinov2-remote-sensing","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fchagmgang%2Fdinov2-remote-sensing","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchagmgang%2Fdinov2-remote-sensing/lists"}