{"id":22035634,"url":"https://github.com/alexstoken/image-matching-models","last_synced_at":"2025-05-16T01:06:53.057Z","repository":{"id":239083145,"uuid":"763818113","full_name":"alexstoken/image-matching-models","owner":"alexstoken","description":"A framework to easily use 32 (and growing) different image matching methods","archived":false,"fork":false,"pushed_at":"2025-05-09T04:14:47.000Z","size":90535,"stargazers_count":423,"open_issues_count":3,"forks_count":38,"subscribers_count":10,"default_branch":"main","last_synced_at":"2025-05-09T05:20:18.272Z","etag":null,"topics":["feature-matching","image-matching","image-retrieval","local-features","visual-localization"],"latest_commit_sha":null,"homepage":"https://earthloc-and-earthmatch.github.io/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/alexstoken.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-02-27T01:06:02.000Z","updated_at":"2025-05-09T04:14:50.000Z","dependencies_parsed_at":"2024-06-03T01:27:55.852Z","dependency_job_id":"6e657602-7bec-414d-a818-51ad2a2824c0","html_url":"https://github.com/alexstoken/image-matching-models","commit_stats":null,"previous_names":["gmberton/image-matching-models","alexstoken/image-matching-models"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alexstoken%2Fimage-matching-models","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alexstoken%2Fimage-matching-models/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alexstoken%2Fimage-matching-models/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alexstoken%2Fimage-matching-models/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/alexstoken","download_url":"https://codeload.github.com/alexstoken/image-matching-models/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254448579,"owners_count":22072764,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["feature-matching","image-matching","image-retrieval","local-features","visual-localization"],"created_at":"2024-11-30T10:25:44.750Z","updated_at":"2025-05-16T01:06:48.038Z","avatar_url":"https://github.com/alexstoken.png","language":"Python","readme":"# Image Matching Models (IMM)\n\nA unified API for quickly and easily trying 34 (and growing!) image matching models.\n\n[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/alexstoken/image-matching-models/blob/main/demo.ipynb)\n\nJump to: [Install](#install) | [Use](#use) | [Models](#available-models) | [Add a Model/Contributing](#adding-a-new-method) | [Acknowledgements](#acknowledgements) | [Cite](#cite)\n\n### Matching Examples\nCompare matching models across various scenes. For example, we show `SIFT-LightGlue` and `LoFTR` matches on pairs: \n\u003cp\u003e(1) outdoor, (2) indoor, (3) satellite remote sensing, (4) paintings, (5) a false positive, and (6) spherical. \u003c/p\u003e\n\u003cdetails open\u003e\u003csummary\u003e\nSIFT-LightGlue\n\u003c/summary\u003e\n\u003cp float=\"left\"\u003e\n  \u003cimg src=\"assets/example_sift-lg/output_3_matches.jpg\" width=\"195\" /\u003e\n  \u003cimg src=\"assets/example_sift-lg/output_2_matches.jpg\" width=\"195\" /\u003e\n  \u003cimg src=\"assets/example_sift-lg/output_4_matches.jpg\" width=\"195\" /\u003e\n  \u003cimg src=\"assets/example_sift-lg/output_1_matches.jpg\" width=\"195\" /\u003e\n  \u003cimg src=\"assets/example_sift-lg/output_0_matches.jpg\" width=\"195\" /\u003e\n    \u003cimg src=\"assets/example_sift-lg/output_5_matches.jpg\" width=\"195\" /\u003e\n\n\u003c/p\u003e\n\u003c/details\u003e\n\n\u003cdetails open\u003e\u003csummary\u003e\nLoFTR\n\u003c/summary\u003e\n\u003cp float=\"left\"\u003e\n  \u003cimg src=\"assets/example_loftr/output_3_matches.jpg\" width=\"195\" /\u003e\n  \u003cimg src=\"assets/example_loftr/output_2_matches.jpg\" width=\"195\" /\u003e\n  \u003cimg src=\"assets/example_loftr/output_4_matches.jpg\" width=\"195\" /\u003e\n  \u003cimg src=\"assets/example_loftr/output_1_matches.jpg\" width=\"195\" /\u003e\n  \u003cimg src=\"assets/example_loftr/output_0_matches.jpg\" width=\"195\" /\u003e\n  \u003cimg src=\"assets/example_loftr/output_5_matches.jpg\" width=\"195\" /\u003e\n\u003c/p\u003e\n\u003c/details\u003e\n\n### Extraction Examples\nYou can also extract keypoints and associated descriptors. \n\u003cdetails open\u003e\u003csummary\u003e\nSIFT and DeDoDe\n\u003c/summary\u003e\n\u003cp float=\"left\"\u003e\n  \u003cimg src=\"assets/example_sift-lg/output_8_kpts.jpg\" width=\"195\" /\u003e\n  \u003cimg src=\"assets/example_dedode/output_8_kpts.jpg\" width=\"195\" /\u003e\n  \u003cimg src=\"assets/example_sift-lg/output_0_kpts.jpg\" width=\"195\" /\u003e\n  \u003cimg src=\"assets/example_dedode/output_0_kpts.jpg\" width=\"195\" /\u003e\n\u003c/p\u003e\n\u003c/details\u003e\n\n## Install\n### From Source [Recommended]\nIf you want to to install from source (easiest to edit, use `benchmark.py`, `demo.ipynb`), \n```bash\ngit clone --recursive https://github.com/alexstoken/image-matching-models\ncd image-matching-models\npip install -e .\n```\nSome models (`omniglue`, LoFTR family) require one-off dependencies (`tensorflow`, `pytorch-lightning`), which are not included in the default list. To install these, use \n```\npip install .[all]\n```\nThis will install all dependencies needed to run all models.\n\nWe recommend using torch\u003e=2.2, we didn't test most models with older versions.\n\n### As package (simplest)\nYou can install directly to your package directory with \n```bash\npip install git+https://github.com/alexstoken/image-matching-models.git\n```\nSimilar to the above, to get all optional dependencies, use the `[all]` addendum:\n```bash\npip install \"image-matching-models[all] @ git+https://github.com/alexstoken/image-matching-models.git\"\n```\n\n\u003e [!Note]  \n\u003e SphereGlue depends on `torch-geometric` and `torch-cluster` which require that you pass an additional parameter given your installed versions of torch and CUDA like so: `pip install .[all] -f https://data.pyg.org/whl/torch-2.5.0+cu124.html` (replace `cu124` with `cpu` for CPU version). See [PyTorch Geometric installation docs](https://pytorch-geometric.readthedocs.io/en/latest/install/installation.html) for more information \n\n\n## Use\n\nYou can use any of the matchers with \n\n```python\nfrom matching import get_matcher\nfrom matching.viz import plot_matches\n\ndevice = 'cuda'  # 'cpu'\nmatcher = get_matcher('superpoint-lg', device=device)  # Choose any of our ~30+ matchers listed below\nimg_size = 512  # optional\n\nimg0 = matcher.load_image('assets/example_pairs/outdoor/montmartre_close.jpg', resize=img_size)\nimg1 = matcher.load_image('assets/example_pairs/outdoor/montmartre_far.jpg', resize=img_size)\n\nresult = matcher(img0, img1)\nnum_inliers, H, inlier_kpts0, inlier_kpts1 = result['num_inliers'], result['H'], result['inlier_kpts0'], result['inlier_kpts1']\n# result.keys() = ['num_inliers', 'H', 'all_kpts0', 'all_kpts1', 'all_desc0', 'all_desc1', 'matched_kpts0', 'matched_kpts1', 'inlier_kpts0', 'inlier_kpts1']\nplot_matches(img0, img1, result, save_path='plot_matches.png')\n```\n\nYou can also run this as a standalone script, which will perform inference on the the examples inside `./assets`. You may also resolution (`im_size`) and number of keypoints (`n_kpts`). This will take a few seconds on a laptop's CPU, and will produce the same images that you see above.\n\n```bash\npython main_matcher.py --matcher sift-lg --device cpu --out_dir output_sift-lg\n```\nwhere `sift-lg` will use `SIFT + LightGlue`.\n\nThe script will generate an image with the matching keypoints for each pair, under `./output_sift-lg`.\n\n### Use on your own images\n\nTo use on your images you have three options:\n1. create a directory with sub-directories, with two images per sub-directory, just like `./assets/example_pairs`. Then use as `python main_matcher.py --input path/to/dir`\n2. create a file with pairs of paths, separate by a space, just like `assets/example_pairs_paths.txt`. Then use as `python main_matcher.py --input path/to/file.txt`\n3. import the matcher package into a script/notebook and use from there, as in the example above\n\n### Keypoint Extraction and Description\nTo extract keypoints and descriptions (when available) from a single image, use the `extract()` method.\n\n```python\nfrom matching import get_matcher\n\ndevice = 'cuda' # 'cpu'\nmatcher = get_matcher('xfeat', device=device)  # Choose any of our ~30+ matchers listed below\nimg_size = 512 # optional\n\nimg = matcher.load_image('assets/example_pairs/outdoor/montmartre_close.jpg', resize=img_size)\n\nresult = matcher.extract(img)\n# result.keys() = ['all_kpts0', 'all_desc0']\nplot_kpts(img, result)\n```\n\nAs with matching, you can also run extraction from the command line\n```bash\npython main_extractor.py --matcher sift-lg --device cpu --out_dir output_sift-lg --n_kpts 2048\n```\n\n\n## Available Models\nYou can choose any of the following methods (input to `get_matcher()`):\n\n**Dense**: ```roma, tiny-roma, dust3r, mast3r, minima-roma```\n\n**Semi-dense**: ```loftr, eloftr, se2loftr, xoftr, minima-loftr, aspanformer, matchformer, xfeat-star, xfeat-star-steerers[-perm/-learned], ```\n\n**Sparse**: ```[sift, superpoint, disk, aliked, dedode, doghardnet, gim, xfeat]-lg, dedode, steerers, affine-steerers, xfeat-steerers[-perm/learned], dedode-kornia, [sift, orb, doghardnet]-nn, patch2pix, superglue, r2d2, d2net,  gim-dkm, xfeat, omniglue, [dedode, xfeat, aliked]-subpx, [sift, superpoint]-sphereglue, minima-splg```\n\n\n\u003e [!TIP]\n\u003e You can pass a list of matchers, i.e. `get_matcher([xfeat, tiny-roma])` to run both matchers and concatenate their keypoints.\n\nMost matchers can run on CPU and GPU. MPS is not tested. See [Model Details](#model-details) for runtimes. If a runtime is ❌, it means that model can not run on that device. \n\n## Model Details\n\u003e [!IMPORTANT]\n\u003e Check the LICENSE of each model/original code base before use in your application. Some are heavily restricted. \n\n| Model | Code | Paper | GPU Runtime (s/img)| CPU Runtime (s/img) |\n|-------|------|-------|----|----|\n| MINIMA (arxiv '24)| [Official](https://github.com/LSXI7/MINIMA/tree/main) | [arxiv](https://arxiv.org/abs/2412.19412) | | 11.32 / 7.355 / 96.6 / 0.561\n| XoFTR (CVPR IMW '24) | [Official](https://github.com/OnderT/XoFTR?tab=readme-ov-file) | [pdf](https://openaccess.thecvf.com/content/CVPR2024W/IMW/papers/Tuzcuoglu_XoFTR_Cross-modal_Feature_Matching_Transformer_CVPRW_2024_paper.pdf) | | 8.954 |\n| Affine Steerers (ECCV '24) | [Official](https://github.com/georg-bn/affine-steerers?tab=readme-ov-file) | [arxiv](https://arxiv.org/abs/2408.14186)| 0.677 | ❌ |\n| Keypt2Subpx* (ECCV '24) | [Official](https://github.com/KimSinjeong/keypt2subpx) | [arxiv](https://arxiv.org/abs/2407.11668)| 0.055 /0.164 / 0.033 / 0.291  | -- |\n| MASt3R (ArXiv '24) | [Official](https://github.com/naver/mast3r?tab=readme-ov-file) | [arxiv](https://arxiv.org/abs/2406.09756) | 0.699 | -- |\n| Efficient-LoFTR (CVPR '24) | [Official](https://github.com/zju3dv/efficientloftr) | [pdf](https://zju3dv.github.io/efficientloftr/files/EfficientLoFTR.pdf) | 0.1026 | 2.117 |\n| OmniGlue (CVPR '24) | [Official](https://github.com/google-research/omniglue) | [arxiv](https://arxiv.org/abs/2405.12979) | ❌ | 6.351 |\n| xFeat-Steerers | [Official](https://colab.research.google.com/drive/1ZFifMqUAOQhky1197-WAquEV1K-LhDYP?usp=sharing) | [Discussion](https://github.com/verlab/accelerated_features/issues/32) | 0.037 | 0.124 | \n| xFeat (CVPR '24) | [Official](https://github.com/verlab/accelerated_features) | [arxiv](https://arxiv.org/abs/2404.19174) | 0.027 | 0.048 | \n| GIM (ICLR '24) | [Official](https://github.com/xuelunshen/gim?tab=readme-ov-file) | [arxiv](https://arxiv.org/abs/2402.11095)  |  0.077 (+LG) /  1.627 (+DKMv3) | 5.321 (+LG) /  20.301 (+DKMv3) |\n| RoMa / Tiny-RoMa (CVPR '24) | [Official](https://github.com/Parskatt/RoMa) | [arxiv](https://arxiv.org/abs/2305.15404) |  0.453 / 0.0456 |  18.950 |\n| DUSt3R (CVPR '24) | [Official](https://github.com/naver/dust3r) | [arxiv](https://arxiv.org/abs/2312.14132) | 3.639 |  26.813 |\n| DeDoDe (3DV '24) | [Official](https://github.com/Parskatt/DeDoDe/tree/main) | [arxiv](https://arxiv.org/abs/2308.08479) |  0.311 (+MNN)/ 0.218 (+LG) | ❌ |\n| Steerers (CVPR '24) | [Official](https://github.com/georg-bn/rotation-steerers) | [arxiv](https://arxiv.org/abs/2312.02152) | 0.150 | 26.425 |\n| SphereGlue* (CVPRW '23) | [Official](https://github.com/vishalsharbidar/SphereGlue) | [pdf](https://openaccess.thecvf.com/content/CVPR2023W/IMW/papers/Gava_SphereGlue_Learning_Keypoint_Matching_on_High_Resolution_Spherical_Images_CVPRW_2023_paper.pdf) | 0.548 / 0.070  | 0.804 / 7.407  |\n| LightGlue* (ICCV '23) | [Official](https://github.com/cvg/LightGlue) | [arxiv](https://arxiv.org/abs/2306.13643) | 0.417 / 0.093 / 0.184 / 0.128 | 2.828 / 8.852 / 8.100 / 8.128 |\n| SE2-LoFTR (CVPRW '22) | [Official](https://github.com/georg-bn/se2-loftr) | [arxiv](https://arxiv.org/abs/2204.10144) | 0.133 | 2.378 | \n| Aspanformer (ECCV '22) | [Official](https://github.com/apple/ml-aspanformer) | [arxiv](https://arxiv.org/abs/2208.14201) | 0.384 | 11.73 | \n| Matchformer (ACCV '22) | [Official](https://github.com/jamycheung/MatchFormer) | [arxiv](https://arxiv.org/abs/2203.09645) | 0.232 | 6.101 | \n| LoFTR (CVPR '21) | [Official](https://github.com/zju3dv/LoFTR) / [Kornia](https://kornia.readthedocs.io/en/stable/feature.html#kornia.feature.LoFTR) | [arxiv](https://arxiv.org/abs/2104.00680) | 0.722 | 2.36 | \n| Patch2Pix (CVPR '21) | [Official](https://github.com/GrumpyZhou/patch2pix)  / [IMT](https://github.com/GrumpyZhou/image-matching-toolbox) | [arxiv](https://arxiv.org/abs/2012.01909) | 0.145 | 4.97 | \n| SuperGlue (CVPR '20) | [Official](https://github.com/magicleap/SuperGluePretrainedNetwork) / [IMT](https://github.com/GrumpyZhou/image-matching-toolbox/blob/main/immatch/modules/superglue.py) | [arxiv](https://arxiv.org/abs/1911.11763)  | 0.0894 | 2.178 | \n| R2D2 (NeurIPS '19) | [Official](https://github.com/naver/r2d2) / [IMT](https://github.com/GrumpyZhou/image-matching-toolbox/blob/main/immatch/modules/r2d2.py) | [arxiv](https://arxiv.org/abs/1906.06195) | 0.429 | 6.79 | \n| D2Net (CVPR '19) | [Official](https://github.com/mihaidusmanu/d2-net) / [IMT](https://github.com/GrumpyZhou/image-matching-toolbox/blob/main/immatch/modules/d2net.py) | [arxiv](https://arxiv.org/abs/1905.03561) | 0.600 | 1.324 | \n| SIFT-NN (IJCV '04)| [OpenCV](https://docs.opencv.org/4.x/d7/d60/classcv_1_1SIFT.html) | [pdf](https://www.cs.ubc.ca/~lowe/papers/ijcv04.pdf) |0.124 | 0.117 | \n| ORB-NN (ICCV '11)| [OpenCV](https://docs.opencv.org/3.4/db/d95/classcv_1_1ORB.html) | [ResearchGate](https://www.researchgate.net/publication/221111151_ORB_an_efficient_alternative_to_SIFT_or_SURF) |0.088 | 0.092 |\n| DoGHardNet (NeurIPS '17) | [IMT](https://github.com/GrumpyZhou/image-matching-toolbox/blob/main/immatch/modules/dogaffnethardnet.py) / [Kornia](https://kornia.readthedocs.io/en/stable/feature.html#kornia.feature.HardNet) | [arxiv](https://arxiv.org/abs/1705.10872v4) | 2.697 (+NN) / 0.526 (+LG) | 2.438(+NN) / 4.528 (+LG) |\n\u003c!--| SiLK (ICCV '23) | [Official](https://github.com/facebookresearch/silk) | [arxiv](https://arxiv.org/abs/2304.06194) | 0.694 | 3.733 | --\u003e\n\nOur implementation of Patch2Pix (+ Patch2PixSuperGlue), R2D2, and D2Net are based on the [Image Matching Toolbox](https://github.com/GrumpyZhou/image-matching-toolbox/tree/main) (IMT). LoFTR and DeDoDe-Lightglue are from [Kornia](https://github.com/kornia/kornia). Other models are based on the offical repos above.\n\nRuntime benchmark is the average of 5 iterations over the 5 pairs of examples in the `assets/example_pairs` folder at image size 512x512. Benchmark is done using `benchmark.py` on an NVIDIA RTX A4000 GPU. Results rounded to the hundredths place.\n\n\\* `SphereGlue` model runtimes are listed in the order: SIFT, SuperPoint\n\n\\* `LightGlue` model runtimes are listed in the order: SIFT, SuperPoint, Disk, ALIKED\n\n\\* `Keypt2Subpx` model runtimes are listed in the order: superpoint-lg, aliked-lg, xfeat, dedode\n\n\\* `MINIMA` model runtimes are listed in the order: superpoint-lg, LoFTR, RoMa (large), RoMa (tiny)\n##\n\n### Adding a new method\nSee [CONTRIBUTING.md](CONTRIBUTING.md) for details. \n\n\n    \n\u003e [!Note]  \n\u003e This repo is optimized for usability, not necessarily for speed or performance. Ideally you can use this repo to find the matcher that best suits your needs, and then use the original code (or a modified version of this code) to get maximize performance. Default hyperparameters used here **may not be optimal for your use case!**\n\n    \n### Acknowledgements\n  \nSpecial thanks to the authors of the respective works that are included in this repo (see their papers above). Additional thanks to [@GrumpyZhou](https://github.com/GrumpyZhou) for developing and maintaining the [Image Matching Toolbox](https://github.com/GrumpyZhou/image-matching-toolbox/tree/main), which we have wrapped in this repo, and the [maintainers](https://github.com/kornia/kornia?tab=readme-ov-file#community) of [Kornia](https://github.com/kornia/kornia).\n\n\n## Cite\nThis repo was created as part of the EarthMatch paper. Please consider citing EarthMatch if this repo is helpful to you!\n\n```\n@InProceedings{Berton_2024_EarthMatch,\n    author    = {Berton, Gabriele and Goletto, Gabriele and Trivigno, Gabriele and Stoken, Alex and Caputo, Barbara and Masone, Carlo},\n    title     = {EarthMatch: Iterative Coregistration for Fine-grained Localization of Astronaut Photography},\n    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops},\n    month     = {June},\n    year      = {2024},\n}\n```\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falexstoken%2Fimage-matching-models","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Falexstoken%2Fimage-matching-models","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falexstoken%2Fimage-matching-models/lists"}