{"id":19401060,"url":"https://github.com/google-research/omniglue","last_synced_at":"2025-04-30T13:22:40.579Z","repository":{"id":240993745,"uuid":"797399636","full_name":"google-research/omniglue","owner":"google-research","description":"Code release for CVPR'24 submission 'OmniGlue'","archived":false,"fork":false,"pushed_at":"2024-05-22T01:13:29.000Z","size":12045,"stargazers_count":81,"open_issues_count":0,"forks_count":2,"subscribers_count":3,"default_branch":"main","last_synced_at":"2024-05-22T20:56:28.403Z","etag":null,"topics":["image-matching","multi-view-geometry"],"latest_commit_sha":null,"homepage":"https://hwjiang1510.github.io/OmniGlue","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/google-research.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-05-07T18:50:52.000Z","updated_at":"2024-06-17T18:37:38.053Z","dependencies_parsed_at":"2024-06-17T18:53:34.223Z","dependency_job_id":null,"html_url":"https://github.com/google-research/omniglue","commit_stats":null,"previous_names":["google-research/omniglue"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google-research%2Fomniglue","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google-research%2Fomniglue/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google-research%2Fomniglue/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google-research%2Fomniglue/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/google-research","download_url":"https://codeload.github.com/google-research/omniglue/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":223942653,"owners_count":17229118,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["image-matching","multi-view-geometry"],"created_at":"2024-11-10T11:16:59.258Z","updated_at":"2024-11-10T11:16:59.847Z","avatar_url":"https://github.com/google-research.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n\n# \\[CVPR'24\\] Code release for OmniGlue\n\n\u003cp align=\"center\"\u003e\n    \u003ca href=\"https://hwjiang1510.github.io/\"\u003eHanwen Jiang\u003c/a\u003e,\n    \u003ca href=\"https://scholar.google.com/citations?user=jgSItF4AAAAJ\"\u003eArjun Karpur\u003c/a\u003e,\n    \u003ca href=\"https://scholar.google.com/citations?user=7EeSOcgAAAAJ\"\u003eBingyi Cao\u003c/a\u003e,\n    \u003ca href=\"https://www.cs.utexas.edu/~huangqx/\"\u003eQixing Huang\u003c/a\u003e,\n    \u003ca href=\"https://andrefaraujo.github.io/\"\u003eAndre Araujo\u003c/a\u003e\n\u003c/p\u003e\n\n\u003c/div\u003e\n\n--------------------------------------------------------------------------------\n\n\u003cdiv align=\"center\"\u003e\n    \u003ca href=\"https://hwjiang1510.github.io/OmniGlue/\"\u003e\u003cstrong\u003eProject Page\u003c/strong\u003e\u003c/a\u003e |\n    \u003ca href=\"https://arxiv.org/abs/2405.12979\"\u003e\u003cstrong\u003ePaper\u003c/strong\u003e\u003c/a\u003e |\n    \u003ca href=\"#installation\"\u003e\u003cstrong\u003eUsage\u003c/strong\u003e\u003c/a\u003e |\n    \u003ca href=\"https://huggingface.co/spaces/qubvel-hf/omniglue\"\u003e\u003cstrong\u003eDemo\u003c/strong\u003e\u003c/a\u003e\n\u003c/div\u003e\n\n\u003cbr\u003e\n\n\u003cdiv align=\"center\"\u003e\n\n[![Open in Spaces](https://huggingface.co/datasets/huggingface/badges/resolve/main/open-in-hf-spaces-sm.svg)](https://huggingface.co/spaces/qubvel-hf/omniglue)\n\n\u003c/div\u003e\n\n\u003cbr\u003e\n\nOfficial code release for the CVPR 2024 paper: **OmniGlue: Generalizable Feature\nMatching with Foundation Model Guidance**.\n\n![og_diagram.png](res/og_diagram.png \"og_diagram.png\")\n\n**Abstract:** The image matching field has been witnessing a continuous\nemergence of novel learnable feature matching techniques, with ever-improving\nperformance on conventional benchmarks. However, our investigation shows that\ndespite these gains, their potential for real-world applications is restricted\nby their limited generalization capabilities to novel image domains. In this\npaper, we introduce OmniGlue, the first learnable image matcher that is designed\nwith generalization as a core principle. OmniGlue leverages broad knowledge from\na vision foundation model to guide the feature matching process, boosting\ngeneralization to domains not seen at training time. Additionally, we propose a\nnovel keypoint position-guided attention mechanism which disentangles spatial\nand appearance information, leading to enhanced matching descriptors. We perform\ncomprehensive experiments on a suite of 6 datasets with varied image domains,\nincluding scene-level, object-centric and aerial images. OmniGlue’s novel\ncomponents lead to relative gains on unseen domains of 18.8% with respect to a\ndirectly comparable reference model, while also outperforming the recent\nLightGlue method by 10.1% relatively.\n\n\n## Installation\n\nFirst, use pip to install `omniglue`:\n\n```sh\nconda create -n omniglue pip\nconda activate omniglue\n\ngit clone https://github.com/google-research/omniglue.git\ncd omniglue\npip install -e .\n```\n\nThen, download the following models to `./models/`\n\n```sh\n# Download to ./models/ dir.\nmkdir models\ncd models\n\n# SuperPoint.\ngit clone https://github.com/rpautrat/SuperPoint.git\nmv SuperPoint/pretrained_models/sp_v6.tgz . \u0026\u0026 rm -rf SuperPoint\ntar zxvf sp_v6.tgz \u0026\u0026 rm sp_v6.tgz\n\n# DINOv2 - vit-b14.\nwget https://dl.fbaipublicfiles.com/dinov2/dinov2_vitb14/dinov2_vitb14_pretrain.pth\n\n# OmniGlue.\nwget https://storage.googleapis.com/omniglue/og_export.zip\nunzip og_export.zip \u0026\u0026 rm og_export.zip\n```\n\nDirect download links:\n\n-   [[SuperPoint weights]](https://github.com/rpautrat/SuperPoint/tree/master/pretrained_models): from [github.com/rpautrat/SuperPoint](https://github.com/rpautrat/SuperPoint)\n-   [[DINOv2 weights]](https://dl.fbaipublicfiles.com/dinov2/dinov2_vitb14/dinov2_vitb14_pretrain.pth): from [github.com/facebookresearch/dinov2](https://github.com/facebookresearch/dinov2) (ViT-B/14 distilled backbone without register).\n-   [[OmniGlue weights]](https://storage.googleapis.com/omniglue/og_export.zip)\n\n## Usage\nThe code snippet below outlines how you can perform OmniGlue inference in your\nown python codebase.\n\n```py\n\nimport omniglue\n\nimage0 = ... # load images from file into np.array\nimage1 = ...\n\nog = omniglue.OmniGlue(\n  og_export='./models/og_export',\n  sp_export='./models/sp_v6',\n  dino_export='./models/dinov2_vitb14_pretrain.pth',\n)\n\nmatch_kp0s, match_kp1s, match_confidences = og.FindMatches(image0, image1)\n# Output:\n#   match_kp0: (N, 2) array of (x,y) coordinates in image0.\n#   match_kp1: (N, 2) array of (x,y) coordinates in image1.\n#   match_confidences: N-dim array of each of the N match confidence scores.\n```\n\n## Demo\n\n`demo.py` contains example usage of the `omniglue` module. To try with your own\nimages, replace `./res/demo1.jpg` and `./res/demo2.jpg` with your own\nfilepaths.\n\n```sh\nconda activate omniglue\npython demo.py ./res/demo1.jpg ./res/demo2.jpg\n# \u003csee output in './demo_output.png'\u003e\n```\n\nExpected output:\n![demo_output.png](res/demo_output.png \"demo_output.png\")\n\n\n## Repo TODOs\n\n- ~~Provide `demo.py` example usage script.~~\n- ~~Add to image matching webui~~ (credit: [@Vincentqyw](https://github.com/Vincentqyw))\n- Support matching for pre-extracted features.\n- Release eval pipelines for in-domain (MegaDepth).\n- Release eval pipelines for all out-of-domain datasets.\n\n## BibTex\n```\n@inproceedings{jiang2024Omniglue,\n   title={OmniGlue: Generalizable Feature Matching with Foundation Model Guidance},\n   author={Jiang, Hanwen and Karpur, Arjun and Cao, Bingyi and Huang, Qixing and Araujo, Andre},\n   booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},\n   year={2024},\n}\n```\n\n--------------------------------------------------------------------------------\n\nThis is not an officially supported Google product.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgoogle-research%2Fomniglue","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgoogle-research%2Fomniglue","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgoogle-research%2Fomniglue/lists"}