{"id":15034899,"url":"https://github.com/zju3dv/loftr","last_synced_at":"2025-05-15T04:04:21.507Z","repository":{"id":37426960,"uuid":"343635240","full_name":"zju3dv/LoFTR","owner":"zju3dv","description":"Code for \"LoFTR: Detector-Free Local Feature Matching with Transformers\", CVPR 2021, T-PAMI 2022","archived":false,"fork":false,"pushed_at":"2024-05-31T06:54:41.000Z","size":31584,"stargazers_count":2531,"open_issues_count":87,"forks_count":378,"subscribers_count":42,"default_branch":"master","last_synced_at":"2025-05-15T04:03:09.468Z","etag":null,"topics":["3d-vision","feature-matching","pose-estimation"],"latest_commit_sha":null,"homepage":"https://zju3dv.github.io/loftr/","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/zju3dv.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-03-02T03:36:29.000Z","updated_at":"2025-05-13T10:12:29.000Z","dependencies_parsed_at":"2023-01-21T07:15:40.109Z","dependency_job_id":"628e0527-288d-46d1-9e5c-7caa10c6ec4a","html_url":"https://github.com/zju3dv/LoFTR","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zju3dv%2FLoFTR","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zju3dv%2FLoFTR/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zju3dv%2FLoFTR/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zju3dv%2FLoFTR/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/zju3dv","download_url":"https://codeload.github.com/zju3dv/LoFTR/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254270641,"owners_count":22042858,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["3d-vision","feature-matching","pose-estimation"],"created_at":"2024-09-24T20:26:44.529Z","updated_at":"2025-05-15T04:04:21.472Z","avatar_url":"https://github.com/zju3dv.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# LoFTR: Detector-Free Local Feature Matching with Transformers\n### [Project Page](https://zju3dv.github.io/loftr) | [Paper](https://arxiv.org/pdf/2104.00680.pdf)\n\u003cbr/\u003e\n\n\u003e LoFTR: Detector-Free Local Feature Matching with Transformers  \n\u003e [Jiaming Sun](https://jiamingsun.ml)\u003csup\u003e\\*\u003c/sup\u003e, [Zehong Shen](https://zehongs.github.io/)\u003csup\u003e\\*\u003c/sup\u003e, [Yu'ang Wang](https://github.com/angshine)\u003csup\u003e\\*\u003c/sup\u003e, [Hujun Bao](http://www.cad.zju.edu.cn/home/bao/), [Xiaowei Zhou](http://www.cad.zju.edu.cn/home/xzhou/)  \n\u003e CVPR 2021\n\n![demo_vid](assets/loftr-github-demo.gif)\n\n## TODO List and ETA\n- [x] Inference code and pretrained models (DS and OT) (2021-4-7)\n- [x] Code for reproducing the test-set results (2021-4-7)\n- [x] Webcam demo to reproduce the result shown in the GIF above (2021-4-13)\n- [x] Training code and training data preparation (expected 2021-6-10)\n\nDiscussions about the paper are welcomed in the [discussion panel](https://github.com/zju3dv/LoFTR/discussions).\n\n:thinking: **FAQ**\n\n1. Undistorted images from D2Net are not available anymore.  \n   For a temporal alternative, please use the undistorted images provided by the MegaDepth_v1 (should be downloaded along with the required depth files). We numerically compared these images and only found very subtle difference.\n\n:triangular_flag_on_post: **Updates**\n- Check out [QuadTreeAttention](https://github.com/Tangshitao/QuadTreeAttention), a new attention machanism that improves the efficiency and performance of LoFTR with less demanding GPU requirements for training.\n- :white_check_mark: Integrated to [Huggingface Spaces](https://huggingface.co/spaces) with [Gradio](https://github.com/gradio-app/gradio). See [Gradio Web Demo](https://huggingface.co/spaces/akhaliq/Kornia-LoFTR)\n## Colab demo\nWant to run LoFTR with custom image pairs without configuring your own GPU environment? Try the Colab demo:\n[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1BgNIOjFHauFoNB95LGesHBIjioX74USW?usp=sharing)\n\n## Using from kornia\n\nLoFTR is integrated into [kornia](https://github.com/kornia/kornia) library since version 0.5.11.\n\n```\npip install kornia\n```\n\nThen you can import it as \n```python3\nfrom kornia.feature import LoFTR\n```\n\nSee tutorial on using LoFTR from kornia [here](https://kornia-tutorials.readthedocs.io/en/latest/image_matching.html).\n\n\n## Installation\n```shell\n# For full pytorch-lightning trainer features (recommended)\nconda env create -f environment.yaml\nconda activate loftr\n\n# For the LoFTR matcher only\npip install torch einops yacs kornia\n```\n\nWe provide the [download link](https://drive.google.com/drive/folders/1DOcOPZb3-5cWxLqn256AhwUVjBPifhuf?usp=sharing) to \n  - the scannet-1500-testset (~1GB).\n  - the megadepth-1500-testset (~600MB).\n  - 4 pretrained models of indoor-ds, indoor-ot, outdoor-ds and outdoor-ot (each ~45MB).\n\nBy now, the environment is all set and the LoFTR-DS model is ready to go! \nIf you want to run LoFTR-OT, some extra steps are needed:\n\n\u003cdetails\u003e\n  \u003csummary\u003e[Requirements for LoFTR-OT]\u003c/summary\u003e\n\n  We use the code from [SuperGluePretrainedNetwork](https://github.com/magicleap/SuperGluePretrainedNetwork) for optimal transport. However, we can't provide the code directly due its strict LICENSE requirements. We recommend downloading it with the following command instead. \n\n  ```shell\n  cd src/loftr/utils  \n  wget https://raw.githubusercontent.com/magicleap/SuperGluePretrainedNetwork/master/models/superglue.py \n  ```\n\u003c/details\u003e\n\n\n## Run LoFTR demos\n\n### Match image pairs with LoFTR\n\n\u003cdetails\u003e\n  \u003csummary\u003e[code snippets]\u003c/summary\u003e\n\n  ```python\n  from src.loftr import LoFTR, default_cfg\n\n  # Initialize LoFTR\n  matcher = LoFTR(config=default_cfg)\n  matcher.load_state_dict(torch.load(\"weights/indoor_ds.ckpt\")['state_dict'])\n  matcher = matcher.eval().cuda()\n\n  # Inference\n  with torch.no_grad():\n      matcher(batch)    # batch = {'image0': img0, 'image1': img1}\n      mkpts0 = batch['mkpts0_f'].cpu().numpy()\n      mkpts1 = batch['mkpts1_f'].cpu().numpy()\n  ```\n\n\u003c/details\u003e\n\nAn example is given in `notebooks/demo_single_pair.ipynb`.\n\n### Online demo\nRun the online demo with a webcam or video to reproduce the result shown in the GIF above.\n```bash\ncd demo\n./run_demo.sh\n```\n\u003cdetails\u003e\n  \u003csummary\u003e[run_demo.sh]\u003c/summary\u003e\n\n  ```bash\n  #!/bin/bash\n  set -e\n  # set -x\n\n  if [ ! -f utils.py ]; then\n      echo \"Downloading utils.py from the SuperGlue repo.\"\n      echo \"We cannot provide this file directly due to its strict licence.\"\n      wget https://raw.githubusercontent.com/magicleap/SuperGluePretrainedNetwork/master/models/utils.py\n  fi\n\n  # Use webcam 0 as input source. \n  input=0\n  # or use a pre-recorded video given the path.\n  # input=/home/sunjiaming/Downloads/scannet_test/$scene_name.mp4\n\n  # Toggle indoor/outdoor model here.\n  model_ckpt=../weights/indoor_ds.ckpt\n  # model_ckpt=../weights/outdoor_ds.ckpt\n\n  # Optionally assign the GPU ID.\n  # export CUDA_VISIBLE_DEVICES=0\n\n  echo \"Running LoFTR demo..\"\n  eval \"$(conda shell.bash hook)\"\n  conda activate loftr\n  python demo_loftr.py --weight $model_ckpt --input $input\n  # To save the input video and output match visualizations.\n  # python demo_loftr.py --weight $model_ckpt --input $input --save_video --save_input\n\n  # Running on remote GPU servers with no GUI.\n  # Save images first.\n  # python demo_loftr.py --weight $model_ckpt --input $input --no_display --output_dir=\"./demo_images/\"\n  # Then convert them to a video.\n  # ffmpeg -framerate 15 -pattern_type glob -i '*.png' -c:v libx264 -r 30 -pix_fmt yuv420p out.mp4\n\n  ```\n\u003c/details\u003e\n\n### Reproduce the testing results with pytorch-lightning\nYou need to setup the testing subsets of ScanNet and MegaDepth first. We create symlinks from the previously downloaded datasets to `data/{{dataset}}/test`.\n\n```shell\n# set up symlinks\nln -s /path/to/scannet-1500-testset/* /path/to/LoFTR/data/scannet/test\nln -s /path/to/megadepth-1500-testset/* /path/to/LoFTR/data/megadepth/test\n```\n\n```shell\nconda activate loftr\n# with shell script\nbash ./scripts/reproduce_test/indoor_ds.sh\n\n# or\npython test.py configs/data/scannet_test_1500.py configs/loftr/loftr_ds.py --ckpt_path weights/indoor_ds.ckpt --profiler_name inference --gpus=1 --accelerator=\"ddp\"\n```\n\nFor visualizing the results, please refer to `notebooks/visualize_dump_results.ipynb`.\n\n\u003cbr/\u003e\n\n\n\u003c!-- ### Image pair info for training on ScanNet\nYou can download the data at [here](https://drive.google.com/file/d/1fC2BezUSsSQy7_H65A0ZfrYK0RB3TXXj/view?usp=sharing).\n\n\u003cdetails\u003e\n  \u003csummary\u003e[data format]\u003c/summary\u003e\n\n```python\nIn [14]: npz_path = './cfg_1513_-1_0.2_0.8_0.15/scene_data/train/scene0000_01.npz'\n\nIn [15]: data = np.load(npz_path)\n\nIn [16]: data['name']\nOut[16]:\narray([[   0,    1,  276,  567],\n       [   0,    1, 1147, 1170],\n       [   0,    1,  541, 5757],\n       ...,\n       [   0,    1, 5366, 5393],\n       [   0,    1, 2607, 5278],\n       [   0,    1,  736, 5844]], dtype=uint16)\n\nIn [17]: data['score']\nOut[17]: array([0.2903, 0.7715, 0.5986, ..., 0.7227, 0.5527, 0.4148], dtype=float16)\n\nIn [18]: len(data['name'])\nOut[18]: 1684276\n\nIn [19]: len(data['score'])\nOut[19]: 1684276\n```\n`data['name']` is the image pair info, organized as [`scene_id`, `seq_id`, `image0_id`, `image1_id`].\n\n`data['score']` is the overlapping score defined in [SuperGlue](https://arxiv.org/pdf/1911.11763) (Page 12).\n\u003c/details\u003e --\u003e\n\n\n## Training\nSee [Training LoFTR](./docs/TRAINING.md) for more details.\n\n## Citation\n\nIf you find this code useful for your research, please use the following BibTeX entry.\n\n```bibtex\n@article{sun2021loftr,\n  title={{LoFTR}: Detector-Free Local Feature Matching with Transformers},\n  author={Sun, Jiaming and Shen, Zehong and Wang, Yuang and Bao, Hujun and Zhou, Xiaowei},\n  journal={{CVPR}},\n  year={2021}\n}\n```\n\n## Copyright\nThis work is affiliated with ZJU-SenseTime Joint Lab of 3D Vision, and its intellectual property belongs to SenseTime Group Ltd.\n\n```\nCopyright SenseTime. All Rights Reserved.\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use this file except in compliance with the License.\nYou may obtain a copy of the License at\n\n    http://www.apache.org/licenses/LICENSE-2.0\n\nUnless required by applicable law or agreed to in writing, software\ndistributed under the License is distributed on an \"AS IS\" BASIS,\nWITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nSee the License for the specific language governing permissions and\nlimitations under the License.\n```\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzju3dv%2Floftr","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fzju3dv%2Floftr","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzju3dv%2Floftr/lists"}