{"id":20516346,"url":"https://github.com/nianticlabs/mickey","last_synced_at":"2025-05-15T20:03:44.922Z","repository":{"id":232433112,"uuid":"771059828","full_name":"nianticlabs/mickey","owner":"nianticlabs","description":"[CVPR 2024 - Oral] Matching 2D Images in 3D: Metric Relative Pose from Metric Correspondences","archived":false,"fork":false,"pushed_at":"2024-12-02T10:39:34.000Z","size":819,"stargazers_count":547,"open_issues_count":2,"forks_count":35,"subscribers_count":12,"default_branch":"main","last_synced_at":"2025-04-08T04:11:39.657Z","etag":null,"topics":["computer-vision","cvpr2024","dsac","metric-correspondences","mickey","pose-estimation","ransac","ransac-algorithm"],"latest_commit_sha":null,"homepage":"https://nianticlabs.github.io/mickey/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/nianticlabs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-03-12T16:06:42.000Z","updated_at":"2025-04-05T02:46:51.000Z","dependencies_parsed_at":"2024-04-09T20:34:10.161Z","dependency_job_id":"bde2e88c-dbd8-4a1c-a06f-e3474474fa9c","html_url":"https://github.com/nianticlabs/mickey","commit_stats":null,"previous_names":["nianticlabs/mickey"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nianticlabs%2Fmickey","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nianticlabs%2Fmickey/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nianticlabs%2Fmickey/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nianticlabs%2Fmickey/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/nianticlabs","download_url":"https://codeload.github.com/nianticlabs/mickey/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254414493,"owners_count":22067271,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["computer-vision","cvpr2024","dsac","metric-correspondences","mickey","pose-estimation","ransac","ransac-algorithm"],"created_at":"2024-11-15T21:28:22.754Z","updated_at":"2025-05-15T20:03:43.108Z","avatar_url":"https://github.com/nianticlabs.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv style=\"text-align: center;\"\u003e\n  \u003ch1\u003eMatching 2D Images in 3D: Metric Relative Pose from Metric Correspondences\u003c/h1\u003e\n    \u003cp\u003e\n    \u003ca href=\"https://scholar.google.com/citations?user=m_SPRGUAAAAJ\u0026hl=en\"\u003eAxel Barroso-Laguna\u003c/a\u003e\n    ·\n    \u003ca href=\"https://scholar.google.com/citations?user=l-zRzDEAAAAJ\u0026hl=en\"\u003eSowmya Munukutla\u003c/a\u003e\n    ·\n    \u003ca href=\"https://www.robots.ox.ac.uk/~victor/\"\u003eVictor Adrian Prisacariu\u003c/a\u003e\n    ·\n    \u003ca href=\"https://ebrach.github.io/\"\u003eEric Brachmann\u003c/a\u003e\n  \u003c/p\u003e\n  \u003ch2 style=\"font-size:1.7em; margin-top: -0.5rem; margin-bottom: -0.5rem;\"\u003eCVPR 2024 (Oral)\u003c/h2\u003e  \n  \u003ch3\u003e\u003ca href=\"https://nianticlabs.github.io/mickey/\"\u003eProject Page\u003c/a\u003e | \u003ca href=\"https://storage.googleapis.com/niantic-lon-static/research/mickey/mickey_main_paper.pdf\"\u003ePaper\u003c/a\u003e | \u003ca href=\"https://arxiv.org/abs/2404.06337\"\u003earXiv\u003c/a\u003e | \u003ca href=\"https://storage.googleapis.com/niantic-lon-static/research/mickey/mickey_supp.pdf\"\u003eSupplemental\u003c/a\u003e\u003c/h3\u003e\n\u003c/div\u003e\n\nThis is the reference implementation of the paper **\"Matching 2D Images in 3D: Metric Relative Pose from Metric Correspondences\"** presented at **CVPR 2024**.\n\nThe paper introduces **M**etr**ic Key**points (MicKey), a feature detection pipeline that regresses keypoint positions in camera space.\nMicKey presents a differentiable approach to establish metric correspondences via descriptor matching. From the metric correspondences, MicKey recovers metric relative poses.\nMicKey is trained in an end-to-end fashion using differentiable pose optimization and requires only image pairs and their ground truth relative poses for supervision.\n\n\u003cp align=\"center\"\u003e\n    \u003cimg src=\"resources/teaser_mickey.png\" alt=\"teaser\" width=\"90%\"\u003e\n\u003c/p\u003e\n\n## Setup\n\nAssuming a fresh [Anaconda](https://www.anaconda.com/download/) distribution, you can install dependencies with:\n```shell\nconda env create -f resources/environment.yml\nconda activate mickey\n```\nWe ran our experiments with PyTorch 2.0.1, CUDA 11.6, Python 3.8.17 and Debian GNU/Linux 11.\n\n## Evaluating MicKey\nMicKey aims at addressing the problem of instant Augmented Reality (AR) introduced in the [Map-free benchmark](https://research.nianticlabs.com/mapfree-reloc-benchmark).\nIn the Map-free set up, instead of building 3D maps from hundreds of images and scale calibrations, they propose to use only one photo of a scene as the map.\nThe Map-free benchmark then evaluates how accurate is the estimated metric relative pose between the reference image (the map)\nand the query image (the user).\n\n### Download Map-free dataset\nYou can find the Map-free dataset in [their project page](https://research.nianticlabs.com/mapfree-reloc-benchmark/dataset).\nExtract the test.zip file into `data/mapfree`. Optionally, if you want to train MicKey, also download train and val zip files. \n\n### Pre-trained Models\nWe provide two [MicKey models](https://storage.googleapis.com/niantic-lon-static/research/mickey/assets/mickey_weights.zip).\n  * _mickey.ckpt_: These are the default weights for MicKey, without using the overlapping scores provides in Map-free dataset and following the curriculum learning strategy described in the paper.\n  * _mickey_sc.ckpt_: These are the weights when training MicKey using the min and max overlapping scores defined in Map-free.\n\nExtract mickey_weights.zip into `weights/`. In the zip file, we also provide the default configuration needed to run the evaluation. \n\n### Run the submission script\nSimilar to Map-free code base, we provide a [submission script](submission.py) to generate submission files:\n\n```shell\npython submission.py --config path/to/config --checkpoint path/to/checkpoint --o results/your_method\n```\nThe resulting file `results/your_method/submission.zip` can be uploaded to the Map-free [online benchmark website](https://research.nianticlabs.com/mapfree-reloc-benchmark) and compared against existing methods in the [leaderboard](https://research.nianticlabs.com/mapfree-reloc-benchmark/leaderboard).\n\n### Run the local evaluation\nThe Map-free benchmark does not provide ground-truth poses for the test set. But we can still evaluate our method locally on the validation set.\n```shell\npython submission.py --config path/to/config --checkpoint path/to/checkpoint --o results/your_method --split val\n```\nand evaluate it as:\n```shell\npython -m benchmark.mapfree --submission_path results/your_method/submission.zip --split val\n```\n\n### Download MicKey correspondences and depth files\nWe provide the depth maps and correspondences computed by MicKey.\n- Download [MicKey depth maps](https://storage.googleapis.com/niantic-lon-static/research/map-free-reloc/assets/mickey_depths.tar.gz).\n- Download [MicKey correspondences](https://storage.googleapis.com/niantic-lon-static/research/map-free-reloc/assets/mickey_correspondences.zip).\n- Extract the contents of both files to `data/mapfree`\n\nRefer to the [Map-free benchmark](https://github.com/nianticlabs/map-free-reloc/tree/main?tab=readme-ov-file#feature-matching--scale-from-depth-baselines) to learn how to load precomputed correspondes and depth maps in their feature matching pipeline. \n  \n### Running MicKey in custom images\nWe provide a [demo script](demo_inference.py) to run the relative pose estimation pipeline on custom image pairs.\nAs an example, we store in `data/toy_example` two images with their respective intrinsics.\nThe script computes their metric relative pose and saves the corresponding depth and keypoint score maps in the image folder.\nRun the demo script as:\n```shell\npython demo_inference.py --im_path_ref data/toy_example/im0.jpg \\\n                         --im_path_dst data/toy_example/im1.jpg \\\n                         --intrinsics data/toy_example/intrinsics.txt \\\n                         --checkpoint path/to/checkpoint \\\n                         --config path/to/config\n```\n\nTo generate the 3D assets as in [MicKey's webpage](https://nianticlabs.github.io/mickey/), you can turn on the\n`--generate_3D_vis` flag. This will generate a rendered image with the input images, their computed 3D camera positions,\nand the set of 3D point inliers.\n\n## Training MicKey\nBesides the test scripts, we also provide the training code to train MicKey. \n\nWe provide two default configurations in `config/MicKey/`:\n  * _curriculum_learning.yaml_: This configuration follows the curriculum learning approach detailed in the paper. \n   It hence does not use any image overlapping information but only relative ground truth poses during training. \n  * _overlap_score.yaml_: This configuration relies on the image overlapping information to only choose solvable image pairs during training.\n\nBesides the two default configurations, we also provide a configuration to speed up their training.\nThese configurations use low-resolution images and do not add the null hypothesis (refer to Section 3.1.4 for details).\nWe recommend initializing MicKey with these configurations and then\nfine-tuning the network with the default ones (which use high-resolution images and the null hypothesis).\nThey can be found under `config/MicKey/`:\n  * _curriculum_learning_warm_up.yaml_\n  * _overlap_score_warm_up.yaml_\n\nTo train MicKey default model, use:\n```shell\npython train.py --config config/MicKey/curriculum_learning.yaml \\\n                --dataset_config config/datasets/mapfree.yaml \\\n                --experiment experiment_name \\\n                --path_weights path/to/checkpoint/folder\n```\nResume training from a checkpoint by adding `--resume {path_to_checkpoint}`.\n\nThe top models, according to the validation loss, the VCRE metric, and the pose AUC score, are saved during training.\nTensorboard results and checkpoints are saved into the folder `dir/to/weights/experiment_name`.\n\nNote that by default, the configuration is set to use 4 GPUs. \nYou can reduce the expected number of GPUs in the config file (e.g., _NUM_GPUS: 1_). \n\n## Changelog\n- 19 September 2024: Added vectorized RANSAC and warm up configurations.\n- 13 August 2024: Added visualization code.\n- 7 June 2024: Added precomputed depth maps and keypoint correspondences.\n\n## BibTeX\nIf you use this code in your research, please consider citing our paper:\n\n```bibtex\n@inproceedings{barroso2024mickey,\n  title={Matching 2D Images in 3D: Metric Relative Pose from Metric Correspondences},\n  author={Barroso-Laguna, Axel and Munukutla, Sowmya and Prisacariu, Victor and Brachmann, Eric},\n  booktitle={CVPR},\n  year={2024}\n}\n```\n\n## License\nCopyright © Niantic, Inc. 2024. Patent Pending. All rights reserved. This code is for non-commercial use. Please see the [license](LICENSE) file for terms.\n\n## Acknowledgements\nWe use parts of code from different repositories. We thank the authors and maintainers of the following repositories.\n- [Map-free](https://research.nianticlabs.com/mapfree-reloc-benchmark)\n- [ACE](https://github.com/nianticlabs/ace)\n- [ACE0](https://github.com/nianticlabs/acezero)\n- [DUSt3R](https://github.com/naver/dust3r)\n- [RoMa](https://github.com/Parskatt/RoMa)\n- [DINOv2](https://github.com/facebookresearch/dinov2)\n- [LoFTR](https://github.com/zju3dv/LoFTR)\n- [DPT](https://github.com/isl-org/DPT)\n- [ExtremeRotation](https://github.com/RuojinCai/ExtremeRotation_code)\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnianticlabs%2Fmickey","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnianticlabs%2Fmickey","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnianticlabs%2Fmickey/lists"}