{"id":18792516,"url":"https://github.com/prbonn/descriptor-dr","last_synced_at":"2025-04-13T14:31:20.266Z","repository":{"id":59443683,"uuid":"533213956","full_name":"PRBonn/descriptor-dr","owner":"PRBonn","description":"Learning-Based Dimensionality Reduction for Computing Compact and Effective Local Feature Descriptors (ICRA' 23)","archived":false,"fork":false,"pushed_at":"2023-07-10T08:16:45.000Z","size":15407,"stargazers_count":53,"open_issues_count":0,"forks_count":1,"subscribers_count":10,"default_branch":"main","last_synced_at":"2023-11-17T04:03:13.744Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/PRBonn.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-09-06T07:39:52.000Z","updated_at":"2023-10-06T01:31:36.000Z","dependencies_parsed_at":"2023-01-19T00:01:07.899Z","dependency_job_id":null,"html_url":"https://github.com/PRBonn/descriptor-dr","commit_stats":null,"previous_names":[],"tags_count":0,"template":null,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PRBonn%2Fdescriptor-dr","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PRBonn%2Fdescriptor-dr/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PRBonn%2Fdescriptor-dr/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PRBonn%2Fdescriptor-dr/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/PRBonn","download_url":"https://codeload.github.com/PRBonn/descriptor-dr/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":223589798,"owners_count":17170037,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-07T21:20:20.576Z","updated_at":"2024-11-07T21:20:21.135Z","avatar_url":"https://github.com/PRBonn.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Descriptors Dimensionality Reduction\n\nThis repository contains the implementation of the paper:\n\n**Learning-Based Dimensionality Reduction for Computing Compact and Effective Local Feature Descriptors (ICRA2023)**  \n[Hao Dong](https://sites.google.com/view/dong-hao/), [Xieyuanli Chen](https://www.ipb.uni-bonn.de/people/xieyuanli-chen/), [Mihai Dusmanu](https://dsmn.ml/), [Viktor Larsson](https://vlarsson.github.io/), [Marc Pollefeys](https://www.inf.ethz.ch/personal/pomarc/) and [Cyrill Stachniss](http://www.ipb.uni-bonn.de/people/cyrill-stachniss/)  \n[Link](https://arxiv.org/abs/2209.13586) to the arXiv version of the paper is available.\n\nWe propose and evaluate an MLP-based network for descriptor dimensionality reduction and show its superiority over PCA on multiple descriptors in various tasks.\n\n\u003cimg src=\"pics/overview.png\" width=\"400\"\u003e\nOverview of our approach. We first compute descriptors of given image patches. Then an MLP-based network is used for dimensionality reduction. We aim to learn an MLP-based projection better than PCA to generate lower-dimensional descriptors.\n\n## Abstract\nA distinctive representation of image patches in form of features is a key component of many computer vision and robotics tasks, such as image matching, image retrieval, and visual localization. State-of-the-art descriptors, from hand-crafted descriptors such as SIFT to learned ones such as HardNet, are usually high dimensional; 128 dimensions or even more. The higher the dimensionality, the larger the memory consumption and computational time for approaches using such descriptors. In this paper, we investigate multi-layer perceptrons (MLPs) to extract low-dimensional but high-quality descriptors. We thoroughly analyze our method in unsupervised, self-supervised, and supervised settings, and evaluate the dimensionality reduction results on four representative descriptors. We consider different applications, including visual localization, patch verification, image matching and retrieval. The experiments show that our lightweight MLPs achieve better dimensionality reduction than PCA. The lower-dimensional descriptors generated by our approach outperform the original higher-dimensional descriptors in downstream tasks, especially for the hand-crafted ones. \n\n## Installation\n```bash\npip install numpy kornia tqdm torch torchvision scipy faiss tensorboard_logger tabulate\n```\n## Example\nA simple example for SIFT-SV-64 with pre-trained model:\n```bash\npython example.py\n```\n\n## Training\n### PCA Reduction\nFor SIFT:\n```bash\npython dr_pca.py --descriptor SIFT\n```\nFor HardNet:\n```bash\npython dr_pca.py --descriptor HardNet\n```\n\n### Unsupervised Reduction\nFor SIFT:\n```bash\npython dr_ae_SIFT.py\n```\nFor HardNet:\n```bash\npython dr_ae_HardNet.py\n```\n\n### Self-supervised Reduction\nFor SIFT:\n```bash\npython dr_ss_SIFT.py\n```\nFor HardNet:\n```bash\npython dr_ss_HardNet.py\n```\n\n### Supervised Reduction\nFor SIFT:\n```bash\npython dr_sv_SIFT.py\n```\nFor HardNet:\n```bash\npython dr_sv_HardNet.py\n```\n\n### Evaluate on HPatches dataset\nDownload HPatches dataset and benchmark:\n```bash\ngit clone https://github.com/hpatches/hpatches-benchmark.git\ncd hpatches-benchmark/\nsh download.sh hpatches\n```\nExtract descriptors on HPatches dataset and evaluate:\n```bash\ncd ..\npython hpatches_extract_SIFT_64.py /path/to/HPatches/dataset\nmkdir hpatches-benchmark/data/descriptors\nmv SIFT_sv_dim64/ hpatches-benchmark/data/descriptors/\ncd hpatches-benchmark/python/\npython hpatches_eval.py --descr-name=SIFT_sv_dim64 --task=matching --delimiter=\",\"\npython hpatches_results.py --descr=SIFT_sv_dim64 --results-dir=results/ --task=matching\npython hpatches_eval.py --descr-name=SIFT_sv_dim64 --task=verification --delimiter=\",\"\npython hpatches_results.py --descr=SIFT_sv_dim64 --results-dir=results/ --task=verification\npython hpatches_eval.py --descr-name=SIFT_sv_dim64 --task=retrieval --delimiter=\",\"\npython hpatches_results.py --descr=SIFT_sv_dim64 --results-dir=results/ --task=retrieval\n```\n\n\n\n## Citation\nIf you use our implementation in your academic work, please cite the corresponding [paper](https://arxiv.org/abs/2209.13586):\n\n\t@inproceedings{dong2022dr,\n\t\ttitle={Learning-Based Dimensionality Reduction for Computing Compact and Effective Local Feature Descriptors},\n\t\tauthor={Dong, Hao and Chen, Xieyuanli and Dusmanu, Mihai and Larsson, Viktor and Pollefeys, Marc and Stachniss, Cyrill},\n\t\tbooktitle={Proceedings of the IEEE International Conference on Robotics and Automation (ICRA)},\n  \t\tyear={2023}\n\t}\n\n## Localization on Aachen Day-Night v1.1 and InLoc datasets\n\u003cimg src=\"pics/localization.PNG\" width=\"450\"\u003e\n\n## Supplementary\n\n### Embedding visualization of the descriptors\n\n\u003cimg src=\"pics/SIFT.svg\" width=\"230\"\u003e\u003cimg src=\"pics/SIFT-PCA-64.svg\" width=\"230\"\u003e\u003cimg src=\"pics/SIFT-Ours-SV-64.svg\" width=\"230\"\u003e  \u003cbr /\u003e\n\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;SIFT\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;SIFT-PCA-64\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;SIFT-Ours-SV-64 \u003cbr /\u003e\n\u003cimg src=\"pics/MKD.svg\" width=\"230\"\u003e\u003cimg src=\"pics/MKD-PCA-64.svg\" width=\"230\"\u003e\u003cimg src=\"pics/MKD-Ours-SV-64.svg\" width=\"230\"\u003e  \u003cbr /\u003e\n\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;MKD\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;MKD-PCA-64\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;MKD-Ours-SV-64 \u003cbr /\u003e\n\u003cimg src=\"pics/TFeat.svg\" width=\"230\"\u003e\u003cimg src=\"pics/TFeat-PCA-64.svg\" width=\"230\"\u003e\u003cimg src=\"pics/TFeat-Ours-SV-64.svg\" width=\"230\"\u003e  \u003cbr /\u003e\n\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;TFeat\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;TFeat-PCA-64\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;TFeat-Ours-SV-64 \u003cbr /\u003e\n\u003cimg src=\"pics/HardNet.svg\" width=\"230\"\u003e\u003cimg src=\"pics/HardNet-PCA-64.svg\" width=\"230\"\u003e\u003cimg src=\"pics/HardNet-Ours-SV-64.svg\" width=\"230\"\u003e  \u003cbr /\u003e\n\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;HardNet\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;HardNet-PCA-64\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;HardNet-Ours-SV-64 \u003cbr /\u003e\n\nWe provide the t-SNE embedding visualization of the descriptors on UBC Phototour Liberty. We visualize the embeddings of SIFT, MKD, TFeat, and HardNet generated using PCA and supervised methods by mapping high-dimensional descriptors (128 and 64) into 2D using t-SNE visualization. We pass the image patches through the descriptor extractor, followed by PCA or MLPs, to get lower-dimensional descriptors and determine their 2D locations using t-SNE transformation. Finally, we visualize the entire patch at each location.\n\nFrom the visualization, we can observe similar results as we discussed in the paper. For SIFT and MKD, the original descriptor space is irregular, and similar and dissimilar features are overlapped. Therefore, PCA projection will keep this irregular structure of the descriptor space. However, after learning a more discriminative representation using triplet loss, similar image patches in the descriptor space are close to each other while dissimilar ones have distances from each other. For TFeat and HardNet, since the outputting space is already optimized for the $\\ell_2$ metric, image patches in the descriptor space are already separated well based on their appearance. Therefore, a simple PCA can preserve this distinctive structure and perform on par compared with the learned projection.  \n\n\n### Experiments for patch verification, image matching, and patch retrieval on HPatches dataset\n\u003cimg src=\"pics/SIFT-Hpatches.png\" width=\"450\"\u003e  \u003cbr /\u003e\n\u003cimg src=\"pics/MKD-Hpatches.PNG\" width=\"450\"\u003e  \u003cbr /\u003e\n\u003cimg src=\"pics/TFeat-Hpatches.PNG\" width=\"450\"\u003e  \u003cbr /\u003e\n\u003cimg src=\"pics/HardNet-Hpatches.PNG\" width=\"450\"\u003e  \u003cbr /\u003e\n\n\n## Acknowledgment\nWe thank greatly for the authors of the following opensource projects:\n\n- [deepcluster](https://github.com/facebookresearch/deepcluster) \n- [hardnet](https://github.com/DagnyT/hardnet)","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fprbonn%2Fdescriptor-dr","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fprbonn%2Fdescriptor-dr","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fprbonn%2Fdescriptor-dr/lists"}