{"id":13528909,"url":"https://github.com/naver/deep-image-retrieval","last_synced_at":"2025-04-04T22:07:22.305Z","repository":{"id":37625292,"uuid":"192671738","full_name":"naver/deep-image-retrieval","owner":"naver","description":"End-to-end learning of deep visual representations for image retrieval","archived":false,"fork":false,"pushed_at":"2021-05-19T09:07:28.000Z","size":116,"stargazers_count":653,"open_issues_count":14,"forks_count":101,"subscribers_count":23,"default_branch":"master","last_synced_at":"2025-03-28T21:07:04.179Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://europe.naverlabs.com/Research/Computer-Vision/Learning-Visual-Representations/Deep-Image-Retrieval/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/naver.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-06-19T06:25:31.000Z","updated_at":"2025-02-25T11:28:26.000Z","dependencies_parsed_at":"2022-07-19T18:09:17.732Z","dependency_job_id":null,"html_url":"https://github.com/naver/deep-image-retrieval","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/naver%2Fdeep-image-retrieval","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/naver%2Fdeep-image-retrieval/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/naver%2Fdeep-image-retrieval/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/naver%2Fdeep-image-retrieval/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/naver","download_url":"https://codeload.github.com/naver/deep-image-retrieval/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247256112,"owners_count":20909240,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-01T07:00:27.984Z","updated_at":"2025-04-04T22:07:22.282Z","avatar_url":"https://github.com/naver.png","language":"Python","funding_links":[],"categories":["5. Learning based SLAM","Python"],"sub_categories":["5.3 Deep Features"],"readme":"# Deep Image Retrieval\n\nThis repository contains the models and the evaluation scripts (in Python3 and Pytorch 1.0+) of the papers:\n\n**[1] End-to-end Learning of Deep Visual Representations for Image Retrieval**\nAlbert Gordo, Jon Almazan, Jerome Revaud, Diane Larlus, IJCV 2017 [\\[PDF\\]](https://arxiv.org/abs/1610.07940)\n\n**[2] Learning with Average Precision: Training Image Retrieval with a Listwise Loss**\nJerome Revaud, Jon Almazan, Rafael S. Rezende, Cesar de Souza, ICCV 2019 [\\[PDF\\]](https://arxiv.org/abs/1906.07589)\n\n\nBoth papers tackle the problem of image retrieval and explore different ways to learn deep visual representations for this task. In both cases, a CNN is used to extract a feature map that is aggregated into a compact, fixed-length representation by a global-aggregation layer*. Finally, this representation is first projected using a FC layer, and L2 normalized so images can be efficiently compared with the dot product.\n\n\n![dir_network](https://user-images.githubusercontent.com/228798/59742085-aae19f80-9221-11e9-8063-e5f2528c304a.png)\n\nAll components in this network, including the aggregation layer, are differentiable, which makes it end-to-end trainable for the end task. In [1], a Siamese architecture that combines three streams with a triplet loss was proposed to train this network.  In [2], this work was extended by replacing the triplet loss with a new loss that directly optimizes for Average Precision.\n\n![Losses](https://user-images.githubusercontent.com/228798/59742025-7a9a0100-9221-11e9-9d58-1494716e9071.png)\n\n\\* Originally, [1] used R-MAC pooling [3] as the global-aggregation layer. However, due to its efficiency and better performace we have replaced the R-MAC pooling layer with the Generalized-mean pooling layer (GeM) proposed in [4]. You can find the original implementation of [1] in Caffe following [this link](https://europe.naverlabs.com/Research/Computer-Vision/Learning-Visual-Representations/Deep-Image-Retrieval/).\n\n\n## News\n\n- **(6/9/2019)** AP loss, Tie-aware AP loss, Triplet Margin loss, and Triplet LogExp loss added for reference\n- **(5/9/2019)** Update evaluation and AP numbers for all the benchmarks\n- **(22/7/2019)** Paper **_Learning with Average Precision: Training Image Retrieval with a Listwise Loss_** accepted at ICCV 2019\n\n\n## Pre-requisites\n\nIn order to run this toolbox you will need:\n\n- Python3 (tested with Python 3.7.3)\n- PyTorch (tested with version 1.4)\n- The following packages: numpy, matplotlib, tqdm, scikit-learn\n\nWith conda you can run the following commands:\n\n```\nconda install numpy matplotlib tqdm scikit-learn\nconda install pytorch torchvision -c pytorch\n```\n\n## Installation\n\n```\n# Download the code\ngit clone https://github.com/naver/deep-image-retrieval.git\n\n# Create env variables\ncd deep-image-retrieval\nexport DIR_ROOT=$PWD\nexport DB_ROOT=/PATH/TO/YOUR/DATASETS\n# for example: export DB_ROOT=$PWD/dirtorch/data/datasets\n```\n\n\n## Evaluation\n\n\n### Pre-trained models\n\nThe table below contains the pre-trained models that we provide with this library, together with their mAP performance on some of the most well-know image retrieval benchmakrs: [Oxford5K](http://www.robots.ox.ac.uk/~vgg/data/oxbuildings/), [Paris6K](http://www.robots.ox.ac.uk/~vgg/data/parisbuildings/), and their Revisited versions ([ROxford5K and RParis6K](https://github.com/filipradenovic/revisitop)).\n\n\n| Model | Oxford5K | Paris6K |  ROxford5K (med/hard) | RParis6K (med/hard) |\n|---\t|:-:|:-:|:-:|:-:|\n|  [Resnet101-TL-MAC](https://drive.google.com/file/d/13MUGNwn_CYGZvqDBD8FGD8fVYxThsSDg/view?usp=sharing) |  85.6\t| 90.1 |  63.3 / 35.7 \t|   76.6 / 55.5  |\n|  [Resnet101-TL-GeM](https://drive.google.com/open?id=1vhm1GYvn8T3-1C4SPjPNJOuTU9UxKAG6) | 85.7 | **93.4** | 64.5 / 40.9 |  78.8 / 59.2  |\n|  [Resnet50-AP-GeM](https://drive.google.com/file/d/1oPtE_go9tnsiDLkWjN4NMpKjh-_md1G5/view?usp=sharing) | 87.7 \t| 91.9 |  65.5 / 41.0 | 77.6 / 57.1 |\n|  [Resnet101-AP-GeM](https://drive.google.com/open?id=1UWJGDuHtzaQdFhSMojoYVQjmCXhIwVvy) | **89.1** | **93.0** | **67.1** / **42.3** |  **80.3**/**60.9** |\n|  [Resnet101-AP-GeM-LM18](https://drive.google.com/open?id=1r76NLHtJsH-Ybfda4aLkUIoW3EEsi25I)** |  88.1\t| **93.1** | 66.3 / **42.5**\t|   **80.2** / **60.8**  |\n\n\nThe name of the model encodes the backbone architecture of the network and the loss that has been used to train it (TL for triplet loss and AP for Average Precision loss). All models use **Generalized-mean pooling (GeM)** [3] as the global pooling mechanism, except for the model in the first row that uses MAC [3] \\(i.e. max-pooling), and have been trained on the **Landmarks-clean** [1] dataset (the clean version of the [Landmarks dataset](http://sites.skoltech.ru/compvision/projects/neuralcodes/)) directly **fine-tuning from ImageNet**. These numbers have been obtained using a **single resolution** and applying **whitening** to the output features (which has also been learned on Landmarks-clean). For a detailed explanation of all the hyper-parameters see [1] and [2] for the triplet loss and AP loss models, respectively.\n\n** For the sake of completeness, we have added an extra model, `Resnet101-AP-LM18`, which has been trained on the [Google-Landmarks Dataset](https://www.kaggle.com/google/google-landmarks-dataset), a large dataset consisting of more than 1M images and 15K classes.\n\n### Reproducing the results\n\nThe script `test_dir.py` can be used to evaluate the pre-trained models provided and to reproduce the results above:\n\n```\npython -m dirtorch.test_dir --dataset DATASET --checkpoint PATH_TO_MODEL \\\n\t\t[--whiten DATASET] [--whitenp POWER] [--aqe ALPHA-QEXP] \\\n\t\t[--trfs TRANSFORMS] [--gpu ID] [...]\n```\n\n- `--dataset`: selects the dataset (eg.: Oxford5K, Paris6K, ROxford5K, RParis6K) [**required**]\n- `--checkpoint`: path to the model weights [**required**]\n- `--whiten`: applies whitening to the output features [default 'Landmarks_clean']\n- `--whitenp`: whitening power [default: 0.25]\n- `--aqe`: alpha-query expansion parameters [default: None]\n- `--trfs`: input image transformations (can be used to apply multi-scale) [default: None]\n- `--gpu`: selects the GPU ID (-1 selects the CPU)\n\nFor example, to reproduce the results of the Resnet101-AP_loss model on the RParis6K dataset download the model `Resnet-101-AP-GeM.pt` from [here](https://drive.google.com/open?id=1mi50tG6oXY1eE9yJnmGCPdTmlIjG7mr0) and run:\n\n```\ncd $DIR_ROOT\nexport DB_ROOT=/PATH/TO/YOUR/DATASETS\n\npython -m dirtorch.test_dir --dataset RParis6K \\\n\t\t--checkpoint dirtorch/data/Resnet101-AP-GeM.pt \\\n\t\t--whiten Landmarks_clean --whitenp 0.25 --gpu 0\n```\n\nAnd you should see the following output:\n\n```\n\u003e\u003e Evaluation...\n * mAP-easy = 0.907568\n * mAP-medium = 0.803098\n * mAP-hard = 0.608556\n```\n\n**Note:** this script integrates an automatic downloader for the Oxford5K, Paris6K, ROxford5K, and RParis6K datasets (kudos to Filip Radenovic ;)). The datasets will be saved in `$DB_ROOT`.\n\n## Feature extractor\n\nYou can also use the pre-trained models to extract features from your own datasets or collection of images. For that we provide the script `feature_extractor.py`:\n\n```\npython -m dirtorch.extract_features --dataset DATASET --checkpoint PATH_TO_MODEL \\\n\t\t--output PATH_TO_FILE [--whiten DATASET] [--whitenp POWER] \\\n\t\t[--trfs TRANSFORMS] [--gpu ID] [...]\n```\n\nwhere `--output` is used to specify the destination where the features will be saved. The rest of the parameters are the same as seen above.\n\nFor example, this is how the script can be used to extract a feature representation for each one of the images in the RParis6K dataset using the `Resnet-101-AP-GeM.pt` model, and storing them in `rparis6k_features.npy`:\n\n```\ncd $DIR_ROOT\nexport DB_ROOT=/PATH/TO/YOUR/DATASETS\n\npython -m dirtorch.extract_features --dataset RParis6K \\\n\t\t--checkpoint dirtorch/data/Resnet101-AP-GeM.pt \\\n\t\t--output rparis6k_features.npy \\\n\t\t--whiten Landmarks_clean --whitenp 0.25 --gpu 0\n```\n\nThe library also provides a **generic class dataset** (`ImageList`) that allows you to specify the list of images by providing a simple text file.\n\n```\n--dataset 'ImageList(\"PATH_TO_TEXTFILE\" [, \"IMAGES_ROOT\"])'\n```\n\nEach row of the text file should contain a single path to a given image:\n\n```\n/PATH/TO/YOUR/DATASET/images/image1.jpg\n/PATH/TO/YOUR/DATASET/images/image2.jpg\n/PATH/TO/YOUR/DATASET/images/image3.jpg\n/PATH/TO/YOUR/DATASET/images/image4.jpg\n/PATH/TO/YOUR/DATASET/images/image5.jpg\n```\n\nAlternatively, you can also use relative paths, and use `IMAGES_ROOT` to specify the root folder.\n\n## Feature extraction with kapture datasets\n\nKapture is a pivot file format, based on text and binary files, used to describe SFM (Structure From Motion) and more generally sensor-acquired data.\n\nIt is available at https://github.com/naver/kapture.\nIt contains conversion tools for popular formats and several popular datasets are directly available in kapture.\n\nIt can be installed with:\n```bash\npip install kapture\n```\n\nDatasets can be downloaded with:\n```bash\nkapture_download_dataset.py update\nkapture_download_dataset.py list\n# e.g.: install mapping and query of Extended-CMU-Seasons_slice22\nkapture_download_dataset.py install \"Extended-CMU-Seasons_slice22_*\"\n```\nIf you want to convert your own dataset into kapture, please find some examples [here](https://github.com/naver/kapture/blob/master/doc/datasets.adoc).\n\nOnce installed, you can extract global features for your kapture dataset with:\n```bash\ncd $DIR_ROOT\npython -m dirtorch.extract_kapture --kapture-root pathto/yourkapturedataset --checkpoint dirtorch/data/Resnet101-AP-GeM-LM18.pt --gpu 0\n```\n\nRun `python -m dirtorch.extract_kapture --help` for more information on the extraction parameters. \n\n## Citations\n\nPlease consider citing the following papers in your publications if this helps your research.\n\n```\n@article{GARL17,\n title = {End-to-end Learning of Deep Visual Representations for Image Retrieval},\n author = {Gordo, A. and Almazan, J. and Revaud, J. and Larlus, D.}\n journal = {IJCV},\n year = {2017}\n}\n\n@inproceedings{RARS19,\n title = {Learning with Average Precision: Training Image Retrieval with a Listwise Loss},\n author = {Revaud, J. and Almazan, J. and Rezende, R.S. and de Souza, C.R.}\n booktitle = {ICCV},\n year = {2019}\n}\n```\n\n## Contributors\n\nThis library has been developed by Jerome Revaud, Rafael de Rezende, Cesar de Souza, Diane Larlus, and Jon Almazan at **[Naver Labs Europe](https://europe.naverlabs.com)**.\n\n\n**Special thanks to [Filip Radenovic](https://github.com/filipradenovic).** In this library, we have used the ROxford5K and RParis6K downloader from his awesome **[CNN-imageretrieval repository](https://github.com/filipradenovic/cnnimageretrieval-pytorch)**. Consider checking it out if you want to train your own models for image retrieval!\n\n## References\n\n[1] Gordo, A., Almazan, J., Revaud, J., Larlus, D., [End-to-end Learning of Deep Visual Representations for Image Retrieval](https://arxiv.org/abs/1610.07940). IJCV 2017\n\n[2] Revaud, J., Almazan, J., Rezende, R.S., de Souza, C., [Learning with Average Precision: Training Image Retrieval with a Listwise Loss](https://arxiv.org/abs/1906.07589). ICCV 2019\n\n[3] Tolias, G., Sicre, R., Jegou, H., [Particular object retrieval with integral max-pooling of CNN activations](https://arxiv.org/abs/1511.05879). ICLR 2016\n\n[4] Radenovic, F., Tolias, G., Chum, O., [Fine-tuning CNN Image Retrieval with No Human Annotation](https://arxiv.org/pdf/1711.02512). TPAMI 2018\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnaver%2Fdeep-image-retrieval","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnaver%2Fdeep-image-retrieval","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnaver%2Fdeep-image-retrieval/lists"}