{"id":13738457,"url":"https://github.com/Confusezius/Revisiting_Deep_Metric_Learning_PyTorch","last_synced_at":"2025-05-08T16:34:05.860Z","repository":{"id":41870093,"uuid":"262594770","full_name":"Confusezius/Revisiting_Deep_Metric_Learning_PyTorch","owner":"Confusezius","description":"(ICML 2020) This repo contains code for our paper \"Revisiting Training Strategies and Generalization Performance in Deep Metric Learning\" (https://arxiv.org/abs/2002.08473) to facilitate consistent research in the field of Deep Metric Learning.","archived":false,"fork":false,"pushed_at":"2022-04-25T16:12:32.000Z","size":837,"stargazers_count":342,"open_issues_count":2,"forks_count":51,"subscribers_count":8,"default_branch":"master","last_synced_at":"2024-08-04T03:12:35.690Z","etag":null,"topics":["cars196","cub200-2011","deep-learning","deep-metric-learning","icml2020","image-retrieval","metric-learning","stanford-online-products","wandb"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Confusezius.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-05-09T14:59:01.000Z","updated_at":"2024-07-17T03:25:51.000Z","dependencies_parsed_at":"2022-08-11T19:50:18.141Z","dependency_job_id":null,"html_url":"https://github.com/Confusezius/Revisiting_Deep_Metric_Learning_PyTorch","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Confusezius%2FRevisiting_Deep_Metric_Learning_PyTorch","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Confusezius%2FRevisiting_Deep_Metric_Learning_PyTorch/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Confusezius%2FRevisiting_Deep_Metric_Learning_PyTorch/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Confusezius%2FRevisiting_Deep_Metric_Learning_PyTorch/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Confusezius","download_url":"https://codeload.github.com/Confusezius/Revisiting_Deep_Metric_Learning_PyTorch/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224746755,"owners_count":17363107,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cars196","cub200-2011","deep-learning","deep-metric-learning","icml2020","image-retrieval","metric-learning","stanford-online-products","wandb"],"created_at":"2024-08-03T03:02:23.049Z","updated_at":"2024-11-15T07:31:10.243Z","avatar_url":"https://github.com/Confusezius.png","language":"Python","readme":"# Deep Metric Learning Research in PyTorch\n\n---\n## What can I find here?\n\nThis repository contains all code and implementations used in:\n\n```\nRevisiting Training Strategies and Generalization Performance in Deep Metric Learning\n```\n\naccepted to **ICML 2020**.\n\n**Link**: https://arxiv.org/abs/2002.08473\n\nThe code is meant to serve as a research starting point in Deep Metric Learning.\nBy implementing key baselines under a consistent setting and logging a vast set of metrics, it should be easier to ensure that method gains are not due to implementational variations, while better understanding driving factors.\n\nIt is set up in a modular way to allow for fast and detailed prototyping, but with key elements written in a way that allows the code to be directly copied into other pipelines. In addition, multiple training and test metrics are logged in W\u0026B to allow for easy and large-scale evaluation.\n\nFinally, please find a public W\u0026B repo with key runs performed in the paper here: https://app.wandb.ai/confusezius/RevisitDML.\n\n**Contact**: Karsten Roth, karsten.rh1@gmail.com  \n\n*Suggestions are always welcome!*\n\n---\n## Some Notes:\n\nIf you use this code in your research, please cite\n```\n@misc{roth2020revisiting,\n    title={Revisiting Training Strategies and Generalization Performance in Deep Metric Learning},\n    author={Karsten Roth and Timo Milbich and Samarth Sinha and Prateek Gupta and Björn Ommer and Joseph Paul Cohen},\n    year={2020},\n    eprint={2002.08473},\n    archivePrefix={arXiv},\n    primaryClass={cs.CV}\n}\n```\n\nThis repository contains (in parts) code that has been adapted from:\n* https://github.com/idstcv/SoftTriple\n* https://github.com/bnu-wangxun/Deep_Metric\n* https://github.com/valerystrizh/pytorch-histogram-loss\n* https://github.com/Confusezius/Deep-Metric-Learning-Baselines\n\nMake sure to also check out the following repo with a great plug-and-play implementation of DML methods:\n* https://github.com/KevinMusgrave/pytorch-metric-learning\n\n---\n\n**[All implemented methods and metrics are listed at the bottom!](#-implemented-methods)**\n\n---\n\n## Paper-related Information\n\n#### Reproduce results from our paper **[Revisiting Training Strategies and Generalization Performance in Deep Metric Learning](https://arxiv.org/pdf/2002.08473.pdf)**\n\n* *ALL* standardized Runs that were used are available in `Revisit_Runs.sh`.\n* These runs are also logged in this public W\u0026B repo: https://app.wandb.ai/confusezius/RevisitDML.\n* All Runs and their respective metrics can be downloaded and evaluated to generate the plots in our paper by following `Result_Evaluations.py`. This also allows for potential introspection of other relations. It also converts results directly into Latex-table format with mean and standard deviations.\n* To utilize different batch-creation methods, simply set the flag `--data_sampler` to the method of choice. Allowed flags are listed in `datasampler/__init__.py`.\n* To use the proposed spectral regularization for tuple-based methods, set `--batch_mining rho_distance` with flip probability `--miner_rho_distance_cp e.g. 0.2`.\n* A script to run the toy experiments in the paper is provided in `toy_experiments`.\n\n**Note**: There may be small deviations in results based on the Hardware (e.g. between P100 and RTX GPUs) and Software (different PyTorch/Cuda versions) used to run these experiments, but they should be covered in the standard deviations reported in the paper.\n\n---\n\n## How to use this Repo\n\n### Requirements:\n\n* PyTorch 1.2.0+ \u0026 Faiss-Gpu\n* Python 3.6+\n* pretrainedmodels, torchvision 0.3.0+\n\nAn exemplary setup of a virtual environment containing everything needed:\n```\n(1) wget  https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh\n(2) bash Miniconda3-latest-Linux-x86_64.sh (say yes to append path to bashrc)\n(3) source .bashrc\n(4) conda create -n DL python=3.6\n(5) conda activate DL\n(6) conda install matplotlib scipy scikit-learn scikit-image tqdm pandas pillow\n(7) conda install pytorch torchvision faiss-gpu cudatoolkit=10.0 -c pytorch\n(8) pip install wandb pretrainedmodels\n(9) Run the scripts!\n```\n\n### Datasets:\nData for\n* CUB200-2011 (http://www.vision.caltech.edu/visipedia/CUB-200.html)\n* CARS196 (https://ai.stanford.edu/~jkrause/cars/car_dataset.html)\n* Stanford Online Products (http://cvgl.stanford.edu/projects/lifted_struct/)\n\ncan be downloaded either from the respective project sites or directly via Dropbox:\n\n* CUB200-2011 (1.08 GB): https://www.dropbox.com/s/tjhf7fbxw5f9u0q/cub200.tar?dl=0\n* CARS196 (1.86 GB): https://www.dropbox.com/s/zi2o92hzqekbmef/cars196.tar?dl=0\n* SOP (2.84 GB): https://www.dropbox.com/s/fu8dgxulf10hns9/online_products.tar?dl=0\n\n**The latter ensures that the folder structure is already consistent with this pipeline and the dataloaders**.   \n\nOtherwise, please make sure that the datasets have the following internal structure:\n\n* For CUB200-2011/CARS196:\n```\ncub200/cars196\n└───images\n|    └───001.Black_footed_Albatross\n|           │   Black_Footed_Albatross_0001_796111\n|           │   ...\n|    ...\n```\n\n* For Stanford Online Products:\n```\nonline_products\n└───images\n|    └───bicycle_final\n|           │   111085122871_0.jpg\n|    ...\n|\n└───Info_Files\n|    │   bicycle.txt\n|    │   ...\n```\n\nAssuming your folder is placed in e.g. `\u003c$datapath/cub200\u003e`, pass `$datapath` as input to `--source`.\n\n### Training:\nTraining is done by using `main.py` and setting the respective flags, all of which are listed and explained in `parameters.py`. A vast set of exemplary runs is provided in `Revisit_Runs.sh`.\n\n**[I.]** **A basic sample run using default parameters would like this**:\n\n```\npython main.py --loss margin --batch_mining distance --log_online \\\n              --project DML_Project --group Margin_with_Distance --seed 0 \\\n              --gpu 0 --bs 112 --data_sampler class_random --samples_per_class 2 \\\n              --arch resnet50_frozen_normalize --source $datapath --n_epochs 150 \\\n              --lr 0.00001 --embed_dim 128 --evaluate_on_gpu\n```\n\nThe purpose of each flag explained:\n\n* `--loss \u003closs_name\u003e`: Name of the training objective used. See folder `criteria` for implementations of these methods.\n* `--batch_mining \u003cbatchminer_name\u003e`: Name of the batch-miner to use (for tuple-based ranking methods). See folder `batch_mining` for implementations of these methods.\n* `--log_online`: Log metrics online via either W\u0026B (Default) or CometML. Regardless, plots, weights and parameters are all stored offline as well.\n*  `--project`, `--group`: Project name as well as name of the run. Different seeds will be logged into the same `--group` online. The group as well as the used seed also define the local savename.\n* `--seed`, `--gpu`, `--source`: Basic Parameters setting the training seed, the used GPU and the path to the parent folder containing the respective Datasets.\n* `--arch`: The utilized backbone, e.g. ResNet50. You can append `_frozen` and `_normalize` to the name to ensure that BatchNorm layers are frozen and embeddings are normalized, respectively.\n* `--data_sampler`, `--samples_per_class`: How to construct a batch. The default method, `class_random`, selects classes at random and places `\u003csamples_per_class\u003e` samples into the batch until the batch is filled.\n* `--lr`, `--n_epochs`, `--bs` ,`--embed_dim`: Learning rate, number of training epochs, the batchsize and the embedding dimensionality.  \n* `--evaluate_on_gpu`: If set, all metrics are computed using the gpu - requires Faiss-GPU and may need additional GPU memory.\n\n#### Some Notes:\n* During training, metrics listed in `--evaluation_metrics` will be logged for both training and validation/test set. If you do not care about detailed training metric logging, simply set the flag `--no_train_metrics`. A checkpoint is saved for improvements in metrics listed in `--storage_metrics` on training, validation or test sets. Detailed information regarding the available metrics can be found at the bottom of this `README`.\n* If one wishes to use a training/validation split, simply set `--use_tv_split` and `--tv_split_perc \u003ctrain/val split percentage\u003e`.\n\n\n**[II.]** **Advanced Runs**:\n\n```\npython main.py --loss margin --batch_mining distance --loss_margin_beta 0.6 --miner_distance_lower_cutoff 0.5 ... (basic parameters)\n```\n\n* To use specific parameters that are loss, batchminer or e.g. datasampler-related, simply set the respective flag.\n* For structure and ease of use, parameters relating to a specifc loss function/batchminer etc. are marked as e.g. `--loss_\u003clossname\u003e_\u003cparameter_name\u003e`, see `parameters.py`.\n* However, every parameter can be called from every class, as all parameters are stored in a shared namespace that is passed to all methods. This makes it easy to create novel fusion losses and the likes.\n\n\n### Evaluating Results with W\u0026B\nHere some information on using W\u0026B (highly encouraged!)\n\n* Create an account here (free): https://wandb.ai\n* After the account is set, make sure to include your API key in `parameters.py` under `--wandb_key`.\n* To make sure that W\u0026B data can be stored, ensure to run `wandb on` in the folder pointed to by `--save_path`.\n* When data is logged online to W\u0026B, one can use `Result_Evaluations.py` to download all data, create named metric and correlation plots and output a summary in the form of a latex-ready table with mean and standard deviations of all metrics. **This ensures that there are no errors between computed and reported results.**\n\n\n### Creating custom methods:\n\n1. **Create custom objectives**: Simply take a look at e.g. `criteria/margin.py`, and ensure that the used methods has the following properties:\n  * Inherit from `torch.nn.Module` and define a custom `forward()` function.\n  * When using trainable parameters, make sure to either provide a `self.lr` to set the learning rate of the loss-specific parameters, or set `self.optim_dict_list`, which is a list containing optimization dictionaries passed to the optimizer (see e.g `criteria/proxynca.py`). If both are set, `self.optim_dict_list` has priority.\n  * Depending on the loss, remember to set the variables `ALLOWED_MINING_OPS  = None or list of allowed mining operations`, `REQUIRES_BATCHMINER = False or True`, `REQUIRES_OPTIM = False or True` to denote if the method needs a batchminer or optimization of internal parameters.\n\n\n2. **Create custom batchminer**: Simply take a look at e.g. `batch_mining/distance.py` - The miner needs to be a class with a defined `__call__()`-function, taking in a batch and labels and returning e.g. a list of triplets.\n\n3. **Create custom datasamplers**:Simply take a look at e.g. `datasampler/class_random_sampler.py`. The sampler needs to inherit from `torch.utils.data.sampler.Sampler` and has to provide a `__iter__()` and a `__len__()` function. It has to yield a set of indices that are used to create the batch.\n\n\n---\n\n# Implemented Methods\n\nFor a detailed explanation of everything, please refer to the supplementary of our paper!\n\n### DML criteria\n\n* **Angular** [[Deep Metric Learning with Angular Loss](https://arxiv.org/pdf/1708.01682.pdf)] `--loss angular`\n* **ArcFace** [[ArcFace: Additive Angular Margin Loss for Deep Face Recognition](https://arxiv.org/pdf/1801.07698.pdf)] `--loss arcface`\n* **Contrastive** [[Dimensionality Reduction by Learning an Invariant Mapping](http://yann.lecun.com/exdb/publis/pdf/hadsell-chopra-lecun-06.pdf)] `--loss contrastive`\n* **Generalized Lifted Structure** [[In Defense of the Triplet Loss for Person Re-Identification](https://arxiv.org/abs/1703.07737)] `--loss lifted`\n* **Histogram** [[Learning Deep Embeddings with Histogram Loss](https://arxiv.org/pdf/1611.00822.pdf)] `--loss histogram`\n* **Marginloss** [[Sampling Matters in Deep Embeddings Learning](https://arxiv.org/abs/1706.07567)] `--loss margin`\n* **MultiSimilarity** [[Multi-Similarity Loss with General Pair Weighting for Deep Metric Learning](https://arxiv.org/abs/1904.06627)] `--loss multisimilarity`\n* **N-Pair** [[Improved Deep Metric Learning with Multi-class N-pair Loss Objective](https://papers.nips.cc/paper/6200-improved-deep-metric-learning-with-multi-class-n-pair-loss-objective)] `--loss npair`\n* **ProxyNCA** [[No Fuss Distance Metric Learning using Proxies](https://arxiv.org/pdf/1703.07464.pdf)] `--loss proxynca`\n* **Quadruplet** [[Beyond triplet loss: a deep quadruplet network for person re-identification](https://arxiv.org/abs/1704.01719)] `--loss quadruplet`\n* **Signal-to-Noise Ratio (SNR)** [[Signal-to-Noise Ratio: A Robust Distance Metric for Deep Metric Learning](https://arxiv.org/pdf/1904.02616.pdf)] `--loss snr`\n* **SoftTriple** [[SoftTriple Loss: Deep Metric Learning Without Triplet Sampling](https://arxiv.org/abs/1909.05235)] `--loss softtriplet`\n* **Normalized Softmax** [[Classification is a Strong Baseline for Deep Metric Learning](https://arxiv.org/abs/1811.12649)] `--loss softmax`\n* **Triplet** [[Facenet: A unified embedding for face recognition and clustering](https://arxiv.org/abs/1503.03832)] `--loss triplet`\n\n### DML batchminer\n\n* **Random** [[Facenet: A unified embedding for face recognition and clustering](https://arxiv.org/abs/1503.03832)] `--batch_mining random`\n* **Semihard** [[Facenet: A unified embedding for face recognition and clustering](https://arxiv.org/abs/1503.03832)] `--batch_mining semihard`\n* **Softhard** [https://github.com/Confusezius/Deep-Metric-Learning-Baselines] `--batch_mining softhard`\n* **Distance-based** [[Sampling Matters in Deep Embeddings Learning](https://arxiv.org/abs/1706.07567)] `--batch_mining distance`\n* **Rho-Distance** [[Revisiting Training Strategies and Generalization Performance in Deep Metric Learning](https://arxiv.org/abs/2002.08473)] `--batch_mining rho_distance`\n* **Parametric** [[PADS: Policy-Adapted Sampling for Visual Similarity Learning](https://arxiv.org/abs/2003.11113)] `--batch_mining parametric`\n\n### Architectures\n\n* **ResNet50** [[Deep Residual Learning for Image Recognition](https://arxiv.org/abs/1512.03385)] e.g. `--arch resnet50_frozen_normalize`.\n* **Inception-BN** [[Going Deeper with Convolutions](https://arxiv.org/abs/1409.4842)] e.g. `--arch bninception_normalize_frozen`.\n* **GoogLeNet** (torchvision variant w/ BN) [[Going Deeper with Convolutions](https://arxiv.org/abs/1409.4842)] e.g. `--arch googlenet`.\n\n### Datasets\n* **CUB200-2011** [[Caltech-UCSD Birds-200-2011](http://www.vision.caltech.edu/visipedia/CUB-200-2011.html)]  `--dataset cub200`.\n* **CARS196** [[Cars Dataset](https://ai.stanford.edu/~jkrause/cars/car_dataset.html)] `--dataset cars196`.\n* **Stanford Online Products** [[Deep Metric Learning via Lifted Structured Feature Embedding](https://cvgl.stanford.edu/projects/lifted_struct/)] `--dataset online_products`.\n\n\n\n### Evaluation Metrics\n**Metrics based on Euclidean Distances**\n* **Recall@k**: Include R@1 e.g. with `e_recall@1` into the list of evaluation metrics `--evaluation_metrics`.\n* **Normalized Mutual Information (NMI)**: Include with `nmi`.\n* **F1**: include with `f1`.\n* **mAP (class-averaged)**: Include standard mAP at Recall with `mAP_lim`. You may also include `mAP_1000` for mAP limited to Recall@1000, and `mAP_c` limited to mAP at Recall@Max_Num_Samples_Per_Class. Note that all of these are heavily correlated.\n\n**Metrics based on Cosine Similarities** *(not included by default)*\n* **Cosine Recall@k**: Cosine-Similarity variant of Recall@k. Include with `c_recall@k` in `--evaluation_metrics`.\n* **Cosine Normalized Mutual Information (NMI)**: Include with `c_nmi`.\n* **Cosine F1**: include with `c_f1`.\n* **Cosine mAP (class-averaged)**: Include cosine similarity mAP at Recall variants with `c_mAP_lim`. You may also include `c_mAP_1000` for mAP limited to Recall@1000, and `c_mAP_c` limited to mAP at Recall@Max_Num_Samples_Per_Class.\n\n**Embedding Space Metrics**\n* **Spectral Variance**: This metric refers to the spectral decay metric used in our ICML paper. Include it with `rho_spectrum@1`. To exclude the `k` largest spectral values for a more robust estimate, simply include `rho_spectrum@k+1`. Adding `rho_spectrum@0` logs the whole singular value distribution, and `rho_spectrum@-1` computes KL(q,p) instead of KL(p,q).\n* **Mean Intraclass Distance**: Include the mean intraclass distance via `dists@intra`.\n* **Mean Interclass Distance**: Include the mean interlcass distance via `dists@inter`.\n* **Ratio Intra- to Interclass Distance**: Include the ratio of distances via `dists@intra_over_inter`.\n","funding_links":[],"categories":["Python"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FConfusezius%2FRevisiting_Deep_Metric_Learning_PyTorch","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FConfusezius%2FRevisiting_Deep_Metric_Learning_PyTorch","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FConfusezius%2FRevisiting_Deep_Metric_Learning_PyTorch/lists"}