{"id":18317357,"url":"https://github.com/compvis/characterizing_generalization_in_dml","last_synced_at":"2026-03-04T17:33:10.074Z","repository":{"id":65983440,"uuid":"431521367","full_name":"CompVis/Characterizing_Generalization_in_DML","owner":"CompVis","description":null,"archived":false,"fork":false,"pushed_at":"2021-11-24T15:33:41.000Z","size":9759,"stargazers_count":5,"open_issues_count":0,"forks_count":1,"subscribers_count":6,"default_branch":"main","last_synced_at":"2025-09-10T05:24:25.645Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/CompVis.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-11-24T14:46:06.000Z","updated_at":"2025-05-27T03:51:06.000Z","dependencies_parsed_at":"2023-02-19T19:15:50.060Z","dependency_job_id":null,"html_url":"https://github.com/CompVis/Characterizing_Generalization_in_DML","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/CompVis/Characterizing_Generalization_in_DML","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CompVis%2FCharacterizing_Generalization_in_DML","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CompVis%2FCharacterizing_Generalization_in_DML/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CompVis%2FCharacterizing_Generalization_in_DML/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CompVis%2FCharacterizing_Generalization_in_DML/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/CompVis","download_url":"https://codeload.github.com/CompVis/Characterizing_Generalization_in_DML/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CompVis%2FCharacterizing_Generalization_in_DML/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30087400,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-04T15:40:14.053Z","status":"ssl_error","status_checked_at":"2026-03-04T15:40:13.655Z","response_time":59,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-05T18:05:54.046Z","updated_at":"2026-03-04T17:33:10.056Z","avatar_url":"https://github.com/CompVis.png","language":"Python","readme":"# Characterizing Generalization under Out-of-Distribution shifts in Deep Metric Learning\n\n__Links:__ [[Paper]](https://arxiv.org/abs/2107.09562v1)\n\n---\n\n## Description\nThis repository contains code and implementations used in our Generalization Study on Deep Metric Learning under OOD Shifts.\nThe underlying DML pipeline is adapted from [this repository](https://github.com/Confusezius/Revisiting_Deep_Metric_Learning_PyTorch), and introduces the following novel key elements from our paper:\n* It introduces ___ooDML___, a DML benchmark comprising new and progressively harder data distribution shifts based on existing DML benchmarks (see image: black line is default split 'ood-ness', blue the possible split progression generated. Red denotes even harder splits generated with class removal). We also provide `create_dataset_splits.py` to apply this process to potentially other benchmarks.\n\n\u003cimg src=\"images/progression.png\"\n     alt=\"Markdown Monster icon\"\n     width=800px\n     style=\"float: bottom;margin-left: 5%\" /\u003e\n\n* ___ooDML___ allows us to evaluate how conceptual differences hold up in more realistic and especially more challenging train-to-test distribution shifts. The change in 'Out-of-Distribution-ness' is visually highlighted when looking at each benchmark UMAP for different splits:\n\n\u003cimg src=\"images/umaps.png\"\n     alt=\"Markdown Monster icon\"\n     height=300px\n     style=\"float: bottom;margin-left: 5%\" /\u003e\n\n* Different methods and approaches to Deep Metric Learning can then simply be evaluated against a progressively more out-of-distribution train-to-test split:\n\n\u003cimg src=\"images/progression_comp.png\"\n     alt=\"Markdown Monster icon\"\n     height=400px\n     style=\"float: bottom;margin-left: 5%\" /\u003e\n\n* It also introduces few-shot DML as a cost-effective approach to help with retrieval under especially hard OOD shifts.\n\n\u003cimg src=\"images/fewshot.png\"\n     alt=\"Markdown Monster icon\"\n     height=300px\n     style=\"float: bottom;margin-left: 5%\" /\u003e\n\n* It provides the respective scripts to run a large and diverse set, included more recent objectives and extension to DML on our proposed benchmark, ___ooDML___.\n\nIf you wish to use the ___ooDML___ splits with your own codebase, simply copy the corresponding pickle-files from `datasplits`. The internal structure of the pickled dicts is explained below!\n\n___Expect further updates in the next weeks!___  \n\n\n---\n\n## Quick Usage\n* Example runs as used to produce key results in our paper are made available in `OOD-Study_ZeroShot.sh` for zero-shot generalization experiments and `OOD-Study_FewShot.sh` for few-shot generalization experiments.\n* Note that for readability, runs with two different seeds are shown, but for conclusive experiments, more should be used.\n* In addition, results may vary a bit, as the results published in the original paper were computed over multiple different hardware settings.\n* To download the benchmark data (this repository only provides the splits as lists of train and test classes to use), simply use the download links or guides provided [here](https://github.com/Confusezius/Revisiting_Deep_Metric_Learning_PyTorch). The data structure is assumed to be the same for this repository.\n\n---\n\n## Requirements:\n\n* PyTorch 1.2.0+ \u0026 Faiss-Gpu\n* Python 3.6+\n* pretrainedmodels, timm, torchvision 0.3.0+\n\nAn exemplary setup of a virtual environment containing everything needed:\n```\n(1) wget  https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh\n(2) bash Miniconda3-latest-Linux-x86_64.sh (say yes to append path to bashrc)\n(3) source .bashrc\n(4) conda create -n DL python=3.6\n(5) conda activate DL\n(6) conda install matplotlib scipy scikit-learn scikit-image tqdm pandas pillow\n(7) conda install pytorch torchvision faiss-gpu cudatoolkit=10.0 -c pytorch\n(8) pip install wandb pretrainedmodels\n(9) Run the scripts!\n```\n\n---\n\n## The _ooDML_ benchmark\nInformation about the utilised split progressions is fully available in the `.pkl`-files located in the `datasplits` directory.\nFor each base benchmark, the respective `.pkl`-file contains a dictionary with the following structure:\n```\nSplit_ID: 1, 2, ..., 8/9 (depending on benchmark)\n└─── train: list of training classes.\n|       └─── String of classname\n|       ...\n└─── test: List of test classes.\n|       └─── String of classname\n|       ...\n└─── fid: FID score (using R50) for given split.\n└─── test_episodes: Episode data for few-shot evaluation of DML, given per Shot-setup.\n|       └─── 2\n|       |    └─── Episode-ID: 1, ..., 10\n|       |    |       └─── classname\n|       |    |       |       └─── Support samples to use for few-shot adaptation. The complement will be used to generate the query data.\n|       |    |       |       ...\n|       └─── 5 (see 2-Shot setting)\n|       ...\n|       └─── 10 (not used in the paper)\n|       ...\n└─── split_train, split_val: Train/val splits for hyperparameter tuning before running on final test set. Also used in few-shot experiments to provide default train/validation splits.\n|       ...\n```\n\n---\n\n## Detailed Usage\nThere are four key scripts in this repository:\n* `ood_main.py` and `ood_diva_main.py` to run zero-shot ___ooDML___-experiments on all DML criteria.\n* `fewshot_ood_main.py` and `fewshot_ood_diva_main.py` to run fewshot-shot ___ooDML___-experiments on all DML criteria.\n\nHere are exemplary uses for each script:\n\n\n1.\n```\npython ood_main.py --checkpoint --data_hardness $split_id --kernels 6 --source $datapath --n_epochs 200 --log_online --project DML_OOD-Shift_Study --group CUB_ID-1_Margin_b12_Distance --seed 0 --gpu $gpu --bs 112 --samples_per_class 2 --loss margin --batch_mining distance --arch resnet50_frozen_normalize --embed_dim 512\n```\n\n2.\n```\npython ood_diva_main.py --checkpoint --data_hardness $split_id --kernels 6 --source $datapath --n_epochs 200 --log_online --project DML_OOD-Shift_Study --group CUB_ID-1_DiVA --seed 0 --gpu $gpu --bs 108 --samples_per_class 2 --loss margin --batch_mining distance --diva_rho_decorrelation 1500 1500 1500 --diva_alpha_ssl 0.3 --diva_alpha_intra 0.3 --diva_alpha_shared 0.3 --arch resnet50_frozen_normalize --embed_dim 128\n```\n\n3.\n```\npython fewshot_ood_main.py --dataset cars196 --finetune_criterion margin --finetune_shots 2 --finetune_lr_multi 10 --finetune_iter 1000 --finetune_only_last --checkpoint --data_hardness -20 --kernels 6 --source $datapath --n_epochs 200 --log_online --project DML_OOD-Shift_FewShot_Study --group CAR_Shots-2_ID-1_Multisimilarity --seed 0 --gpu $gpu --bs 112 --samples_per_class 2 --loss multisimilarity --arch resnet50_frozen_normalize --embed_dim 512\n```\n\n4.\n```\npython fewshot_ood_main.py --dataset cars196 --finetune_criterion margin --finetune_shots 2 --finetune_lr_multi 10 --finetune_iter 1000 --finetune_only_last --checkpoint --data_hardness -20 --kernels 6 --source $datapath --n_epochs 200 --log_online --project DML_OOD-Shift_FewShot_Study --group CAR_Shots-2_ID-1_Multisimilarity --seed 0 --gpu $gpu --bs 112 --samples_per_class 2 --loss multisimilarity --arch resnet50_frozen_normalize --embed_dim 512\n```\n\nAn explanation for all utilised flags is provided in the respective help-string in `parameters.py`.\n\nAs noted previously, the data is assumed to have the following structure:\n\n* For CUB200-2011/CARS196:\n```\ncub200/cars196\n└───images\n|    └───001.Black_footed_Albatross\n|           │   Black_Footed_Albatross_0001_796111\n|           │   ...\n|    ...\n```\n\n* For Stanford Online Products:\n```\nonline_products\n└───images\n|    └───bicycle_final\n|           │   111085122871_0.jpg\n|    ...\n|\n└───Info_Files\n|    │   bicycle.txt\n|    │   ...\n```\n\nAssuming your folder is placed in e.g. `\u003c$datapath/cub200\u003e`, pass `$datapath` as input to `--source`.\n\n---\n\n## Citation\nIf you use the code in this repository, insights and/or the ooDML benchmark, we kindly ask that you cite\n```\n@inproceedings{milbich2021characterizing,\n      title={Characterizing Generalization under Out-Of-Distribution Shifts in Deep Metric Learning},\n      author={Timo Milbich and Karsten Roth and Samarth Sinha and Ludwig Schmidt and Marzyeh Ghassemi and Björn Ommer},\n      year={2021},\n      booktitle={Thirty-fifth Conference on Neural Information Processing Systems (NeurIPS)}\n}\n```\nThanks! :)\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcompvis%2Fcharacterizing_generalization_in_dml","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcompvis%2Fcharacterizing_generalization_in_dml","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcompvis%2Fcharacterizing_generalization_in_dml/lists"}