{"id":24111259,"url":"https://github.com/oml-team/open-metric-learning","last_synced_at":"2025-04-14T19:56:17.123Z","repository":{"id":44150808,"uuid":"499830255","full_name":"OML-Team/open-metric-learning","owner":"OML-Team","description":"Metric learning and retrieval pipelines, models and zoo.","archived":false,"fork":false,"pushed_at":"2025-04-12T13:27:14.000Z","size":2635,"stargazers_count":920,"open_issues_count":31,"forks_count":68,"subscribers_count":11,"default_branch":"main","last_synced_at":"2025-04-12T14:38:36.656Z","etag":null,"topics":["computer-vision","data-science","deep-learning","hacktoberfest","hacktoberfest-2023","hacktoberfest2023","metric-learning","pytorch","pytorch-lightning","representation-learning","similarity-learning"],"latest_commit_sha":null,"homepage":"https://open-metric-learning.readthedocs.io/en/latest/index.html","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/OML-Team.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-06-04T13:12:25.000Z","updated_at":"2025-04-12T13:27:14.000Z","dependencies_parsed_at":"2023-10-15T21:08:38.986Z","dependency_job_id":"de5dae0c-e76e-460d-81cf-ff1b76a13ac9","html_url":"https://github.com/OML-Team/open-metric-learning","commit_stats":{"total_commits":241,"total_committers":17,"mean_commits":"14.176470588235293","dds":"0.43568464730290457","last_synced_commit":"03b45c33e0270e224d9005459dfc2b4d78d3e4c9"},"previous_names":[],"tags_count":24,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OML-Team%2Fopen-metric-learning","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OML-Team%2Fopen-metric-learning/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OML-Team%2Fopen-metric-learning/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OML-Team%2Fopen-metric-learning/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/OML-Team","download_url":"https://codeload.github.com/OML-Team/open-metric-learning/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248952208,"owners_count":21188424,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["computer-vision","data-science","deep-learning","hacktoberfest","hacktoberfest-2023","hacktoberfest2023","metric-learning","pytorch","pytorch-lightning","representation-learning","similarity-learning"],"created_at":"2025-01-11T02:21:03.185Z","updated_at":"2025-04-14T19:56:17.115Z","avatar_url":"https://github.com/OML-Team.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n\u003cimg src=\"https://i.ibb.co/wsmD5r4/photo-2022-06-06-17-40-52.jpg\" width=\"400px\"\u003e\n\n\n[![Documentation Status](https://readthedocs.org/projects/open-metric-learning/badge/?version=latest)](https://open-metric-learning.readthedocs.io/en/latest/?badge=latest)\n[![PyPI Status](https://pepy.tech/badge/open-metric-learning)](https://pepy.tech/project/open-metric-learning)\n[![Pipi version](https://img.shields.io/pypi/v/open-metric-learning.svg)](https://pypi.org/project/open-metric-learning/)\n[![python](https://img.shields.io/badge/python_3.10-passing-success)](https://github.com/OML-Team/open-metric-learning/actions/workflows/tests.yaml/badge.svg?)\n[![python](https://img.shields.io/badge/python_3.11-passing-success)](https://github.com/OML-Team/open-metric-learning/actions/workflows/tests.yaml/badge.svg?)\n[![python](https://img.shields.io/badge/python_3.12-passing-success)](https://github.com/OML-Team/open-metric-learning/actions/workflows/tests.yaml/badge.svg?)\n\n\nOML is a PyTorch-based framework to train and validate the models producing high-quality embeddings.\n\n### Trusted by\n\n\u003cdiv align=\"center\"\u003e\n\u003ca href=\"https://docs.neptune.ai/integrations/community_developed/\" target=\"_blank\"\u003e\u003cimg src=\"https://i.ibb.co/bMDShKDx/neptune-logo-less-margin-e1611939742683.png\" width=\"100\"/\u003e\u003c/a\u003eㅤㅤ\n\u003ca href=\"https://www.newyorker.de/\" target=\"_blank\"\u003e\u003cimg src=\"https://upload.wikimedia.org/wikipedia/commons/thumb/d/d8/New_Yorker.svg/1280px-New_Yorker.svg.png\" width=\"100\"/\u003e\u003c/a\u003eㅤㅤ\n\u003ca href=\"https://www.epoch8.co/\" target=\"_blank\"\u003e\u003cimg src=\"https://i.ibb.co/GdNVTyt/Screenshot-2023-07-04-at-11-19-24.png\" width=\"100\"/\u003e\u003c/a\u003eㅤㅤ\n\u003ca href=\"https://www.meituan.com\" target=\"_blank\"\u003e\u003cimg src=\"https://upload.wikimedia.org/wikipedia/commons/6/61/Meituan_English_Logo.png\" width=\"100\"/\u003e\u003c/a\u003eㅤㅤ\n\u003ca href=\"https://constructor.io/\" target=\"_blank\"\u003e\u003cimg src=\"https://rethink.industries/wp-content/uploads/2022/04/constructor.io-logo.png\" width=\"100\"/\u003e\u003c/a\u003eㅤㅤ\n\u003ca href=\"https://edgify.ai/\" target=\"_blank\"\u003e\u003cimg src=\"https://edgify.ai/wp-content/uploads/2024/04/new-edgify-logo.svg\" width=\"100\" height=\"30\"/\u003e\u003c/a\u003eㅤㅤ\n\u003ca href=\"https://inspector-cloud.ru/\" target=\"_blank\"\u003e\u003cimg src=\"https://thumb.tildacdn.com/tild6533-6433-4137-a266-613963373637/-/resize/540x/-/format/webp/photo.png\" width=\"150\" height=\"30\"/\u003e\u003c/a\u003eㅤㅤ\n\u003ca href=\"https://yango-tech.com/\" target=\"_blank\"\u003e\u003cimg src=\"https://yango-backend.sborkademo.com/media/pages/home/205f66f309-1717169752/opengr4-1200x630-crop-q85.jpg\" width=\"100\" height=\"30\"/\u003e\u003c/a\u003eㅤㅤ\n\u003ca href=\"https://www.adagrad.ai/\" target=\"_blank\"\u003e\u003cimg src=\"https://cdn.prod.website-files.com/619cafd224a31d1835ece5bd/66b5e8e2818e2231e4734805_Frame%201.2.png\" width=\"100\" height=\"30\"/\u003e\u003c/a\u003e\n\n\u003ca href=\"https://www.ox.ac.uk/\" target=\"_blank\"\u003e\u003cimg src=\"https://i.ibb.co/zhWL6tD/21-05-2019-16-08-10-6922268.png\" width=\"120\"/\u003e\u003c/a\u003eㅤㅤ\n\u003ca href=\"https://www.hse.ru/en/\" target=\"_blank\"\u003e\u003cimg src=\"https://www.hse.ru/data/2020/11/16/1367274044/HSE_University_blue.jpg.(230x86x123).jpg\" width=\"100\"/\u003e\u003c/a\u003e\n\nThere is a number of people from\n[Oxford](https://www.ox.ac.uk/) and\n[HSE](https://www.hse.ru/en/)\nuniversities who have used OML in their theses.\n[[1]](https://github.com/nilomr/open-metric-learning/tree/great-tit/great-tit-train)\n[[2]](https://github.com/nastygorodi/PROJECT-Deep_Metric_Learning)\n[[3]](https://github.com/nik-fedorov/term_paper_metric_learning)\n\n\n\u003cdiv align=\"left\"\u003e\n\n\n\n## [Documentation](https://open-metric-learning.readthedocs.io/en/latest/index.html)\n\n\u003cdetails\u003e\n\u003csummary\u003eFAQ\u003c/summary\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003eWhy do I need OML?\u003c/summary\u003e\n\u003cp\u003e\n\nYou may think *\"If I need image embeddings I can simply train a vanilla classifier and take its penultimate layer\"*.\nWell, it makes sense as a starting point. But there are several possible drawbacks:\n\n* If you want to use embeddings to perform searching you need to calculate some distance among them (for example, cosine or L2).\n  Usually, **you don't directly optimize these distances during the training** in the classification setup. So, you can only hope that\n  final embeddings will have the desired properties.\n\n* **The second problem is the validation process**.\n  In the searching setup, you usually care how related your top-N outputs are to the query.\n  The natural way to evaluate the model is to simulate searching requests to the reference set\n  and apply one of the retrieval metrics.\n  So, there is no guarantee that classification accuracy will correlate with these metrics.\n\n* Finally, you may want to implement a metric learning pipeline by yourself.\n  **There is a lot of work**: to use triplet loss you need to form batches in a specific way,\n  implement different kinds of triplets mining, tracking distances, etc. For the validation, you also need to\n  implement retrieval metrics,\n  which include effective embeddings accumulation during the epoch, covering corner cases, etc.\n  It's even harder if you have several gpus and use DDP.\n  You may also want to visualize your search requests by highlighting good and bad search results.\n  Instead of doing it by yourself, you can simply use OML for your purposes.\n\n\u003c/p\u003e\n\u003c/details\u003e\n\n\n\u003cdetails\u003e\n\u003csummary\u003eWhat is the difference between Open Metric Learning and PyTorch Metric Learning?\u003c/summary\u003e\n\u003cp\u003e\n\n[PML](https://github.com/KevinMusgrave/pytorch-metric-learning) is the popular library for Metric Learning,\nand it includes a rich collection of losses, miners, distances, and reducers; that is why we provide straightforward\n[examples](https://open-metric-learning.readthedocs.io/en/latest/feature_extraction/python_examples.html#usage-with-pytorch-metric-learning) of using them with OML.\nInitially, we tried to use PML, but in the end, we came up with our library, which is more pipeline / recipes oriented.\nThat is how OML differs from PML:\n\n* OML has [Pipelines](https://github.com/OML-Team/open-metric-learning/tree/main/pipelines)\n  which allows training models by preparing a config and your data in the required format\n  (it's like converting data into COCO format to train a detector from [mmdetection](https://github.com/open-mmlab/mmdetection)).\n\n* OML focuses on end-to-end pipelines and practical use cases.\n  It has config based examples on popular benchmarks close to real life (like photos of products of thousands ids).\n  We found some good combinations of hyperparameters on these datasets, trained and published models and their configs.\n  Thus, it makes OML more recipes oriented than PML, and its author\n  [confirms](https://github.com/KevinMusgrave/pytorch-metric-learning/issues/169#issuecomment-670814393)\n  this saying that his library is a set of tools rather the recipes, moreover, the examples in PML are mostly for CIFAR and MNIST datasets.\n\n* OML has the [Zoo](https://github.com/OML-Team/open-metric-learning#zoo) of pretrained models that can be easily accessed from\n  the code in the same way as in `torchvision` (when you type `resnet50(pretrained=True)`).\n\n* OML is integrated with [PyTorch Lightning](https://www.pytorchlightning.ai/), so, we can use the power of its\n  [Trainer](https://pytorch-lightning.readthedocs.io/en/stable/common/trainer.html).\n  This is especially helpful when we work with DDP, so, you compare our\n  [DDP example](https://open-metric-learning.readthedocs.io/en/latest/feature_extraction/python_examples.html#usage-with-pytorch-lightning)\n  and the\n  [PMLs one](https://github.com/KevinMusgrave/pytorch-metric-learning/blob/master/examples/notebooks/DistributedTripletMarginLossMNIST.ipynb).\n  By the way, PML also has [Trainers](https://kevinmusgrave.github.io/pytorch-metric-learning/trainers/), but it's not\n  widely used in the examples and custom `train` / `test` functions are used instead.\n\nWe believe that having Pipelines, laconic examples, and Zoo of pretrained models sets the entry threshold to a really low value.\n\n\u003c/p\u003e\n\u003c/details\u003e\n\n\n\u003cdetails\u003e\n\u003csummary\u003eWhat is Metric Learning?\u003c/summary\u003e\n\u003cp\u003e\n\nMetric Learning problem (also known as *extreme classification* problem) means a situation in which we\nhave thousands of ids of some entities, but only a few samples for every entity.\nOften we assume that during the test stage (or production) we will deal with unseen entities\nwhich makes it impossible to apply the vanilla classification pipeline directly. In many cases obtained embeddings\nare used to perform search or matching procedures over them.\n\nHere are a few examples of such tasks from the computer vision sphere:\n* Person/Animal Re-Identification\n* Face Recognition\n* Landmark Recognition\n* Searching engines for online shops\n and many others.\n\u003c/p\u003e\n\u003c/details\u003e\n\n\n\u003cdetails\u003e\n\u003csummary\u003eGlossary (Naming convention) \u003c/summary\u003e\n\u003cp\u003e\n\n* `embedding` - model's output (also known as `features vector` or `descriptor`).\n* `query` - a sample which is used as a request in the retrieval procedure.\n* `gallery set` - the set of entities to search items similar to `query` (also known as `reference` or `index`).\n* `Sampler` - an argument for `DataLoader` which is used to form batches\n* `Miner` - the object to form pairs or triplets after the batch was formed by `Sampler`. It's not necessary to form\n  the combinations of samples only inside the current batch, thus, the memory bank may be a part of `Miner`.\n* `Samples`/`Labels`/`Instances` - as an example let's consider DeepFashion dataset. It includes thousands of\n  fashion item ids (we name them `labels`) and several photos for each item id\n  (we name the individual photo as `instance` or `sample`). All of the fashion item ids have their groups like\n  \"skirts\", \"jackets\", \"shorts\" and so on (we name them `categories`).\n  Note, we avoid using the term `class` to avoid misunderstanding.\n* `training epoch` - batch samplers which we use for combination-based losses usually have a length equal to\n  `[number of labels in training dataset] / [numbers of labels in one batch]`. It means that we don't observe all of\n  the available training samples in one epoch (as opposed to vanilla classification),\n  instead, we observe all of the available labels.\n\n\u003c/p\u003e\n\u003c/details\u003e\n\n\n\u003cdetails\u003e\n\u003csummary\u003eHow good may be a model trained with OML? \u003c/summary\u003e\n\u003cp\u003e\n\nIt may be comparable with the current (2022 year) [SotA](https://paperswithcode.com/task/metric-learning) methods,\nfor example, [Hyp-ViT](https://arxiv.org/pdf/2203.10833.pdf).\n*(Few words about this approach: it's a ViT architecture trained with contrastive loss,\nbut the embeddings were projected into some hyperbolic space.\nAs the authors claimed, such a space is able to describe the nested structure of real-world data.\nSo, the paper requires some heavy math to adapt the usual operations for the hyperbolical space.)*\n\nWe trained the same architecture with triplet loss, fixing the rest of the parameters:\ntraining and test transformations, image size, and optimizer. See configs in [Models Zoo](https://github.com/OML-Team/open-metric-learning#zoo).\nThe trick was in heuristics in our miner and sampler:\n\n* [Category Balance Sampler](https://open-metric-learning.readthedocs.io/en/latest/contents/samplers.html#categorybalancesampler)\n  forms the batches limiting the number of categories *C* in it.\n  For instance, when *C = 1* it puts only jackets in one batch and only jeans into another one (just an example).\n  It automatically makes the negative pairs harder: it's more meaningful for a model to realise why two jackets\n  are different than to understand the same about a jacket and a t-shirt.\n\n* [Hard Triplets Miner](https://open-metric-learning.readthedocs.io/en/latest/contents/miners.html#hardtripletsminer)\n  makes the task even harder keeping only the hardest triplets (with maximal positive and minimal negative distances).\n\nHere are *CMC@1* scores for 2 popular benchmarks.\nSOP dataset: Hyp-ViT — 85.9, ours — 86.6. DeepFashion dataset: Hyp-ViT — 92.5, ours — 92.1.\nThus, utilising simple heuristics and avoiding heavy math we are able to perform on SotA level.\n\n\u003c/p\u003e\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003eWhat about Self-Supervised Learning?\u003c/summary\u003e\n\u003cp\u003e\n\nRecent research in SSL definitely obtained great results. The problem is that these approaches\nrequired an enormous amount of computing to train the model. But in our framework, we consider the most common case\nwhen the average user has no more than a few GPUs.\n\nAt the same time, it would be unwise to ignore success in this sphere, so we still exploit it in two ways:\n* As a source of checkpoints that would be great to start training with. From publications and our experience,\n  they are much better as initialisation than the default supervised model trained on ImageNet. Thus, we added the possibility\n  to initialise your models using these pretrained checkpoints only by passing an argument in the config or the constructor.\n* As a source of inspiration. For example, we adapted the idea of a memory bank from *MoCo* for the *TripletLoss*.\n\n\u003c/p\u003e\n\u003c/details\u003e\n\n\n\u003cdetails\u003e\n\u003csummary\u003eDo I need to know other frameworks to use OML?\u003c/summary\u003e\n\u003cp\u003e\n\nNo, you don't. OML is a framework-agnostic. Despite we use PyTorch Lightning as a loop\nrunner for the experiments, we also keep the possibility to run everything on pure PyTorch.\nThus, only the tiny part of OML is Lightning-specific and we keep this logic separately from\nother code (see `oml.lightning`). Even when you use Lightning, you don't need to know it, since\nwe provide ready to use [Pipelines](https://github.com/OML-Team/open-metric-learning/blob/main/pipelines/).\n\nThe possibility of using pure PyTorch and modular structure of the code leaves a room for utilizing\nOML with your favourite framework after the implementation of the necessary wrappers.\n\n\u003c/p\u003e\n\u003c/details\u003e\n\n\n\u003cdetails\u003e\n\u003csummary\u003eCan I use OML without any knowledge in DataScience?\u003c/summary\u003e\n\u003cp\u003e\n\nYes. To run the experiment with [Pipelines](https://github.com/OML-Team/open-metric-learning/blob/main/pipelines/)\nyou only need to write a converter\nto our format (it means preparing the\n`.csv` table with a few predefined columns).\nThat's it!\n\nProbably we already have a suitable pre-trained model for your domain\nin our *Models Zoo*. In this case, you don't even need to train it.\n\u003c/p\u003e\n\u003c/details\u003e\n\n\n\u003cdetails\u003e\n\u003csummary\u003eCan I export models to ONNX?\u003c/summary\u003e\n\u003cp\u003e\n\nCurrently, we don't support exporting models to ONNX directly.\nHowever, you can use the built-in PyTorch capabilities to achieve this. For more information, please refer to this [issue](https://github.com/OML-Team/open-metric-learning/issues/592).\n\u003c/p\u003e\n\u003c/details\u003e\n\n\u003c/details\u003e\n\n\n[DOCUMENTATION](https://open-metric-learning.readthedocs.io/en/latest/index.html)\n\nTUTORIAL TO START WITH:\n[English](https://medium.com/@AlekseiShabanov/practical-metric-learning-b0410cda2201) |\n[Russian](https://habr.com/ru/company/ods/blog/695380/) |\n[Chinese](https://zhuanlan.zhihu.com/p/683102241)\n\n\u003cdetails\u003e\n\u003csummary\u003eMORE\u003c/summary\u003e\n\n* The\n[DEMO](https://dapladoc-oml-postprocessing-demo-srcappmain-pfh2g0.streamlit.app/)\nfor our paper\n[STIR: Siamese Transformers for Image Retrieval Postprocessing](https://arxiv.org/abs/2304.13393)\n\n* Meet OpenMetricLearning (OML) on\n[Marktechpost](https://www.marktechpost.com/2023/12/26/meet-openmetriclearning-oml-a-pytorch-based-python-framework-to-train-and-validate-the-deep-learning-models-producing-high-quality-embeddings/)\n\n* The report for Berlin-based meetup: \"Computer Vision in production\". November, 2022.\n[Link](https://drive.google.com/drive/folders/1uHmLU8vMrMVMFodt36u0uXAgYjG_3D30?usp=share_link)\n\n\u003c/details\u003e\n\n## [Installation](https://open-metric-learning.readthedocs.io/en/latest/oml/installation.html)\n\n```shell\npip install -U open-metric-learning  # minimum dependencies\npip install -U open-metric-learning[nlp]\npip install -U open-metric-learning[audio]\npip install -U open-metric-learning[pipelines]\n\n# in the case of conflicts install without dependencies and manage versions manually:\npip install --no-deps open-metric-learning\n```\n\n\u003cdetails\u003e\u003csummary\u003eDockerHub\u003c/summary\u003e\n\n```shell\ndocker pull omlteam/oml:gpu\ndocker pull omlteam/oml:cpu\n```\n\n\u003c/details\u003e\n\n\n## OML features\n\n\u003cdiv style=\"overflow-x: auto;\"\u003e\n\n\u003ctable style=\"width: 100%; border-collapse: collapse; border-spacing: 0; margin: 0; padding: 0;\"\u003e\n\n\u003ctr\u003e\n\u003c/tr\u003e\n\n\u003ctr\u003e\n\u003ctd style=\"text-align: left;\"\u003e\n\u003ca href=\"https://open-metric-learning.readthedocs.io/en/latest/contents/losses.html\"\u003e \u003cb\u003eLosses\u003c/b\u003e\u003c/a\u003e |\n\u003ca href=\"https://open-metric-learning.readthedocs.io/en/latest/contents/miners.html\"\u003e \u003cb\u003eMiners\u003c/b\u003e\u003c/a\u003e\n\n```python\nminer = AllTripletsMiner()\nminer = NHardTripletsMiner()\nminer = MinerWithBank()\n...\ncriterion = TripletLossWithMiner(0.1, miner)\ncriterion = ArcFaceLoss()\ncriterion = SurrogatePrecision()\n```\n\n\u003c/td\u003e\n\u003ctd style=\"text-align: left;\"\u003e\n\u003ca href=\"https://open-metric-learning.readthedocs.io/en/latest/contents/samplers.html\"\u003e \u003cb\u003eSamplers\u003c/b\u003e\u003c/a\u003e\n\n```python\nlabels = train.get_labels()\nl2c = train.get_label2category()\n\n\nsampler = BalanceSampler(labels)\nsampler = CategoryBalanceSampler(labels, l2c)\nsampler = DistinctCategoryBalanceSampler(labels, l2c)\n```\n\n\u003c/td\u003e\n\u003c/tr\u003e\n\n\u003ctr\u003e\n\u003c/tr\u003e\n\n\u003ctr\u003e\n\u003ctd style=\"text-align: left;\"\u003e\n\u003ca href=\"https://github.com/OML-Team/open-metric-learning/tree/main/pipelines/\"\u003e\u003cb\u003eConfigs support\u003c/b\u003e\u003c/a\u003e\n\n```yaml\nmax_epochs: 10\nsampler:\n  name: balance\n  args:\n    n_labels: 2\n    n_instances: 2\n```\n\n\u003c/td\u003e\n\u003ctd style=\"text-align: left;\"\u003e\n\u003ca href=\"https://github.com/OML-Team/open-metric-learning?tab=readme-ov-file#zoo\"\u003e\u003cb\u003ePre-trained models of different modalities\u003c/b\u003e\u003c/a\u003e\n\n```python\nmodel_hf = AutoModel.from_pretrained(\"roberta-base\")\ntokenizer = AutoTokenizer.from_pretrained(\"roberta-base\")\nextractor_txt = HFWrapper(model_hf)\n\nextractor_img = ViTExtractor.from_pretrained(\"vits16_dino\")\ntransforms, _ = get_transforms_for_pretrained(\"vits16_dino\")\n\nextractor_audio = ECAPATDNNExtractor.from_pretrained()\n```\n\n\u003c/td\u003e\n\u003c/tr\u003e\n\n\u003ctr\u003e\n\u003c/tr\u003e\n\n\u003ctr\u003e\n\u003ctd style=\"text-align: left;\"\u003e\n\u003ca href=\"https://open-metric-learning.readthedocs.io/en/latest/postprocessing/algo_examples.html\"\u003e\u003cb\u003ePost-processing\u003c/b\u003e\u003c/a\u003e\n\n```python\nemb = inference(extractor, dataset)\nrr = RetrievalResults.from_embeddings(emb, dataset)\n\npostprocessor = AdaptiveThresholding()\nrr_upd = postprocessor.process(rr, dataset)\n```\n\n\u003c/td\u003e\n\u003ctd style=\"text-align: left;\"\u003e\n\u003ca href=\"https://open-metric-learning.readthedocs.io/en/latest/postprocessing/siamese_examples.html\"\u003e\u003cb\u003ePost-processing by NN\u003c/b\u003e\u003c/a\u003e |\n\u003ca href=\"https://github.com/OML-Team/open-metric-learning/tree/main/pipelines/postprocessing/pairwise_postprocessing\"\u003e\u003cb\u003ePaper\u003c/b\u003e\u003c/a\u003e\n\n```python\nembeddings = inference(extractor, dataset)\nrr = RetrievalResults.from_embeddings(embeddings, dataset)\n\npostprocessor = PairwiseReranker(ConcatSiamese(), top_n=3)\nrr_upd = postprocessor.process(rr, dataset)\n```\n\n\u003c/td\u003e\n\u003c/tr\u003e\n\n\u003ctr\u003e\n\u003c/tr\u003e\n\n\u003ctr\u003e\n\u003ctd style=\"text-align: left;\"\u003e\n\u003ca href=\"https://open-metric-learning.readthedocs.io/en/latest/oml/logging.html#\"\u003e\u003cb\u003eLogging\u003c/b\u003e\u003c/a\u003e\u003cbr\u003e\n\n```python\nlogger = TensorBoardPipelineLogger()\nlogger = NeptunePipelineLogger()\nlogger = WandBPipelineLogger()\nlogger = MLFlowPipelineLogger()\nlogger = ClearMLPipelineLogger()\n```\n\n\u003c/td\u003e\n\u003ctd style=\"text-align: left;\"\u003e\n\u003ca href=\"https://open-metric-learning.readthedocs.io/en/latest/feature_extraction/python_examples.html#usage-with-pytorch-metric-learning\"\u003e\u003cb\u003ePML\u003c/b\u003e\u003c/a\u003e\u003cbr\u003e\n\n```python\nfrom pytorch_metric_learning import losses\n\ncriterion = losses.TripletMarginLoss(0.2, \"all\")\npred = ViTExtractor()(data)\ncriterion(pred, gts)\n```\n\n\u003c/td\u003e\n\u003c/tr\u003e\n\n\u003ctr\u003e\n\u003c/tr\u003e\n\n\u003ctr\u003e\n\u003ctd style=\"text-align: left;\"\u003e\u003ca href=\"https://open-metric-learning.readthedocs.io/en/latest/feature_extraction/python_examples.html#handling-categories\"\u003e\u003cb\u003eCategories support\u003c/b\u003e\u003c/a\u003e\n\n```python\n# train\nloader = DataLoader(CategoryBalanceSampler())\n\n# validation\nrr = RetrievalResults.from_embeddings()\nm.calc_retrieval_metrics_rr(rr, query_categories)\n```\n\n\u003c/td\u003e\n\u003ctd style=\"text-align: left;\"\u003e\u003ca href=\"https://open-metric-learning.readthedocs.io/en/latest/contents/metrics.html\"\u003e\u003cb\u003eMisc metrics\u003c/b\u003e\u003c/a\u003e\n\n```python\nembeddigs = inference(model, dataset)\nrr = RetrievalResults.from_embeddings(embeddings, dataset)\n\nm.calc_retrieval_metrics_rr(rr, precision_top_k=(5,))\nm.calc_fnmr_at_fmr_rr(rr, fmr_vals=(0.1,))\nm.calc_topological_metrics(embeddings, pcf_variance=(0.5,))\n```\n\n\u003c/td\u003e\n\u003c/tr\u003e\n\n\u003ctr\u003e\n\u003c/tr\u003e\n\n\u003ctr\u003e\n\u003ctd style=\"text-align: left;\"\u003e\n\u003ca href=\"https://open-metric-learning.readthedocs.io/en/latest/feature_extraction/python_examples.html#usage-with-pytorch-lightning\"\u003e\u003cb\u003eLightning\u003c/b\u003e\u003c/a\u003e\u003cbr\u003e\n\n```python\nimport pytorch_lightning as pl\n\nmodel = ViTExtractor.from_pretrained(\"vits16_dino\")\nclb = MetricValCallback(EmbeddingMetrics(dataset))\nmodule = ExtractorModule(model, criterion, optimizer)\n\ntrainer = pl.Trainer(max_epochs=3, callbacks=[clb])\ntrainer.fit(module, train_loader, val_loader)\n```\n\n\u003c/td\u003e\n\u003ctd style=\"text-align: left;\"\u003e\n\u003ca href=\"https://open-metric-learning.readthedocs.io/en/latest/feature_extraction/python_examples.html#usage-with-pytorch-lightning\"\u003e\u003cb\u003eLightning DDP\u003c/b\u003e\u003c/a\u003e\u003cbr\u003e\n\n```python\nclb = MetricValCallback(EmbeddingMetrics(val))\nmodule = ExtractorModuleDDP(\n    model, criterion, optimizer, train, val\n)\n\nddp = {\"devices\": 2, \"strategy\": DDPStrategy()}\ntrainer = pl.Trainer(max_epochs=3, callbacks=[clb], **ddp)\ntrainer.fit(module)\n```\n\n\u003c/td\u003e\n\u003c/tr\u003e\n\n\u003c/table\u003e\n\n\u003c/div\u003e\n\n## [Examples](https://open-metric-learning.readthedocs.io/en/latest/feature_extraction/python_examples.html#)\n\nHere is an example of how to train, validate and post-process the model\non a tiny dataset of\n[images](https://drive.google.com/drive/folders/1plPnwyIkzg51-mLUXWTjREHgc1kgGrF4),\n[texts](https://github.com/OML-Team/open-metric-learning/blob/main/oml/utils/download_mock_dataset.py#L83),\nor\n[audios](https://drive.google.com/drive/folders/1NcKnyXqDyyYARrDETmhJcTTXegO3W0Ju).\nSee more details on dataset\n[format](https://open-metric-learning.readthedocs.io/en/latest/oml/data.html).\n\nSCROLL RIGHT FOR **IMAGES** \u003e **TEXTS** \u003e **AUDIOS**\n\n\u003cdiv style=\"overflow-x: auto;\"\u003e\n\n\u003ctable style=\"width: 100%; border-collapse: collapse; border-spacing: 0; margin: 0; padding: 0;\"\u003e\n\n\u003ctr\u003e\n\u003c/tr\u003e\n\n\u003ctr\u003e\n    \u003ctd style=\"text-align: left; padding: 0;\"\u003e\u003cb\u003eIMAGES\u003c/b\u003e\u003c/td\u003e\n    \u003ctd style=\"text-align: left; padding: 0;\"\u003e\u003cb\u003eTEXTS\u003c/b\u003e\u003c/td\u003e\n    \u003ctd style=\"text-align: left; padding: 0;\"\u003e\u003cb\u003eAUDIOS\u003c/b\u003e\u003c/td\u003e\n\u003c/tr\u003e\n\n\u003ctr\u003e\n\u003c/tr\u003e\n\n\u003ctr\u003e\n\n\u003ctd\u003e\n\n[comment]:train-val-img-start\n```python\nfrom torch.optim import Adam\nfrom torch.utils.data import DataLoader\n\nfrom oml import datasets as d\nfrom oml.inference import inference\nfrom oml.losses import TripletLossWithMiner\nfrom oml.metrics import calc_retrieval_metrics_rr\nfrom oml.miners import HardTripletsMiner\nfrom oml.models import ViTExtractor\nfrom oml.registry import get_transforms_for_pretrained\nfrom oml.retrieval import RetrievalResults, AdaptiveThresholding\nfrom oml.samplers import BalanceSampler\nfrom oml.utils import get_mock_images_dataset\n\nmodel = ViTExtractor.from_pretrained(\"vits16_dino\").to(\"cpu\").train()\ntransform, _ = get_transforms_for_pretrained(\"vits16_dino\")\n\ndf_train, df_val = get_mock_images_dataset(global_paths=True)\ntrain = d.ImageLabeledDataset(df_train, transform=transform)\nval = d.ImageQueryGalleryLabeledDataset(df_val, transform=transform)\n\noptimizer = Adam(model.parameters(), lr=1e-4)\ncriterion = TripletLossWithMiner(0.1, HardTripletsMiner(), need_logs=True)\nsampler = BalanceSampler(train.get_labels(), n_labels=2, n_instances=2)\n\n\n# training 1 epoch\nfor batch in DataLoader(train, batch_sampler=sampler):\n    embeddings = model(batch[\"input_tensors\"])\n    loss = criterion(embeddings, batch[\"labels\"])\n    loss.backward()\n    optimizer.step()\n    optimizer.zero_grad()\n    print(criterion.last_logs)\n\n\n# validation by retrieving relevant items\nembeddings = inference(model, val, batch_size=4, num_workers=0)\nrr = RetrievalResults.from_embeddings(embeddings, val, n_items=3)\nrr = AdaptiveThresholding(n_std=2).process(rr)\nrr.visualize(query_ids=[2, 1], dataset=val, show=True)\nprint(calc_retrieval_metrics_rr(rr, map_top_k=(3,), cmc_top_k=(1,)))\n\n\n\n```\n[comment]:train-val-img-end\n\n\u003c/td\u003e\n\n\u003ctd\u003e\n\n[comment]:train-val-txt-start\n```python\nfrom torch.optim import Adam\nfrom torch.utils.data import DataLoader\nfrom transformers import AutoModel, AutoTokenizer\n\nfrom oml import datasets as d\nfrom oml.inference import inference\nfrom oml.losses import TripletLossWithMiner\nfrom oml.metrics import calc_retrieval_metrics_rr\nfrom oml.miners import NHardTripletsMiner\nfrom oml.models import HFWrapper\nfrom oml.retrieval import RetrievalResults, AdaptiveThresholding\nfrom oml.samplers import BalanceSampler\nfrom oml.utils import get_mock_texts_dataset\n\nmodel = HFWrapper(AutoModel.from_pretrained(\"bert-base-uncased\"), 768).to(\"cpu\").train()\ntokenizer = AutoTokenizer.from_pretrained(\"bert-base-uncased\")\n\ndf_train, df_val = get_mock_texts_dataset()\ntrain = d.TextLabeledDataset(df_train, tokenizer=tokenizer)\nval = d.TextQueryGalleryLabeledDataset(df_val, tokenizer=tokenizer)\n\noptimizer = Adam(model.parameters(), lr=1e-4)\ncriterion = TripletLossWithMiner(\n    0.1, NHardTripletsMiner(n_positive=2, n_negative=2), need_logs=True\n)\nsampler = BalanceSampler(train.get_labels(), n_labels=2, n_instances=2)\n\n\n# training 1 epoch\nfor batch in DataLoader(train, batch_sampler=sampler):\n    embeddings = model(batch[\"input_tensors\"])\n    loss = criterion(embeddings, batch[\"labels\"])\n    loss.backward()\n    optimizer.step()\n    optimizer.zero_grad()\n    print(criterion.last_logs)\n\n\n# validation by retrieving relevant items\nembeddings = inference(model, val, batch_size=4, num_workers=0)\nrr = RetrievalResults.from_embeddings(embeddings, val, n_items=3)\nrr = AdaptiveThresholding(n_std=2).process(rr)\nrr.visualize(query_ids=[2, 1], dataset=val, show=True)\nprint(calc_retrieval_metrics_rr(rr, map_top_k=(3,), cmc_top_k=(1,)))\n\n\n\n```\n[comment]:train-val-txt-end\n\u003c/td\u003e\n\n\u003ctd\u003e\n\n[comment]:train-val-audio-start\n```python\nfrom torch.optim import Adam\nfrom torch.utils.data import DataLoader\n\nfrom oml import datasets as d\nfrom oml.inference import inference\nfrom oml.losses import ArcFaceLoss\nfrom oml.metrics import calc_retrieval_metrics_rr\nfrom oml.models import ECAPATDNNExtractor\nfrom oml.retrieval import AdaptiveThresholding, RetrievalResults\nfrom oml.samplers import BalanceSampler\nfrom oml.utils import get_mock_audios_dataset\n\nmodel = ECAPATDNNExtractor.from_pretrained(\"ecapa_tdnn_taoruijie\").to(\"cpu\").train()\n\ndf_train, df_val = get_mock_audios_dataset(global_paths=True)\ntrain = d.AudioLabeledDataset(df_train)\nval = d.AudioQueryGalleryLabeledDataset(df_val)\n\noptimizer = Adam(model.parameters(), lr=1e-4)\ncriterion = ArcFaceLoss(m=0.2, s=30, in_features=192, num_classes=4)  # similar to paper\nsampler = BalanceSampler(train.get_labels(), n_labels=2, n_instances=2)\n\n\n# training 1 epoch\nfor batch in DataLoader(train, batch_sampler=sampler):\n    embeddings = model(batch[\"input_tensors\"])\n    loss = criterion(embeddings, batch[\"labels\"])\n    loss.backward()\n    optimizer.step()\n    optimizer.zero_grad()\n    print(criterion.last_logs)\n\n\n# validation by retrieving relevant items\nembeddings = inference(model, val, batch_size=4, num_workers=0)\nrr = RetrievalResults.from_embeddings(embeddings, val, n_items=3)\nrr = AdaptiveThresholding(n_std=2).process(rr)\nrr.visualize_as_html(query_ids=[2, 1], dataset=val, show=True)\nprint(calc_retrieval_metrics_rr(rr, map_top_k=(3,), cmc_top_k=(1,)))\n\n\n\n```\n[comment]:train-val-audio-end\n\u003c/td\u003e\n\n\u003c/tr\u003e\n\n\u003ctr\u003e\n\u003c/tr\u003e\n\n\u003ctr\u003e\n\n\u003ctd\u003e\n\n\u003cdetails style=\"padding-bottom: 10px\"\u003e\n\u003csummary\u003eOutput\u003c/summary\u003e\n\n```python\n{'active_tri': 0.125, 'pos_dist': 82.5, 'neg_dist': 100.5}  # batch 1\n{'active_tri': 0.0, 'pos_dist': 36.3, 'neg_dist': 56.9}     # batch 2\n\n{'cmc': {1: 0.75}, 'precision': {5: 0.75}, 'map': {3: 0.8}}\n\n```\n\n\u003cimg src=\"https://i.ibb.co/MVxBf80/retrieval-img.png\" height=\"200px\"\u003e\n\n\u003c/details\u003e\n\n[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1Fr4HhDOqmjx1hCFS30G3MlYjeqBW5vDg?usp=sharing)\n\n\u003c/td\u003e\n\n\u003ctd\u003e\n\n\u003cdetails style=\"padding-bottom: 10px\"\u003e\n\u003csummary\u003eOutput\u003c/summary\u003e\n\n```python\n{'active_tri': 0.0, 'pos_dist': 8.5, 'neg_dist': 11.0}  # batch 1\n{'active_tri': 0.25, 'pos_dist': 8.9, 'neg_dist': 9.8}  # batch 2\n\n{'cmc': {1: 0.8}, 'precision': {5: 0.7}, 'map': {3: 0.9}}\n\n```\n\n\u003cimg src=\"https://i.ibb.co/HqfXdYd/text-retrieval.png\" height=\"200px\"\u003e\n\n\u003c/details\u003e\n\n[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/19o2Ox2VXZoOWOOXIns7mcs0aHJZgJWeO?usp=sharing)\n\n\u003c/td\u003e\n\n\u003ctd\u003e\n\n\u003cdetails style=\"padding-bottom: 10px\"\u003e\n\u003csummary\u003eOutput\u003c/summary\u003e\n\n```python\n{'active_tri': 0.25, 'pos_dist': 17.3, 'neg_dist': 18.4}  # batch 1\n{'active_tri': 0.0, 'pos_dist': 17.1, 'neg_dist': 18.5}   # batch 2\n\n{'cmc': {1: 1.0}, 'precision': {5: 1.0}, 'map': {3: 1.0}}\n\n```\n\n\u003cimg src=\"https://i.ibb.co/VWFNkJr0/audio.jpg\" height=\"200px\"\u003e\n\n\u003c/details\u003e\n\n[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1Sfz7xMdjXg634-3KmBPq8Zs6i_gbsWD1?usp=sharing)\n\n\u003c/td\u003e\n\n\u003c/tr\u003e\n\n\u003c/table\u003e\n\n\u003c/div\u003e\n\n\u003cbr\u003e\n\n[Extra illustrations, explanations and tips](https://github.com/OML-Team/open-metric-learning/tree/main/pipelines/features_extraction#training)\nfor the code above.\n\n### Retrieval by trained model\n\nHere is an inference time example (in other words, retrieval on test set).\nThe code below works for both texts and images.\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eSee example\u003c/b\u003e\u003c/summary\u003e\n\u003cp\u003e\n\n[comment]:usage-retrieval-start\n```python\nfrom oml.datasets import ImageQueryGalleryDataset\nfrom oml.inference import inference\nfrom oml.models import ViTExtractor\nfrom oml.registry import get_transforms_for_pretrained\nfrom oml.utils import get_mock_images_dataset\nfrom oml.retrieval import RetrievalResults, AdaptiveThresholding\n\n_, df_test = get_mock_images_dataset(global_paths=True)\ndel df_test[\"label\"]  # we don't need gt labels for doing predictions\n\nextractor = ViTExtractor.from_pretrained(\"vits16_dino\").to(\"cpu\")\ntransform, _ = get_transforms_for_pretrained(\"vits16_dino\")\n\ndataset = ImageQueryGalleryDataset(df_test, transform=transform)\nembeddings = inference(extractor, dataset, batch_size=4, num_workers=0)\n\nrr = RetrievalResults.from_embeddings(embeddings, dataset, n_items=5)\nrr = AdaptiveThresholding(n_std=3.5).process(rr)\nrr.visualize(query_ids=[0, 1], dataset=dataset, show=True)\n\n# you get the ids of retrieved items and the corresponding distances\nprint(rr)\n```\n[comment]:usage-retrieval-end\n\n\u003c/details\u003e\n\n\n\n### Retrieval by trained model: streaming \u0026 txt2im\n\nHere is an example where queries and galleries processed separately.\n* First, it may be useful for **streaming retrieval**, when a gallery (index) set is huge and fixed, but\n  queries are coming in batches.\n* Second, queries and galleries have different natures, for examples, **queries are texts, but galleries are images**.\n\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eSee example\u003c/b\u003e\u003c/summary\u003e\n\u003cp\u003e\n\n[comment]:usage-streaming-retrieval-start\n```python\nimport pandas as pd\n\nfrom oml.datasets import ImageBaseDataset\nfrom oml.inference import inference\nfrom oml.models import ViTExtractor\nfrom oml.registry import get_transforms_for_pretrained\nfrom oml.retrieval import RetrievalResults, ConstantThresholding\nfrom oml.utils import get_mock_images_dataset\n\nextractor = ViTExtractor.from_pretrained(\"vits16_dino\").to(\"cpu\")\ntransform, _ = get_transforms_for_pretrained(\"vits16_dino\")\n\npaths = pd.concat(get_mock_images_dataset(global_paths=True))[\"path\"]\ngalleries, queries1, queries2 = paths[:20], paths[20:22], paths[22:24]\n\n# gallery is huge and fixed, so we only process it once\ndataset_gallery = ImageBaseDataset(galleries, transform=transform)\nembeddings_gallery = inference(extractor, dataset_gallery, batch_size=4, num_workers=0)\n\n# queries come \"online\" in stream\nfor queries in [queries1, queries2]:\n    dataset_query = ImageBaseDataset(queries, transform=transform)\n    embeddings_query = inference(extractor, dataset_query, batch_size=4, num_workers=0)\n\n    # for the operation below we are going to provide integrations with vector search DB like QDrant or Faiss\n    rr = RetrievalResults.from_embeddings_qg(\n        embeddings_query=embeddings_query, embeddings_gallery=embeddings_gallery,\n        dataset_query=dataset_query, dataset_gallery=dataset_gallery\n    )\n    rr = ConstantThresholding(th=80).process(rr)\n    rr.visualize_qg([0, 1], dataset_query=dataset_query, dataset_gallery=dataset_gallery, show=True)\n    print(rr)\n```\n[comment]:usage-streaming-retrieval-end\n\n\u003c/details\u003e\n\n## [Pipelines](https://github.com/OML-Team/open-metric-learning/tree/main/pipelines)\n\nPipelines provide a way to run metric learning experiments via changing only the config file.\nAll you need is to prepare your dataset in a required format.\n\nSee [Pipelines](https://github.com/OML-Team/open-metric-learning/blob/main/pipelines/) folder for more details:\n* Feature extractor [pipeline](https://github.com/OML-Team/open-metric-learning/tree/main/pipelines/features_extraction)\n* Retrieval re-ranking [pipeline](https://github.com/OML-Team/open-metric-learning/tree/main/pipelines/postprocessing)\n\n## [Zoo: Images](https://open-metric-learning.readthedocs.io/en/latest/feature_extraction/zoo.html#zoo-images)\n\nYou can use an image model from our Zoo or\nuse other arbitrary models after you inherited it from [IExtractor](https://open-metric-learning.readthedocs.io/en/latest/contents/interfaces.html#iextractor).\n\n\u003cdetails style=\"padding-bottom: 15px\"\u003e\n\u003csummary\u003e\u003cb\u003eSee how to use models\u003c/b\u003e\u003c/summary\u003e\n\u003cp\u003e\n\n[comment]:zoo-image-start\n```python\nfrom oml.const import CKPT_SAVE_ROOT as CKPT_DIR, MOCK_DATASET_PATH as DATA_DIR\nfrom oml.models import ViTExtractor\nfrom oml.registry import get_transforms_for_pretrained\n\nmodel = ViTExtractor.from_pretrained(\"vits16_dino\").eval()\ntransforms, im_reader = get_transforms_for_pretrained(\"vits16_dino\")\n\nimg = im_reader(DATA_DIR / \"images\" / \"circle_1.jpg\")  # put path to your image here\nimg_tensor = transforms(img)\n# img_tensor = transforms(image=img)[\"image\"]  # for transforms from Albumentations\n\nfeatures = model(img_tensor.unsqueeze(0))\n\n# Check other available models:\nprint(list(ViTExtractor.pretrained_models.keys()))\n\n# Load checkpoint saved on a disk:\nmodel_ = ViTExtractor(weights=CKPT_DIR / \"vits16_dino.ckpt\", arch=\"vits16\", normalise_features=False)\n```\n[comment]:zoo-image-end\n\n\u003c/p\u003e\n\u003c/details\u003e\n\n### Image models zoo\n\nModels, trained by us.\nThe metrics below are for **224 x 224** images:\n\n|                      model                      | cmc1  |         dataset          |                                              weights                                              |                                                    experiment                                                     |\n|:-----------------------------------------------:|:-----:|:------------------------:|:-------------------------------------------------------------------------------------------------:|:-----------------------------------------------------------------------------------------------------------------:|\n| `ViTExtractor.from_pretrained(\"vits16_inshop\")` | 0.921 |    DeepFashion Inshop    |    [link](https://drive.google.com/file/d/1niX-TC8cj6j369t7iU2baHQSVN3MVJbW/view?usp=sharing)     | [link](https://github.com/OML-Team/open-metric-learning/tree/main/pipelines/features_extraction/extractor_inshop) |\n|  `ViTExtractor.from_pretrained(\"vits16_sop\")`   | 0.866 | Stanford Online Products |   [link](https://drive.google.com/file/d/1zuGRHvF2KHd59aw7i7367OH_tQNOGz7A/view?usp=sharing)      |  [link](https://github.com/OML-Team/open-metric-learning/tree/main/pipelines/features_extraction/extractor_sop)   |\n| `ViTExtractor.from_pretrained(\"vits16_cars\")`   | 0.907 |         CARS 196         |   [link](https://drive.google.com/drive/folders/17a4_fg94dox2sfkXmw-KCtiLBlx-ut-1?usp=sharing)    |  [link](https://github.com/OML-Team/open-metric-learning/tree/main/pipelines/features_extraction/extractor_cars)  |\n|  `ViTExtractor.from_pretrained(\"vits16_cub\")`   | 0.837 |       CUB 200 2011       |   [link](https://drive.google.com/drive/folders/1TPCN-eZFLqoq4JBgnIfliJoEK48x9ozb?usp=sharing)    |  [link](https://github.com/OML-Team/open-metric-learning/tree/main/pipelines/features_extraction/extractor_cub)   |\n\nModels, trained by other researchers.\nNote, that some metrics on particular benchmarks are so high because they were part of the training dataset (for example `unicom`).\nThe metrics below are for 224 x 224 images:\n\n|                            model                             | Stanford Online Products | DeepFashion InShop | CUB 200 2011 | CARS 196 |\n|:------------------------------------------------------------:|:------------------------:|:------------------:|:------------:|:--------:|\n|    `ViTUnicomExtractor.from_pretrained(\"vitb16_unicom\")`     |          0.700           |       0.734        |    0.847     |  0.916   |\n|    `ViTUnicomExtractor.from_pretrained(\"vitb32_unicom\")`     |          0.690           |       0.722        |    0.796     |  0.893   |\n|    `ViTUnicomExtractor.from_pretrained(\"vitl14_unicom\")`     |          0.726           |       0.790        |    0.868     |  0.922   |\n| `ViTUnicomExtractor.from_pretrained(\"vitl14_336px_unicom\")`  |          0.745           |       0.810        |    0.875     |  0.924   |\n|    `ViTCLIPExtractor.from_pretrained(\"sber_vitb32_224\")`     |          0.547           |       0.514        |    0.448     |  0.618   |\n|    `ViTCLIPExtractor.from_pretrained(\"sber_vitb16_224\")`     |          0.565           |       0.565        |    0.524     |  0.648   |\n|    `ViTCLIPExtractor.from_pretrained(\"sber_vitl14_224\")`     |          0.512           |       0.555        |    0.606     |  0.707   |\n|   `ViTCLIPExtractor.from_pretrained(\"openai_vitb32_224\")`    |          0.612           |       0.491        |    0.560     |  0.693   |\n|   `ViTCLIPExtractor.from_pretrained(\"openai_vitb16_224\")`    |          0.648           |       0.606        |    0.665     |  0.767   |\n|   `ViTCLIPExtractor.from_pretrained(\"openai_vitl14_224\")`    |          0.670           |       0.675        |    0.745     |  0.844   |\n|        `ViTExtractor.from_pretrained(\"vits16_dino\")`         |          0.648           |       0.509        |    0.627     |  0.265   |\n|         `ViTExtractor.from_pretrained(\"vits8_dino\")`         |          0.651           |       0.524        |    0.661     |  0.315   |\n|        `ViTExtractor.from_pretrained(\"vitb16_dino\")`         |          0.658           |       0.514        |    0.541     |  0.288   |\n|         `ViTExtractor.from_pretrained(\"vitb8_dino\")`         |          0.689           |       0.599        |    0.506     |  0.313   |\n|       `ViTExtractor.from_pretrained(\"vits14_dinov2\")`        |          0.566           |       0.334        |    0.797     |  0.503   |\n|     `ViTExtractor.from_pretrained(\"vits14_reg_dinov2\")`      |          0.566           |       0.332        |    0.795     |  0.740   |\n|       `ViTExtractor.from_pretrained(\"vitb14_dinov2\")`        |          0.565           |       0.342        |    0.842     |  0.644   |\n|     `ViTExtractor.from_pretrained(\"vitb14_reg_dinov2\")`      |          0.557           |       0.324        |    0.833     |  0.828   |\n|       `ViTExtractor.from_pretrained(\"vitl14_dinov2\")`        |          0.576           |       0.352        |    0.844     |  0.692   |\n|     `ViTExtractor.from_pretrained(\"vitl14_reg_dinov2\")`      |          0.571           |       0.340        |    0.840     |  0.871   |\n|    `ResnetExtractor.from_pretrained(\"resnet50_moco_v2\")`     |          0.493           |       0.267        |    0.264     |  0.149   |\n| `ResnetExtractor.from_pretrained(\"resnet50_imagenet1k_v1\")`  |          0.515           |       0.284        |    0.455     |  0.247   |\n\n*The metrics may be different from the ones reported by papers,\nbecause the version of train/val split and usage of bounding boxes may differ.*\n\n## [Zoo: Texts](https://open-metric-learning.readthedocs.io/en/latest/feature_extraction/zoo.html#zoo-texts)\n\nHere is a lightweight integration with [HuggingFace Transformers](https://github.com/huggingface/transformers) models.\nYou can replace it with other arbitrary models inherited from [IExtractor](https://open-metric-learning.readthedocs.io/en/latest/contents/interfaces.html#iextractor).\n\n```shell\npip install open-metric-learning[nlp]\n```\n\n\u003cdetails style=\"padding-bottom: 15px\"\u003e\n\u003csummary\u003e\u003cb\u003eSee how to use models\u003c/b\u003e\u003c/summary\u003e\n\u003cp\u003e\n\n[comment]:zoo-text-start\n```python\nfrom transformers import AutoModel, AutoTokenizer\n\nfrom oml.models import HFWrapper\n\nmodel = AutoModel.from_pretrained('bert-base-uncased').eval()\ntokenizer = AutoTokenizer.from_pretrained('bert-base-uncased')\nextractor = HFWrapper(model=model, feat_dim=768)\n\ninp = tokenizer(text=\"Hello world\", return_tensors=\"pt\", add_special_tokens=True)\nembeddings = extractor(inp)\n```\n[comment]:zoo-text-end\n\n\u003c/p\u003e\n\u003c/details\u003e\n\nNote, we don't have our own text models zoo at the moment.\n\n## [Zoo: Audios](https://open-metric-learning.readthedocs.io/en/latest/feature_extraction/zoo.html#zoo-audios)\n\n\nYou can use an audio model from our Zoo or\nuse other arbitrary models after you inherited it from [IExtractor](https://open-metric-learning.readthedocs.io/en/latest/contents/interfaces.html#iextractor).\n\n```shell\npip install open-metric-learning[audio]\n```\n\n\u003cdetails style=\"padding-bottom: 15px\"\u003e\n\u003csummary\u003e\u003cb\u003eSee how to use models\u003c/b\u003e\u003c/summary\u003e\n\u003cp\u003e\n\n[comment]:zoo-audio-start\n```python\nimport torchaudio\n\nfrom oml.models import ECAPATDNNExtractor\nfrom oml.const import CKPT_SAVE_ROOT as CKPT_DIR, MOCK_AUDIO_DATASET_PATH as DATA_DIR\n\n# replace it by your actual paths\nckpt_path = CKPT_DIR / \"ecapa_tdnn_taoruijie.pth\"\nfile_path = DATA_DIR / \"voices\" / \"voice0_0.wav\"\n\nmodel = ECAPATDNNExtractor(weights=ckpt_path, arch=\"ecapa_tdnn_taoruijie\", normalise_features=False).to(\"cpu\").eval()\naudio, sr = torchaudio.load(file_path)\n\nif audio.shape[0] \u003e 1:\n    audio = audio.mean(dim=0, keepdim=True)  # mean by channels\nif sr != 16000:\n    audio = torchaudio.functional.resample(audio, sr, 16000)\n\nembeddings = model.extract(audio)\n```\n[comment]:zoo-audio-end\n\n\u003c/p\u003e\n\u003c/details\u003e\n\n### Audio models zoo\n\n|                            model                             | Vox1_O | Vox1_E | Vox1_H |\n|:------------------------------------------------------------:|:------:|:------:|:------:|\n| `ECAPATDNNExtractor.from_pretrained(\"ecapa_tdnn_taoruijie\")` |  0.86  |  1.18  |  2.17  |\n\n*The metrics above represent Equal Error Rate (EER). Lower is better.*\n\n## [Contributing guide](https://open-metric-learning.readthedocs.io/en/latest/oml/contributing.html)\n\nWe welcome new contributors! Please, see our:\n* [Contributing guide](https://open-metric-learning.readthedocs.io/en/latest/oml/contributing.html)\n* [Kanban board](https://github.com/OML-Team/open-metric-learning/projects/1)\n\n## Acknowledgments\n\n\u003ca href=\"https://github.com/catalyst-team/catalyst\" target=\"_blank\"\u003e\u003cimg src=\"https://raw.githubusercontent.com/catalyst-team/catalyst-pics/master/pics/catalyst_logo.png\" width=\"100\"/\u003e\u003c/a\u003e\n\nThe project was started in 2020 as a module for [Catalyst](https://github.com/catalyst-team/catalyst) library.\nI want to thank people who worked with me on that module:\n[Julia Shenshina](https://github.com/julia-shenshina),\n[Nikita Balagansky](https://github.com/elephantmipt),\n[Sergey Kolesnikov](https://github.com/Scitator)\nand others.\n\nI would like to thank people who continue working on this pipeline when it became a separate project:\n[Julia Shenshina](https://github.com/julia-shenshina),\n[Misha Kindulov](https://github.com/b0nce),\n[Aron Dik](https://github.com/dapladoc),\n[Aleksei Tarasov](https://github.com/DaloroAT) and\n[Verkhovtsev Leonid](https://github.com/leoromanovich).\n\n\u003ca href=\"https://www.newyorker.de/\" target=\"_blank\"\u003e\u003cimg src=\"https://upload.wikimedia.org/wikipedia/commons/thumb/d/d8/New_Yorker.svg/1280px-New_Yorker.svg.png\" width=\"100\"/\u003e\u003c/a\u003e\n\nI also want to thank NewYorker, since the part of functionality was developed (and used) by its computer vision team led by me.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foml-team%2Fopen-metric-learning","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Foml-team%2Fopen-metric-learning","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foml-team%2Fopen-metric-learning/lists"}