{"id":13618923,"url":"https://github.com/google-research/meta-dataset","last_synced_at":"2025-10-09T01:38:30.331Z","repository":{"id":35891347,"uuid":"174000899","full_name":"google-research/meta-dataset","owner":"google-research","description":"A dataset of datasets for learning to learn from few examples","archived":false,"fork":false,"pushed_at":"2024-05-24T13:39:34.000Z","size":3335,"stargazers_count":778,"open_issues_count":45,"forks_count":140,"subscribers_count":22,"default_branch":"main","last_synced_at":"2025-04-09T06:08:58.464Z","etag":null,"topics":["benchmark","few-shot-learning","machine-learning","maml","meta-learning","tensorflow"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/google-research.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":"AUTHORS","dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-03-05T18:39:08.000Z","updated_at":"2025-04-08T08:59:37.000Z","dependencies_parsed_at":"2024-11-08T05:31:41.447Z","dependency_job_id":"117a9a73-7ecd-4c8f-ac2b-462e22514c74","html_url":"https://github.com/google-research/meta-dataset","commit_stats":{"total_commits":302,"total_committers":29,"mean_commits":"10.413793103448276","dds":0.6390728476821192,"last_synced_commit":"13ca9ed2533056909f232168c759c096ae291740"},"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google-research%2Fmeta-dataset","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google-research%2Fmeta-dataset/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google-research%2Fmeta-dataset/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google-research%2Fmeta-dataset/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/google-research","download_url":"https://codeload.github.com/google-research/meta-dataset/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254527071,"owners_count":22085917,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["benchmark","few-shot-learning","machine-learning","maml","meta-learning","tensorflow"],"created_at":"2024-08-01T21:00:32.295Z","updated_at":"2025-10-09T01:38:25.297Z","avatar_url":"https://github.com/google-research.png","language":"Jupyter Notebook","funding_links":[],"categories":["Python","Few-Shot Learning (Classification)","Few-shot Learning with Multi-scale Self-supervision. arXiv'2001","Datasets"],"sub_categories":["Papers","[few-shot 知乎](https://zhuanlan.zhihu.com/p/58298920)","[Meta Reinforcement Learning]()"],"readme":"# NEW! TFDS API for Meta-Dataset\n\nTo accompany the presentation of the [VTAB+MD paper](https://openreview.net/pdf?id=Q0hm0_G1mpH)\nat NeurIPS 2021's Datasets and Benchmarks track, we are releasing a TensorFlow\nDatasets-based implementation of Meta-Dataset's input pipeline which is\ncompatible with both the original Meta-Dataset protocol (MD-v1) and the updated\nprotocol designed for VTAB+MD (MD-v2). See [the documentation page](meta_dataset/data/tfds/README.md)\nfor more information and example code snippets.\n\n# Meta-Dataset\n\nThis repository contains accompanying code for the article introducing\nMeta-Dataset, [arxiv.org/abs/1903.03096](https://arxiv.org/abs/1903.03096) and the follow-up paper that proposes the VTAB+MD merged benchmark [arxiv.org/abs/2104.02638](http://arxiv.org/abs/2104.02638). It also contains accompanying code and checkpoints for the CrossTransformers\n[https://arxiv.org/abs/2007.11498](https://arxiv.org/abs/2007.11498) and FLUTE [https://arxiv.org/abs/2105.07029](https://arxiv.org/abs/2105.07029) follow-up works, which improve performance.\n\nThis code is provided here in order to give more details on the implementation\nof the data-providing pipeline, our back-bones and models, as well as the\nexperimental setting.\n\nSee below for [user instructions](#user-instructions), including how to:\n\n1.  [install](#installation) the software,\n2.  [download and convert](#downloading-and-converting-datasets) the data, and\n3.  [train](#training) implemented models.\n\nSee this\n[introduction notebook](https://github.com/google-research/meta-dataset/blob/main/Intro_to_Metadataset.ipynb)\nfor a demonstration of how to sample data from the pipeline (episodes or\nbatches).\n\nIn order to run the experiments described in the first version of the arXiv\narticle, [arxiv.org/abs/1903.03096v1](https://arxiv.org/abs/1903.03096v1),\nplease use the instructions, code, and configuration files at version\n[arxiv_v1](https://github.com/google-research/meta-dataset/tree/arxiv_v1) of\nthis repository.\n\nWe are currently working on updating the instructions, code, and configuration\nfiles to reproduce the results in the second version of the article,\n[arxiv.org/abs/1903.03096v2](https://arxiv.org/abs/1903.03096v2). You can follow\nthe progess in branch\n[arxiv_v2_dev](https://github.com/google-research/meta-dataset/tree/arxiv_v2_dev)\nof this repository.\n\nThis is not an officially supported Google product.\n\n## Meta-Dataset: A Dataset of Datasets for Learning to Learn from Few Examples\n\n_Eleni Triantafillou, Tyler Zhu, Vincent Dumoulin, Pascal Lamblin, Utku Evci,\nKelvin Xu, Ross Goroshin, Carles Gelada, Kevin Swersky, Pierre-Antoine Manzagol,\nHugo Larochelle_\n\nFew-shot classification refers to learning a classifier for new classes given\nonly a few examples. While a plethora of models have emerged to tackle it, we\nfind the procedure and datasets that are used to assess their progress lacking.\nTo address this limitation, we propose Meta-Dataset: a new benchmark for\ntraining and evaluating models that is large-scale, consists of diverse\ndatasets, and presents more realistic tasks. We experiment with popular\nbaselines and meta-learners on Meta-Dataset, along with a competitive method\nthat we propose. We analyze performance as a function of various characteristics\nof test tasks and examine the models' ability to leverage diverse training\nsources for improving their generalization. We also propose a new set of\nbaselines for quantifying the benefit of meta-learning in Meta-Dataset. Our\nextensive experimentation has uncovered important research challenges and we\nhope to inspire work in these directions.\n\n## CrossTransformers: spatially-aware few-shot transfer\n\n_Carl Doersch, Ankush Gupta, Andrew Zisserman_\n\nThis is a Transformer-based neural network architecture which can find coarse\nspatial correspondence between the query and the support images, and then infer\nclass membership by computing distances between spatially-corresponding\nfeatures.  The paper also introduces SimCLR episodes, which are episodes that\nrequire SimCLR-style instance recognition, and therefore encourage features\nwhich capture more than just the training-set categories.  This algorithm is\nSOTA on Meta-Dataset (train-on-ILSVRC) as of NeurIPS 2020.\n\nConfiguration files for CrossTransformers with and without SimCLR episodes (CTX\nand CTX+SimCLR Eps from the paper) can be found in\n`learn/gin/default/crosstransformer*`.  We also have pretrained checkpoints for\nthese two configurations:\n[CTX](https://storage.googleapis.com/dm_crosstransformer/ctx.zip),\nand\n[CTX+SimCLR Eps](https://storage.googleapis.com/dm_crosstransformer/ctx_simclreps.zip),\nas well as\n[CTX+SimCLR Eps+BOHB Aug](https://storage.googleapis.com/dm_crosstransformer/ctx_simclreps_bohbaug.zip).\nNote that these were retrained from the versions reported in the paper, but\ntheir performance should be on-par.  The network structure is the same for all\nthree models, and so they can be loaded using either of the CrossTransformer\nconfig files.\n\n## Learning a Universal Template for Few-shot Dataset Generalization (FLUTE)\n_Eleni Triantafillou, Hugo Larochelle, Richard Zemel, Vincent Dumoulin\n\nFew-shot Learning with a Universal TEmplate (FLUTE) is a model designed for the \nstrong generalization challenge of few-shot learning classes from unseen\ndatasets. At the time of publication (ICML 2021), it achieved SOTA on\nMeta-Dataset (train-on-all). It works by leveraging the training datasets to \nlearn a 'universal template' that can be repurposed to solve diverse test tasks, \nby appropriately 'filling in the blanks' of the template each time, with an \nappropriate set of FiLM parameters that are learned with gradient descent in \neach test task.\n\nConfiguration files for training FLUTE, as well as the dataset classifier used\nin FLUTE's Blender network can be found in `learn/gin/default/flute.gin` and\n`learn/gin/default/flute_dataset_classifier.gin`, respectively. Configuration\nfiles for testing different variants of FLUTE can be found in\n`learn/gin/best/flute*` The results reported in the paper were obtained with\n`learn/gin/best/flute.gin`.\n\nThe training script for FLUTE is `train_flute.py`. We also have pre-trained\ncheckpoints for FLUTE and its Blender network: https://console.cloud.google.com/storage/gresearch/flute\n\n# Leaderboard (in progress)\n\nThe tables below were generated by\n[this notebook](https://github.com/google-research/meta-dataset/blob/main/Leaderboard.ipynb).\n\n## Adding a new model to the leaderboard\n\n1.  Gather accuracy results and 95% confidence intervals, as well as the number\n    of episodes used for the CI (minimum 600).\n2.  If you were affected by\n    [#54](https://github.com/google-research/meta-dataset/issues/54),\n    make sure the evaluation on Traffic Sign is done on shuffled samples. We\n    encourage you to re-train your best model (or at least perform validation\n    again) as well.\n3.  Create an\n    [issue](https://github.com/google-research/meta-dataset/issues/new),\n    with the name of the model, results, as well as the article to cite or any\n    other relevant information to include, and label it \"leaderboard\".\n    Alternatively, submit a PR with an update to the notebook.\n\n\u003c!-- Beginning of content generated by `Leaderboard.ipynb` --\u003e\n\n## Training on ImageNet only\n\nMethod                     |Avg rank                   |ILSVRC (test)              |Omniglot                   |Aircraft                   |Birds                      |Textures                   |QuickDraw                  |Fungi                      |VGG Flower                 |Traffic signs              |MSCOCO                     \n---------------------------|---------------------------|---------------------------|---------------------------|---------------------------|---------------------------|---------------------------|---------------------------|---------------------------|---------------------------|---------------------------|---------------------------\nk-NN [[1]]                 |14.6                       |41.03±1.01\u0026nbsp;(15)       |37.07±1.15\u0026nbsp;(16)       |46.81±0.89\u0026nbsp;(15)       |50.13±1.00\u0026nbsp;(15.5)     |66.36±0.75\u0026nbsp;(13)       |32.06±1.08\u0026nbsp;(16)       |36.16±1.02\u0026nbsp;(13)       |83.10±0.68\u0026nbsp;(12)       |44.59±1.19\u0026nbsp;(15)       |30.38±0.99\u0026nbsp;(15.5)     \nFinetune [[1]]             |10.45                      |45.78±1.10\u0026nbsp;(13)       |60.85±1.58\u0026nbsp;(11.5)     |68.69±1.26\u0026nbsp;(5)        |57.31±1.26\u0026nbsp;(14)       |69.05±0.90\u0026nbsp;(9.5)      |42.60±1.17\u0026nbsp;(13.5)     |38.20±1.02\u0026nbsp;(11)       |85.51±0.68\u0026nbsp;(9)        |66.79±1.31\u0026nbsp;(5)        |34.86±0.97\u0026nbsp;(13)       \nMatchingNet [[1]]          |13.55                      |45.00±1.10\u0026nbsp;(13)       |52.27±1.28\u0026nbsp;(14)       |48.97±0.93\u0026nbsp;(13)       |62.21±0.95\u0026nbsp;(12.5)     |64.15±0.85\u0026nbsp;(15)       |42.87±1.09\u0026nbsp;(13.5)     |33.97±1.00\u0026nbsp;(14)       |80.13±0.71\u0026nbsp;(15)       |47.80±1.14\u0026nbsp;(12.5)     |34.99±1.00\u0026nbsp;(13)       \nProtoNet [[1]]             |10.75                      |50.50±1.08\u0026nbsp;(10.5)     |59.98±1.35\u0026nbsp;(11.5)     |53.10±1.00\u0026nbsp;(10.5)     |68.79±1.01\u0026nbsp;(8.5)      |66.56±0.83\u0026nbsp;(13)       |48.96±1.08\u0026nbsp;(11)       |39.71±1.11\u0026nbsp;(9)        |85.27±0.77\u0026nbsp;(9)        |47.12±1.10\u0026nbsp;(14)       |41.00±1.10\u0026nbsp;(10.5)     \nfo-MAML [[1]]              |12.25                      |45.51±1.11\u0026nbsp;(13)       |55.55±1.54\u0026nbsp;(13)       |56.24±1.11\u0026nbsp;(8.5)      |63.61±1.06\u0026nbsp;(12.5)     |68.04±0.81\u0026nbsp;(9.5)      |43.96±1.29\u0026nbsp;(13.5)     |32.10±1.10\u0026nbsp;(15)       |81.74±0.83\u0026nbsp;(14)       |50.93±1.51\u0026nbsp;(10.5)     |35.30±1.23\u0026nbsp;(13)       \nRelationNet [[1]]          |15.55                      |34.69±1.01\u0026nbsp;(16)       |45.35±1.36\u0026nbsp;(15)       |40.73±0.83\u0026nbsp;(16)       |49.51±1.05\u0026nbsp;(15.5)     |52.97±0.69\u0026nbsp;(16)       |43.30±1.08\u0026nbsp;(13.5)     |30.55±1.04\u0026nbsp;(16)       |68.76±0.83\u0026nbsp;(16)       |33.67±1.05\u0026nbsp;(16)       |29.15±1.01\u0026nbsp;(15.5)     \nfo-Proto-MAML [[1]]        |9.25                       |49.53±1.05\u0026nbsp;(10.5)     |63.37±1.33\u0026nbsp;(8.5)      |55.95±0.99\u0026nbsp;(8.5)      |68.66±0.96\u0026nbsp;(8.5)      |66.49±0.83\u0026nbsp;(13)       |51.52±1.00\u0026nbsp;(9.5)      |39.96±1.14\u0026nbsp;(6.5)      |87.15±0.69\u0026nbsp;(6)        |48.83±1.09\u0026nbsp;(12.5)     |43.74±1.12\u0026nbsp;(9)        \nALFA+fo-Proto-MAML [[3]]   |7.1                        |52.80±1.11\u0026nbsp;(8.5)      |61.87±1.51\u0026nbsp;(8.5)      |63.43±1.10\u0026nbsp;(6)        |69.75±1.05\u0026nbsp;(6.5)      |70.78±0.88\u0026nbsp;(7)        |59.17±1.16\u0026nbsp;(5.5)      |41.49±1.17\u0026nbsp;(6.5)      |85.96±0.77\u0026nbsp;(9)        |60.78±1.29\u0026nbsp;(8)        |48.11±1.14\u0026nbsp;(5.5)      \nProtoNet (large) [[4]]     |7.25                       |53.69±1.07\u0026nbsp;(6)        |68.50±1.27\u0026nbsp;(5.5)      |58.04±0.96\u0026nbsp;(7)        |74.07±0.92\u0026nbsp;(4.5)      |68.76±0.77\u0026nbsp;(9.5)      |53.30±1.06\u0026nbsp;(8)        |40.73±1.15\u0026nbsp;(6.5)      |86.96±0.73\u0026nbsp;(6)        |58.11±1.05\u0026nbsp;(9)        |41.70±1.08\u0026nbsp;(10.5)     \nCTX [[4]]                  |2.75                       |62.76±0.99\u0026nbsp;(2.5)      |82.21±1.00\u0026nbsp;(2.5)      |79.49±0.89\u0026nbsp;(2.5)      |80.63±0.88\u0026nbsp;(3)        |75.57±0.64\u0026nbsp;(4)        |72.68±0.82\u0026nbsp;(2)        |51.58±1.11\u0026nbsp;(2.5)      |95.34±0.37\u0026nbsp;(2)        |82.65±0.76\u0026nbsp;(3)        |59.90±1.02\u0026nbsp;(3.5)      \nBOHB [[5]]                 |7.85                       |51.92±1.05\u0026nbsp;(8.5)      |67.57±1.21\u0026nbsp;(5.5)      |54.12±0.90\u0026nbsp;(10.5)     |70.69±0.90\u0026nbsp;(6.5)      |68.34±0.76\u0026nbsp;(9.5)      |50.33±1.04\u0026nbsp;(9.5)      |41.38±1.12\u0026nbsp;(6.5)      |87.34±0.59\u0026nbsp;(6)        |51.80±1.04\u0026nbsp;(10.5)     |48.03±0.99\u0026nbsp;(5.5)      \nSimpleCNAPS [[14],[7]]     |8.75                       |54.80±1.20\u0026nbsp;(6)        |62.00±1.30\u0026nbsp;(8.5)      |49.20±0.90\u0026nbsp;(13)       |66.50±1.00\u0026nbsp;(10.5)     |71.60±0.70\u0026nbsp;(5.5)      |56.60±1.00\u0026nbsp;(7)        |37.50±1.20\u0026nbsp;(11)       |82.10±0.90\u0026nbsp;(12)       |63.10±1.10\u0026nbsp;(6.5)      |45.80±1.00\u0026nbsp;(7.5)      \nTransductiveCNAPS [[14],[8]]|8.6                        |54.10±1.10\u0026nbsp;(6)        |62.90±1.30\u0026nbsp;(8.5)      |48.40±0.90\u0026nbsp;(13)       |67.30±0.90\u0026nbsp;(10.5)     |72.50±0.70\u0026nbsp;(5.5)      |58.00±1.00\u0026nbsp;(5.5)      |37.70±1.10\u0026nbsp;(11)       |82.80±0.80\u0026nbsp;(12)       |61.80±1.10\u0026nbsp;(6.5)      |45.80±1.00\u0026nbsp;(7.5)      \nTSA_resnet18 [[12]]        |3.8                        |59.50±1.10\u0026nbsp;(4)        |78.20±1.20\u0026nbsp;(4)        |72.20±1.00\u0026nbsp;(4)        |74.90±0.90\u0026nbsp;(4.5)      |77.30±0.70\u0026nbsp;(3)        |67.60±0.90\u0026nbsp;(4)        |44.70±1.00\u0026nbsp;(4)        |90.90±0.60\u0026nbsp;(4)        |82.50±0.80\u0026nbsp;(3)        |59.00±1.00\u0026nbsp;(3.5)      \nTSA_resnet34 [[12]]        |2.5                        |63.73±0.99\u0026nbsp;(2.5)      |82.58±1.11\u0026nbsp;(2.5)      |80.13±1.01\u0026nbsp;(2.5)      |83.39±0.80\u0026nbsp;(2)        |79.61±0.68\u0026nbsp;(2)        |71.03±0.84\u0026nbsp;(3)        |51.38±1.17\u0026nbsp;(2.5)      |94.05±0.45\u0026nbsp;(3)        |81.71±0.95\u0026nbsp;(3)        |61.67±0.95\u0026nbsp;(2)        \nDIPA [[15]]                |**1**                      |**71.40**±0.90\u0026nbsp;(1)    |**84.30**±1.20\u0026nbsp;(1)    |**86.70**±1.00\u0026nbsp;(1)    |**88.20**±0.90\u0026nbsp;(1)    |**87.10**±0.60\u0026nbsp;(1)    |**74.60**±0.80\u0026nbsp;(1)    |**61.40**±1.20\u0026nbsp;(1)    |**97.40**±0.40\u0026nbsp;(1)    |**88.90**±1.00\u0026nbsp;(1)    |**65.20**±1.00\u0026nbsp;(1)    \n\n## Training on all datasets\n\nMethod                     |Avg rank                   |ILSVRC (test)              |Omniglot                   |Aircraft                   |Birds                      |Textures                   |QuickDraw                  |Fungi                      |VGG Flower                 |Traffic signs              |MSCOCO                     \n---------------------------|---------------------------|---------------------------|---------------------------|---------------------------|---------------------------|---------------------------|---------------------------|---------------------------|---------------------------|---------------------------|---------------------------\nk-NN [[1]]                 |16.85                      |38.55±0.94\u0026nbsp;(16.5)     |74.60±1.08\u0026nbsp;(18)       |64.98±0.82\u0026nbsp;(19)       |66.35±0.92\u0026nbsp;(14.5)     |63.58±0.79\u0026nbsp;(15.5)     |44.88±1.05\u0026nbsp;(19)       |37.12±1.06\u0026nbsp;(15.5)     |83.47±0.61\u0026nbsp;(15.5)     |40.11±1.10\u0026nbsp;(18)       |29.55±0.96\u0026nbsp;(17)       \nFinetune [[1]]             |14.1                       |43.08±1.08\u0026nbsp;(14.5)     |71.11±1.37\u0026nbsp;(19)       |72.03±1.07\u0026nbsp;(15.5)     |59.82±1.15\u0026nbsp;(17)       |69.14±0.85\u0026nbsp;(9.5)      |47.05±1.16\u0026nbsp;(18)       |38.16±1.04\u0026nbsp;(15.5)     |85.28±0.69\u0026nbsp;(14)       |66.74±1.23\u0026nbsp;(3)        |35.17±1.08\u0026nbsp;(15)       \nMatchingNet [[1]]          |16.4                       |36.08±1.00\u0026nbsp;(18)       |78.25±1.01\u0026nbsp;(16.5)     |69.17±0.96\u0026nbsp;(17.5)     |56.40±1.00\u0026nbsp;(18)       |61.80±0.74\u0026nbsp;(17)       |60.81±1.03\u0026nbsp;(15.5)     |33.70±1.04\u0026nbsp;(18)       |81.90±0.72\u0026nbsp;(17)       |55.57±1.08\u0026nbsp;(9.5)      |28.79±0.96\u0026nbsp;(17)       \nProtoNet [[1]]             |14.5                       |44.50±1.05\u0026nbsp;(14.5)     |79.56±1.12\u0026nbsp;(16.5)     |71.14±0.86\u0026nbsp;(15.5)     |67.01±1.02\u0026nbsp;(14.5)     |65.18±0.84\u0026nbsp;(13.5)     |64.88±0.89\u0026nbsp;(14)       |40.26±1.13\u0026nbsp;(14)       |86.85±0.71\u0026nbsp;(13)       |46.48±1.00\u0026nbsp;(16)       |39.87±1.06\u0026nbsp;(13.5)     \nfo-MAML [[1]]              |16.15                      |37.83±1.01\u0026nbsp;(16.5)     |83.92±0.95\u0026nbsp;(13.5)     |76.41±0.69\u0026nbsp;(13)       |62.43±1.08\u0026nbsp;(16)       |64.16±0.83\u0026nbsp;(15.5)     |59.73±1.10\u0026nbsp;(17)       |33.54±1.11\u0026nbsp;(18)       |79.94±0.84\u0026nbsp;(18)       |42.91±1.31\u0026nbsp;(17)       |29.37±1.08\u0026nbsp;(17)       \nRelationNet [[1]]          |17.7                       |30.89±0.93\u0026nbsp;(19)       |86.57±0.79\u0026nbsp;(12)       |69.71±0.83\u0026nbsp;(17.5)     |54.14±0.99\u0026nbsp;(19)       |56.56±0.73\u0026nbsp;(19)       |61.75±0.97\u0026nbsp;(15.5)     |32.56±1.08\u0026nbsp;(18)       |76.08±0.76\u0026nbsp;(19)       |37.48±0.93\u0026nbsp;(19)       |27.41±0.89\u0026nbsp;(19)       \nfo-Proto-MAML [[1]]        |12.65                      |46.52±1.05\u0026nbsp;(13)       |82.69±0.97\u0026nbsp;(15)       |75.23±0.76\u0026nbsp;(14)       |69.88±1.02\u0026nbsp;(12.5)     |68.25±0.81\u0026nbsp;(11.5)     |66.84±0.94\u0026nbsp;(13)       |41.99±1.17\u0026nbsp;(13)       |88.72±0.67\u0026nbsp;(11)       |52.42±1.08\u0026nbsp;(12.5)     |41.74±1.13\u0026nbsp;(11)       \nCNAPs [[2]]                |11.15                      |50.80±1.10\u0026nbsp;(11.5)     |91.70±0.50\u0026nbsp;(8)        |83.70±0.60\u0026nbsp;(8.5)      |73.60±0.90\u0026nbsp;(11)       |59.50±0.70\u0026nbsp;(18)       |74.70±0.80\u0026nbsp;(12)       |50.20±1.10\u0026nbsp;(8.5)      |88.90±0.50\u0026nbsp;(11)       |56.50±1.10\u0026nbsp;(9.5)      |39.40±1.00\u0026nbsp;(13.5)     \nSUR [[6]]                  |8.45                       |56.10±1.10\u0026nbsp;(8)        |93.10±0.50\u0026nbsp;(5.5)      |84.60±0.70\u0026nbsp;(6.5)      |70.60±1.00\u0026nbsp;(12.5)     |71.00±0.80\u0026nbsp;(7.5)      |81.30±0.60\u0026nbsp;(4)        |64.20±1.10\u0026nbsp;(4.5)      |82.80±0.80\u0026nbsp;(15.5)     |53.40±1.00\u0026nbsp;(12.5)     |50.10±1.00\u0026nbsp;(8)        \nSUR-pnf [[6]]              |9                          |56.00±1.10\u0026nbsp;(8)        |90.00±0.60\u0026nbsp;(10.5)     |79.70±0.80\u0026nbsp;(11.5)     |75.90±0.90\u0026nbsp;(8.5)      |72.50±0.70\u0026nbsp;(5.5)      |76.70±0.70\u0026nbsp;(8.5)      |49.80±1.10\u0026nbsp;(8.5)      |90.00±0.60\u0026nbsp;(8.5)      |52.20±0.80\u0026nbsp;(12.5)     |50.20±1.10\u0026nbsp;(8)        \nSimpleCNAPS [[14],[7]]     |8.25                       |56.50±1.10\u0026nbsp;(8)        |91.90±0.60\u0026nbsp;(8)        |83.80±0.60\u0026nbsp;(8.5)      |76.10±0.90\u0026nbsp;(8.5)      |70.00±0.80\u0026nbsp;(9.5)      |78.30±0.70\u0026nbsp;(6.5)      |49.10±1.20\u0026nbsp;(8.5)      |91.30±0.60\u0026nbsp;(7)        |59.20±1.00\u0026nbsp;(7)        |42.40±1.10\u0026nbsp;(11)       \nTransductiveCNAPS [[14],[8]]|6.85                       |57.90±1.10\u0026nbsp;(3.5)      |94.30±0.40\u0026nbsp;(3.5)      |84.70±0.50\u0026nbsp;(6.5)      |78.80±0.70\u0026nbsp;(4.5)      |66.20±0.80\u0026nbsp;(13.5)     |77.90±0.60\u0026nbsp;(6.5)      |48.90±1.20\u0026nbsp;(8.5)      |92.30±0.40\u0026nbsp;(4)        |59.70±1.10\u0026nbsp;(7)        |42.50±1.10\u0026nbsp;(11)       \nURT [[9]]                  |6.85                       |55.70±1.00\u0026nbsp;(8)        |94.40±0.40\u0026nbsp;(3.5)      |85.80±0.60\u0026nbsp;(5)        |76.30±0.80\u0026nbsp;(8.5)      |71.80±0.70\u0026nbsp;(5.5)      |**82.50**±0.60\u0026nbsp;(2)    |63.50±1.00\u0026nbsp;(4.5)      |88.20±0.60\u0026nbsp;(11)       |51.10±1.10\u0026nbsp;(15)       |52.20±1.10\u0026nbsp;(5.5)      \nURT-pf [[9]]               |8.4                        |55.50±1.10\u0026nbsp;(8)        |90.20±0.60\u0026nbsp;(10.5)     |79.80±0.70\u0026nbsp;(11.5)     |77.50±0.80\u0026nbsp;(6)        |73.50±0.70\u0026nbsp;(4)        |75.80±0.70\u0026nbsp;(10.5)     |48.10±0.90\u0026nbsp;(11.5)     |91.90±0.50\u0026nbsp;(4)        |52.00±1.40\u0026nbsp;(12.5)     |52.10±1.00\u0026nbsp;(5.5)      \nFLUTE [[10]]               |6.65                       |51.80±1.10\u0026nbsp;(11.5)     |93.20±0.50\u0026nbsp;(5.5)      |87.20±0.50\u0026nbsp;(3.5)      |79.20±0.80\u0026nbsp;(4.5)      |68.80±0.80\u0026nbsp;(11.5)     |79.50±0.70\u0026nbsp;(5)        |58.10±1.10\u0026nbsp;(6)        |91.60±0.60\u0026nbsp;(4)        |58.40±1.10\u0026nbsp;(7)        |50.00±1.00\u0026nbsp;(8)        \nURL [[11]]                 |2.75                       |57.51±1.08\u0026nbsp;(3.5)      |**94.51**±0.41\u0026nbsp;(1.5)  |88.59±0.46\u0026nbsp;(2)        |80.54±0.69\u0026nbsp;(2.5)      |76.17±0.67\u0026nbsp;(2.5)      |**81.94**±0.56\u0026nbsp;(2)    |**68.75**±0.95\u0026nbsp;(1.5)  |92.11±0.48\u0026nbsp;(4)        |63.34±1.19\u0026nbsp;(4.5)      |54.03±0.96\u0026nbsp;(3.5)      \nTSA [[12]]                 |**2.25**                   |57.35±1.05\u0026nbsp;(3.5)      |**94.96**±0.38\u0026nbsp;(1.5)  |**89.33**±0.44\u0026nbsp;(1)    |81.42±0.74\u0026nbsp;(2.5)      |76.74±0.72\u0026nbsp;(2.5)      |**82.01**±0.57\u0026nbsp;(2)    |**67.40**±0.99\u0026nbsp;(1.5)  |92.18±0.52\u0026nbsp;(4)        |83.55±0.90\u0026nbsp;(2)        |55.75±1.06\u0026nbsp;(2)        \nTriM [[13]]                |7.4                        |58.60±1.00\u0026nbsp;(3.5)      |92.00±0.60\u0026nbsp;(8)        |82.80±0.70\u0026nbsp;(10)       |75.30±0.80\u0026nbsp;(8.5)      |71.20±0.80\u0026nbsp;(7.5)      |77.30±0.70\u0026nbsp;(8.5)      |48.50±1.00\u0026nbsp;(11.5)     |90.50±0.50\u0026nbsp;(8.5)      |63.00±1.00\u0026nbsp;(4.5)      |52.80±1.10\u0026nbsp;(3.5)      \nDIPA [[15]]                |3.65                       |**70.90**±1.00\u0026nbsp;(1)    |84.70±1.10\u0026nbsp;(13.5)     |86.30±1.00\u0026nbsp;(3.5)      |**90.80**±0.80\u0026nbsp;(1)    |**88.60**±0.50\u0026nbsp;(1)    |75.30±0.80\u0026nbsp;(10.5)     |66.60±1.10\u0026nbsp;(3)        |**97.90**±0.30\u0026nbsp;(1)    |**91.30**±1.00\u0026nbsp;(1)    |**64.80**±1.00\u0026nbsp;(1)    \n\n## References\n\n[1]: #1-triantafillou-et-al-2020\n[2]: #2-requeima-et-al-2019\n[3]: #3-baik-et-al-2020\n[4]: #4-doersch-et-al-2020\n[5]: #5-saikia-et-al-2020\n[6]: #6-dvornik-et-al-2020\n[7]: #7-bateni-et-al-2020\n[8]: #8-bateni-et-al-2022a\n[9]: #9-liu-et-al-2021a\n[10]: #10-triantafillou-et-al-2021\n[11]: #11-li-et-al-2021a\n[12]: #12-li-et-al-2021b\n[13]: #13-liu-et-al-2021b\n[14]: #14-bateni-et-al-2022b\n[15]: #15-perera-halgamuge-2024\n\n###### \\[1\\] Triantafillou et al. (2020)\n\nEleni Triantafillou, Tyler Zhu, Vincent Dumoulin, Pascal Lamblin, Utku Evci, Kelvin Xu, Ross Goroshin, Carles Gelada, Kevin Swersky, Pierre-Antoine Manzagol, Hugo Larochelle; [_Meta-Dataset: A Dataset of Datasets for Learning to Learn from Few Examples_](https://arxiv.org/abs/1903.03096); ICLR 2020.\n\n\n###### \\[2\\] Requeima et al. (2019)\n\nJames Requeima, Jonathan Gordon, John Bronskill, Sebastian Nowozin, Richard E. Turner; [_Fast and Flexible Multi-Task Classification Using Conditional Neural Adaptive Processes_](https://arxiv.org/abs/1906.07697); NeurIPS 2019.\n\n\n###### \\[3\\] Baik et al. (2020)\n\nSungyong Baik, Myungsub Choi, Janghoon Choi, Heewon Kim, Kyoung Mu Lee; [_Meta-Learning with Adaptive Hyperparameters_](https://papers.nips.cc/paper/2020/hash/ee89223a2b625b5152132ed77abbcc79-Abstract.html); NeurIPS 2020.\n\n\n###### \\[4\\] Doersch et al. (2020)\n\nCarl Doersch, Ankush Gupta, Andrew Zisserman; [_CrossTransformers: spatially-aware few-shot transfer_](https://arxiv.org/abs/2007.11498); NeurIPS 2020.\n\n\n###### \\[5\\] Saikia et al. (2020)\n\nTonmoy Saikia, Thomas Brox, Cordelia Schmid; [_Optimized Generic Feature Learning for Few-shot Classification across Domains_](https://arxiv.org/abs/2001.07926); arXiv 2020.\n\n\n###### \\[6\\] Dvornik et al. (2020)\n\nNikita Dvornik, Cordelia Schmid, Julien Mairal; [_Selecting Relevant Features from a Multi-domain Representation for Few-shot Classification_](https://arxiv.org/abs/2003.09338); ECCV 2020.\n\n\n###### \\[7\\] Bateni et al. (2020)\n\nPeyman Bateni, Raghav Goyal, Vaden Masrani, Frank Wood, Leonid Sigal; [_Improved Few-Shot Visual Classification_](https://openaccess.thecvf.com/content_CVPR_2020/html/Bateni_Improved_Few-Shot_Visual_Classification_CVPR_2020_paper.html); CVPR 2020.\n\n\n###### \\[8\\] Bateni et al. (2022a)\n\nPeyman Bateni, Jarred Barber, Jan-Willem van de Meent, Frank Wood; [_Enhancing Few-Shot Image Classification with Unlabelled Examples_](https://openaccess.thecvf.com/content/WACV2022/html/Bateni_Enhancing_Few-Shot_Image_Classification_With_Unlabelled_Examples_WACV_2022_paper.html); WACV 2022.\n\n\n###### \\[9\\] Liu et al. (2021a)\n\nLu Liu, William Hamilton, Guodong Long, Jing Jiang, Hugo Larochelle; [_Universal Representation Transformer Layer for Few-Shot Image Classification_](https://arxiv.org/abs/2006.11702); ICLR 2021.\n\n\n###### \\[10\\] Triantafillou et al. (2021)\n\nEleni Triantafillou, Hugo Larochelle, Richard Zemel, Vincent Dumoulin; [_Learning a Universal Template for Few-shot Dataset Generalization_](https://arxiv.org/abs/2105.07029); ICML 2021.\n\n\n###### \\[11\\] Li et al. (2021a)\n\nWei-Hong Li, Xialei Liu, Hakan Bilen; [_Universal Representation Learning from Multiple Domains for Few-shot Classification_](https://arxiv.org/pdf/2103.13841.pdf); ICCV 2021.\n\n\n###### \\[12\\] Li et al. (2021b)\n\nWei-Hong Li, Xialei Liu, Hakan Bilen; [_Cross-domain Few-shot Learning with Task-specific Adapters_](https://arxiv.org/pdf/2107.00358.pdf); arXiv 2021.\n\n\n###### \\[13\\] Liu et al. (2021b)\n\nYanbin Liu, Juho Lee, Linchao Zhu, Ling Chen, Humphrey Shi, Yi Yang; [_A Multi-Mode Modulator for Multi-Domain Few-Shot Classification_](https://openaccess.thecvf.com/content/ICCV2021/papers/Liu_A_Multi-Mode_Modulator_for_Multi-Domain_Few-Shot_Classification_ICCV_2021_paper.pdf); ICCV 2021.\n\n\n###### \\[14\\] Bateni et al. (2022b)\n\nBateni Peyman, Jarred Barber, Raghav Goyal, Vaden Masrani, Jan-Willem van de Meent, Leonid Sigal, and Frank Wood.; [_Beyond Simple Meta-Learning: Multi-Purpose Models for Multi-Domain, Active and Continual Few-Shot Learning._](https://arxiv.org/abs/2201.05151); arXiv 2022.\n\n\n###### \\[15\\] Perera \u0026 Halgamuge (2024)\n\nRashindrie Perera, Saman Halgamuge;[Discriminative Sample-Guided and Parameter-Efficient Feature Space Adaptation for Cross-Domain Few-Shot Learning](https://arxiv.org/abs/2403.04492);To appear in CVPR 2024.\n\n\n\u003c!-- End of content generated by `Leaderboard.ipynb` --\u003e\n\n# User instructions\n\n## Installation\n\nMeta-Dataset is generally compatible with Python 2 and Python 3, but some parts\nof the code may require Python 3. The code works with TensorFlow 2, although it\nmakes extensive use of `tf.compat.v1` internally.\n\n-   We recommend you follow\n    [these instructions](https://www.tensorflow.org/install/pip) to install\n    TensorFlow.\n-   A list of packages to install is available in `requirements.txt`, you can\n    install them using `pip`.\n-   Clone the `meta-dataset` repository. Most command lines start with `python\n    -m meta_dataset.\u003csomething\u003e`, and should be typed from within that clone\n    (where a `meta_dataset` Python module should be visible).\n-   To reproduce the CrossTransformers training, you will need data augmentation\n    code from [simclr](https://github.com/google-research/simclr), which is\n    autimatically downloaded by `setup.py`.\n\n## Downloading and converting datasets\n\nMeta-Dataset uses several established datasets, that are available from\ndifferent sources. You can find below a summary of these datasets, as well as\ninstructions to download them and convert them into a common format.\n\nIn addition to the datasets below, the FLUTE paper reported results on 3 extra\ndatasets, following recent work. You can find instructions for downloading and \nconverting those 3 additional datasets (MNIST, CIFAR-10 and CIFAR-100) in the\nCNAPs [repo](https://github.com/cambridge-mlg/cnaps).\n\nFor brevity of the command line examples, we assume the following environment\nvariables are defined:\n\n-   `$DATASRC`: root of where the original data is downloaded and potentially\n    extracted from compressed files. This directory does not need to be\n    available after the data conversion is done.\n-   `$SPLITS`: directory where `*_splits.json` files will be created, one per\n    dataset. For instance, `$SPLITS/fungi_splits.json` contains information\n    about which classes are part of the meta-training, meta-validation, and\n    meta-test set. These files are only used during the dataset conversion\n    phase, but can help troubleshooting later. To re-use the\n    [canonical splits](https://github.com/google-research/meta-dataset/tree/main/meta_dataset/dataset_conversion/splits)\n    instead of re-generating them, you can make it point to\n    `meta_dataset/dataset_conversion` in your checkout.\n-   `$RECORDS`: root directory that will contain the converted datasets (one per\n    sub-directory). This directory needs to be available during training and\n    evaluation.\n\n### Dataset summary\n\nDataset (other names)                                                                                                                        | Number of classes (train/valid/test)    | Size on disk                 | Conversion time\n-------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------- | ---------------------------- | ---------------\nilsvrc\\_2012 (ImageNet, ILSVRC) \\[[instructions](doc/dataset_conversion.md#ilsvrc_2012)\\]                                                  | 1000 (712/158/130, hierarchical)        | \\~140 GiB                    | 5 to 13 hours\nomniglot \\[[instructions](doc/dataset_conversion.md#omniglot)\\]                                                                            | 1623 (883/81/659, by alphabet: 25/5/20) | \\~60 MiB                     | few seconds\naircraft (FGVC-Aircraft) \\[[instructions](doc/dataset_conversion.md#aircraft)\\]                                                            | 100 (70/15/15)                          | \\~470 MiB (2.6 GiB download) | 5 to 10 minutes\ncu\\_birds (Birds, CUB-200-2011) \\[[instructions](doc/dataset_conversion.md#cu_birds)\\]                                                     | 200 (140/30/30)                         | \\~1.1 GiB                    | \\~1 minute\ndtd (Describable Textures, DTD) \\[[instructions](doc/dataset_conversion.md#dtd)\\]                                                          | 47 (33/7/7)                             | \\~600 MiB                    | few seconds\nquickdraw (Quick, Draw!) \\[[instructions](doc/dataset_conversion.md#quickdraw)\\]                                                           | 345 (241/52/52)                         | \\~50 GiB                     | 3 to 4 hours\nfungi (FGVCx Fungi) \\[[instructions](doc/dataset_conversion.md#fungi)\\]                                                                    | 1394 (994/200/200)                      | \\~13 GiB                     | 5 to 15 minutes\nvgg\\_flower (VGG Flower) \\[[instructions](doc/dataset_conversion.md#vgg_flower)\\]                                                          | 102 (71/15/16)                          | \\~330 MiB                    | \\~1 minute\ntraffic\\_sign (Traffic Signs, German Traffic Sign Recognition Benchmark, GTSRB) \\[[instructions](doc/dataset_conversion.md#traffic_sign)\\] | 43 (0/0/43, test only)                  | \\~50 MiB (263 MiB download)  | \\~1 minute\nmscoco (Common Objects in Context, COCO) \\[[instructions](doc/dataset_conversion.md#mscoco)\\]                                              | 80 (0/40/40, validation and test only)  | \\~5.3 GiB (18 GiB download)  | 4 hours\n*Total (All datasets)*                                                                                                                       | *4934 (3144/598/1192)*                  | *\\~210 GiB*                  | *12 to 24 hours*\n\n### Meta-Dataset-v2\nIn order to make the combined benchmark (VTAB+MD) compatible with each other, Meta-Dataset-v2 makes some changes on the existing pipelines. When converting the ImageNet dataset please use `ilsvrc\\_2012\\_v2` \\([instructions](doc/dataset_conversion.md#ilsvrc_2012)\\) in order to make it a training only dataset. Also,`VGG Flowers` is reserved as a VTAB task in VTAB+MD, so there is no need to convert it. For more details check the [paper](http://arxiv.org/abs/2104.02638).\n\nIn order to run existing meta-learners with the updated training, validation and test classes you can refer to the `learn/gin/setups/imagenet_v2.gin` `learn/gin/setups/all_v2.gin`. These files are meant to be drop in replacements for `learn/gin/setups/imagenet.gin` and `learn/gin/setups/all.gin` files respectively.\n## Training\n\nExperiments are defined via [gin](google/gin-config) configuration files, that\nare under `meta_dataset/learn/gin/`:\n\n-   `setups/` contain generic setups for classes of experiment, for instance\n    which datasets to use (`imagenet` or `all`), parameters for sampling the\n    number of ways and shots of episodes.\n-   `models/` define settings for different meta-learning algorithms (baselines,\n    prototypical networks, MAML...)\n-   `default/` contains files that each correspond to one experiment, mostly\n    defining a setup and a model, with default values for training\n    hyperparameters.\n-   `best/` contains files with values for training hyperparameters that\n    achieved the best performance during hyperparameter search.\n\nThere are three main architectures, also called \"backbones\" (or \"embedding\nnetworks\"): `four_layer_convnet` (sometimes `convnet` for short), `resnet`, and\n`wide_resnet`. These architectures can be used by all baselines and episodic\nmodels. Another backbone, `relationnet_convnet` (similar to `four_layer_convnet`\nbut without pooling on the last layer), is only used by RelationNet (and\nbaseline, for pre-training purposes).  CrossTransformers use a larger backbone\n`resnet34`, which is similar to `resnet` but with more layers.\n\n### Reproducing results\n\nSee [Reproducing best results](doc/reproducing_best_results.md) for\ninstructions to launch training experiments with the values of hyperparameters\nthat were selected in the paper. The hyperparameters (including the backbone,\nwhether to train from scratch or from pre-trained weights, and the number of\ntraining updates) were selected using only the validation classes of the ILSVRC\n2012 dataset for all experiments. Even when training on \"all\" datasets, the\nvalidation classes of the other datasets were not used.\n\n### Adding `task_adaptation` code to the path\nIn order to use `data.read_episodes` module you need to get task_adaptation\ncode. You can do that by running following code.\n```bash\ngit clone https://github.com/google-research/task_adaptation.git\nexport PYTHONPATH=$PYTHONPATH:$PWD\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgoogle-research%2Fmeta-dataset","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgoogle-research%2Fmeta-dataset","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgoogle-research%2Fmeta-dataset/lists"}