{"id":13737851,"url":"https://github.com/dbolya/parc","last_synced_at":"2025-07-11T12:35:58.864Z","repository":{"id":107926062,"uuid":"418781286","full_name":"dbolya/parc","owner":"dbolya","description":"A benchmark suite for Scalable Diverse Model Selection for Accessible Transfer Learning from our NeurIPS 2021 paper.","archived":false,"fork":false,"pushed_at":"2022-12-14T04:02:36.000Z","size":42,"stargazers_count":15,"open_issues_count":1,"forks_count":3,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-04-09T15:27:23.152Z","etag":null,"topics":["deep-learning","machine-learning","model-selection","transfer-learning"],"latest_commit_sha":null,"homepage":"https://dbolya.github.io/parc","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dbolya.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2021-10-19T05:21:11.000Z","updated_at":"2024-01-04T17:02:29.000Z","dependencies_parsed_at":"2023-04-01T14:03:04.790Z","dependency_job_id":null,"html_url":"https://github.com/dbolya/parc","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/dbolya/parc","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dbolya%2Fparc","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dbolya%2Fparc/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dbolya%2Fparc/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dbolya%2Fparc/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dbolya","download_url":"https://codeload.github.com/dbolya/parc/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dbolya%2Fparc/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":264808153,"owners_count":23666953,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deep-learning","machine-learning","model-selection","transfer-learning"],"created_at":"2024-08-03T03:02:03.197Z","updated_at":"2025-07-11T12:35:58.840Z","avatar_url":"https://github.com/dbolya.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# PARC for Scalable Diverse Model Selection\n\nHere, we present a set of benchmarks for Scalable Diverse Model Selection.\nWe also include our method, PARC, as a good baseline on this benchmark.\n\nThis is the code for our NeurIPS 2021 paper available [here](https://arxiv.org/abs/2111.06977).\n\n\n## Installing the Benchmark\nTo install, first clone this repo:\n```\ngit clone https://github.com/dbolya/parc.git\ncd parc\n```\n\nMake sure your python version is at least 3.7.\nThen install the requirements:\n```\npip install -r requirements.txt\n```\n\nFinally, download the cached probe sets from here:   [500 Image Probe Sets](https://www.dropbox.com/s/l08n4ejuip2b1h6/probes.zip?dl=0)\n\nThen extract the probes into `./cache/` (or symlink it there):\n```\nunzip probes.zip -d ./cache/\n```\n\nVerify that the probe set exists\n```\nls ./cache/probes/fixed_budget_500 | head\n```\nYou should see a couple of probe sets `pkl` files.\n\nAnd you're done! If you want to create your own probe sets see the `Advanced` section below.\n\n\n\n## Evaluation\nSee `demo.py` for an example of how to perform evaluation:\n```py\nfrom evaluate import Experiment\nfrom metrics import MetricEval\n\nfrom methods import PARC, kNN\n\n\n# Set up the methods to use.\n# To define your own method, inherit methods.TransferabilityMethod. See the methods in methods.py for more details.\nmy_methods = {\n\t'PARC f=32': PARC(n_dims=32),\n\t'1-NN CV'  : kNN(k=1)\n}\n\nexperiment = Experiment(my_methods, name='test', append=False) # Set up an experiment with those methods named \"test\".\n                                                               # Append=True skips evaluations that already happend. Setting it to False will overwrite.\nexperiment.run()                                               # Run the experiment and save results to ./results/{name}.csv\n\nmetric = MetricEval(experiment.out_file)                       # Load the experiment file we just created with the default oracle\nmetric.add_plasticity()                                        # Adds the \"capacity to learn\" heuristic defined in the paper\nmean, variance, _all = metric.aggregate()                      # Compute metrics and aggregate them\n\n# Prints {'PARC f=32': 70.27800205353863, '1-NN CV': 68.01407390300884}. Same as Table 4 in the paper.\nprint(mean)\n```\n\n\n\nIf you wish to use the extended set of transfers (using crowd-sourced models), pass `model_bank='all'` to the experiment and pass `oracle_path='./oracles/all.csv'` when creating the metric evaluation object.\n\n\n\n## PARC\nIf you wish to use PARC to recommend models for you, PARC is defined in `methods.py`. We don't have a well supported way for you to pass arbitrary data in yet, but as long as you pass in everything required in `TransferabilityMethod`, you should be fine.\n\n\n## Advanced\nIf you want the trained models, they are available here:\n[All Trained Models](https://www.dropbox.com/s/gk32wdqmf19lnmt/models.zip?dl=0). Note that this only includes the models we trained from scratch, not the crowd sourced models.\n\nIf you want to create the probe sets yourself, put / symlink the datasets as `./data/{dataset}/`. Then put the models above in `./models/`. This is not necessary if you use the pre-extracted probe sets instead.\n\n\n## Citation\nIf you used PARC, this benchmark, or this code your work, please cite:\n```\n@inproceedings{parc-neurips2021,\n  author    = {Daniel Bolya and Rohit Mittapalli and Judy Hoffman},\n  title     = {Scalable Diverse Model Selection for Accessible Transfer Learning},\n  booktitle = {NeurIPS},\n  year      = {2021},\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdbolya%2Fparc","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdbolya%2Fparc","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdbolya%2Fparc/lists"}