{"id":13738301,"url":"https://github.com/KevinMusgrave/powerful-benchmarker","last_synced_at":"2025-05-08T16:33:10.864Z","repository":{"id":37096667,"uuid":"220663367","full_name":"KevinMusgrave/powerful-benchmarker","owner":"KevinMusgrave","description":"A library for ML benchmarking. It's powerful.","archived":false,"fork":false,"pushed_at":"2024-01-10T10:56:01.000Z","size":25587,"stargazers_count":438,"open_issues_count":3,"forks_count":43,"subscribers_count":10,"default_branch":"domain-adaptation","last_synced_at":"2025-04-13T00:41:30.318Z","etag":null,"topics":["benchmarking","computer-vision","deep-learning","domain-adaptation","machine-learning","metric-learning","pytorch","transfer-learning"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/KevinMusgrave.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2019-11-09T15:36:55.000Z","updated_at":"2025-03-12T00:00:17.000Z","dependencies_parsed_at":"2024-01-13T10:40:54.609Z","dependency_job_id":null,"html_url":"https://github.com/KevinMusgrave/powerful-benchmarker","commit_stats":{"total_commits":861,"total_committers":3,"mean_commits":287.0,"dds":"0.0023228803716608404","last_synced_commit":"009150a46497cd5738a9d794e102665f44de0a3b"},"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KevinMusgrave%2Fpowerful-benchmarker","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KevinMusgrave%2Fpowerful-benchmarker/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KevinMusgrave%2Fpowerful-benchmarker/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KevinMusgrave%2Fpowerful-benchmarker/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/KevinMusgrave","download_url":"https://codeload.github.com/KevinMusgrave/powerful-benchmarker/tar.gz/refs/heads/domain-adaptation","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253105423,"owners_count":21855023,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["benchmarking","computer-vision","deep-learning","domain-adaptation","machine-learning","metric-learning","pytorch","transfer-learning"],"created_at":"2024-08-03T03:02:17.626Z","updated_at":"2025-05-08T16:33:10.365Z","avatar_url":"https://github.com/KevinMusgrave.png","language":"Jupyter Notebook","readme":"\u003ch1\u003e\n\u003ca href=\"https://github.com/KevinMusgrave/powerful-benchmarker\"\u003e\n\u003cimg alt=\"Powerful Benchmarker\" src=\"https://github.com/KevinMusgrave/powerful-benchmarker/blob/domain-adaptation/imgs/Logo.png\"\u003e\n\u003c/a\u003e\n\u003c/h1\u003e\n\n## Which git branch should you checkout?\n\n- The [domain-adaptation](https://github.com/KevinMusgrave/powerful-benchmarker/tree/domain-adaptation) branch contains code for:\n  - [Three New Validators and a Large-Scale Benchmark Ranking for Unsupervised Domain Adaptation](https://arxiv.org/pdf/2208.07360.pdf)\n- The [metric-learning](https://github.com/KevinMusgrave/powerful-benchmarker/tree/metric-learning) branch contains code for:\n  - [A Metric Learning Reality Check](https://arxiv.org/pdf/2003.08505.pdf)\n  \nCurrently I can provide technical support (help with code, bug fixes etc.) for the `domain-adaptation` branch only.\n\n## Installation\n\nClone this repo:\n```bash\ngit clone https://github.com/KevinMusgrave/powerful-benchmarker.git\n```\n\nThen go into the folder and install the required packages:\n```bash\ncd powerful-benchmarker\npip install -r requirements.txt\n```\n\n## Set paths in `constants.yaml`\n\n- `exp_folder`: experiments will be saved as sub-folders inside of `exp_folder`\n- `dataset_folder`: datasets will be downloaded here. For example, `\u003cdataset_folder\u003e/mnistm`\n- `conda_env`: (optional) the conda environment that will be activated for slurm jobs\n- `slurm_folder`: slurm logs will be saved to `\u003cexp_folder\u003e/.../\u003cslurm_folder\u003e`\n- `gdrive_folder`: (optional) the google drive folder to which logs can be uploaded\n\n\n## Folder organization\n\nVisit each folder to view its readme file.\n\n| Folder | Description |\n| - | - |\n| [`latex`](https://github.com/KevinMusgrave/powerful-benchmarker/tree/domain-adaptation/latex) | Code for creating latex tables from experiment data.\n| [`notebooks`](https://github.com/KevinMusgrave/powerful-benchmarker/tree/domain-adaptation/notebooks) | Jupyter notebooks\n| [`powerful_benchmarker`](https://github.com/KevinMusgrave/powerful-benchmarker/tree/domain-adaptation/powerful_benchmarker) | Code for hyperparameter searches for training models.\n| [`scripts`](https://github.com/KevinMusgrave/powerful-benchmarker/tree/domain-adaptation/scripts) | Various bash scripts, including scripts for uploading logs to google drive.\n| [`unit_tests`](https://github.com/KevinMusgrave/powerful-benchmarker/tree/domain-adaptation/unit_tests) | Tests to check if there are bugs.\n| [`validator_tests`](https://github.com/KevinMusgrave/powerful-benchmarker/tree/domain-adaptation/validator_tests) | Code for evaluating validation methods (validators).\n\n\n## Useful top-level scripts\n\n### delete_slurm_logs.py\nDelete all slurm logs:\n```bash\npython delete_slurm_logs.py --delete\n```\n\nOr delete slurm logs for specific experiments groups. For example, delete slurm logs for all experiment groups starting with \"officehome\":\n```bash\npython delete_slurm_logs.py --delete --exp_group_prefix officehome\n```\n---\n### kill_all.py\nKill all model training jobs:\n```bash\npython kill_all.py\n```\nOr kill all validator test jobs:\n```bash\npython kill_all.py --validator_tests\n```\n---\n### print_progress.py\nPrint how many hyperparameter trials are done:\n```bash\npython print_progress.py\n```\n\nInclude a detailed summary of validator test jobs:\n```bash\npython print_progress.py --with_validator_progress\n```\n\nSave to `progress.txt` instead of printing to screen:\n```bash\npython print_progress.py --save_to_file progress.txt\n```\n---\n### simple_slurm.py\nA simple way to run a program via slurm. \n\nFor example, run `collect_dfs.py` for all experiment groups starting with \"office31\", using a separate slurm job for each experiment group:\n```bash\npython simple_slurm.py --command \"python validator_tests/collect_dfs.py\" --slurm_config_folder validator_tests \\\n--slurm_config a100 --job_name=collect_dfs --cpus-per-task=16 --exp_group_prefix office31\n```\n\nOr run a program without considering experiment groups at all:\n```bash\npython simple_slurm.py --command \"python validator_tests/zip_dfs.py\" --slurm_config_folder validator_tests \\\n--slurm_config a100 --job_name=zip_dfs --cpus-per-task=16\n```\n---\n### upload_logs.py\nUpload slurm logs and experiment progress to a google drive folder at regular intervals (the default is every 2 hours):\n```bash\npython upload_logs.py\n```\nSet the google drive folder in `constants.yaml`.\n\n\n## Logo\nThanks to [Jeff Musgrave](https://www.designgenius.ca/) for designing the logo.\n\n\n## Citing the paper\n\n```bibtex\n@article{Musgrave2022ThreeNew,\n  title={Three New Validators and a Large-Scale Benchmark Ranking for Unsupervised Domain Adaptation},\n  author={Kevin Musgrave and Serge J. Belongie and Ser Nam Lim},\n  journal={ArXiv},\n  year={2022},\n  volume={abs/2208.07360}\n}\n```\n","funding_links":[],"categories":["Python"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FKevinMusgrave%2Fpowerful-benchmarker","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FKevinMusgrave%2Fpowerful-benchmarker","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FKevinMusgrave%2Fpowerful-benchmarker/lists"}