{"id":23951386,"url":"https://github.com/psteinb/sota_on_uncertainties","last_synced_at":"2025-07-26T13:36:59.173Z","repository":{"id":145285352,"uuid":"457442418","full_name":"psteinb/sota_on_uncertainties","owner":"psteinb","description":"trying to obtain uncertainties from training accuracies using timm","archived":false,"fork":false,"pushed_at":"2022-06-17T09:11:31.000Z","size":5439,"stargazers_count":9,"open_issues_count":2,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-04-12T23:42:00.494Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/psteinb.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2022-02-09T16:38:40.000Z","updated_at":"2023-10-29T01:11:53.000Z","dependencies_parsed_at":"2023-04-21T09:46:34.997Z","dependency_job_id":null,"html_url":"https://github.com/psteinb/sota_on_uncertainties","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/psteinb/sota_on_uncertainties","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/psteinb%2Fsota_on_uncertainties","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/psteinb%2Fsota_on_uncertainties/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/psteinb%2Fsota_on_uncertainties/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/psteinb%2Fsota_on_uncertainties/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/psteinb","download_url":"https://codeload.github.com/psteinb/sota_on_uncertainties/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/psteinb%2Fsota_on_uncertainties/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":267175581,"owners_count":24047884,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-26T02:00:08.937Z","response_time":62,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-01-06T12:59:40.151Z","updated_at":"2025-07-26T13:36:59.165Z","avatar_url":"https://github.com/psteinb.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# SOTA with uncertainties\n\nObtain uncertainties from training accuracies using [timm](https://github.com/rwightman/pytorch-image-models/) using ensembles from an automated workflow. More details are available in [our paper](https://ml-eval.github.io/accepted-papers/#11).\n\n# Required Environment\n\n## instructions for a vanilla python installation\n\nWe assume you have some form of GPU available including the required runtime environment available. If not, you can try to execute the workflow on CPU-only hardware. Note though that should you wish to train the networks, running on a CPU-only hardware can be very slow.\n\nChecking the python version:\n\n```bash\npython --version\n```\nshould give:\n```\n3.8.5\n```\nSet up the `venv`:\n```bash\npython -m venv \u003csome-name\u003e\n```\nSource the `venv`:\n```bash\nsource \u003csome-name\u003e/bin/activate\n```\n\n## instructions for Jusuf\n\nFor development:\n\n```bash\nsalloc -N 1 -p gpus -A \u003comitted\u003e -t 01:00:00\n```\n\n```bash\nsrun --cpu_bind=none --pty /bin/bash -i\n```\n\nSetup the software environment:\n\n```bash\nml add Stages/2020 GCCcore/.10.3.0 CUDA/11.3 Python/3.8.5\n```\nThis will setup the environment to:\n\n```bash\nml\n```\nthis produces:\n```\nCurrently Loaded Modules:\n  1) Stages/2020            (S)     9) ncurses/.6.2     (H)  17) util-linux/.2.36    (H)  25) libxml2/.2.9.10  (H)  33) libspatialindex/.1.9.3 (H)\n  2) StdEnv/2020                   10) libreadline/.8.0 (H)  18) fontconfig/.2.13.92 (H)  26) libxslt/.1.1.34  (H)  34) NASM/.2.15.03          (H)\n  3) GCCcore/.10.3.0        (H)    11) Tcl/8.6.10            19) xorg-macros/.1.19.2 (H)  27) libffi/.3.3      (H)  35) libjpeg-turbo/.2.0.5   (H)\n  4) binutils/.2.36.1       (H)    12) SQLite/.3.32.3   (H)  20) libpciaccess/.0.16  (H)  28) libyaml/.0.2.5   (H)  36) Python/3.8.5\n  5) zlib/.1.2.11           (H)    13) expat/.2.2.9     (H)  21) X11/20200222             29) Java/15.0.1\n  6) nvidia-driver/.default (H,g)  14) libpng/.1.6.37   (H)  22) Tk/.8.6.10          (H)  30) PostgreSQL/12.3\n  7) CUDA/11.3              (g)    15) freetype/.2.10.1 (H)  23) GMP/6.2.0                31) protobuf/.3.13.0 (H)\n  8) bzip2/.1.0.8           (H)    16) gperf/.3.1       (H)  24) XZ/.5.2.5           (H)  32) gflags/.2.2.2    (H)\n\n  Where:\n   S:  Module is Sticky, requires --force to unload or purge\n   g:  built for GPU\n   H:             Hidden Module\n\n```\nChecking the python version:\n\n```bash\npython --version\n```\nshould give:\n```\n3.8.5\n```\nSet up the `venv`:\n```bash\npython -m venv \u003csome-name\u003e\n```\nSource the `venv`:\n```bash\nsource \u003csome-name\u003e/bin/activate\n```\n\nFor the sake of good practice, update `pip` whenever possible:\n\n```bash\npython -m pip install -U pip\n```\n\n\n# Required Python Packages\n\n### Prepare for full training\n\nTo prepare the environment and set up `timm` for complete training, we need to install our own pytorch including support for Nvidia GPUs. As `timm` supports exclusively cuda at the moment, we follow suite and require our users to have access to Nvidia GPUs :\n\n```bash\npython -m pip install torch==1.10.2+cu113 torchvision==0.11.3+cu113  -f https://download.pytorch.org/whl/cu113/torch_stable.html\n```\n\nTo conduct training, you need to install the full software stack:\n\n```bash\npython -m pip install -r requirements-full.txt\n```\n\n**NB** We did not test our workflow in this scenario.\n\n### Prepare for reproducing the figures only\n\n**Note**: If you are not interested to rerun the machine learning traing, you are fine to go without `pytorch` and `timm` such as:\n\n```bash\npython -m pip install -r requirements.txt\n```\n\n\n# Running the experiments\n\n## Getting the data\n\nNote, this repo involves 360 1h runs on a Nvidia V100. If you'd like to repeat the experiments, you need to download `imagenette2` the dataset as documented in the timmdocs. Our workflow can do that automatically. Just in case you want to look around first, here is how to obtain it:\n\n```bash\nmkdir data\ncd data\nwget https://s3.amazonaws.com/fast-ai-imageclas/imagenette2-320.tgz\ntar xf imagenette2-320.tgz\n```\n\n## training all models\n\nTo run the 360 experiments sequentially, do\n\n```bash\ncd /root/to/repo\n```\nto switch the current working directory. Then run the training:\n\n```bash\nsnakemake -j1 imagenette2_train\n```\n\nPlease use the issue tracker to report any shortcomings.\n\n## parallel execution on a cluster\n\nThis workflow setup is prepared with a [slurm](https://slurm.schedm.com) cluster in mind. JUSUF at JSC is managed by [slurm](https://slurm.schedm.com). \n\nBefore you do anything, the destination folders for the slurm logs have to be created.\n\n```bash\ncd /path/to/repo\nmkdir -p outputs/slurm/logs/\n```\n\nOn Jusuf, you can run all model trainings as\n\n```bash\nsnakemake -j40 -p --profile config/slurm/jusuf imagenette2_train\n```\n\nNote, this will submit `360` jobs in total, but only run `40` jobs at a time. You can only invoke this command from the `venv` described above. If you'd like to run this on another cluster, adjust `config/slurm/jusuf/config.yaml` to your needs (see [slurm profile](https://github.com/Snakemake-Profiles/slurm) for the api documentation of `config.yaml`).\n\n## Inference (default workflow target)\n\nThe default workflow target is to run inference on the validation datasets created. You need at least one GPU for this and all `last.pth.tar` model files generated by `timm` in a folder structure which the workflow expects. In other words:\n\n```bash\nsnakemake -j80 -p --profile config/slurm/jusuf imagenette2_inference_last\n```\n\n# Viewing the execution graph\n\n```bash\nsnakemake -j1 -F --dag results/figures/imagenette2_compare_meanstd_approx.png| dot -Tsvg \u003e ~/imagenette2_compare_meanstd_approx_dag.svg\n```\n\n# Citation\n\n```\n@article{https://doi.org/10.48550/arxiv.2204.05173,\n  doi = {10.48550/ARXIV.2204.05173},\n  url = {https://ml-eval.github.io/accepted-papers/#11},\n  author = {Steinbach, Peter and Gernhardt, Felicita and Tanveer, Mahnoor and Schmerler, Steve and Starke, Sebastian},\n  keywords = {Machine Learning (cs.LG), Computer Vision and Pattern Recognition (cs.CV), FOS: Computer and information sciences, FOS: Computer and information sciences},\n  title = {Machine Learning State-of-the-Art with Uncertainties},\n  publisher = {arXiv},\n  year = {2022},\n  copyright = {Creative Commons Attribution 4.0 International},\n  note = {ICLR22, ML Evaluation Standards workshop}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpsteinb%2Fsota_on_uncertainties","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpsteinb%2Fsota_on_uncertainties","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpsteinb%2Fsota_on_uncertainties/lists"}