{"id":16916782,"url":"https://github.com/ternaus/imagenet18","last_synced_at":"2025-03-20T21:41:01.032Z","repository":{"id":87572878,"uuid":"211390685","full_name":"ternaus/imagenet18","owner":"ternaus","description":null,"archived":false,"fork":false,"pushed_at":"2019-09-27T21:25:44.000Z","size":565,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-01-25T18:43:10.619Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"unlicense","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ternaus.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-09-27T19:44:24.000Z","updated_at":"2020-04-17T19:35:01.000Z","dependencies_parsed_at":null,"dependency_job_id":"c44968da-72ac-4c83-a1fb-e5407d5d3053","html_url":"https://github.com/ternaus/imagenet18","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ternaus%2Fimagenet18","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ternaus%2Fimagenet18/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ternaus%2Fimagenet18/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ternaus%2Fimagenet18/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ternaus","download_url":"https://codeload.github.com/ternaus/imagenet18/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244693644,"owners_count":20494487,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-13T19:29:50.786Z","updated_at":"2025-03-20T21:41:01.011Z","avatar_url":"https://github.com/ternaus.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"Code to reproduce ImageNet in 18 minutes, by Andrew Shaw, Yaroslav Bulatov, and Jeremy Howard. High-level overview of techniques used is [here](http://fast.ai/2018/08/10/fastai-diu-imagenet/)\n\n\nPre-requisites: Python 3.6 or higher\n\n- Set your `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_DEFAULT_REGION` (example [instructions](https://docs.google.com/document/d/1Z8lCZVWXs7XORbiNmBAsBDtouV3KwrtH8-UL5M-zHus/edit))\n\n```\npip install -r requirements.txt\n\nncluster spot_prices p3                            # check spot prices for regions to find valid zone for p3 instances\nexport NCLUSTER_ZONE=us-east-1                     # set to a zone with cheap p3's\npython tools/replicate_imagenet.py --replicas=4   # configure 16 high performance disks\npython train.py --machines=4\npython tools/replicate_imagenet.py --replicas=4 --delete  # delete high performance disks\n```\n\nTo run with smaller number of machines:\n\n```\npython train.py --machines=1\npython train.py --machines=2\npython train.py --machines=4\npython train.py --machines=8\npython train.py --machines=16\n```\n\nTo run as spot prices, add `--spot` argument, ie `train.py --spot`\n\nYour AWS account needs to have high enough limit in order to reserve this number of p3.16xlarge instances. The code will set up necessary infrastructure like EFS, VPC, subnets, keypairs and placement groups. Therefore permissions to create these those resources are needed. Note that high performance disks cost about $1/hour, so make sure to delete them after using.\n\n\n# Checking progress\n\nMachines print progress to local stdout, log TensorBoard event files to EFS under unique directory and also send data to wandb if WANDB_API_KEY env var is set to API key (it's under https://app.wandb.ai/settings).\n\n\n## TensorBoard\n1. launch tensorboard using `python tools/launch_tensorboard.py`\n\nThat will provide a link to tensorboard instance which has loss graph under \"losses\" group. You'll see something like this under \"Losses\" tab\n\u003cimg src='https://raw.githubusercontent.com/diux-dev/imagenet18/master/tensorboard.png'\u003e\n\n## Console\nYou can connect to one of the instances using instructions printed during launch. Look for something like this\n\n```\n2019-07-29 15:58:10.653377 0.monday-quad: To connect to 0.monday-quad do \"ncluster connect 0.monday-quad\" or\n    ssh ubuntu@184.73.100.7\n    tmux a\n```\n\nThis will connect you to tmux session and you will see something like this\n\n```\n.997 (65.102)   Acc@5 85.854 (85.224)   Data 0.004 (0.035)      BW 2.444 2.445\nEpoch: [21][175/179]    Time 0.318 (0.368)      Loss 1.4276 (1.4767)    Acc@1 66.169 (65.132)   Acc@5 86.063 (85.244)   Data 0.004 (0.035)      BW 2.464 2.466\nChanging LR from 0.4012569832402235 to 0.40000000000000013\nEpoch: [21][179/179]    Time 0.336 (0.367)      Loss 1.4457 (1.4761)    Acc@1 65.473 (65.152)   Acc@5 86.061 (85.252)   Data 0.004 (0.034)      BW 2.393 2.397\nTest:  [21][5/7]        Time 0.106 (0.563)      Loss 1.3254 (1.3187)    Acc@1 67.508 (67.693)   Acc@5 88.644 (88.315)\nTest:  [21][7/7]        Time 0.105 (0.432)      Loss 1.4089 (1.3346)    Acc@1 67.134 (67.462)   Acc@5 87.257 (88.124)\n~~21    0.31132         67.462          88.124\n```\n\nThe last number indicates that at epoch 21 the run got 67.462 top-1 test accuracy and 88.124 top-5 test accuracy.\n\n## Weights and Biases\n\nRuns will show up under under \"imagenet18\" project in your Weights and Biases page, is https://app.wandb.ai/yaroslavvb/imagenet18/runs/8fv3xosq\n\n# Other notes\nIf you run locally, you may need to download imagenet yourself from [here](https://s3.amazonaws.com/yaroslavvb2/data/imagenet18.tar)\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fternaus%2Fimagenet18","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fternaus%2Fimagenet18","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fternaus%2Fimagenet18/lists"}