{"id":19463004,"url":"https://github.com/libffcv/ffcv-imagenet","last_synced_at":"2025-04-25T08:32:02.329Z","repository":{"id":38979892,"uuid":"449178383","full_name":"libffcv/ffcv-imagenet","owner":"libffcv","description":"Train ImageNet *fast* in 500 lines of code with FFCV","archived":false,"fork":false,"pushed_at":"2024-05-10T18:32:18.000Z","size":58,"stargazers_count":141,"open_issues_count":14,"forks_count":35,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-04-03T19:05:35.378Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/libffcv.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2022-01-18T07:10:26.000Z","updated_at":"2025-03-26T09:28:43.000Z","dependencies_parsed_at":"2023-12-07T03:26:02.386Z","dependency_job_id":"6f869dc4-5dec-41b8-bb0b-2297258566ea","html_url":"https://github.com/libffcv/ffcv-imagenet","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/libffcv%2Fffcv-imagenet","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/libffcv%2Fffcv-imagenet/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/libffcv%2Fffcv-imagenet/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/libffcv%2Fffcv-imagenet/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/libffcv","download_url":"https://codeload.github.com/libffcv/ffcv-imagenet/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250782206,"owners_count":21486407,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-10T18:07:44.758Z","updated_at":"2025-04-25T08:32:02.090Z","avatar_url":"https://github.com/libffcv.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# `ffcv` ImageNet Training\nA minimal, single-file PyTorch ImageNet training script designed for hackability. Run `train_imagenet.py` to get...\n- ...high accuracies on ImageNet\n- ...with as many lines of code as the PyTorch ImageNet example\n- ...in 1/10th the time.\n\n## Results\nTrain models more efficiently, either with 8 GPUs in parallel or by training 8 ResNet-18's at once.\n\u003cimg src=\"assets/perf_scatterplot.svg\" width='830px'/\u003e\n\nSee benchmark setup here: [https://docs.ffcv.io/benchmarks.html](https://docs.ffcv.io/benchmarks.html).\n\n## Citation\nIf you use this setup in your research, cite:\n\n```\n@misc{leclerc2022ffcv,\n    author = {Guillaume Leclerc and Andrew Ilyas and Logan Engstrom and Sung Min Park and Hadi Salman and Aleksander Madry},\n    title = {ffcv},\n    year = {2022},\n    howpublished = {\\url{https://github.com/libffcv/ffcv/}},\n    note = {commit xxxxxxx}\n}\n```\n(Make sure to replace ``xxxxxxx`` above with the hash of the commit used!)\n\n## Configurations\nThe configuration files corresponding to the above results are:\n\n| Link to Config                                                                                                                         |   top_1 |   top_5 |   # Epochs |   Time (mins) | Architecture   | Setup    |\n|:---------------------------------------------------------------------------------------------------------------------------------------|--------:|--------:|-----------:|--------------:|:---------------|:---------|\n| \u003ca href='https://github.com/libffcv/ffcv-imagenet/tree/main/rn50_configs/rn50_88_epochs.yaml'\u003eLink\u003c/a\u003e | 0.784 | 0.941  |         88 |       77.2 | ResNet-50      | 8 x A100 |\n| \u003ca href='https://github.com/libffcv/ffcv-imagenet/tree/main/rn50_configs/rn50_56_epochs.yaml'\u003eLink\u003c/a\u003e | 0.780 | 0.937 |         56 |       49.4 | ResNet-50      | 8 x A100 |\n| \u003ca href='https://github.com/libffcv/ffcv-imagenet/tree/main/rn50_configs/rn50_40_epochs.yaml'\u003eLink\u003c/a\u003e | 0.772 | 0.932 |         40 |       35.6 | ResNet-50      | 8 x A100 |\n| \u003ca href='https://github.com/libffcv/ffcv-imagenet/tree/main/rn50_configs/rn50_32_epochs.yaml'\u003eLink\u003c/a\u003e | 0.766 | 0.927 |         32 |       28.7 | ResNet-50      | 8 x A100 |\n| \u003ca href='https://github.com/libffcv/ffcv-imagenet/tree/main/rn50_configs/rn50_24_epochs.yaml'\u003eLink\u003c/a\u003e | 0.756 | 0.921 |         24 |       21.7  | ResNet-50      | 8 x A100 |\n| \u003ca href='https://github.com/libffcv/ffcv-imagenet/tree/main/rn50_configs/rn50_16_epochs.yaml'\u003eLink\u003c/a\u003e | 0.738 | 0.908 |         16 |       14.9 | ResNet-50      | 8 x A100 |\n| \u003ca href='https://github.com/libffcv/ffcv-imagenet/tree/main/rn18_configs/rn18_88_epochs.yaml'\u003eLink\u003c/a\u003e | 0.724 | 0.903   |         88 |      187.3  | ResNet-18      | 1 x A100 |\n| \u003ca href='https://github.com/libffcv/ffcv-imagenet/tree/main/rn18_configs/rn18_56_epochs.yaml'\u003eLink\u003c/a\u003e | 0.713  | 0.899 |         56 |      119.4   | ResNet-18      | 1 x A100 |\n| \u003ca href='https://github.com/libffcv/ffcv-imagenet/tree/main/rn18_configs/rn18_40_epochs.yaml'\u003eLink\u003c/a\u003e | 0.706 | 0.894 |         40 |       85.5 | ResNet-18      | 1 x A100 |\n| \u003ca href='https://github.com/libffcv/ffcv-imagenet/tree/main/rn18_configs/rn18_32_epochs.yaml'\u003eLink\u003c/a\u003e | 0.700 | 0.889 |         32 |       68.9   | ResNet-18      | 1 x A100 |\n| \u003ca href='https://github.com/libffcv/ffcv-imagenet/tree/main/rn18_configs/rn18_24_epochs.yaml'\u003eLink\u003c/a\u003e | 0.688  | 0.881 |         24 |       51.6 | ResNet-18      | 1 x A100 |\n| \u003ca href='https://github.com/libffcv/ffcv-imagenet/tree/main/rn18_configs/rn18_16_epochs.yaml'\u003eLink\u003c/a\u003e | 0.669 | 0.868 |         16 |       35.0 | ResNet-18      | 1 x A100 |\n\n## Training Models\n\nFirst pip install the requirements file in this directory:\n```\npip install -r requirements.txt\n```\nThen, generate an ImageNet dataset; make the dataset used for the results above with the following command (`IMAGENET_DIR` should point to a PyTorch style [ImageNet dataset](https://github.com/MadryLab/pytorch-imagenet-dataset):\n\n```bash\n# Required environmental variables for the script:\nexport IMAGENET_DIR=/path/to/pytorch/format/imagenet/directory/\nexport WRITE_DIR=/your/path/here/\n\n# Starting in the root of the Git repo:\ncd examples;\n\n# Serialize images with:\n# - 500px side length maximum\n# - 50% JPEG encoded\n# - quality=90 JPEGs\n./write_imagenet.sh 500 0.50 90\n```\nThen, choose a configuration from the [configuration table](#configurations). With the config file path in hand, train as follows:\n```bash\n# 8 GPU training (use only 1 for ResNet-18 training)\nexport CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7\n\n# Set the visible GPUs according to the `world_size` configuration parameter\n# Modify `data.in_memory` and `data.num_workers` based on your machine\npython train_imagenet.py --config-file rn50_configs/\u003cyour config file\u003e.yaml \\\n    --data.train_dataset=/path/to/train/dataset.ffcv \\\n    --data.val_dataset=/path/to/val/dataset.ffcv \\\n    --data.num_workers=12 --data.in_memory=1 \\\n    --logging.folder=/your/path/here\n```\nAdjust the configuration by either changing the passed YAML file or by specifying arguments via [fastargs](https://github.com/GuillaumeLeclerc/fastargs) (i.e. how the dataset paths were passed above).\n\n## Training Details\n\u003cp\u003e\u003cb\u003eSystem setup.\u003c/b\u003e We trained on p4.24xlarge ec2 instances (8 A100s).\n\u003c/p\u003e\n\n\u003cp\u003e\u003cb\u003eDataset setup. Generally larger side length will aid in accuracy but decrease\nthroughput:\u003c/b\u003e\n\n - ResNet-50 training: 50% JPEG 500px side length\n - ResNet-18 training: 10% JPEG 400px side length\n\n\u003c/p\u003e\n\n\n\u003cp\u003e\u003cb\u003eAlgorithmic details.\u003c/b\u003e We use a standard ImageNet training pipeline (à la the PyTorch ImageNet example) with only the following differences/highlights:\n\n- SGD optimizer with momentum and weight decay on all non-batchnorm parameters\n- Test-time augmentation over left/right flips\n- Progressive resizing from 160px to 192px: 160px training until 75% of the way through training (by epochs), then 192px until the end of training.\n- Validation set sizing according to [\"Fixing the train-test resolution discrepancy\"](https://arxiv.org/abs/1906.06423): 224px at test time.\n- Label smoothing\n- Cyclic learning rate schedule\n\u003c/p\u003e\n\nRefer to the code and configuration files for a more exact specification.\nTo obtain configurations we first gridded for hyperparameters at a 30 epoch schedule. Fixing these parameters, we then varied only the number of epochs (stretching the learning rate schedule across the number of epochs as motivated by [Budgeted Training](https://arxiv.org/abs/1905.04753)) and plotted the results above.\n\n## FAQ\n### Why is the first epoch slow?\nThe first epoch can be slow for the first epoch if the dataset hasn't been cached in memory yet.\n\n### What if I can't fit my dataset in memory?\nSee this [guide here](https://docs.ffcv.io/parameter_tuning.html#scenario-large-scale-datasets).\n\n### Other questions\nPlease open up a [GitHub discussion](https://github.com/MadryLab/ffcv/discussions) for non-bug related questions; if you find a bug please report it on [GitHub issues](https://github.com/MadryLab/ffcv/issues).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flibffcv%2Fffcv-imagenet","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flibffcv%2Fffcv-imagenet","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flibffcv%2Fffcv-imagenet/lists"}