{"id":13935896,"url":"https://github.com/cgnorthcutt/benchmarking-keras-pytorch","last_synced_at":"2025-04-07T05:11:21.506Z","repository":{"id":95967727,"uuid":"171768165","full_name":"cgnorthcutt/benchmarking-keras-pytorch","owner":"cgnorthcutt","description":"🔥 Reproducibly benchmarking Keras and PyTorch models","archived":false,"fork":false,"pushed_at":"2021-01-28T08:41:58.000Z","size":48272,"stargazers_count":367,"open_issues_count":3,"forks_count":43,"subscribers_count":20,"default_branch":"master","last_synced_at":"2025-03-30T22:11:07.286Z","etag":null,"topics":["benchmarking","cnn","cnn-classification","cnn-keras","deep-learning","imagenet","keras","keras-tutorials","pytorch","pytorch-tutorial"],"latest_commit_sha":null,"homepage":"https://l7.curtisnorthcutt.com/towards-reproducibility-benchmarking-keras-pytorch","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cgnorthcutt.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-02-21T00:00:06.000Z","updated_at":"2024-05-23T01:19:21.000Z","dependencies_parsed_at":"2023-04-26T09:47:19.337Z","dependency_job_id":null,"html_url":"https://github.com/cgnorthcutt/benchmarking-keras-pytorch","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cgnorthcutt%2Fbenchmarking-keras-pytorch","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cgnorthcutt%2Fbenchmarking-keras-pytorch/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cgnorthcutt%2Fbenchmarking-keras-pytorch/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cgnorthcutt%2Fbenchmarking-keras-pytorch/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cgnorthcutt","download_url":"https://codeload.github.com/cgnorthcutt/benchmarking-keras-pytorch/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247595335,"owners_count":20963943,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["benchmarking","cnn","cnn-classification","cnn-keras","deep-learning","imagenet","keras","keras-tutorials","pytorch","pytorch-tutorial"],"created_at":"2024-08-07T23:02:10.826Z","updated_at":"2025-04-07T05:11:21.480Z","avatar_url":"https://github.com/cgnorthcutt.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# Benchmarking Keras and PyTorch Pre-Trained Models\n\nBenchmarks for **every** pre-trained model in PyTorch and Keras-Tensorflow. Benchmarks are reproducible.\n\n## Why this is helpful\n\nCombining Keras and PyTorch benchmarks into a single framework lets researchers decide which platform is best for a given model. For example `resnet` architectures perform better in PyTorch and `inception` architectures perform better in Keras (see below). These benchmarks serve as a standard from which to start new projects or debug current implementations. \n\nFor researchers exploring Keras and PyTorch models, these benchmarks serve as a standard from which to start new projects or debug current implementations. \n\nMany researchers struggle with reproducible accuracy benchmarks of pre-trained Keras (Tensorflow) models on ImageNet. Examples of issues are [here1](https://github.com/keras-team/keras/issues/10040), [here2](https://github.com/keras-team/keras/issues/10979), [here3](http://blog.datumbox.com/the-batch-normalization-layer-of-keras-is-broken/), [here4](https://github.com/keras-team/keras/issues/8672), and [here5](https://github.com/keras-team/keras/issues/7848). \n\nIn Keras, the [published benchmarks](https://keras.io/applications/#documentation-for-individual-models) on [Keras Applications](https://keras.io/applications/) cannot be reproduced by exactly copying the associated code. In fact, the reported accuracies are usually higher than the actual accuries.\n\nI dive slightly deeper into the reproducibility issues of Keras in the associated [blog post](http://l7.curtisnorthcutt.com/towards-reproducibility-benchmarking-keras-pytorch).\n\n## Benchmark Results on ImageNet\n\nThe **actual** validation set accuracies on ImageNet for all Keras and PyTorch models (verified on macOS 10.11.6, Linux Debian 9, and Ubuntu 18.04).\n\n| Platform    | Model             | Acc@1 | Acc@5 | Rank@1 | Rank@5 |\n|-------------|-------------------|-------|-------|--------|--------|\n| Keras 2.2.4 | nasnetlarge       | 80.83 | 95.27 | 1      | 1      |\n| Keras 2.2.4 | inceptionresnetv2 | 78.93 | 94.45 | 2      | 2      |\n| PyTorch 1.0 | resnet152         | 77.62 | 93.81 | 3      | 3      |\n| Keras 2.2.4 | xception          | 77.18 | 93.49 | 4      | 4      |\n| PyTorch 1.0 | densenet161       | 76.92 | 93.49 | 5      | 5      |\n| PyTorch 1.0 | resnet101         | 76.64 | 93.30 | 6      | 6      |\n| PyTorch 1.0 | densenet201       | 76.41 | 93.18 | 7      | 7      |\n| Keras 2.2.4 | inceptionv3       | 76.02 | 92.90 | 8      | 8      |\n| PyTorch 1.0 | densenet169       | 75.59 | 92.69 | 9      | 9      |\n| PyTorch 1.0 | resnet50          | 75.06 | 92.48 | 10     | 10     |\n| Keras 2.2.4 | densenet201       | 74.77 | 92.32 | 11     | 11     |\n| PyTorch 1.0 | densenet121       | 74.07 | 91.94 | 12     | 12     |\n| Keras 2.2.4 | densenet169       | 73.92 | 91.76 | 13     | 13     |\n| PyTorch 1.0 | vgg19_bn          | 72.90 | 91.35 | 14     | 14     |\n| PyTorch 1.0 | resnet34          | 72.34 | 90.84 | 15     | 16     |\n| PyTorch 1.0 | vgg16_bn          | 72.29 | 91.01 | 16     | 15     |\n| Keras 2.2.4 | densenet121       | 72.09 | 90.70 | 17     | 17     |\n| Keras 2.2.4 | nasnetmobile      | 71.59 | 90.19 | 18     | 19     |\n| PyTorch 1.0 | vgg19             | 71.19 | 90.40 | 19     | 18     |\n| PyTorch 1.0 | vgg16             | 70.66 | 89.93 | 20     | 20     |\n| Keras 2.2.4 | resnet50          | 70.35 | 89.55 | 21     | 22     |\n| PyTorch 1.0 | vgg13_bn          | 70.12 | 89.56 | 22     | 21     |\n| Keras 2.2.4 | mobilenetV2       | 69.98 | 89.49 | 23     | 23     |\n| PyTorch 1.0 | vgg11_bn          | 69.36 | 89.06 | 24     | 24     |\n| PyTorch 1.0 | inception_v3      | 69.25 | 88.69 | 25     | 25     |\n| Keras 2.2.4 | mobilenet         | 69.02 | 88.48 | 26     | 27     |\n| PyTorch 1.0 | vgg13             | 68.69 | 88.65 | 27     | 28     |\n| PyTorch 1.0 | resnet18          | 68.37 | 88.56 | 28     | 26     |\n| PyTorch 1.0 | vgg11             | 67.85 | 88.11 | 29     | 29     |\n| Keras 2.2.4 | vgg19             | 65.58 | 86.54 | 30     | 30     |\n| Keras 2.2.4 | vgg16             | 65.24 | 86.20 | 31     | 31     |\n| PyTorch 1.0 | squeezenet1_0     | 56.49 | 79.06 | 32     | 33     |\n| PyTorch 1.0 | squeezenet1_1     | 56.42 | 79.21 | 33     | 32     |\n| PyTorch 1.0 | alexnet           | 54.50 | 77.64 | 34     | 34     |\n\n\n\n## To Reproduce\n\n### Get the ImageNet validation dataset \n\n* Download\n  * Download the Imagenet 2012 Validation dataset from [http://image-net.org/download-images](http://image-net.org/download-images) or [other options](http://academictorrents.com). This dataset contains 50000 images\n\n* Preprocess/Extract validation data\n  * Once `ILSVRC2012_img_val.tar` is downloaded, run:\n  ```bash\n  # Credit to Soumith: https://github.com/soumith/imagenet-multiGPU.torch\n  $ cd ../ \u0026\u0026 mkdir val \u0026\u0026 mv ILSVRC2012_img_val.tar val/ \u0026\u0026 cd val \u0026\u0026 tar -xvf ILSVRC2012_img_val.tar\n  $ wget -qO- https://raw.githubusercontent.com/soumith/imagenetloader.torch/master/valprep.sh | bash\n  ```\n\n### Reproduce in 10 seconds\n\nThe top 5 predictions for every example in the ImageNet validation set have been pre-computed for you [here for Keras models](https://github.com/cgnorthcutt/benchmarking-keras-pytorch/tree/master/keras_imagenet) and [here for PyTorch models](https://github.com/cgnorthcutt/benchmarking-keras-pytorch/tree/master/pytorch_imagenet). These are automatically used by the following code which takes a few seconds to run:\n\n```bash\n$ git clone https://github.com:cgnorthcutt/imagenet-benchmarking.git\n$ cd benchmarking-keras-pytorch\n$ python imagenet_benchmarking.py /path/to/imagenet_val_data\n```\n\n### Reproduce model outputs (hours)\n\nYou can also reproduce the inference-time output of each Keras and PyTorch model without using the pre-computed data. Inference for Keras takes a long time (5-10 hours) because I compute the forward pass on each example one at a time and avoid vectorized operations: this was the only approach I found would reliably reproduce the same accuracies. PyTorch is fairly quick (less than one hour). To reproduce:\n\n```bash\n$ git clone https://github.com:cgnorthcutt/imagenet-benchmarking.git\n$ cd benchmarking-keras-pytorch\n$ # Compute outputs of PyTorch models (1 hour)\n$ ./imagenet_pytorch_get_predictions.py /path/to/imagenet_val_data\n$ # Compute outputs of Keras models (5-10 hours)\n$ ./imagenet_keras_get_predictions.py /path/to/imagenet_val_data\n$ # View benchmark results\n$ ./imagenet_benchmarking.py /path/to/imagenet_val_data\n```\n\nYou can control *GPU usage*, *batch size*, *output storage directories*, and more. Run the files with the `-h` flag to see command line argument options.\n\n#### Tips for Keras\n\nOne of the goals of this project is to help reconcile issues with reproducibility in Keras pre-trained models. The way I deal with these issues is three-fold. In Keras I \n1. avoid batches during inference.\n2. run each example one at a time. This is silly slow, but yields a reproducible output for every model.\n3. only run models in local functions or use `with` clauses to ensure no aspects of a previous model persist in memory when the next model is loaded.\n\n#### Example use cases from Pervasive Label Errors Paper\n\n\n```bash\n# Compare standard benchmarking versus cleaned and corrected benchmarking\n# Requires providing index of all test examples EXCEPT non-consensus errors (ambiguous or multi-class)\n# Requires providing the labels of the remaining test examples, including the corrected labels of the honeypot.\nfor consensus in {3..5}; do python imagenet_benchmarking.py -p pytorch_imagenet -o imagenet_benchmarks/cleaned_and_corrected_$consensus.csv -j /home/cgn/cgn/label-errors/experiments/mturk_processing/imagenet_masks/cleaned_and_corrected_labels_idx_$consensus.npy -l /home/cgn/cgn/label-errors/experiments/mturk_processing/imagenet_masks/cleaned_and_corrected_labels_$consensus.npy /datasets/datasets/imagenet/val; done\n\n# Honeypot benchmarking -- consensus corected labels versus original labels\nfor consensus in {3..5}; do python imagenet_benchmarking.py -p pytorch_imagenet -o imagenet_benchmarks/consensus$consensus.csv -j /home/cgn/cgn/label-errors/experiments/mturk_processing/imagenet_masks/mturk_actual_labels_idx_$consensus.npy -l /home/cgn/cgn/label-errors/experiments/mturk_processing/imagenet_masks/mturk_actual_labels_$consensus.npy /datasets/datasets/imagenet/val; done\n\n# ANOTHER AN EXAMPLE -- THIS RESULT IS NOT IN THE PAPER\n# Compare standard benchmarking versus cleaned benchmarking\nfor consensus in {3..5}; do python imagenet_benchmarking.py -p pytorch_imagenet -o imagenet_benchmarks/test_noise_removed_$consensus.csv -i /home/cgn/cgn/label-errors/experiments/mturk_processing/imagenet_masks/mturk_noise_indices_$consensus.npy /datasets/datasets/imagenet/val; done\n```\n\n## License\n\nCopyright (c) 2019 Curtis Northcutt. Released under the MIT License. See [LICENSE](https://github.com/cgnorthcutt/imagenet_benchmarking/blob/master/LICENSE) for details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcgnorthcutt%2Fbenchmarking-keras-pytorch","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcgnorthcutt%2Fbenchmarking-keras-pytorch","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcgnorthcutt%2Fbenchmarking-keras-pytorch/lists"}