{"id":19176528,"url":"https://github.com/deeplite/deeplite-profiler","last_synced_at":"2025-05-07T19:24:02.585Z","repository":{"id":40689040,"uuid":"354927391","full_name":"Deeplite/deeplite-profiler","owner":"Deeplite","description":"A collection of metrics to profile a single deep learning model or compare two different deep learning models","archived":false,"fork":false,"pushed_at":"2023-11-07T00:37:51.000Z","size":3260,"stargazers_count":26,"open_issues_count":12,"forks_count":9,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-04-15T04:51:33.643Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Deeplite.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-04-05T18:10:43.000Z","updated_at":"2025-01-21T09:25:42.000Z","dependencies_parsed_at":"2024-11-09T10:29:45.455Z","dependency_job_id":"2e5ca5cc-02d8-44de-8ae5-b1b1d1436e02","html_url":"https://github.com/Deeplite/deeplite-profiler","commit_stats":{"total_commits":101,"total_committers":8,"mean_commits":12.625,"dds":0.6237623762376238,"last_synced_commit":"fc8d19de532f284edec104dc16c9a2359496a7b8"},"previous_names":[],"tags_count":29,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Deeplite%2Fdeeplite-profiler","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Deeplite%2Fdeeplite-profiler/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Deeplite%2Fdeeplite-profiler/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Deeplite%2Fdeeplite-profiler/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Deeplite","download_url":"https://codeload.github.com/Deeplite/deeplite-profiler/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252942110,"owners_count":21829002,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-09T10:28:55.272Z","updated_at":"2025-05-07T19:24:02.559Z","avatar_url":"https://github.com/Deeplite.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://docs.deeplite.ai/neutrino/_static/content/deeplite-logo-color.png\" /\u003e\n\u003c/p\u003e\n\n[![Build Status](https://travis-ci.com/Deeplite/deeplite-profiler.svg?token=KrazyWqBHDFfVzGZSU9X\u0026branch=master)](https://travis-ci.com/Deeplite/deeplite-profiler) [![codecov](https://codecov.io/gh/Deeplite/deeplite-profiler/branch/master/graph/badge.svg?token=D1RMWA1TDC)](https://codecov.io/gh/Deeplite/deeplite-profiler)\n\n# Deeplite Profiler\n\nTo be able to use a deep learning model in research and production, it is essential to understand different performance metrics of the model beyond just the model's accuracy.  ``deeplite-profiler`` helps to easily and effectively measure the different performance metrics of a deep learning model. In addition to the existing metrics in the ``deeplite-profiler``, users could seamlessly contribute any custom metric to measure using the profiler. ``deeplite-profiler`` could also be used to compare the performance between two different deep learning models, for example, a teacher and a student model. ``deeplite-profiler`` currently supports PyTorch and TensorFlow Keras (v1) as two different backend frameworks.\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://docs.deeplite.ai/neutrino/_images/profiler.png\" /\u003e\n\u003c/p\u003e\n\n- [Deeplite Profiler](#deeplite-profiler)\n- [Installation](#installation)\n  - [Install using pip](#install-using-pip)\n  - [Install from source](#install-from-source)\n  - [Install in Dev mode](#install-in-dev-mode)\n- [How to Use](#how-to-use)\n  - [For a PyTorch Model](#for-a-pytorch-model)\n  - [For a TensorFlow Model](#for-a-tensorflow-model)\n  - [Output Display](#output-display)\n- [Examples](#examples)\n- [Contribute a Custom Metric](#contribute-a-custom-metric)\n\n\n# Installation\n\n## Install using pip\n\nUse following command to install the package from our internal PyPI repository.\n\n```console\n$ pip install --upgrade pip\n$ pip install deeplite-profiler[`backend`]\n```\n\n## Install from source\n\n```console\n$ git clone https://github.com/Deeplite/deeplite-profiler.git\n$ pip install .[`backend`]\n```\n\nOne can install specific ``backend`` modules, depending on the required framework and compute support. ``backend`` could be one of the following values\n- ``torch``: to install a ``torch`` specific profiler\n- ``tf``: to install a ``TensorFlow`` specific profiler (this supports only CPU compute)\n- ``tf-gpu``: to install a ``TensorFlow-gpu`` specific profiler (this supports both CPU and GPU compute)\n- ``all``: to install both ``torch`` and ``TensorFlow`` specific profiler (this supports only CPU compute for TensorFlow)\n- ``all-gpu``: to install both ``torch`` and ``TensorFlow-gpu`` specific profiler (for GPU environment) (this supports both CPU and GPU compute for TensorFlow)\n\n\n## Install in Dev mode\n\n```console\n$ git clone https://github.com/Deeplite/deeplite-profiler.git\n$ pip install -e .[`backend`]\n$ pip install -r requirements-test.txt\n```\n\nTo test the installation, one can run the basic tests using `pytest` command in the root folder.\n\n\u003e **_NOTE:_**  Currently, we support Tensorflow 1.14 and 1.15 versions, for Python 3.6 and 3.7. We _do not_ support Python 3.8+.\n\n\n# How to Use\n\n## For a PyTorch Model\n\n```python\n# Step 1: Define native pytorch dataloaders and model\n# 1a. data_splits = {\"train\": train_dataloder, \"test\": test_dataloader}\ndata_splits = /* ... load iterable data loaders ... */\nmodel = /* ... load native deep learning model ... */\n\n# Step 2: Create Profiler class and register the profiling functions\ndata_loader = TorchProfiler.enable_forward_pass_data_splits(data_splits)\nprofiler = TorchProfiler(model, data_splits, name=\"Original Model\")\nprofiler.register_profiler_function(ComputeComplexity())\nprofiler.register_profiler_function(ComputeExecutionTime())\nprofiler.register_profiler_function(ComputeEvalMetric(get_accuracy, 'accuracy', unit_name='%'))\n\n# Step 3: Compute the registered profiler metrics for the PyTorch Model\nprofiler.compute_network_status(batch_size=1, device=Device.CPU, short_print=False,\n                                                 include_weights=True, print_mode='debug')\n\n# Step 4: Compare two different models or profilers.\nprofiler2 = profiler.clone(model=deepcopy(model)) # Creating a dummy clone of the current profiler\nprofiler2.name = \"Clone Model\"\nprofiler.compare(profiler2, short_print=False, batch_size=1, device=Device.CPU, print_mode='debug')\n```\n\n## For a TensorFlow Model\n\n```python\n# Step 1: Define native tensorflow dataloaders and model\n# 1a. data_splits = {\"train\": train_dataloder, \"test\": test_dataloader}\ndata_splits = /* ... load iterable data loaders ... */\nmodel = /* ... load native deep learning model ... */\n\n# Step 2: Create Profiler class and register the profiling functions\ndata_loader = TFProfiler.enable_forward_pass_data_splits(data_splits)\nprofiler = TFProfiler(model, data_splits, name=\"Original Model\")\nprofiler.register_profiler_function(ComputeFlops())\nprofiler.register_profiler_function(ComputeSize())\nprofiler.register_profiler_function(ComputeParams())\nprofiler.register_profiler_function(ComputeLayerwiseSummary())\nprofiler.register_profiler_function(ComputeExecutionTime())\nprofiler.register_profiler_function(ComputeEvalMetric(get_accuracy, 'accuracy', unit_name='%'))\n\n# Step 3: Compute the registered profiler metrics for the Tensorflow Keras Model\nprofiler.compute_network_status(batch_size=1, device=Device.CPU, short_print=False,\n                                                 include_weights=True, print_mode='debug')\n\n# Step 4: Compare two different models or profilers.\nprofiler2 = profiler.clone(model=model) # Creating a dummy clone of the current profiler\nprofiler2.name = \"Clone Model\"\nprofiler.compare(profiler2, short_print=False, batch_size=1, device=Device.CPU, print_mode='debug')\n```\n\n## Output Display\n\nAn example output of the ``deeplite-profiler`` for ``resnet18`` model using the standard ``CIFAR100`` dataset using ``PyTorch`` backend looks as follows\n\n```console\n+---------------------------------------------------------------+\n|                    deeplite Model Profiler                    |\n+-----------------------------------------+---------------------+\n|             Param Name (Original Model) |                Value|\n|                   Backend: TorchBackend |                     |\n+-----------------------------------------+---------------------+\n|                   Evaluation Metric (%) |              76.8295|\n|                         Model Size (MB) |              42.8014|\n|     Computational Complexity (GigaMACs) |               0.5567|\n|             Total Parameters (Millions) |              11.2201|\n|                   Memory Footprint (MB) |              48.4389|\n|                     Execution Time (ms) |               2.6537|\n+-----------------------------------------+---------------------+\n```\n\n- **Evaluation Metric:** Computed performance of the model on the given data\n- **Model Size:** Memory consumed by the parameters (weights and biases) of the model\n- **Computational Complexity:** Summation of Multiply-Add Cumulations (MACs) per single image (batch_size=1)\n- **#Total Parameters:** Total number of parameters (trainable and non-trainable) in the model\n- **Memory Footprint:** Total memory consumed by parameters and activations per single image (batch_size=1)\n- **Execution Time:** On `NVIDIA TITAN V \u003chttps://www.nvidia.com/en-us/titan/titan-v/\u003e`_ GPU, time required for the forward pass per single image (batch_size=1)\n\n# Examples\n\nA list of different examples to use ``deeplite-profiler`` to profiler different PyTorch and TensorFlow models can be found [here](./examples)\n\n# Contribute a Custom Metric\n\n\u003e **_NOTE:_**  If you looking for an SDK documentation, please head over [here](https://deeplite.github.io/deeplite-profiler/).\n\nWe always welcome community contributions to expand the scope of `deeplite-profiler` and also to have additional new metrics. Please refer to the [documentation](https://docs.deeplite.ai/neutrino/profiler.html) for the detailed steps on how to design a new metrics. In general, we follow the `fork-and-pull` Git workflow.\n\n1. **Fork** the repo on GitHub\n2. **Clone** the project to your own machine\n3. **Commit** changes to your own branch\n4. **Push** your work back up to your fork\n5. Submit a **Pull request** so that we can review your changes\n\nNOTE: Be sure to merge the latest from \"upstream\" before making a pull request!","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdeeplite%2Fdeeplite-profiler","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdeeplite%2Fdeeplite-profiler","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdeeplite%2Fdeeplite-profiler/lists"}