{"id":13633372,"url":"https://github.com/octoml/octoml-profile","last_synced_at":"2025-04-18T10:34:37.688Z","repository":{"id":66963056,"uuid":"601425509","full_name":"octoml/octoml-profile","owner":"octoml","description":"Home for OctoML PyTorch Profiler","archived":false,"fork":false,"pushed_at":"2023-04-24T17:58:51.000Z","size":2833,"stargazers_count":107,"open_issues_count":0,"forks_count":7,"subscribers_count":5,"default_branch":"main","last_synced_at":"2024-11-09T02:34:12.327Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/octoml.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-02-14T03:07:21.000Z","updated_at":"2024-10-31T07:28:45.000Z","dependencies_parsed_at":null,"dependency_job_id":"e2adb664-291b-407c-86a5-f12302152a52","html_url":"https://github.com/octoml/octoml-profile","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/octoml%2Foctoml-profile","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/octoml%2Foctoml-profile/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/octoml%2Foctoml-profile/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/octoml%2Foctoml-profile/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/octoml","download_url":"https://codeload.github.com/octoml/octoml-profile/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249479056,"owners_count":21279187,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-01T23:00:35.615Z","updated_at":"2025-04-18T10:34:37.657Z","avatar_url":"https://github.com/octoml.png","language":null,"funding_links":[],"categories":["Performance","Profiling"],"sub_categories":["Profiling"],"readme":"## octoml-profile\n\n*octoml-profile* is a python library and cloud service that enables ML\nengineers to easily assess the performance and cost of PyTorch models on cloud\nhardware with state-of-the-art ML acceleration technology.\n\nWhether you're building machine learning models for research, development, or\nproduction, benchmarking your AI applications is a necessary step before\ndeployment. An optimally chosen hardware + runtime deployment strategy can\nreduce cloud costs by more than 10x over default solutions.\n\nWith *octoml-profile*, within minutes, you can measure the performance and\ncost of your PyTorch models on different cloud hardware. Our ML\nacceleration technology ensures that you get the most accurate and efficient\nresults, so you can make informed decisions about how to optimize your AI\napplications.\n\n*Note: this tool is not designed for profiling individual PyTorch\nops on the local machine. Please use `torch.profiler` for such purpose.*\n\n\n### Key Features\n- 🔧 Magic remote execution with only a few additional lines of code\n- 💻 Runs on local development environment without any GPU requirement\n- 💪 Absolves tedious tasks such as model export, hardware provisioning, and dependency preparation\n- 🚀 Provides performance and cost insights within seconds (or minutes for larger models)\n- ⚙️  Supports diverse hardware and state-of-the-art software backends\n- 🌟 Supports the latest generative AI models with dynamic shapes\n- 📊 Uses the same data and workflow as your training and experiment tracking\n\n\n### Limitation\n- Only supports inference workload\n\n\n### Demos\n- [SentenceTransformers](examples/sentence_transformers/SentenceTransformerEval.ipynb)\n- [Stable Diffusion](examples/stable_diffusion.py)\n- [GPTNeo](examples/gpt_neo_125m.py)\n- [Whisper](examples/whisper.py)\n- [T5](examples/t5.py)\n\n\n### Latest\n- [04-25-2023] Client update `v0.2.2` with enhanced terminal output and more examples\n- [03-22-2023] Initial release of `v0.2.0`\n\n\n### Documentation quick links\n* [Installation and Getting Started](#installation-and-getting-started)\n* [Dynamic shapes](#dynamic-shapes)\n* [How it works](#how-it-works)\n* [Data privacy](#data-privacy)\n* [Known issues](#known-issues)\n* [Contact the team](#contact-the-team)\n\n### \"Hello World\" example\n\nLet's say you have a PyTorch model that performs sentiment analysis using a\nDistilBert model, and you want to optimize it for cloud deployment. With\noctoml-profile, you can easily benchmark the predict function on various cloud\nhardware and use different acceleration techniques to find the optimal\ndeployment strategy.\n\n![Distilbert Example](assets/distilbert-diff.png)\n\nWithin a few seconds, you will find the runtime and cost that help you pick the\noptimal hardware and inference engine for deployment.\n\n```\nFunction `predict` has 1 profile:\n- Profile `predict[1/1]` ran 3 times. (1 discarded because compilation happened)\n\nInstance     Processor           Backend              Backend Time (ms)  Total Time (ms)  Cost ($/MReq)\n=======================================================================================================\nr6i.large    Intel Ice Lake CPU  torch-eager-cpu                 24.735           52.009          $1.82\ng4dn.xlarge  Nvidia T4 GPU       torch-eager-cuda                 5.336           32.610          $4.76\ng4dn.xlarge  Nvidia T4 GPU       torch-inductor-cuda              3.249           30.523          $4.46\ng4dn.xlarge  Nvidia T4 GPU       onnxrt-cuda                      1.399           28.673          $4.19\n-------------------------------------------------------------------------------------------------------\nTotal time above is `remote backend time + local python code time`,\n in which local python code run time is 27.274 ms.\nGraph level profile is located at /tmp/octoml_profile_n603dewx/0/predict_1*\n```\n\n## Installation and Getting Started\n- Create and activate a python virtual environment. `Python 3.8` is recommended\n  and tested on both `Ubuntu` and `macOS`. `Python 3.10.9` is tested on `macOS`\n  with Apple silicon.\n\n  ```\n  python3 -m venv env\n  source env/bin/activate\n  ```\n- Install dependencies\n\n  PyTorch 2.0 and above is required. Below we install the cpu version for\n  simplicity; CUDA version works too.\n\n  ```\n  pip install --upgrade pip\n  pip install \"torch\u003e=2.0.0\" torchvision torchaudio --index-url https://download.pytorch.org/whl/cpu\n  pip install \"octoml-profile\u003e=0.2.0\"\n  ```\n\n  You've completed installation! (If you have trouble, see [issues with installation](#issues-with-installation))\n\n- Next, try running this very simple example that shows how to integrate octoml-profile into your model code.\n\n  ```python\n  import torch\n  from torch.nn import Linear, ReLU, Sequential\n  from octoml_profile import accelerate, remote_profile\n\n  model = Sequential(Linear(100, 200), ReLU())\n\n  @accelerate\n  def predict(x: torch.Tensor):\n      return model(x)\n\n  with remote_profile():\n      for _ in range(3):\n          x = torch.randn(1, 100)\n          predict(x)\n  ```\n\n- The first time you run this, you'll be prompted to supply your API key.  \n\n  ```\n      ,-\"\"-.\n    /      \\    Welcome to OctoML Profiler!\n    :        ;\n    \\      /    It looks like you don't have an access token configured.\n      `.  .'     Please go to https://profiler.app.octoml.ai/ to generate one\n    '._.'`._.'   and then paste it here.\n\n  Access token: \n  ```\n  (\u003ca href=\"https://profiler.app.octoml.ai/\"\u003eSign up\u003c/a\u003e so that you can generate an API token when prompted)\n\n- Once you've provided credentials, running this results in the following\n  output that shows times of the function being executed remotely on each\n  backend.\n\n  ```\n  Function `predict` has 1 profile:\n  - Profile `predict[1/1]` ran 3 times. (1 discarded because compilation happened)\n\n  Instance     Processor           Backend              Backend Time (ms)  Total Time (ms)  Cost ($/MReq)\n  =======================================================================================================\n  r6i.large    Intel Ice Lake CPU  torch-eager-cpu                  0.024            0.086          $0.00\n  g4dn.xlarge  Nvidia T4 GPU       torch-eager-cuda                 0.097            0.159          $0.02\n  g4dn.xlarge  Nvidia T4 GPU       torch-inductor-cuda              0.177            0.239          $0.03\n  -------------------------------------------------------------------------------------------------------\n  Total time above is `remote backend time + local python code time`,\n  in which local python code run time is 0.062 ms.\n  Graph level profile is located at /tmp/octoml_profile_8o45fe39/0/predict_1*\n  ```\n  To see more examples, see [examples/](examples).\n\n\n### Issues with installation\n\n - If you are on macOS with Apple silicon and seeing `symbol not found in flat\n   namespace '_CFRelease'`, it is likely that you created a `venv` with python\n   installed by `conda`. Please make sure to deactivate any `conda`\n   environment(s) and use the system-shipped python on macOS to create `venv`.\n   Or follow the instructions below to create a conda environment.\n\n    ```\n    conda create -n octoml python=3.8\n    conda activate octoml\n    ```\n\n- If you see a version conflict, please install the pip dependencies above with `--force-reinstall`.\n\n- For any other problems, please file a github issue.\n\n\n### Dynamic shapes\n\nThis is an experimental feature that requires installing nightly of PyTorch.\nThe dynamic shape graph capture feature is still under active development by\nthe PyTorch team, so your results may vary. If you find any problems, please\nreport via github issue.\n\n```\npip install --pre torch==2.1.0.dev20230416 torchaudio==2.1.0.dev20230416 torchvision==0.16.0.dev20230416 --index-url https://download.pytorch.org/whl/nightly/cpu\n```\n\nBy default, the `@accelerate` decorator will recompile a new graph if the input\nshapes to the graph is changed. For generative model cases such as text\ngeneration, it is inefficient to have to compile a separate graph for each\nsequence length. The solution is to turn on \"dynamic-shapes\" for the compiler,\nwhich means the graph compilation will be agnostic to the input shapes,\nresulting in drastically fewer graphs to be compiled and lower memory to run\nend to end.\n\nAs an toy example:\n\n```python\nimport torch\nfrom octoml_profile import accelerate, remote_profile\n\nconv = torch.nn.Conv2d(16, 16, 3)\n\n# With `dynamic=True` any model inside will not be specialized to the input shape\n@accelerate(dynamic=True)\ndef predict(x: torch.Tensor):\n    return conv(x)\n\nwith remote_profile(backends=[\"r6i.large/onnxrt-cpu\"]):\n    # batch size is different but compilation only \n    # happens once\n    for i in range(1, 5):\n        predict(torch.randn(i, 16, 10, 10))\n```\n\nSet `@accelerate(dynamic=True)` on any `accelerate` usage.\n\n\n## How it works\n\n* [How octoml-profile works](#how-octoml-profile-works)\n* [Where `@accelerate` should be applied](#where-accelerate-should-be-applied)\n* [The profile report](#the-profile-report)\n* [Local python segments](#local-python-segments)\n* [Quota](#quota)\n* [Supported backends](#supported-backends)\n\n### How octoml-profile works\n\noctoml-profile consists of two main components: a Python library and a cloud\nservice. The Python library is used to automatically extract PyTorch models on\nyour local machine and send them to the cloud service for remote benchmarking.\nThe cloud service provides access to different cloud hardware targets that are\nprepared with various deep learning inference engines. This enables users to\noptimize and measure their PyTorch models in a variety of deployment\nconfigurations.\n\n![Architecture Illustration](assets/architecture.png)\n\nIn various examples above, we first `import octoml-profile` python library, and then\ndecorate a `predict` function with the `@accelerate` decorator. By default, it\nbehaves like `@torch.compile`:\n[TorchDynamo](https://pytorch.org/docs/stable/dynamo/index.html) is used to\nextract one or more computation graphs, optimize them, and replace the bytecode\ninside the function with the optimized version.\n\nWhen the code is surrounded with `remote_profile()` context manager, the\nbehavior of the `@accelerate` decorator changes. Instead of running the\nextracted graphs on the local machine, the graphs are sent to one or more\nremote inference workers for execution and measurement. The run time of the\noffloaded graphs are referred to as \"remote backend run time\" in the output\nabove.\n\nCode that cannot be captured as a computation graph is not offloaded -- such code\nruns locally and is shown as \"local python\". For more details see\nthe [local python code section](#local-python-segments) below.\n\nWhen the `remote_profile()` context manager is entered, it reserves\nexclusive access to hardware specified in the optional `backends` keyword argument\n(or to a set of default hardware targets if the argument is omitted).\nIf there are multiple backends, they will run in parallel.\n\nThe `predict` function may contain pre/post processing code, non tensor logic\nlike control flows, side effects, and multiple models. Only eligible graphs\nwill be intelligently extracted and offloaded for remote execution.\n\nAs a result, the estimated end to end run time of the decorated function for a\nparticular hardware and acceleration engine is `remote backend run\ntime + local python run time`. If `local python run time` is much smaller\ncomparing to the total time, the estimate is fairly accurate because the\nimpact of potential difference between local and remote machine for local python \ncode is minimal.\n\n\n### Where to apply `@accelerate`\nIn general, `@accelerate` is a drop-in replacement for `@torch.compile` and\nshould be applied to function which contains PyTorch Model that performs\ninference. When the function is called under the context manager of `with\nremote_profile()`, the remote execution and profiling activated. When called\nwithout `remote_profile()` it behaves just as `torch.compile`. By default,\n`torch.no_grad()` is set in the `remote_profile()` context, because the remote\nexecution does not support training yet.\n\nIf you expect the input shape to change especially for generative models,\nsee [Dynamic Shapes](dynamic-shapes).\n\nLast but not least, `@accelerate` should not be used to decorate a function\nthat has already been decorated with `@accelerate` or `@torch.compile`.\n\n### The profile report\n\nMost users should be satisfied with the output of just total run time and cost\nof using different hardware and software backends.\nHowever, advanced users like ML compiler engineers may be interested in diving\ninto graph level performance analysis or reducing the number of graph breaks. This\nsection shows you where to find the next level details.\n\nThe location of the `Profile` report is printed at the end of total runtime\ntable, for instance:\n```\nGraph level profile is located at /tmp/octoml_profile_8o45fe39/0/predict_1*\n```\n\nFor each decorated function, the profile report is suffixed with `.profile.txt`,\ni.e. `/tmp/octoml_profile_8o45fe39/0/predict_1.profile.txt` is for function `predict`.\n\n\nAn example report for DistilBert is the following:\n```\n   Segment                            Samples  Avg ms  Failures\n===============================================================\n0  Local python                             2  27.227\n\n1  Graph #1\n     r6i.large/torch-eager-cpu             20  25.648         0\n     g4dn.xlarge/torch-eager-cuda          20   5.381         0\n     g4dn.xlarge/torch-inductor-cuda       20   3.208         0\n     g4dn.xlarge/onnxrt-cuda               20   1.400         0\n\n2  Local python                             2   0.110\n---------------------------------------------------------------\n```\n\nTo understand this report, let's first define some terminology.\n\nTerminology:\n- `function` is a python function decorated with `@accelerate` to be profiled.\n- `run` is one execution of the function.\n- `subgraph` is a computation graph of tensor operations auto captured by TorchDynamo as\n  you run the function. A subgraph is a logical portion of the function.\n- `call` is one execution of a subgraph. A segment in a profile is a result of a call.\n- `repeats` is the number of times a graph is measured on a remote backend for each call.\n- `samples` is the total number an execution of a subgraph or local python segment is measured.\n  For each subgraph, `samples = repeats * call`.\n\nOn function, subgraph, and profile:\n- A single run of `predict` can have more than one subgraphs due to graph breaks.\n- Runs with different arguments may produce different subgraphs because the\n  computation may change. Runs that have the same sequence of graph execution\n  are merged into a \"profile\". For example, if `f(x1)`, `f(x2)`\nruns graph \"1,2,3\", and `f(x3)` runs graph \"1,3,4\", the segments of `f(x1)` and\n`f(x2)` will be merged into one profile and `f(x3)` will be its own profile.\n\nTo find out what operations each graph contains and where in the source code\ndoes each graph come from, take a look at `graph_dump.txt` under the same\ndirectory.\n\nLike the DistilBert example above, when there are only a few segments, a profile \nwill show the linear sequence of subgraph segment runs. \n\nWhat happens when there are tens or hundreds of segments? When too many segments\nare run, we collapse the linear segment sequence into an abridged summary where\nonly a few subgraphs that have the highest aggregate run times across their run\nsegments are shown.\n\nFor example, a generative encoder-decoder based model that produces a large number of\nrun segments will display an abridged report displaying \n**runtime by subgraph** instead of **runtime by segment** by default.\nIn cases like this, you'll see:\n\n```\nTop subgraph               Avg ms/call  Avg ms/run  Runtime % of e2e  Failures\n==============================================================================\nGraph #7 (17 calls)      \n  r6i.large/onnxrt-cpu          36.979     628.645              70.2         0\n  g4dn.xlarge/onnxrt-cuda        5.612      95.403              37.9         0\n\nGraph #4 (1 calls)       \n  r6i.large/onnxrt-cpu          43.823      43.823               4.9         0\n  g4dn.xlarge/onnxrt-cuda        5.002       5.002               2.0         0\n\nGraph #2 (1 calls)       \n  r6i.large/onnxrt-cpu          43.357      43.357               4.8         0\n  g4dn.xlarge/onnxrt-cuda        3.154       3.154               1.3         0\n\n4 other subgraphs        \n  r6i.large/onnxrt-cpu                      35.892               4.0         0\n  g4dn.xlarge/onnxrt-cuda                    4.833               1.9         0\n\n42 local python segments                   143.621\n------------------------------------------------------------------------------\n```\n\nOther graphs are hidden. If your output has been abridged in this way\nbut you want to see the full, sequential results of your profiling run, you can print\nthe report with `verbose`:\n\n```python\n# print_results_to=None silences the default output profile report.\nwith remote_profile(print_results_to=None) as prof:\n    ...\nprof.report().print(verbose=True)\n```\n\n### Local python segments\n\nYou will see `Local python` in the profiling report because TorchDynamo only\ncaptures PyTorch code as graph. Where it cannot continue capturing, a \"graph\nbreak\" happens and the uncaptured code runs in python eagerly.\n\nLet's look at the DistilBert example again:\n```python\n@accelerate\ndef predict(input: str):\n    inputs = tokenizer(input, return_tensors=\"pt\")\n    logits = model(**inputs).logits\n    predicted_class_id = logits.argmax().item()\n    return model.config.id2label[predicted_class_id]\n```\n\n```\n   Segment                            Samples  Avg ms  Failures\n===============================================================\n0  Local python                             2  27.227\n\n1  Graph #1\n     r6i.large/torch-eager-cpu             20  25.648         0\n     g4dn.xlarge/torch-eager-cuda          20   5.381         0\n     g4dn.xlarge/torch-inductor-cuda       20   3.208         0\n     g4dn.xlarge/onnxrt-cuda               20   1.400         0\n\n2  Local python                             2   0.110\n---------------------------------------------------------------\n```\n\nSegment 1 is runs `Graph #1` which corresponds to the `model`.\nSegment 0 corresponds to the `tokenizer` function, and Segment 2 the logits and\nlabel mapping.\n\nLocal Python segment is run locally once for every invocation. The total estimated time \nequals total remote backend time for all the graphs and total local python time.\n\nTo print graph breaks and understand more of what TorchDynamo is doing under the hood, see\nthe [TorchDynamo Deeper Dive](https://pytorch.org/docs/stable/dynamo/deep-dive.html), \n[Torch.FX](https://pytorch.org/docs/stable/fx.html) and\n[PyTorch\nTroubleshooting](https://pytorch.org/docs/master/dynamo/troubleshooting.html#torchdynamo-troubleshooting)\npages.\n\n\n### Quota\n\nEach user has a limit on the number of concurrent backends held by the user's sessions which automatically get created and closed within `remote_profile()` context manager.\nIf you find yourself hitting quota limits because you need to run more tasks concurrently, please contact us to increase the limit. \n\n### Supported backends\n\nTo programmatically access a list of supported backends, please invoke:\n\n```python\nimport octoml_profile\nprint(octoml_profile.get_supported_backends())\n```\n\n**Supported Cloud Hardware**\n\nAWS\n- g4dn.xlarge (Nvidia T4 GPU)\n- g5.xlarge  (Nvidia A10g GPU)\n- r6i.large (Intel Xeon IceLake CPU)\n- r7g.large (Arm based Graviton3 CPU)\n\n**Supported Acceleration Libraries**\n\nONNXRuntime\n- onnxrt-cpu\n- onnxrt-cuda\n- onnxrt-tensorrt\n\nPyTorch\n- torch-eager-cpu\n- torch-eager-cuda\n- torch-inductor-cpu\n- torch-inductor-cuda\n\nIf no backends are specified while calling `remote_profile(backends=[...])`, then defaults are used,\nwhich are determined by the server. At the moment of writing, the default is\n`[\"r6i.large/torch-eager-cpu\", \"g4dn.xlarge/torch-eager-cuda\", \"g4dn.xlarge/torch-inductor-cuda\"]`.\n\n\n\n## Data privacy\n\nWe know that keeping your model data private is important to you. We guarantee that no other\nuser has access to your data. We do not scrape any model information internally, and we do not use\nyour uploaded data to try to improve our system -- we rely on you filing github issues or otherwise\ncontacting us to understand your use case, and anything else about your model.\nHere's how our system currently works.\n\nWe leverage TorchDynamo's subgraph capture to identify PyTorch-only code, serialize those subgraphs,\nand upload them to our system for benchmark.\ng\nOn model upload, we cache your model in AWS S3. This helps with your development iteration speed --\nevery subsequent time you want profiling results, you won't have to wait for model re-upload on every\nminor tweak to your model or update to your requested backends list. When untouched for four weeks,\nany model subgraphs and constants are automatically removed from S3.\n\nYour model subgraphs are loaded onto our remote workers and are cleaned up on the creation of\nevery subsequent session. Between your session's closure and another session's startup,\nserialized subgraphs may lie around idle. No users can access these subgraphs in this interval.\n\nIf you still have concerns around data privacy, please [contact the team](#contact-the-team).\n\n## Known issues\n\n### Waiting for Session\nWhen you create a session, you get exclusive access to the requested hardware.\nWhen there are no hardware available, new session requests will\nbe queued.\n\n### OOM for large models\nWhen a function contains too many graph breaks or individual\ngraph exceeds the memory on the worker,\nthe remote inference worker may run out of CPU/GPU memory.\nWhen it happens, you may get an \"Error on loading model component\". \nThis is known to happen with models like Stable Diffusion.\nWe are actively working on optimizing the memory allocation of \nmany subgraphs.\n\n### Limitations of TorchDynamo \nTorchDynamo is under active development. You may encounter\nerrors that are TorchDynamo related.\nThese should not be fundamental problems as we believe TorchDynamo\nwill continue to improve its coverage.\nIf you find a broken model, please [file an issue](https://github.com/octoml/octoml-profile/issues).\n\n## Contact the team\n- Discord: [OctoML community Discord](https://discord.gg/Quc8hSxpMe)\n- Github issues: https://github.com/octoml/octoml-profile/issues\n- Email: dynamite@octoml.ai\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foctoml%2Foctoml-profile","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Foctoml%2Foctoml-profile","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foctoml%2Foctoml-profile/lists"}