{"id":23771637,"url":"https://github.com/michurin/pprofiler","last_synced_at":"2025-09-05T15:32:43.741Z","repository":{"id":78009156,"uuid":"103805470","full_name":"michurin/pprofiler","owner":"michurin","description":"Pure Python profiler","archived":false,"fork":false,"pushed_at":"2024-01-21T08:11:17.000Z","size":26,"stargazers_count":5,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2024-01-21T09:23:57.271Z","etag":null,"topics":["profiling","pure-python","python"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/michurin.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2017-09-17T05:29:42.000Z","updated_at":"2023-11-16T00:51:32.000Z","dependencies_parsed_at":"2023-04-03T12:33:43.226Z","dependency_job_id":null,"html_url":"https://github.com/michurin/pprofiler","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/michurin%2Fpprofiler","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/michurin%2Fpprofiler/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/michurin%2Fpprofiler/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/michurin%2Fpprofiler/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/michurin","download_url":"https://codeload.github.com/michurin/pprofiler/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":232047647,"owners_count":18465031,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["profiling","pure-python","python"],"created_at":"2025-01-01T04:20:36.759Z","updated_at":"2025-01-01T04:20:37.929Z","avatar_url":"https://github.com/michurin.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![test](https://github.com/michurin/pprofiler/actions/workflows/ci.yaml/badge.svg)](https://github.com/michurin/pprofiler/actions/workflows/ci.yaml)\n[![codecov](https://codecov.io/gh/michurin/pprofiler/branch/master/graph/badge.svg)](https://codecov.io/gh/michurin/pprofiler)\n[![Python 3.8](https://img.shields.io/badge/python-3.8-blue.svg)](https://www.python.org/)\n[![Python 2.7](https://img.shields.io/badge/python-2.7-blue.svg)](https://www.python.org/)\n\npprofiler\n=========\n\n`pprofiler` (\"Pretty profiler\" or \"Python code level profiler\" or \"Pure Python profiler\")\nis a Python level profiler like `timeit` (against C code level profilers like `cProfile`).\n\nKey features\n------------\n\n* Can be used like\n  * context manager\n  * decorator\n* Provides\n  * total time\n  * mean avarage\n  * standard deviation\n  * min/max\n* Nested scopes\n* Flexible reporting formats\n  * simplest printing to stdout\n  * ouput using `logging` or other engine\n  * data as nested structure to store into some document-oriented database like MongoDB\n  * data as flat structure to store into some relational database like PostgreSQL\n* Easy usage/integration\n\nQuick overview\n--------------\n\nYou can found all examples in `/examples` dir.\n\nTo use `pprofiler` you need to import only one symbol. You can drop `pprofiler.py` into you\nsource tree and use it without installation `pprofiler` package.\n\n### Using as context manager\n\n```python\n#!/usr/bin/python\n# coding: U8\n\n\nimport time\nimport random\n\nfrom pprofiler import profiler\n\n\ndef rand_sleep(dt):\n    time.sleep(dt * (1 + 0.4 * (random.random() - 0.5)))  # dt ± 20%\n\n\ndef main():\n    with profiler('prepare input'):\n        rand_sleep(.1)\n    for n in range(3):\n        with profiler('process chunk'):\n            rand_sleep(.2)\n    with profiler('push result'):\n        rand_sleep(.1)\n    profiler.print_report()\n\n\nif __name__ == '__main__':\n    main()\n```\n\nOutput:\n\n```\nname             perc   sum  n   avg   max   min   dev\n---------------- ---- ----- -- ----- ----- ----- -----\nprocess chunk ..  73%  0.58  3  0.19  0.22  0.18  0.02\npush result ....  14%  0.11  1  0.11  0.11  0.11     -\nprepare input ..  13%  0.10  1  0.10  0.10  0.10     -\n```\n\n### Using as decorator:\n\n```python\n#!/usr/bin/python\n# coding: U8\n\n\nimport time\nimport random\n\nfrom pprofiler import profiler\n\n\ndef rand_sleep(dt):\n    time.sleep(dt * (1 + 0.4 * (random.random() - 0.5)))  # dt ± 20%\n\n\n@profiler('prepare input')\ndef prepare_input():\n    rand_sleep(.1)\n\n\n@profiler('process chunk')\ndef process_chunk():\n    rand_sleep(.2)\n\n\n@profiler('push result')\ndef push_result():\n    rand_sleep(.1)\n\n\ndef main():\n    prepare_input()\n    for n in range(3):\n        process_chunk()\n    push_result()\n    profiler.print_report()\n\n\nif __name__ == '__main__':\n    main()\n```\n\nOutput:\n\n```\nname             perc   sum  n   avg   max   min   dev\n---------------- ---- ----- -- ----- ----- ----- -----\nprocess chunk ..  74%  0.64  3  0.21  0.23  0.19  0.03\nprepare input ..  14%  0.12  1  0.12  0.12  0.12     -\npush result ....  13%  0.11  1  0.11  0.11  0.11     -\n```\n\n### Report formats\n\nYou can get reports in different ways and formats. Look at this simplest example:\n\n```python\n#!/usr/bin/python\n# coding: U8\n\n\nimport time\nimport random\nimport pprint\nimport sys\nimport logging\n\nfrom pprofiler import profiler\n\n\nlogger = logging.getLogger(__name__)\n\n\ndef rand_sleep(dt):\n    time.sleep(dt * (1 + 0.4 * (random.random() - 0.5)))  # dt ± 20%\n\n\n@profiler('demo')\ndef demo():\n    rand_sleep(.2)\n\n\ndef main():\n    logging.basicConfig(\n        format='%(asctime)s [%(levelname)s] [%(process)d] %(message)s',\n        datefmt='%H:%M:%S',\n        level=logging.DEBUG,\n        handlers=[logging.StreamHandler(sys.stdout)])\n\n    demo()\n    demo()\n    profiler.print_report()  # print to stdout\n    profiler.print_report(logger.info)  # print using custom logger\n    pprint.pprint(profiler.report)  # get report as structure\n    pprint.pprint(list(profiler))  # get report as lines\n\n\nif __name__ == '__main__':\n    main()\n```\n\nYou can just print formated report to `stdout` using `profiler.print_report()`:\n\n```\nname     perc   sum  n   avg   max   min   dev\n------- ----- ----- -- ----- ----- ----- -----\ndemo ..  100%  0.37  2  0.19  0.21  0.16  0.03\n```\n\nYou can use you custom logger like this `profiler.print_report(logger.info)`:\n\n```\n13:06:22 [INFO] [27038] name     perc   sum  n   avg   max   min   dev\n13:06:22 [INFO] [27038] ------- ----- ----- -- ----- ----- ----- -----\n13:06:22 [INFO] [27038] demo ..  100%  0.37  2  0.19  0.21  0.16  0.03\n```\n\nYou can get report as nested structure (`profiler.report`) to store it into some document-orienteted storage like MongoDB:\n\n```python\n[{'avg': 0.1868218183517456,\n  'dev': 0.03486269297645324,\n  'max': 0.2114734649658203,\n  'min': 0.1621701717376709,\n  'name': 'demo',\n  'num': 2,\n  'percent': 100.0,\n  'sum': 0.3736436367034912}]\n```\n\nAnd you can get the report as flat list of items to store report line by line to some relational databases, using `profiler` as iterator:\n\n```python\n[{'avg': 0.18060684204101562,\n  'dev': 0.0004245030582017247,\n  'level': 0,\n  'max': 0.1809070110321045,\n  'min': 0.18030667304992676,\n  'name': 'demo',\n  'num': 2,\n  'percent': 100.0,\n  'sum': 0.36121368408203125}]\n```\n\nYou can find more complex exmaples in 'examples/' directory.\n\n### Nested scopes\n\nNested scopes helps you to study your timings deeper. Look at this:\n\n```python\n#!/usr/bin/python\n# coding: U8\n\n\nimport time\nimport random\n\nfrom pprofiler import profiler\n\n\ndef rand_sleep(dt):\n    time.sleep(dt * (1 + 0.4 * (random.random() - 0.5)))  # dt ± 20%\n\n\ndef main():\n    with profiler('cook document'):\n        rand_sleep(.1)  # we do not want to give the individual attention to this\n        with profiler('create title'):\n            rand_sleep(.2)\n        with profiler('create body'):\n            rand_sleep(.3)\n    with profiler('push document'):\n        rand_sleep(.5)\n    profiler.print_report()\n\n\nif __name__ == '__main__':\n    main()\n```\n\nHere we have two high level tasks: (i) 'cook document' and (ii) 'push document'. But we want to pay attention\nto some subtasks in 'cook document': 'create title' and 'create body'.\n\nWe can get report like this:\n\n```\nname              perc   sum  n   avg   max   min dev\n----------------- ---- ----- -- ----- ----- ----- ---\ncook document ...  52%  0.63  1  0.63  0.63  0.63   -\n. create body ...  65%  0.34  1  0.34  0.34  0.34   -\n. create title ..  35%  0.18  1  0.18  0.18  0.18   -\npush document ...  48%  0.59  1  0.59  0.59  0.59   -\n```\n\nYou can read it level by level. This is high level tasks:\n\n```\n...\ncook document ...  52%  0.63  1  0.63  0.63  0.63   -\n...\npush document ...  48%  0.59  1  0.59  0.59  0.59   -\n```\n\nYou can see, that cooking takes 52% of time, and pushing takes 48%. But you can look deeper\ninside of cooking:\n\n```\n...\n. create body ...  65%  0.34  1  0.34  0.34  0.34   -\n. create title ..  35%  0.18  1  0.18  0.18  0.18   -\n...\n```\n\nHere you can analyze parts of 'cook document' task.\n\nMultithreading/multiprocessing\n------------------------------\n\n`pprofiler` is not designed for concurrent computing. But it is possible to\nuse it in multithreading/multiprocessing applications. The simplest way is\nto build autonomous instance of `pprofiler` in each worker:\n\n```python\n#!/usr/bin/python\n# coding: U8\n\n\nimport time\nimport random\nimport multiprocessing\nimport sys\nimport logging\n\nfrom pprofiler import profiler\n\n\nlogger = logging.getLogger()\n\n\ndef rand_sleep(dt):\n    time.sleep(dt * (1 + 0.4 * (random.random() - 0.5)))  # dt ± 20%\n\n\ndef subprocess_worker(x):\n    local_profiler = type(profiler)()  # create and use local profiler in each child process\n    logger.debug('Worker %d start', x)\n    with local_profiler('worker_%d' % x):\n        rand_sleep(3)\n    logger.debug('Worker %d done', x)\n    local_profiler.print_report(logger.info)\n\n\ndef main():\n    logging.basicConfig(\n        format='%(asctime)s,%(msecs)03d [%(process)d] [%(levelname)s] %(message)s',\n        datefmt='%H:%M:%S',\n        level=logging.DEBUG,\n        handlers=[logging.StreamHandler(sys.stdout)])\n    logger.debug('Master start')\n    with profiler('master (total)'):\n        pool = multiprocessing.Pool(processes=2)\n        for i in pool.imap_unordered(subprocess_worker, range(2)):\n            pass\n    logger.debug('Master done')\n    profiler.print_report(logger.info)\n\n\nif __name__ == '__main__':\n    main()\n```\n\nYou can get something like this:\n\n```\n22:53:11,092 [1216] [DEBUG] Master start\n22:53:11,236 [1217] [DEBUG] Worker 0 start\n22:53:11,237 [1218] [DEBUG] Worker 1 start\n22:53:14,189 [1218] [DEBUG] Worker 1 done\n22:53:14,190 [1218] [INFO] name         perc   sum  n   avg   max   min dev\n22:53:14,190 [1218] [INFO] ----------- ----- ----- -- ----- ----- ----- ---\n22:53:14,191 [1218] [INFO] worker_1 ..  100%  2.95  1  2.95  2.95  2.95   -\n22:53:14,402 [1217] [DEBUG] Worker 0 done\n22:53:14,403 [1217] [INFO] name         perc   sum  n   avg   max   min dev\n22:53:14,403 [1217] [INFO] ----------- ----- ----- -- ----- ----- ----- ---\n22:53:14,404 [1217] [INFO] worker_0 ..  100%  3.16  1  3.16  3.16  3.16   -\n22:53:14,405 [1216] [DEBUG] Master done\n22:53:14,406 [1216] [INFO] name               perc   sum  n   avg   max   min dev\n22:53:14,406 [1216] [INFO] ----------------- ----- ----- -- ----- ----- ----- ---\n22:53:14,407 [1216] [INFO] master (total) ..  100%  3.31  1  3.31  3.31  3.31   -\n```\n\nYou can see three processes: one master (PID=1216) and two workers (PID=1217 and 1218).\nEach of them uses a separate instance of `pprofiler` and gets its own report.\n\nThis is not the only solution. You can collect all reports in one process/thread\nand store all results or prepare you custom report:\n\n```python\n#!/usr/bin/python\n# coding: U8\n\n\nimport time\nimport random\nimport multiprocessing\nimport sys\nimport logging\nimport itertools\n\nfrom pprofiler import profiler\n\n\nlogger = logging.getLogger()\n\n\ndef rand_sleep(dt):\n    time.sleep(dt * (1 + 0.4 * (random.random() - 0.5)))  # dt ± 20%\n\n\ndef subprocess_worker(x):\n    local_profiler = type(profiler)()  # create and use local profiler in each child process\n    logger.debug('Worker %d start', x)\n    with local_profiler('worker_%d' % x):\n        rand_sleep(3)\n    logger.debug('Worker %d done', x)\n    return local_profiler\n\n\ndef main():\n    logging.basicConfig(\n        format='%(asctime)s,%(msecs)03d [%(process)d] [%(levelname)s] %(message)s',\n        datefmt='%H:%M:%S',\n        level=logging.DEBUG,\n        handlers=[logging.StreamHandler(sys.stdout)])\n    logger.debug('Master start')\n    with profiler('master (total)'):\n        pool = multiprocessing.Pool(processes=2)\n        workers_reports = list(pool.imap_unordered(subprocess_worker, range(2)))\n    logger.debug('Master done')\n    logger.info('== Native reports:')\n    profiler.print_report(logger.info)\n    for r in workers_reports:\n        r.print_report(logger.info)\n    logger.info('== Simple custom report:')\n    for r in itertools.chain.from_iterable(x.report for x in [profiler] + workers_reports):\n        logger.info('{name:.\u003c20s} {sum:.3f} seconds'.format(**r))\n\n\nif __name__ == '__main__':\n    main()\n```\n\nResult:\n\n```\n10:24:48,085 [1353] [DEBUG] Master start\n10:24:48,136 [1355] [DEBUG] Worker 1 start\n10:24:48,136 [1354] [DEBUG] Worker 0 start\n10:24:51,391 [1355] [DEBUG] Worker 1 done\n10:24:51,705 [1354] [DEBUG] Worker 0 done\n10:24:51,707 [1353] [DEBUG] Master done\n10:24:51,708 [1353] [INFO] == Native reports:\n10:24:51,708 [1353] [INFO] name               perc   sum  n   avg   max   min dev\n10:24:51,709 [1353] [INFO] ----------------- ----- ----- -- ----- ----- ----- ---\n10:24:51,709 [1353] [INFO] master (total) ..  100%  3.62  1  3.62  3.62  3.62   -\n10:24:51,710 [1353] [INFO] name         perc   sum  n   avg   max   min dev\n10:24:51,710 [1353] [INFO] ----------- ----- ----- -- ----- ----- ----- ---\n10:24:51,710 [1353] [INFO] worker_1 ..  100%  3.25  1  3.25  3.25  3.25   -\n10:24:51,711 [1353] [INFO] name         perc   sum  n   avg   max   min dev\n10:24:51,711 [1353] [INFO] ----------- ----- ----- -- ----- ----- ----- ---\n10:24:51,711 [1353] [INFO] worker_0 ..  100%  3.57  1  3.57  3.57  3.57   -\n10:24:51,712 [1353] [INFO] == Simple custom report:\n10:24:51,712 [1353] [INFO] master (total)...... 3.621 seconds\n10:24:51,712 [1353] [INFO] worker_1............ 3.253 seconds\n10:24:51,713 [1353] [INFO] worker_0............ 3.567 seconds\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmichurin%2Fpprofiler","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmichurin%2Fpprofiler","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmichurin%2Fpprofiler/lists"}