{"id":16693433,"url":"https://github.com/paulross/pymemtrace","last_synced_at":"2025-03-17T00:33:25.495Z","repository":{"id":39575284,"uuid":"113045318","full_name":"paulross/pymemtrace","owner":"paulross","description":"Python memory tracing.","archived":false,"fork":false,"pushed_at":"2024-09-12T17:42:46.000Z","size":748,"stargazers_count":22,"open_issues_count":0,"forks_count":2,"subscribers_count":3,"default_branch":"master","last_synced_at":"2024-10-13T16:31:10.046Z","etag":null,"topics":["memory-management","python","python-memory"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/paulross.png","metadata":{"files":{"readme":"README.rst","changelog":"HISTORY.rst","contributing":"CONTRIBUTING.rst","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":"AUTHORS.rst","dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-12-04T13:21:23.000Z","updated_at":"2024-09-12T17:42:50.000Z","dependencies_parsed_at":"2024-10-12T16:40:36.931Z","dependency_job_id":null,"html_url":"https://github.com/paulross/pymemtrace","commit_stats":{"total_commits":99,"total_committers":2,"mean_commits":49.5,"dds":"0.010101010101010055","last_synced_commit":"946b0ec9a79109bcc459d57f6fd47197c42ee7a4"},"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/paulross%2Fpymemtrace","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/paulross%2Fpymemtrace/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/paulross%2Fpymemtrace/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/paulross%2Fpymemtrace/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/paulross","download_url":"https://codeload.github.com/paulross/pymemtrace/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":221669504,"owners_count":16860895,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["memory-management","python","python-memory"],"created_at":"2024-10-12T16:30:29.731Z","updated_at":"2024-10-27T11:36:46.433Z","avatar_url":"https://github.com/paulross.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"*******************\nIntroduction\n*******************\n\n\n``pymemtrace`` provides tools for tracking and understanding Python memory usage at different levels, at different\ngranularities and with different runtime costs.\n\nFull documentation: https://pymemtrace.readthedocs.io\n\npymemtrace Tools\n======================\n\nThe tools provided by ``pymemtrace``:\n\n* ``process`` is a very lightweight way of logging the total memory usage at regular time intervals.\n  It can plot memory over time with plotting programs such as ``gnuplot``.\n  See `some process examples \u003chttps://pymemtrace.readthedocs.io/en/latest/examples/process.html\u003e`_\n* ``cPyMemTrace`` is a memory tracer written in C that can report total memory usage for every function call/return for\n  both C and Python sections.\n  See some `cPyMemTrace examples \u003chttps://pymemtrace.readthedocs.io/en/latest/examples/c_py_mem_trace.html\u003e`_\n  and a `technical note on cPyMemTrace \u003chttps://pymemtrace.readthedocs.io/en/latest/tech_notes/cPyMemTrace.html\u003e`_.\n* DTrace: Here are a number of D scripts that can trace the low level ``malloc()`` and ``free()`` system calls and\n  report how much memory was allocated and by whom.\n  See some `DTrace examples \u003chttps://pymemtrace.readthedocs.io/en/latest/examples/dtrace.html\u003e`_\n  and a `technical note on DTrace \u003chttps://pymemtrace.readthedocs.io/en/latest/tech_notes/dtrace.html\u003e`_.\n* ``trace_malloc`` is a convenience wrapper around the Python standard library `tracemalloc` module.\n  This can report Python memory usage by module and line compensating for the cost of ``tracemalloc``.\n  This can take memory snapshots before and after code blocks and show the change on memory caused by that code.\n  See some `trace_malloc examples \u003chttps://pymemtrace.readthedocs.io/en/latest/examples/trace_malloc.html\u003e`_\n* ``debug_malloc_stats`` is a wrapper around the ``sys._debugmallocstats`` function that can take snapshots of\n  memory before and after code execution and report the significant differences of the Python small object allocator.\n  See some `debug_malloc_stats examples \u003chttps://pymemtrace.readthedocs.io/en/latest/examples/debug_malloc_stats.html\u003e`_\n\n\nTool Characteristics\n======================\n\nEach tool can be characterised by:\n\n- *Memory Granularity*: In how much detail is a memory change is observed.\n  An example of *coarse* memory granularity is measuring the\n  `Resident Set Size \u003chttps://en.wikipedia.org/wiki/Resident_set_size\u003e`_ which is normally in chunks of 4096 bytes.\n  An example of *fine* memory granularity is recording every ``malloc()`` and ``free()``.\n- *Execution Granularity*: In how much code detail is the memory change observed.\n  An example of *coarse* execution granularity is measuring the memory usage every second.\n  An example of *fine* execution granularity is recording the memory usage every Python line.\n- *Memory Cost*: How much extra memory the tool needs.\n- *Execution Cost*: How much the execution time is increased.\n\nClearly there are trade-offs between these depending on the problem you are trying to solve.\n\n.. list-table:: **Tool Characteristics**\n   :widths: 15 30 30 30 30\n   :header-rows: 1\n\n   * - Tool\n     - Memory Granularity\n     - Execution Granularity\n     - Memory Cost\n     - Execution Cost\n   * - ``process``\n     - RSS (total Python and C memory).\n     - Regular time intervals.\n     - Near zero.\n     - Near zero.\n   * - ``cPyMemTrace``\n     - RSS (total Python and C memory).\n     - Per Python line, Python function and C function call.\n     - Near zero.\n     - x10 to x20.\n   * - DTrace\n     - Every ``malloc()`` and ``free()``.\n     - Per function call and return.\n     - Minimal.\n     - x90 to x100.\n   * - ``trace_malloc``\n     - Every Python object.\n     - Per Python line, per function call.\n     - Significant but compensated.\n     - x900 for small objects, x6 for large objects.\n   * - ``debug_malloc_stats``\n     - Python memory pool.\n     - Snapshots the CPython memory pool either side of a block of code.\n     - Minimal.\n     - x2000+ for small objects, x12 for large objects.\n\nPackage Metadata\n=========================\n\n.. image:: https://img.shields.io/pypi/v/pymemtrace.svg\n        :target: https://pypi.python.org/pypi/pymemtrace\n\n.. image:: https://img.shields.io/travis/paulross/pymemtrace.svg\n        :target: https://travis-ci.org/paulross/pymemtrace\n\n.. image:: https://readthedocs.org/projects/pymemtrace/badge/?version=latest\n        :target: https://pymemtrace.readthedocs.io/en/latest/?badge=latest\n        :alt: Documentation Status\n\n.. image:: https://pyup.io/repos/github/paulross/pymemtrace/shield.svg\n     :target: https://pyup.io/repos/github/paulross/pymemtrace/\n     :alt: Updates\n    \n\nLicence\n-----------------------\n\nPython memory tracing.\n\n* Free software: MIT license\n* Documentation: https://pymemtrace.readthedocs.io.\n* Project: https://github.com/paulross/pymemtrace.\n\nCredits\n-----------------\n\nPhil Smith (AHL) with whom a casual lunch time chat lead to the creation of an earlier, but quite different\nimplementation, of ``cPyMemTrace`` in pure Python.\n\nThis package was created with Cookiecutter_ and the `audreyr/cookiecutter-pypackage`_ project template.\n\n.. _Cookiecutter: https://github.com/audreyr/cookiecutter\n.. _`audreyr/cookiecutter-pypackage`: https://github.com/audreyr/cookiecutter-pypackage\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpaulross%2Fpymemtrace","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpaulross%2Fpymemtrace","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpaulross%2Fpymemtrace/lists"}