{"id":19275486,"url":"https://github.com/liran-funaro/objsize","last_synced_at":"2025-04-04T14:06:18.003Z","repository":{"id":44390544,"uuid":"161351886","full_name":"liran-funaro/objsize","owner":"liran-funaro","description":"Traversal over Python's objects subtree and calculate the total size of the subtree in bytes (deep size).","archived":false,"fork":false,"pushed_at":"2025-01-19T01:29:06.000Z","size":186,"stargazers_count":38,"open_issues_count":0,"forks_count":6,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-28T13:05:44.033Z","etag":null,"topics":["exclusive-objects","python","python-objects","traversal"],"latest_commit_sha":null,"homepage":"https://liran-funaro.github.io/objsize/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/liran-funaro.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-12-11T15:09:52.000Z","updated_at":"2025-01-19T01:27:24.000Z","dependencies_parsed_at":"2023-01-18T15:53:41.685Z","dependency_job_id":"57b451b3-f9ad-46d6-9d36-964bd3540221","html_url":"https://github.com/liran-funaro/objsize","commit_stats":{"total_commits":39,"total_committers":4,"mean_commits":9.75,"dds":"0.46153846153846156","last_synced_commit":"b9ab0f5505e30d046f5eaa1e3b6b93706be24a40"},"previous_names":[],"tags_count":15,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/liran-funaro%2Fobjsize","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/liran-funaro%2Fobjsize/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/liran-funaro%2Fobjsize/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/liran-funaro%2Fobjsize/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/liran-funaro","download_url":"https://codeload.github.com/liran-funaro/objsize/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247186440,"owners_count":20898159,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["exclusive-objects","python","python-objects","traversal"],"created_at":"2024-11-09T20:50:56.033Z","updated_at":"2025-04-04T14:06:17.984Z","avatar_url":"https://github.com/liran-funaro.png","language":"Python","readme":"# objsize\n\n[![Coverage Status](https://coveralls.io/repos/github/liran-funaro/objsize/badge.svg?branch=master)](https://coveralls.io/github/liran-funaro/objsize?branch=master) [![Downloads](https://static.pepy.tech/badge/objsize)](https://pepy.tech/project/objsize)\n\nThe `objsize` Python package allows for the exploration and\nmeasurement of an object’s complete memory usage in bytes, including its\nchild objects. This process, often referred to as deep size calculation,\nis achieved through Python’s internal Garbage Collection (GC) mechanism.\n\nThe `objsize` package is designed to ignore shared objects, such as\n`None`, types, modules, classes, functions, and lambdas, because they\nare shared across many instances. One of the key performance features of\n`objsize` is that it avoids recursive calls, ensuring a faster and\nsafer execution.\n\n## Key Features\n\n* Traverse objects’ subtree\n* Calculates the size of objects, including nested objects (deep size), in bytes\n* Exclude non-exclusive objects\n* Exclude specified objects subtree\n* Provides flexibility by allowing users to define custom handlers for:\n  - Object’s size calculation\n  - Object’s referents (i.e., its children)\n  - Object filter (skip specific objects)\n\n## Documentation\n\n| [`objsize`](https://liran-funaro.github.io/objsize/library/objsize.html#module-objsize)   | Traversal over Python's objects subtree and calculating the total size of the subtree (deep size).   |\n|-------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------|\n\n# Install\n\n```bash\npip install objsize==0.7.1\n```\n\n# Basic Usage\n\nCalculate the size of the object including all its members in bytes.\n\n```pycon\n\u003e\u003e\u003e import objsize\n\u003e\u003e\u003e objsize.get_deep_size(dict(arg1='hello', arg2='world'))\n340\n```\n\nIt is possible to calculate the deep size of multiple objects by passing\nmultiple arguments:\n\n```pycon\n\u003e\u003e\u003e objsize.get_deep_size(['hello', 'world'], dict(arg1='hello', arg2='world'), {'hello', 'world'})\n628\n```\n\n# Complex Data\n\n`objsize` can calculate the size of an object’s entire subtree in\nbytes regardless of the type of objects in it, and its depth.\n\nHere is a complex data structure, for example, that include a self\nreference:\n\n```python\nmy_data = list(range(3)), list(range(3, 6))\n\nclass MyClass:\n    def __init__(self, x, y):\n        self.x = x\n        self.y = y\n        self.d = {'x': x, 'y': y, 'self': self}\n\n    def __repr__(self):\n        return f\"{self.__class__.__name__}()\"\n\nmy_obj = MyClass(*my_data)\n```\n\nWe can calculate `my_obj` deep size, including its stored data.\n\n```pycon\n\u003e\u003e\u003e objsize.get_deep_size(my_obj)\n724\n```\n\nWe might want to ignore non-exclusive objects such as the ones stored in\n`my_data`.\n\n```pycon\n\u003e\u003e\u003e objsize.get_deep_size(my_obj, exclude=[my_data])\n384\n```\n\nOr simply let `objsize` detect that automatically:\n\n```pycon\n\u003e\u003e\u003e objsize.get_exclusive_deep_size(my_obj)\n384\n```\n\n# Non Shared Functions or Classes\n\n`objsize` filters functions, lambdas, and classes by default since\nthey are usually shared among many objects. For example:\n\n```pycon\n\u003e\u003e\u003e method_dict = {\"identity\": lambda x: x, \"double\": lambda x: x*2}\n\u003e\u003e\u003e objsize.get_deep_size(method_dict)\n232\n```\n\nSome objects, however, as illustrated in the above example, have unique\nfunctions not shared by other objects. Due to this, it may be useful to\ncount their sizes. You can achieve this by providing an alternative\nfilter function.\n\n```pycon\n\u003e\u003e\u003e objsize.get_deep_size(method_dict, filter_func=objsize.shared_object_filter)\n986\n```\n\nNotes:\n\n* The default filter function is\n  [`objsize.traverse.shared_object_or_function_filter()`](https://liran-funaro.github.io/objsize/library/objsize.traverse.html#objsize.traverse.shared_object_or_function_filter).\n* When using [`objsize.traverse.shared_object_filter()`](https://liran-funaro.github.io/objsize/library/objsize.traverse.html#objsize.traverse.shared_object_filter), shared functions and\n  lambdas are also counted, but builtin functions are still excluded.\n\n# Special Cases\n\nSome objects handle their data in a way that prevents Python’s GC from\ndetecting it. The user can supply a special way to calculate the actual\nsize of these objects.\n\n## Case 1: [`torch`](https://pytorch.org/docs/stable/torch.html#module-torch)\n\nUsing a simple calculation of the object size won’t work for\n[`torch.Tensor`](https://pytorch.org/docs/stable/tensors.html#torch.Tensor).\n\n```pycon\n\u003e\u003e\u003e import torch\n\u003e\u003e\u003e objsize.get_deep_size(torch.rand(200))\n72\n```\n\nSo the user can define its own size calculation handler for such cases:\n\n```python\nimport objsize\nimport sys\nimport torch\n\ndef get_size_of_torch(o):\n    # `objsize.safe_is_instance` catches `ReferenceError` caused by `weakref` objects\n    if objsize.safe_is_instance(o, torch.Tensor):\n        return sys.getsizeof(o) + (o.element_size() * o.nelement())\n    else:\n        return sys.getsizeof(o)\n```\n\nThen use it as follows:\n\n```pycon\n\u003e\u003e\u003e objsize.get_deep_size(\n...   torch.rand(200),\n...   get_size_func=get_size_of_torch\n... )\n872\n```\n\nThe above approach may neglect the object’s internal structure. The user\ncan help `objsize` to find the object’s hidden storage by supplying it\nwith its own referent and filter functions:\n\n```python\nimport objsize\nimport gc\nimport torch\n\ndef get_referents_torch(*objs):\n    # Yield all native referents\n    yield from gc.get_referents(*objs)\n    for o in objs:\n        # If the object is a torch tensor, then also yield its storage\n        if type(o) == torch.Tensor:\n            yield o.untyped_storage()\n\n# `torch.dtype` is a common object like Python's types.\nMySharedObjects = (*objsize.SharedObjectOrFunctionType, torch.dtype)\n\ndef filter_func(o):\n    return not objsize.safe_is_instance(o, MySharedObjects)\n```\n\nThen use these as follows:\n\n```pycon\n\u003e\u003e\u003e objsize.get_deep_size(\n...   torch.rand(200),\n...   get_referents_func=get_referents_torch,\n...   filter_func=filter_func\n... )\n928\n```\n\n## Case 2: [`weakref`](https://docs.python.org/3/library/weakref.html#module-weakref)\n\nUsing a simple calculation of the object size won’t work for\n`weakref.proxy`.\n\n```pycon\n\u003e\u003e\u003e from collections import UserList\n\u003e\u003e\u003e o = UserList([0]*100)\n\u003e\u003e\u003e objsize.get_deep_size(o)\n1032\n\u003e\u003e\u003e import weakref\n\u003e\u003e\u003e o_ref = weakref.proxy(o)\n\u003e\u003e\u003e objsize.get_deep_size(o_ref)\n72\n```\n\nTo mitigate this, you can provide a method that attempts to fetch the\nproxy’s referents:\n\n```python\nimport weakref\nimport gc\n\ndef get_weakref_referents(*objs):\n    yield from gc.get_referents(*objs)\n    for o in objs:\n        if type(o) in weakref.ProxyTypes:\n            try:\n                yield o.__repr__.__self__\n            except ReferenceError:\n                pass\n```\n\nThen use it as follows:\n\n```pycon\n\u003e\u003e\u003e objsize.get_deep_size(o_ref, get_referents_func=get_weakref_referents)\n1104\n```\n\nAfter the referenced object will be collected, then the size of the\nproxy object will be reduced.\n\n```pycon\n\u003e\u003e\u003e del o\n\u003e\u003e\u003e gc.collect()\n\u003e\u003e\u003e # Wait for the object to be collected\n\u003e\u003e\u003e objsize.get_deep_size(o_ref, get_referents_func=get_weakref_referents)\n72\n```\n\n# Object Size Settings\n\nTo avoid repeating the input settings when handling the special cases\nabove, you can use the [`ObjSizeSettings`](https://liran-funaro.github.io/objsize/library/objsize.traverse.html#objsize.traverse.ObjSizeSettings) class.\n\n```pycon\n\u003e\u003e\u003e torch_objsize = objsize.ObjSizeSettings(\n...   get_referents_func=get_referents_torch,\n...   filter_func=filter_func,\n... )\n\u003e\u003e\u003e torch_objsize.get_deep_size(torch.rand(200))\n928\n\u003e\u003e\u003e torch_objsize.get_deep_size(torch.rand(300))\n1328\n```\n\nSee [`ObjSizeSettings`](https://liran-funaro.github.io/objsize/library/objsize.traverse.html#objsize.traverse.ObjSizeSettings) for the\nlist of configurable parameters.\n\n# Traversal\n\nA user can implement its own function over the entire subtree using the\ntraversal method, which traverses all the objects in the subtree.\n\n```pycon\n\u003e\u003e\u003e for o in objsize.traverse_bfs(my_obj):\n...     print(o)\n...\nMyClass()\n{'x': [0, 1, 2], 'y': [3, 4, 5], 'd': {'x': [0, 1, 2], 'y': [3, 4, 5], 'self': MyClass()}}\n[0, 1, 2]\n[3, 4, 5]\n{'x': [0, 1, 2], 'y': [3, 4, 5], 'self': MyClass()}\n2\n1\n0\n5\n4\n3\n```\n\nSimilar to before, non-exclusive objects can be ignored.\n\n```pycon\n\u003e\u003e\u003e for o in objsize.traverse_exclusive_bfs(my_obj):\n...     print(o)\n...\nMyClass()\n{'x': [0, 1, 2], 'y': [3, 4, 5], 'd': {'x': [0, 1, 2], 'y': [3, 4, 5], 'self': MyClass()}}\n{'x': [0, 1, 2], 'y': [3, 4, 5], 'self': MyClass()}\n```\n\n# Alternative\n\n[Pympler](https://pythonhosted.org/Pympler/) also supports\ndetermining an object deep size via `pympler.asizeof()`. There are two\nmain differences between `objsize` and `pympler`.\n\n1. `objsize` has additional features:\n   * Traversing the object subtree: iterating all the object’s\n     descendants one by one.\n   * Excluding non-exclusive objects. That is, objects that are also\n     referenced from somewhere else in the program. This is true for\n     calculating the object’s deep size and for traversing its\n     descendants.\n2. `objsize` has a simple and robust implementation with significantly\n   fewer lines of code, compared to `pympler`. The Pympler\n   implementation uses recursion, and thus have to use a maximal depth\n   argument to avoid reaching Python’s max depth. `objsize`, however,\n   uses BFS which is more efficient and simple to follow. Moreover, the\n   Pympler implementation carefully takes care of any object type.\n   `objsize` archives the same goal with a simple and generic\n   implementation, which has fewer lines of code.\n\n# License: BSD-3\n\nCopyright (c) 2006-2025, Liran Funaro.\nAll rights reserved.\n\nRedistribution and use in source and binary forms, with or without\nmodification, are permitted provided that the following conditions are met:\n\n1. Redistributions of source code must retain the above copyright\n   notice, this list of conditions and the following disclaimer.\n2. Redistributions in binary form must reproduce the above copyright\n   notice, this list of conditions and the following disclaimer in the\n   documentation and/or other materials provided with the distribution.\n3. Neither the name of the copyright holder nor the\n   names of its contributors may be used to endorse or promote products\n   derived from this software without specific prior written permission.\n\nTHIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS “AS IS”\nAND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE\nIMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE\nARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE\nLIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR\nCONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF\nSUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS\nINTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN\nCONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)\nARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE\nPOSSIBILITY OF SUCH DAMAGE.\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fliran-funaro%2Fobjsize","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fliran-funaro%2Fobjsize","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fliran-funaro%2Fobjsize/lists"}