{"id":18591785,"url":"https://github.com/bshillingford/python-torchfile","last_synced_at":"2025-04-05T07:08:16.954Z","repository":{"id":56825682,"uuid":"51490769","full_name":"bshillingford/python-torchfile","owner":"bshillingford","description":"Deserialize Lua torch-serialized objects from Python","archived":false,"fork":false,"pushed_at":"2019-12-27T05:08:53.000Z","size":36,"stargazers_count":216,"open_issues_count":5,"forks_count":25,"subscribers_count":8,"default_branch":"master","last_synced_at":"2025-03-29T06:08:49.212Z","etag":null,"topics":["deep-learning","lua","machine-learning","python","torch"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bshillingford.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2016-02-11T03:02:18.000Z","updated_at":"2025-03-26T10:08:02.000Z","dependencies_parsed_at":"2022-09-13T08:00:35.968Z","dependency_job_id":null,"html_url":"https://github.com/bshillingford/python-torchfile","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bshillingford%2Fpython-torchfile","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bshillingford%2Fpython-torchfile/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bshillingford%2Fpython-torchfile/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bshillingford%2Fpython-torchfile/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bshillingford","download_url":"https://codeload.github.com/bshillingford/python-torchfile/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247299833,"owners_count":20916190,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deep-learning","lua","machine-learning","python","torch"],"created_at":"2024-11-07T01:04:46.414Z","updated_at":"2025-04-05T07:08:16.924Z","avatar_url":"https://github.com/bshillingford.png","language":"Python","readme":"# Torch serialization reader for Python\n[![Build Status](https://travis-ci.org/bshillingford/python-torchfile.svg?branch=master)](https://travis-ci.org/bshillingford/python-torchfile)\n[![Coverage Status](https://coveralls.io/repos/github/bshillingford/python-torchfile/badge.svg)](https://coveralls.io/github/bshillingford/python-torchfile)\n\nMostly direct port of the torch7 Lua and C serialization implementation to \nPython, depending only on `numpy` (and the standard library: `array` \nand `struct`). Sharing of objects including `torch.Tensor`s is preserved.\n\n```python\nimport torchfile\nstuff = torchfile.load('a_bunch_of_stuff.t7')\n```\n\n## Installation:\nInstall from [PyPI](https://pypi.python.org/pypi/torchfile/0.0.2):\n```sh\npip install torchfile\n```\nor clone this repository, then:\n```sh\npython setup.py install\n```\n\nSupports Python 2.7, 3.4, 3.5, 3.6. Probably others too.\n\n## More examples:\n### Write from torch, read from Python:\nLua:\n```lua\n+th\u003e torch.save('/tmp/test.t7', {hello=123, world=torch.rand(1,2,3)})\n```\nPython:\n```python\nIn [3]: o = torchfile.load('/tmp/test.t7')\nIn [4]: print o['world'].shape\n(1, 2, 3)\nIn [5]: o\nOut[5]: \n{'hello': 123, 'world': array([[[ 0.52291083,  0.29261517,  0.11113465],\n         [ 0.01017287,  0.21466237,  0.26572137]]])}\n```\n\n### Arbitary torch classes supported:\n```python\nIn [1]: import torchfile\n\nIn [2]: o = torchfile.load('testfiles_x86_64/gmodule_with_linear_identity.t7')\n\nIn [3]: o.forwardnodes[3].data.module\nOut[3]: TorchObject(nn.Identity, {'output': array([], dtype=float64), 'gradInput': array([], dtype=float64)})\n\nIn [4]: for node in o.forwardnodes: print(repr(node.data.module))                                                                                                            \nNone\nNone\nNone\nTorchObject(nn.Identity, {'output': array([], dtype=float64), 'gradInput': array([], dtype=float64)})\nNone\nTorchObject(nn.Identity, {'output': array([], dtype=float64), 'gradInput': array([], dtype=float64)})\nTorchObject(nn.Linear, {'weight': array([[-0.0248373 ],\n       [ 0.17503954]]), 'gradInput': array([], dtype=float64), 'gradWeight': array([[  1.22317168e-312],\n       [  1.22317168e-312]]), 'bias': array([ 0.05159848, -0.25367146]), 'gradBias': array([  1.22317168e-312,   1.22317168e-312]), 'output': array([], dtype=float64)})\nTorchObject(nn.CAddTable, {'output': array([], dtype=float64), 'gradInput': []})\nNone\n\nIn [5]: o.forwardnodes[6].data.module.weight\nOut[5]: \narray([[-0.0248373 ],\n       [ 0.17503954]])\n\nIn [6]: o.forwardnodes[6].data.module.bias\nOut[6]: array([ 0.05159848, -0.25367146])\n```\n\n### More complex writing from torch:\nLua:\n```lua\n+th\u003e f = torch.DiskFile('/tmp/test.t7', 'w'):binary()\n+th\u003e f:writeBool(false)\n+th\u003e f:writeObject({hello=123})\n+th\u003e f:writeInt(456)\n+th\u003e f:close()\n```\nPython:\n```python\nIn [1]: import torchfile\nIn [2]: with open('/tmp/test.t7','rb') as f:\n   ...:     r = torchfile.T7Reader(f)\n   ...:     print(r.read_boolean())\n   ...:     print(r.read_obj())\n   ...:     print(r.read_int())\n   ...: \nFalse\n{'hello': 123}\n456\n```\n\n\n## Supported types:\n * `nil` to Python `None`\n * numbers to Python floats, or by default a heuristic changes them to ints or\n   longs if they are integral\n * booleans\n * strings: read as byte strings (Python 3) or normal strings (Python 2), like\n   lua strings which don't support unicode, and that can contain null chars\n * tables converted to a special dict (*); if they are list-like (i.e. have\n   numeric keys from 1 through n) they become a python list by default\n * Torch classes: supports Tensors and Storages, and most classes such as \n   modules. Trivially extensible much like the Torch serialization code.\n   Trivial torch classes like most `nn.Module` subclasses become \n   `TorchObject`s. The `torch_readers` dict contains the mapping from class\n   names to reading functions.\n * functions: loaded into the `LuaFunction` `namedtuple`,\n   which simply wraps the raw serialized data, i.e. upvalues and code.\n   These are mostly useless, but exist so you can deserialize anything.\n * tds.Hash, tds.Vec\n\n(*) Since Lua allows you to index a table with a table but Python does not, we \n    replace dicts with a subclass that is hashable, and change its\n    equality comparison behaviour to compare by reference.\n    See `hashable_uniq_dict`.\n\n\n### Test files demonstrating various features:\n```python\nIn [1]: import torchfile\n\nIn [2]: torchfile.load('testfiles_x86_64/list_table.t7')\nOut[2]: ['hello', 'world', 'third item', 123]\n\nIn [3]: torchfile.load('testfiles_x86_64/doubletensor.t7')\nOut[3]: \narray([[ 1. ,  2. ,  3. ],\n       [ 4. ,  5. ,  6.9]])\n\n# ...also other files demonstrating various types.\n```\n\nThe example `t7` files will work on any modern Intel or AMD 64-bit CPU, but the\ncode will use the native byte ordering etc. Currently, the implementation \nassumes the system-dependent binary Torch format, but minor refactoring can \ngive support for the ascii format as well.\n\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbshillingford%2Fpython-torchfile","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbshillingford%2Fpython-torchfile","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbshillingford%2Fpython-torchfile/lists"}