{"id":20447831,"url":"https://github.com/nlpodyssey/gopickle","last_synced_at":"2025-10-18T05:46:12.486Z","repository":{"id":57529793,"uuid":"264263773","full_name":"nlpodyssey/gopickle","owner":"nlpodyssey","description":"Go library for loading Python's data serialized with pickle and PyTorch module files.","archived":false,"fork":false,"pushed_at":"2024-08-01T10:31:08.000Z","size":114,"stargazers_count":142,"open_issues_count":3,"forks_count":19,"subscribers_count":6,"default_branch":"master","last_synced_at":"2025-03-30T03:06:47.924Z","etag":null,"topics":["go","golang","pickle","python"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-2-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/nlpodyssey.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-05-15T18:01:45.000Z","updated_at":"2025-02-25T20:48:15.000Z","dependencies_parsed_at":"2024-03-21T16:30:00.111Z","dependency_job_id":"cc28e002-dec1-4f07-aedb-ee784c9efead","html_url":"https://github.com/nlpodyssey/gopickle","commit_stats":null,"previous_names":[],"tags_count":5,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nlpodyssey%2Fgopickle","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nlpodyssey%2Fgopickle/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nlpodyssey%2Fgopickle/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nlpodyssey%2Fgopickle/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/nlpodyssey","download_url":"https://codeload.github.com/nlpodyssey/gopickle/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247430966,"owners_count":20937875,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["go","golang","pickle","python"],"created_at":"2024-11-15T10:29:48.684Z","updated_at":"2025-10-18T05:46:12.403Z","avatar_url":"https://github.com/nlpodyssey.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# GoPickle\n\nGoPickle is a Go library for loading Python's data serialized with `pickle`\nand PyTorch module files.\n\nThe `pickle` sub-package provides the core functionality for loading data\nserialized with Python `pickle` module, from a file, string, or byte sequence.\nAll _pickle_ protocols from 0 to 5 are supported.\n\nThe `pytorch` sub-package implements types and functions for loading\nPyTorch module files. Both the _modern_ zip-compressed format and the \n_legacy_ non-tar format are supported. Legacy tar-compressed\nfiles and TorchScript archives are _not_ supported.\n\n## Project Status and Contributions\n\nThis project is currently in **alpha** development stage. While we provide\nbetter documentation, tests, and functionalities, above all we'd like to\nbattle-test this library with real data and models. This would tremendously\nhelp us find tricky bugs, add more built-in types, or change the\nlibrary API to make it easier to use.\n\nThe simplest and most useful and way to contribute is to try this library\nyourself and give us your feedback: report bugs, suggest improvements,\nor just tell us your opinion.\nAnd of course, if you feel like it, please go on with your own pull\nrequests! We can discuss any issue and try to find the best\nsolution for it together.\n\n## Usage\n\n### Pickle\n\nSimple usage:\n\n```go\nimport \"github.com/nlpodyssey/gopickle/pickle\"\n\n// ...\n\n// from file\nfoo, err := pickle.Load(\"foo.p\") \n\n// from string\nstringDump := \"I42\\n.\"\nbar, err := pickle.Loads(stringDump)\n\n// ...\n```\n\nAdvanced/custom usage:\n\n```go\nimport \"github.com/nlpodyssey/gopickle/pickle\"\n\nvar r io.Reader\n\n// ...\n\nu := pickle.NewUnpickler(r)\n\n// Handle custom classes\nu.FindClass = func(module, name string) (interface{}, error) {\n    if module == \"foo\" \u0026\u0026 name == \"Bar\" {\n        return myFooBarClass, nil\n    }\n    return nil, fmt.Errorf(\"class not found :(\")\n}\n\n// Resolve objects by persistent ID\nu.PersistentLoad = func(persistentId interface{}) (interface{}, error) {\n    obj := doSomethingWithPersistentId(persistentId)\n    return obj, nil\n}\n\n// Handle custom pickle extensions\nu.GetExtension = func(code int) (interface{}, error) {\n    obj := doSomethingToResolveExtension(code)\n    return obj, nil\n}\n\n// Handle Out-of-band Buffers\n// https://docs.python.org/3/library/pickle.html#out-of-band-buffers\nu.NextBuffer = func() (interface{}, error) {\n    buf := getMyNextBuffer()\n    return buf, nil\n}\n\n// Low-level function to handle pickle protocol 5 READONLY_BUFFER opcode.\n// By default it is completely ignored (sort of no-op); here you have the\n// ability to manipulate objects as you need.\nu.MakeReadOnly = func(obj interface{}) (interface{}, error) {\n    newObj := myReadOnlyTransform(obj)\n    return newObj, nil\n}\n\ndata, err := u.Load()\n\n// ...\n```\n\n### PyTorch\n\nThe library currently provides a high-level function for loading a module file:\n\n```go\nimport \"github.com/nlpodyssey/gopickle/pytorch\"\n\n// ...\n\nmyModel, err := pytorch.Load(\"module.pt\")\n\n// ...\n```\n\nMore features will be provided in the future. \n\n## How it works\n\n### Pickle\n\nUnlike more traditional data serialization formats, (such as JSON or YAML),\na \"pickle\" is a _program_ for a so-called _unpickling machine_, also known\nas _virtual pickle machine_, or _PM_ for short. A program consists in a\nsequence of opcodes which instructs the virtual machine about how to build\narbitrarily complex Python objects. You can learn more  from Python\n`pickletools` [module documentation](https://github.com/python/cpython/blob/3.8/Lib/pickletools.py).\n\nPython PM implementation is straightforward, since it can take advantage\nof the whole environment provided by a running Python interpreter. For this\nGo implementation we want to keep things simple, for example avoiding\ndependencies or foreign bindings, yet we want to provide flexibility, and a way\nfor any user to extend basic functionalities of the library.\n\nThis Go unpickling machine implementation makes use of a set of types defined\nin `types`.\nThis sub-package contains Go types representing classes, instances and common\ninterfaces for some of the most commonly used builtin non-scalar types in \nPython.\nWe chose to provide only minimal functionalities for each type, for the sole \npurpose of making them easy to be handled by the machine.\n\nSince Python's _pickle_ can dump and load _any_ object, the aforementioned types\nare clearly not always sufficient. You can easily handle the loading of any \nmissing class by explicitly providing a `FindClass` callback to an `Unpickler`\nobject. The implementation of your custom classes can be as simple or as\nsophisticated as you need. If a certain class is required but is not found,\nby default a `GenericClass` is used.\nIn some circumstances, this is enough to fully load a _pickle_ program, but\non other occasions the pickle program might require a certain class with\nspecific traits: in this case, the `GenericClass` is not enough and an error\nis returned. You should be able to fix this situation providing\na custom class implementation, that jas to reflect the same basic behaviour\nyou can observe in the original Python implementation.\n\nA similar approach is adopted for other peculiar aspects, such as persistent\nobjects loading, extensions handling, and a couple of protocol-5 opcodes:\nwhenever necessary, you can implement custom behaviours providing one or more\ncallback functions.\n\nOnce resolved, all representation of classes and objects are casted to\n`interface{}` type; then the machine looks for specific types or\ninterfaces to be implemented on an object only where strictly necessary. \n\nThe virtual machine closely follows the original implementation\nfrom Python 3.8 - see the [`Unpickler` class](https://github.com/python/cpython/blob/3.8/Lib/pickle.py#L1134). \n\n### PyTorch\n\n[PyTorch](https://pytorch.org/) machine learning framework allows you to save\nand load Python objects which include (or _are_) [Tensors](https://pytorch.org/docs/stable/tensors.html)\nor other framework-specific elements. Tensors data are handled by the more\nprimitive [Storage](https://pytorch.org/docs/stable/storage.html) classes, which\nare efficiently serialized as raw sequences of bytes. All the rest is dumped\nusing `pickle`. When serializing, the programmer can choose any available pickle\nprotocol, and whether to use zip compression. \n\nThe package `pytorch` implements loading functionalities for data\nfiles serialized with PyTorch (called _modules_). The Go implementation\nstrictly follows the original Python (and C++) [code](https://github.com/pytorch/pytorch/blob/master/torch/serialization.py#L486).\nThe `pickle` and `types` packages are used to read some parts of a given file.\nOther specific types are implemented in the `pytorch` module itself, most\nnotably to reflect the content of PyTorch Tensor and Storage objects. \n\n## License\n\nGoPickle is licensed under a BSD-style license.\nSee [LICENSE](https://github.com/nlpodyssey/gopickle/blob/master/LICENSE) for\nthe full license text.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnlpodyssey%2Fgopickle","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnlpodyssey%2Fgopickle","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnlpodyssey%2Fgopickle/lists"}