{"id":15019046,"url":"https://github.com/ph4r05/monero-serialize","last_synced_at":"2025-10-24T01:31:14.914Z","repository":{"id":29962729,"uuid":"123334770","full_name":"ph4r05/monero-serialize","owner":"ph4r05","description":"Monero serialization python library, supports blockchain, Boost and RPC formats.","archived":false,"fork":false,"pushed_at":"2022-06-20T15:44:48.000Z","size":383,"stargazers_count":10,"open_issues_count":0,"forks_count":3,"subscribers_count":4,"default_branch":"master","last_synced_at":"2024-09-26T20:55:36.256Z","etag":null,"topics":["blockchain-parser","blockchain-parsing","boost","boost-archive","boost-archive-python-parser","boost-serialization","cn","monero","monero-blockchain","monero-boost","monero-serialization","monero-serialize","monero-wallet-parser","python","rpc","serialization"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ph4r05.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-02-28T19:53:18.000Z","updated_at":"2024-05-31T07:28:14.000Z","dependencies_parsed_at":"2022-08-28T10:23:14.243Z","dependency_job_id":null,"html_url":"https://github.com/ph4r05/monero-serialize","commit_stats":null,"previous_names":[],"tags_count":26,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ph4r05%2Fmonero-serialize","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ph4r05%2Fmonero-serialize/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ph4r05%2Fmonero-serialize/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ph4r05%2Fmonero-serialize/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ph4r05","download_url":"https://codeload.github.com/ph4r05/monero-serialize/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":219867198,"owners_count":16555821,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["blockchain-parser","blockchain-parsing","boost","boost-archive","boost-archive-python-parser","boost-serialization","cn","monero","monero-blockchain","monero-boost","monero-serialization","monero-serialize","monero-wallet-parser","python","rpc","serialization"],"created_at":"2024-09-24T19:52:49.591Z","updated_at":"2025-10-24T01:31:13.663Z","avatar_url":"https://github.com/ph4r05.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Monero Python serialization library\n\n[![Build Status](https://travis-ci.org/ph4r05/monero-serialize.svg?branch=master)](https://travis-ci.org/ph4r05/monero-serialize)\n\nThe library provides basic serialization logic for the Monero types,\nused in transaction processing and transaction signing.\n\n- Mainly supports binary serialization equivalent to Monero `BEGIN_SERIALIZE_OBJECT()`.\nThis serialization mechanism is used in the blockchain entity serialization.\n- Boost portable serialization added.\n- Support for `BEGIN_KV_SERIALIZE_MAP` is mainly supported. JSON and binary wire format.\n\nThe binary wire formats use streaming dumping / parsing for better memory efficiency.\n\nFor usage please take a look at [tests](https://github.com/ph4r05/monero-serialize/tree/master/monero_serialize/tests).\n\n```\npip install monero-serialize\n```\n\n## Example usage:\n\n```python\nimport binascii\nfrom monero_serialize import xmrserialize as x\nfrom monero_serialize import xmrtypes as xmr\n\nmsg = xmr.TxinToKey(amount=123, key_offsets=[1, 2, 3, 2**76], k_image=bytearray(range(32)))\n\n# Serialize\nwriter = x.MemoryReaderWriter()\nawait x.dump_message(writer, msg)\nprint(binascii.hexlify(writer.buffer))\n\n# Deserialize\ntest_deser = await x.load_message(x.MemoryReaderWriter(writer.buffer), xmr.TxinGen)\n```\n\n# Donations\nThanks for your support!\n\n```\n86KCjujm1Hp4dD2uyZdXjHQr5e5aNujG6LcunA8iQYjx4AwRorVLDpHMKKmrKQfAPFC5KCKPpjdax3NEbExBRTnSTS1QXWA\n```\n\n\n# Advanced usage\n\n## Archive interface\n\n```python\nimport binascii\nfrom monero_serialize import xmrserialize as x\nfrom monero_serialize import xmrtypes as xmr\n\nmsg = xmr.TxinGen(height=42)\n\n# Serialize\nwriter = x.MemoryReaderWriter()\nar1 = x.Archive(writer, True)\nawait ar1.message(msg)\n\n# Deserialize\nmsg2 = xmr.TxinGen()\nar2 = x.Archive(x.MemoryReaderWriter(writer.buffer), False)\nawait ar2.message(msg2)\n```\n\n## Symmetric Boost archive\n\n```python\nimport binascii\nfrom monero_serialize import xmrserialize as x\nfrom monero_serialize import xmrtypes as xmr\nfrom monero_serialize import xmrboost as xmrb\n\ndata_hex = b'011673657269616c697a6174696f6e3a3a61726368697665000000000134'\ndata_bin = base64.b16decode(data_hex, True)\nreader = x.MemoryReaderWriter(bytearray(data_bin))\nar = xmrb.Archive(reader, False)\n\nmsg = xmr.TxinGen()\nawait ar.root_message(msg)\nself.assertEqual(msg.height, 0x34)\n```\n\n\n## XMR classes\n\n```python\nclass Hash(x.BlobType): pass;\nclass ECKey(x.BlobType): pass;\nclass ECPoint(x.BlobType): pass;\nclass SecretKey(ECKey): pass;\nclass ECPublicKey(ECPoint): pass;\nclass KeyImage(ECPoint): pass;\nclass KeyDerivation(ECPoint): pass;\nclass TxoutToScript(x.MessageType): pass;\nclass TxoutToKey(x.MessageType): pass;\nclass TxoutToScriptHash(x.MessageType): pass;\nclass TxoutTargetV(x.VariantType): pass;\nclass TxinGen(x.MessageType): pass;\nclass TxinToKey(x.MessageType): pass;\nclass TxinToScript(x.MessageType): pass;\nclass TxinToScriptHash(x.MessageType): pass;\nclass TxInV(x.VariantType): pass;\nclass TxOut(x.MessageType): pass;\nclass TransactionPrefix(x.MessageType): pass;\nclass TransactionPrefixExtraBlob(TransactionPrefix): pass;\nclass TxIndex(x.MessageType): pass;\nclass TransactionMetaData(x.MessageType): pass;\n\n#\n# rctTypes.h\n#\n\nclass Key64(x.ContainerType): pass;\nclass KeyV(x.ContainerType): pass;\nclass KeyM(x.ContainerType): pass;\nclass KeyVFix(x.ContainerType): pass;\nclass KeyMFix(x.ContainerType): pass;\nclass CtKey(x.MessageType): pass;\nclass CtkeyV(x.ContainerType): pass;\nclass CtkeyM(x.ContainerType): pass;\nclass MultisigKLRki(x.MessageType): pass;\nclass MultisigOut(x.MessageType): pass;\nclass EcdhTuple(x.MessageType): pass;\nclass BoroSig(x.MessageType): pass;\nclass MgSig(x.MessageType): pass;\nclass RangeSig(x.MessageType): pass;\nclass Bulletproof(x.MessageType): pass;\nclass EcdhInfo(x.ContainerType): pass;\nclass RctSigBase(x.MessageType): pass;\nclass RctSigPrunable(x.MessageType): pass;\nclass RctSig(RctSigBase): pass;\nclass Signature(x.MessageType): pass;\nclass SignatureArray(x.ContainerType): pass;\nclass Transaction(TransactionPrefix): pass;\nclass BlockHeader(x.MessageType): pass;\nclass HashVector(x.ContainerType): pass;\nclass Block(BlockHeader): pass;\nclass AccountPublicAddress(x.MessageType): pass;\nclass SubaddressIndex(x.MessageType): pass;\nclass MultisigLR(x.MessageType): pass;\nclass MultisigInfo(x.MessageType): pass;\nclass MultisigStruct(x.MessageType): pass;\nclass TxExtraPadding(x.MessageType): pass;\nclass TxExtraPubKey(x.MessageType): pass;\nclass TxExtraNonce(x.MessageType): pass;\nclass TxExtraMergeMiningTag(x.MessageType): pass;\nclass TxExtraAdditionalPubKeys(x.MessageType): pass;\nclass TxExtraMysteriousMinergate(x.MessageType): pass;\nclass TxExtraField(x.VariantType): pass;\nclass TxExtraFields(x.ContainerType): pass;\nclass OutputEntry(x.TupleType): pass;\nclass TxSourceEntry(x.MessageType): pass;\nclass TxDestinationEntry(x.MessageType): pass;\nclass TransferDetails(x.MessageType): pass;\nclass TxConstructionData(x.MessageType): pass;\nclass PendingTransaction(x.MessageType): pass;\nclass PendingTransactionVector(x.ContainerType): pass;\nclass UnsignedTxSet(x.MessageType): pass;\nclass SignedTxSet(x.MessageType): pass;\nclass MultisigTxSet(x.MessageType): pass;\n```\n\n\n# Serialization formats\n\n## Blockchain format\n\nThe BC serialization format is scheme-oriented, i.e., you have to provide the scheme according to which \nserialize/deserialize the data. Scheme specifies how are fields composed, whether the size of containers is \nfixed (and then also the size is specified by the scheme) or not.\n\nThe format is not versioned, i.e., serialization format does not store explicit version numbers which would affect serialization scheme.\n\n### Uvarint\n\n- Variable length integer encoded by 7-bit chunks, little endian. \n- The MSB indicates whether there are more octets (1) or it is the last one (0).\n- 0 - 0x7f encoded in 1 byte, 0x80 - 0x3fff encoded in 2 bytes, ...\n\nExample:\n- `0x0f     -\u003e 0f`\n- `0x1000   -\u003e 8020`\n- `0xffff   -\u003e ffff03`\n- `0xffffff -\u003e ffffff07`\n\n### UInt\n\n- Fixed width integer, little endian encoded\n\nExample:\n\n- `0x0f     Width 4 = 0f000000`\n- `0x1000   Width 4 = 001000`\n- `0xffffff Width 4 = ffffff00`\n\n### Blob\n\n- Binary bytes can have either fixed size or variable size.\n- Variable size format: `uvarint(lenght) || data`\n- Fixed-size format: `data`\n- Typical example of a fixed-width blob is EC point or scalar, which has 32 bytes\n\n### Unicode string\n\n- Format: `uvarint(length) || input.encode(\"utf8\")`\n\n### Container\n\n- Variable size format: `uvarint(length) || *elements`\n- Fixed size format: `*elements`\n- Elements are serialized according to the scheme of the element. \n- All elements are of the same type which is specified by the schema of the container\n\n### Tuple\n\n- Tuple is heterogenous\n- Format: `uvarint(length) || *elements`\n- Each element is serialized according to the scheme specified in the tuple\n\n### Variant\n\n- Similar to union from C, stores precisely one data type out of many\n- Variants are identified by 1 byte code\n- Format: `uint(variant_code) || variant_object`\n- Variant object is serialized according to the scheme corresponding for the particular variant\n\n### Message / object\n\n- Collection of heterogenous fields\n- Fields are serialized according to the scheme\n- Field ordering is fixed by the message scheme\n- Field have names, but no name nor number of fields are serialized \n- Messages serialization scheme can vary based on the version number provided from outside\n\n\n\n## Boost serialization\n\n- Mainly used by the wallet and for internal purposes.\n- Scheme oriented format. Scheme is required to understand the serialized data.\n- Versioned serialization format, explicitly storing version numbers to the serialized data.\n- Reference: [Boost Serialization](https://www.boost.org/doc/libs/1_61_0/libs/serialization/doc/index.html)\n- Archive starts with the header `011673657269616c697a6174696f6e3a3a617263686976650000` which translates to `\\x01\\x16serialization::archive\\x00\\x00`. The `[-2]` byte enables tracking (tracking is not supported in this lib), `[-1]` is a version of root element.\n- [Basic archive](https://www.boost.org/doc/libs/1_66_0/libs/serialization/src/basic_oarchive.cpp)\n\n### Versioning\n- Schemes are versioned based on the C++ type, this python lib identifies the object based on its type and parameters (e.g. container + element type). Once the version for particular type has been stored to the stream, it is not stored again. In C++ this is handled by the type system and compiler. In this library we have to explicitly track whether the type version has been already stored. \n- Elementary types are not versioned (int, uvarint, unicode)\n- Versioning can be disabled in the complex types \n- Version is stored as the following tuple: `(uvarint(tracking) || uvarint(version))`\n  - tracking relates to the [Boost Object Tracking](https://www.boost.org/doc/libs/1_61_0/libs/serialization/doc/special.html#objecttracking), advanced construct used with pointers (tracking by memory address), alpha version of tracking is supported for reading the archive.\n  - tracking in C++ code: `BOOST_CLASS_TRACKING`\n  - Versioning in the [Boost docs](https://www.boost.org/doc/libs/1_61_0/libs/serialization/doc/tutorial.html#versioning)\n  - If tracking is enabled, the format is: `(uvarint(tracking) || uvarint(version) || uvarint(object_id))`\n\n### Uvarint\n\n- First byte encodes length and sign. Supports byte widths: 0-8. 2 byte positive number: 0x2, 2 byte negative number: 0xfe\n- Encoding in 8bit chunks, little endian.\n- Uvarint serialization scheme defined by Monero code.\n\n### Unicode \n\n- Format: `uvarint(length) || input.encode(\"utf8\")`\n\n### Blob\n\n- Format: `uvarint(length) || data`\n\n### Container\n\n- Primitive type containers are not versioned, in general, container is versioned.\n- After version follows `uvarint(length)`\n- Version of the element follows as `uvarint(element_version)` (exception for case of raw containers = statically allocated like Key64)\n- Elements follow. If element is serialized for the first time, same rules apply for the versioning - version is stored.\n- Example: `(tr, v), collection_size, element_version, (obj_tr, obj_v), obj1, obj2, ...`\n\n### Tuple\n\n- Versioned\n- Stored without size information - obtained from the scheme\n\n### Variant\n\n- Versioned\n- Format: `uvarint(variant_code) || field`\n- Example: `(tr, v), which, (tr, v), val`\n\n### Message\n\n- Versioned\n- Stores field in a defined order, without storing field names or field types. Types are derived from the scheme.\n- Fields serialized recursively.\n\n## JSONRPC \n\n- Monero serialization format, with either binary or JSON form\n- Defines serialization types supported by the format, each supported serialization primitive has a tag.\n  - signed and unsigned integers of a fixed widths of: 18, 16,32, 64 bits\n  - boolean \n  - double\n  - string\n  - object\n  - array\n\n- Root of archive starts with the `0x01011101 || 0x01020101 || 0x1`, i.e, signature A, signature B, format version\n- Archive can be parsed as JSON, i.e., without scheme.\n\n### Integers\n- Integer to encode is N, then the format encodes `N\u003c\u003c2 | size_flag` as little endian\n- Flags 0, 1, 2, 3 correspond to 1, 2, 4, 8 byte integers correspondingly \n\n### String\n- Format: `uvarint(length) || data`\n\n### Array\n- Serializes `element_type | SerializeType.ARRAY_FLAG` as integer, where `element_type` is the object tag.\n- Serializes length of the container\n- Serializes each element recursively\n\n### Section / object\n- Correspond to JSON dictionaries with string keys\n- Format: `uvarint(length) || serialize_string(section_1) || serialize_storage(value_1)`\n- The `serialize_storage` serializes:\n  - object tag if object type does not have `ARRAY_FLAG` flag\n  - object itself recursively\n  \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fph4r05%2Fmonero-serialize","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fph4r05%2Fmonero-serialize","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fph4r05%2Fmonero-serialize/lists"}