{"id":20285648,"url":"https://github.com/aep/madpack","last_synced_at":"2025-10-15T22:52:58.340Z","repository":{"id":57617278,"uuid":"289062733","full_name":"aep/madpack","owner":"aep","description":"stupid small json binary encoding","archived":false,"fork":false,"pushed_at":"2021-07-13T14:37:03.000Z","size":36,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-02-24T14:54:32.667Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/aep.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-08-20T17:02:28.000Z","updated_at":"2021-07-13T14:37:06.000Z","dependencies_parsed_at":"2022-08-29T05:40:19.688Z","dependency_job_id":null,"html_url":"https://github.com/aep/madpack","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aep%2Fmadpack","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aep%2Fmadpack/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aep%2Fmadpack/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aep%2Fmadpack/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/aep","download_url":"https://codeload.github.com/aep/madpack/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":241780512,"owners_count":20019061,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-14T14:28:04.316Z","updated_at":"2025-10-15T22:52:53.305Z","avatar_url":"https://github.com/aep.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# madpack\n\nstupid small json binary encoding.\n\nuses mad hax (tm) to pack all valid json faster and smaller than **everything else**\n(tested: msgpack, ubjson, protobuf, gzip and zstandard)\n\n\n## building \u0026 usage\n\nget the [zetz.it](https://zetz.it) compiler\n\n    zz build --release\n    ./target/release/bin/madpack \u003c some.json \u003e some.madpack\n    ./target/release/bin/madpack --unpack \u003c some.madpack\n\nzetz can automatically create packages for many languages including npm, python, rust, golang.\n\n\nso lets say we have this small json\n\n```json\n{\n    \"sha256\": \"beep boop yadda\",\n    \"commitmsg\": \"hella\",\n    \"stable\": false,\n    \"contentsize\": 2332\n}\n```\n\n    $ wc -c \u003c some.json\n    108\n\ncompressing it with zstandard (it's really good)\n\n    $ zstd some.json | wc -c\n    99\n\nmadpack beats zstd out of the box for small messages\n\n    $ ./target/release/bin/madpack \u003c some.json | wc -c\n    65\n\nbut preshared indices are the real magic\n\n    $ ./target/release/bin/madpack --make-index \u003c some.json \u003e some.madindex\n    $ ./target/release/bin/madpack --index some.madindex  \u003c some.json | wc -c\n    29\n\n\n\n\n## preshared index for structured data\n\nThe way to beat zipped json is to create an index from a data sample or schema and simply not include that index in the encoded file.\nThis is what protobuf does, but we still beat protobuf with other mad hax;\n\nAlso unlike with protobuf, the message can be fully recovered without the index.\nDesaster recovery or bug hunting is much easier with a fully intact message structure.\n\nto create an index from any json run\n\n    ./target/release/bin/madpack --make-index \u003c some.json \u003e some.madindex\n\n\nyou can then use the index in encoding and decoding\n\n    ./target/release/bin/madpack --index some.madindex \u003c some.json \u003e some.madpack\n    ./target/release/bin/madpack --index some.madindex --unpack \u003c some.madpack\n\nif you ever loose the index, you can still read the message and guess the key names from context\n\n    ./target/release/bin/madpack --unpack \u003c some.madpack\n    {\n        \"1\" : 77,\n        \"2\" : \"0.10.1-4-g2bc35c7-dirty-R3riyLGLmb\",\n        \"3\" : true,\n\n\n\n\n\n\n## encoding\n\n- there's a maximum of 65535 unique strings per file\n- key strings can only be 65535 bytes long\n- in maps,   every member is preceeded by a key byte\n- in arrays, every member is preceeded by a value byte\n\n### key byte\n\n    000x xxxx 0x00 value is u8\n    001x xxxx 0x20 value is u16\n    010x xxxx 0x40 value is f32\n    011x xxxx 0x60 value is bytes  u8\n    100x xxxx 0x80 value is string u8\n    101x xxxx 0xa0 value is map\n    110x xxxx 0xc0 value is array\n    111x xxxx 0xe0 full value byte follows\n\n\n    xxx0 0000 0x00 reserved\n    xxx0 0001 0x01 key number 1\n    xxx1 1010 0x1a key number 26\n    xxx1 1011 0x1b key number as a u8\n    xxx1 1100 0x1c key number as a u16\n    xxx1 1101 0x1d key is a string size u8\n    xxx1 1110 0x1e key is a string size u16\n\n    1111 1111 0xff end\n\n### value byte\n\n    0000 0000 0x00  literal 0\n    0110 1111 0x6f  literal 111\n\n    0111 0000 0x70  u8\n    0111 0001 0x71  u16\n    0111 0010 0x72  u32\n    0111 0011 0x73  u64\n\n    0111 0100 0x74  i8\n    0111 0101 0x75  i16\n    0111 0110 0x76  i32\n    0111 0111 0x77  i64\n\n    0111 1000 0x78  null\n    0111 1001 0x79  true\n    0111 1010 0x7a  false\n    0111 1011 0x7b  map\n\n    0111 1100 0x7c  array\n    0111 1101 0x7d  f32\n    0111 1110 0x7e  f64\n    0111 1111 0x7f  ext\n\n    dynamic size:\n\n    1000 xxxx 0x80 string\n    1001 xxxx 0x90 bytes\n    1010 xxxx 0xa0 reserved\n    1011 xxxx 0xb0 reserved\n    1100 xxxx 0xc0 reserved\n    1101 xxxx 0xd0 reserved\n    1110 xxxx 0xe0 reserved\n    1111 xxxx 0xf0 reserved\n\n         0000 size 0\n         1011 size 11\n         1100 size see next 1 bytes\n         1101 size see next 2 bytes\n         1110 size see next 4 bytes\n         1111 size see next 8 bytes\n\n\n    1111 1111 0xff end\n\n\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faep%2Fmadpack","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Faep%2Fmadpack","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faep%2Fmadpack/lists"}