{"id":16866364,"url":"https://github.com/jprjr/lua-etf","last_synced_at":"2025-03-18T17:43:24.120Z","repository":{"id":153362358,"uuid":"629142717","full_name":"jprjr/lua-etf","owner":"jprjr","description":"Erlang External Term Format encoder and decoder for Lua","archived":false,"fork":false,"pushed_at":"2023-04-17T18:38:39.000Z","size":146,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-01-24T21:13:47.566Z","etag":null,"topics":["erlpack","lua","luajit"],"latest_commit_sha":null,"homepage":"","language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jprjr.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-04-17T17:56:35.000Z","updated_at":"2023-04-17T18:43:50.000Z","dependencies_parsed_at":null,"dependency_job_id":"be16ff2a-9b0b-4d63-a6bf-3f411a08a775","html_url":"https://github.com/jprjr/lua-etf","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jprjr%2Flua-etf","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jprjr%2Flua-etf/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jprjr%2Flua-etf/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jprjr%2Flua-etf/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jprjr","download_url":"https://codeload.github.com/jprjr/lua-etf/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244272544,"owners_count":20426755,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["erlpack","lua","luajit"],"created_at":"2024-10-13T14:50:21.916Z","updated_at":"2025-03-18T17:43:24.100Z","avatar_url":"https://github.com/jprjr.png","language":"C","readme":"# etf\n\nA Lua library for encoding and decoding the Erlang [External Term Format](https://www.erlang.org/doc/apps/erts/erl_ext_dist.html).\n\nTested on Lua 5.1 - 5.4 and LuaJIT.\n\nBy default, this decodes and encodes with logic similar to [erlpack](https://github.com/discord/erlpack),\nbut decoders and encoders have options to customize how values are processed.\n\n## LICENSE\n\nMIT (see file `LICENSE`).\n\nSome files are third-party and contain their own licensing:\n\n* `csrc/thirdparty/bigint/bigint.h`: Zero-clause BSD, details in source file.\n* `csrc/thirdparty/miniz/miniz.h`: MIT license, details in `csrc/thirdparty/miniz/LICENSE`.\n\n## Decoding\n\n```lua\nlocal decoder = etf.decoder(options) -- create a decoder\nlocal decoded = decoder:decode('\\131\\116\\0\\0\\0\\2\\100\\0\\1\\97\\97\\1\\100\\0\\1\\98\\97\\2')\n\n-- decoded will be a table like:\n   { a = 1, b = 2 }\n```\n\n`options` is an optional table with the following keys, all optional:\n\n* `use_integer` - set to `true` to decode all integers as `etf.integer` userdata.\n* `use_float` - set to `true` to decode all floats as `etf.float` userdata.\n* `version` - specify the Erlang Term Format version you wish to decode.\nAs far as I can tell, `131` is the only version in existence.\n* `atom_map` - customize how Atom types are decoded. This can be a table, or\na function that accepts a string (representing the atom name) and a boolean (`true`\nif the atom is a map key, `false` otherwise).\n\nHere's how various Erlang types are mapped to Lua by default:\n\n| Supported | Erlang Type | Lua Type |\n|-----------|-------------|----------|\n| [ ] | `ATOM_CACHE_REF` |  |\n| [x] | `ZLIB` | (automatically decompressed and decoded) |\n| [x] | `SMALL_INTEGER_EXT` | `number` |\n| [x] | `INTEGER_EXT` | `number` or `etf.integer` (based on value) |\n| [x] | `FLOAT_EXT` | `number` |\n| [x] | `PORT_EXT` | `table` |\n| [x] | `NEW_PORT_EXT` | `table` |\n| [x] | `V4_PORT_EXT` | `table` |\n| [x] | `PID_EXT` | `table` |\n| [x] | `NEW_PID_EXT` | `table` |\n| [x] | `SMALL_TUPLE_EXT` | `table` |\n| [x] | `LARGE_TUPLE_EXT` | `table` |\n| [x] | `MAP_EXT` | `table` |\n| [x] | `NIL_EXT` | `table` (empty) |\n| [x] | `STRING_EXT` | `string` |\n| [x] | `LIST_EXT` | `table` |\n| [x] | `BINARY_EXT` | `string` |\n| [x] | `SMALL_BIG_EXT` | `number` or `etf.integer` |\n| [x] | `LARGE_BIG_EXT` | `number` or `etf.integer` |\n| [x] | `REFERENCE_EXT` | `table` |\n| [x] | `NEW_REFERENCE_EXT` | `table` |\n| [x] | `NEWER_REFERENCE_EXT` | `table` |\n| [x] | `FUN_EXT` | `table` |\n| [x] | `NEW_FUN_EXT` | `table` |\n| [x] | `EXPORT_EXT` | `table` |\n| [x] | `BIT_BINARY_EXT` | `string` |\n| [x] | `NEW_FLOAT_EXT` | `number` |\n| [x] | `ATOM_UTF8_EXT` | `string` or `boolean` or `etf.null` |\n| [x] | `SMALL_ATOM_UTF8_EXT` | `string` or `boolean` or `etf.null` |\n| [x] | `ATOM_EXT` | `string` or `boolean` or `etf.null` |\n| [x] | `SMALL_ATOM_EXT` | `string` or `boolean` or `etf.null` |\n\n### Decoding Details\n\n### Integers\n\n`etf` will figure out the maximum and minimum integer values that can be\nsafely handled by Lua at run-time. When any integer is decoded, it will\nuse Lua's `number` type if possible, and a `etf.integer` userdata if it's\noutside the safe range.\n\nYou can opt to have all integers be returned as `etf.integer` userdatas. The\nbenefit of this is all values will use the same type. On Lua 5.2 and later,\nthe `etf.integer` userdatas can be compared to regular Lua numbers, but on\nLua 5.1 you can only compare `etf.integer` values with other `etf.integer`\nvalues.\n\nTo enable this, create the decoder with the `use_integer` option set to `true`:\n\n```lua\nlocal etf = require'etf'\nlocal decoder = etf.decoder({use_integer = true })\nlocal val = decoder:decode('\\131\\97\\1') -- returns a integer\nprint(debug.getmetatable(val).__name)\n-- prints \"etf.integer\"\n```\n\n### Atoms\n\nErlang supports a concept of \"atoms\" which doesn't completely translate to\nLua.\n\nIn Erlang, one can create a map like:\n\n```erlang\nMap = #{ a =\u003e 1, b =\u003e false, c =\u003e hello }\n```\n\nIn that example, `a`, `b`, `false`, and `hello` are all atoms. They're\nessentially small strings that can be used for map keys, enums, etc.\n\nNote that Erlang doesn't have a `boolean` type. `false` is just another atom.\n\nBy default, atoms are decoded with the following logic:\n\n* If the atom is a map key (like `a` and `b` in the example), it's decoded as a string.\n* If the atom is a value (like `false` and `hello` in the example, then:\n    * Atom `true` is decoded as Lua's boolean `true`.\n    * Atom `false` is decoded as Lua's boolean `false`.\n    * Atom `nil` is decoded as `etf.null`, which is an atom userdata.\n    * Anything else is decoded as a string.\n\nThis is meant to be compatible with [erlpack](https://github.com/discord/erlpack), and\nto make decoded data as easy to handle as possible.\n\nIf your application has other atoms that need to be translated into values,\nyou can specify the `atom_map` parameter. This can be a table with string keys,\nor a function. The function should accept a string parameter representing\nthe atom name, and a boolean representing if the atom is a map key or not.\n\nThe default logic can be represented as:\n\n```lua\nlocal function atom_map(str, is_key)\n  if is_key then return str end\n  if str == 'true' then\n    return true\n  elseif str == 'false' then\n    return false\n  elseif str == 'nil' then\n    return etf.null\n  end\n  return str\nend\n```\n\nIf for example, you wanted to keep string keys but keep the values as atoms:\n\n```lua\nlocal function atom_map(str, is_key)\n  if is_key then return str end\n  return etf.atom(str)\nend\n```\n\n### Tuples and Lists\n\n`SMALL_TUPLE_EXT`, `LARGE_TUPLE_EXT`, `LIST_EXT`, and `NIL_EXT`\nwill be decoded into array-like tables (all keys are integers, they're consecutive,\nand they start at 1).\n\nThe table will have a metatable set to indicate the original type - `etf.tuple_mt` for tuples, and `etf.list_mt` for lists.\n\n### Maps\n\n`MAP_EXT` will be decoded into a Lua table. By default, the keys are (probably) strings,\nsee above about how atoms are mapped. Values are mapped into the appropriate Lua type\naccording to the above table.\n\nThe table will have a metatable set to indicate it was a map - `etf.map_mt`.\n\n### Ports\n\nThe various `PORT` types (`PORT_EXT`, `NEW_PORT_EXT`, `V4_PORT_EXT`) will be decoded into\na table with the following fields:\n\n* `node` - a string.\n* `id` - a number or integer.\n* `creation` - a number or integer.\n\nThe table will have a metatable set to `etf.port_mt`.\n\n### PIDs\n\nThe `PID` types (`PID_EXT`, `NEW_PID_EXT`) will be decoded into a table with\nthe following fields:\n\n* `node` - a string.\n* `id` - a number or integer.\n* `serial` - a number or integer.\n* `creation` - a number or integer.\n\nThe table will have a metatable set to `etf.pid_mt`.\n\n### FUN_EXT\n\n`FUN_EXT` will be decoded into a table with the following fields:\n\n* `numfree` - a number or integer.\n* `pid` - the previously-mentioned `PID` type.\n* `module` - a string.\n* `index` - a number or integer.\n* `uniq` - a number or integer.\n* `free_vars` - an array like table of terms.\n\nThe table will have a metatable set to `etf.fun_mt`.\n\n### NEW_FUN_EXT\n\n`NEW_FUN_EXT` will be decoded into a table with the following fields:\n\n* `size` - a number or integer.\n* `arity` - a number.\n* `uniq` - a string.\n* `index` - a number or integer.\n* `numfree` - a number or integer.\n* `module` - a string.\n* `oldindex` - a number or integer.\n* `olduniq` - a number or integer.\n* `pid` - the previously-mentioned `PID` type.\n* `free_vars` - an array like table of terms.\n\nThe table will have a metatable set to `etf.new_fun_mt`.\n\n### EXPORT_EXT\n\n`EXPORT_EXT` will be decoded into a table with the following fields:\n\n* `module` - a string.\n* `function` - a string.\n* `arity` - a number or integer.\n\nThe table will have a metatable set to `etf.export_mt`.\n\n### References\n\nThe `REFERENCE` types (`REFERENCE_EXT`, `NEW_REFERENCE_EXT`, `NEWER_REFERENCE_EXT`) will\nbe decoded into a table with the following fields:\n\n* `node` - a string.\n* `creation` - a number or integer.\n* `id` - an array-like table of numbers or integers.\n\nThe table will have a metatable set to `etf.reference_mt`.\n\n## Encoding\n\n```lua\nlocal encoder = etf.encoder(options) -- create a encoder\nlocal encoded = encoder:encode({ a = 1, b = 2 })\n-- encoded will be a MAP_EXT with BINARY_EXT keys and SMALL_INT_EXT values\n```\n\n`options` is an optional table with the following keys, all optional:\n\n* `version` - specify the Erlang Term Format version you wish to encode.\nAs far as I can tell, `131` is the only version in existence.\n* `compress` - set to `true` to enable compression at the default level, or\n`0` through `9` to specify a compression level.\n* `value_map` - customize how values are encoded, this can be a table or a\nfunction that accepts the value to be encoded, and a boolean indicating if\nthe value is a table key.\n\n### Lua Types\n\nHere's how various Lua types are mapped to Erlang Term Format by default:\n\n| Supported | Lua Type | Erlang Type |\n|-----------|----------|-------------|\n| [x] | `nil` | a `nil` `SMALL_ATOM_UTF8_EXT` |\n| [x] | `number` | `NEW_FLOAT_EXT`, `SMALL_INTEGER_EXT`, `INTEGER_EXT`, `SMALL_BIG_EXT`, `LARGE_BIG_EXT` (as appropriate) |\n| [x] | `boolean` | `SMALL_ATOM_UTF8_EXT` |\n| [x] | `string` | `BINARY_EXT` |\n| [x] | `table` | `NIL_EXT`, `LIST_EXT`, or `MAP_EXT` |\n| [x] | `userdata` | (see details below) |\n\n\n### A note on tables\n\nA table is determined to either be map-like or list-like. If a table\nhas integer keys starting at 1, with no gaps, it's considered to be\nlist-like and will be encoded as a `LIST_EXT`.\n\nIf a table has no keys at all, it will be treated as a list-type\nwith zero items and encoded as a `NIL_EXT` (Erlang's version of an\nempty list).\n\nOtherwise, the table is considered map-like, and will be encoded\nas a `MAP_EXT`. All table keys will be encoded as strings (specifically\n`BINARY_EXT`). This is meant to be compatible with [erlpack](https://github.com/discord/erlpack).\n\n### Userdata types\n\n`etf` allows creating various userdata to force a specific encoding:\n\n| Userdata | Erlang Type |\n|----------|-------------|\n| `etf.integer` | `SMALL_INTEGER_EXT`, `INTEGER_EXT`, `SMALL_BIG_EXT`, `LARGE_BIG_EXT` as appropriate |\n| `etf.float` | `NEW_FLOAT_EXT` |\n| `etf.string` | `STRING_EXT`\n| `etf.binary` | `BINARY_EXT`\n| `etf.atom` | `SMALL_ATOM_UTF8_EXT` or `ATOM_UTF8_EXT` |\n| `etf.tuple` | `TUPLE_EXT` |\n| `etf.list` | `LIST_EXT` |\n| `etf.map` | `MAP_EXT` |\n| `etf.port` | `NEW_PORT_EXT` or `V4_PORT_EXT` |\n| `etf.pid` | `NEW_PID_EXT` |\n| `etf.export` | `EXPORT_EXT`|\n| `etf.reference` | `NEWER_REFERENCE_EXT` |\n\nThe `etf.integer` type will encoded to the smallest-possible integer. So, a `integer`\nin the range of an 8-bit unsigned integer will be encoded as a `SMALL_INTEGER_EXT` value,\na `integer` in the range of a 32-bit signed integer will be encoded as an `INTEGER_EXT` value,\nand so on.\n\nUsing these userdata with a custom `value_map` function allows precise control over\nmapping. For example, if you want to use Atom types for all table keys, you could do:\n\n```lua\nlocal function value_map(val, is_key)\n  if is_key then\n    return etf.atom(val)\n  end\n  return val\nend\n\nlocal encoder = etf.encoder({value_map = value_map })\nlocal binary = encoder:encode({ a = 1, b = 2 })\n-- will return a MAP_EXT with atom keys and integer values\n```\n\n## Full listing of fields in the `etf` module:\n\n### Decoding Functions\n\n* `decoder` - function that returns a `decoder` userdata.\n* `decode` - convenience function to decode without creating a decoder.\n\n### Encoding Functions\n\n* `encoder` - function that returns an `encoder` userdata.\n* `encode` - convenience function to encode without creating an encoder.\n\n### Userdata-creating Functions\n\n#### String-like types\n\n* `atom` - function that returns an `atom` userdata (requires a string).\n* `binary` - function that returns a `binary` userdata (requires a string).\n* `string` - a function that returns a `string` userdata (requires a string).\n\n#### Integer types\n\n* `integer` - function that returns a `integer` userdata (accepts a number, string, or none).\n\n#### Float types\n\n* `float` - function that returns a `float` userdata (accepts a number, string, or none).\n\n#### Table-like types\n\n* `list` - function that returns a `list` userdata (optionally accepts a table).\n* `map` - function that returns a `map` userdata (optionally accepts a table).\n* `tuple` - a function that returns a `tuple` userdata (optionally accepts a table).\n\n#### Other tpyes\n\n* `export` - function that returns an `export` userdata (requires a table matching `EXPORT_EXT` above).\n* `pid` - a function that returns a `pid` userdata (requires a table matching `PID_EXT` above).\n* `port` - a function that returns a `port` userdata (requires a table matching `PORT_EXT` above).\n* `reference` - a function that returns a `reference` userdata (requires a table matching `REFERENCE_EXT` above).\n\n### Pre-created Userdatas\n\n* `maxinteger` - a `integer` value representing the maximum integer that can be represented by Lua natively.\n* `mininteger` - a `integer` value representing the minimum integer that can be represented by Lua natively.\n* `null` - an `atom` that represents a `nil` atom.\n\n### Metatables\n\n* `atom_mt` - the `atom` userdata's metatable.\n* `integer_mt` - the `integer` userdata's metatable.\n* `float_mt` - the `float` userdata's metatable.\n* `binary_mt` - the `binary` userdata's metatable.\n* `decoder_131_mt` - the `decoder` userdata's metatable.\n* `encoder_131_mt` - the `encoder` userdata's metatable.\n* `export_mt` - the `export` userdata's metatable.\n* `fun_mt` - the `fun` userdata's metatable.\n* `list_mt` - the `list` userdata's metatable.\n* `map_mt` - the `map` userdata's metatable.\n* `new_fun_mt` - the `new_fun` userdata's metatable.\n* `pid_mt` - the `pid` userdata's metatable.\n* `port_mt` - the `port` userdata's metatable.\n* `reference_mt` - the `reference` userdata's metatable.\n* `string_mt` - a `string` userdata's metatable.\n* `tuple_mt` - the `tuple` userdata's metatable.\n\n### Version Info\n\n* `_VERSION` - the module version as a string.\n* `_VERSION_MAJOR` - the module's major version as a number.\n* `_VERSION_MINOR` - the module's minor version as a number.\n* `_VERSION_PATCH` - the module's patch version as a number.\n\n### Misc\n\n* `numsize` - the size of a Lua number, in bytes.\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjprjr%2Flua-etf","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjprjr%2Flua-etf","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjprjr%2Flua-etf/lists"}