{"id":16116732,"url":"https://github.com/vitiral/zoa","last_synced_at":"2025-10-17T02:05:52.029Z","repository":{"id":64476276,"uuid":"447308145","full_name":"vitiral/zoa","owner":"vitiral","description":"serialized structured data and it's textual representation","archived":false,"fork":false,"pushed_at":"2023-10-12T13:37:03.000Z","size":194,"stargazers_count":4,"open_issues_count":1,"forks_count":2,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-01-12T06:40:35.466Z","etag":null,"topics":["civboot","data","serialization-format"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"unlicense","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/vitiral.png","metadata":{"files":{"readme":"README.cxt","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2022-01-12T17:22:39.000Z","updated_at":"2023-10-12T13:35:08.000Z","dependencies_parsed_at":"2023-10-11T21:30:00.269Z","dependency_job_id":"6d1ce9b5-7943-42e9-a545-120c10cbd798","html_url":"https://github.com/vitiral/zoa","commit_stats":null,"previous_names":["vitiral/zoa"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vitiral%2Fzoa","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vitiral%2Fzoa/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vitiral%2Fzoa/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vitiral%2Fzoa/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/vitiral","download_url":"https://codeload.github.com/vitiral/zoa/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":234444609,"owners_count":18833658,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["civboot","data","serialization-format"],"created_at":"2024-10-09T20:25:31.196Z","updated_at":"2025-10-17T02:05:46.986Z","avatar_url":"https://github.com/vitiral.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"[!]##########################################################################[/]\n[h1]zoa: serialized data made easy[/]\n\n[\"]Most content has moved to https://github.com/civboot/civlua/tree/main/zoa[\"]\n\n[\"]This document is written using @cxt in README.cxt[/]\n\nzoa is a set of standards related to serialized structured data and it's\ntextual representation and processing. It is part of the @Civboot\nproject, specifically @fngi.\n\n[\"] zoa is named after \"protozoa\", which is itself a nod to @protobuf. Yes, I'm\n    aware protozoa are not fungi, but the name was too good.[/]\n\nzoa encompases the following technologies:[+]\n  * [b]zty[b] `.zty`: specification for structured types similar to @protobuf\n  * [b]zoab[b] `.zoa`: a binary structured data composed of only data and array.\n  * [b]zoat[b] `.zoa`: an extensible text format for representing zoab in a\n      human readable way. Zoat intentionally has little functionality on it's\n      own but allows extensibility and a standardized syntax for the different\n      tools built on top of it.\n  * [b]zoac[b] `.zc`: a configuration lanugage built on zoat and and another\n      langauge (i.e. @fngi), allowing execution of arbitrary (but\n      stateless) user-defined functions.\n  * [b]zoash[b] `.zh`: a text-based shell built on zoac allowing creating of\n      processes, mutation of files or anything else a computer user might want\n      to do.  zoac functions are still callable to process data, but other\n      functions are available as well perform side effects.  File extension:\n      .zh\n[/]\n\n[\"]Yes, zoab and zoat have the same file extension. This is intentional and you\n   will see how below![/]\n\nBoth zoab and zoat are extremely simple.\n\nzoab is about the simplest possible general purpose structured data that can\nexist. Having only two types allows it to be compact and fast.  The use of\nsingle byte signals alows it to be trivially deserialized by even\nembedded system.\n\nzoat is simpler than yaml but can achieve the same functionaly but with far\nless ambiguitity. The only thing it is missing is integer/float/literal-map/[\n]none types, which are not actually necessary for serializing or deserializing\narbitrary structured data. Zoat also has massively increased functionality over\nyaml with features like variable storage and concatenation.\n\n[t set=protobuf r=https://developers.google.com/protocol-buffers]protobuf[/]\n[t set=cxt r=https://github.com/civboot/cxt]cxt[/]\n[t set=Civboot r=https://civboot.org]Civboot[/]\n[t set=fngi r=http://github.com/civboot/fngi]fngi[/]\n\n[!]##########################[/]\n[h2]zoat[/]\n\nzoat (like zoab) has only two data types: data and array. Data start at the\nfirst non-whitespace character (except for `|` or `{`, which we will get to)\nand continue until a pipe. For example:\n\n[###]\nThis is a zoat string. It ends with a pipe|\n[###]\n\n\nAn array is specified using curly braces and can be nested, like this:\n[###]\n{ first string in array|\n  second string in array|\n  { nested array element 1|\n    nested array element 2|\n  }\n}\n[###]\n\n\nLeading whitespace is always ignored until the first character, after which\nnewlines are treated as spaces and special characters use c-like escapes. Here\nare the valid escapes in text:\n[###]\n  \\n          newline\n  \\t          tab\n  \\|          a literal '|' character\n  \\{          a literal '{' character\n  \\}          a literal '}' character\n  \\s\n  \\           a literal space character\n  \\xHH        hex byte of value 0xHH\n  \\\u003cnewline\u003e  line continuation. When \\ is used at the end of a line no\n              space will be insert when going to the next line.\n\n      An example,\n      there is one space after the comma above. \\\n      \\  However there are three spaces after the period since\n      an escape was used both before and after the newline.|\n[###]\n\n\nThe following are how to use commands. Commands must be specified immediately\nfollowing either a pipe or curly brace:\n\n[###]\n  |X          Where X is a non-whitespace character. Runs a command, some of\n              which are defined below. Note that multiple '|' together are\n              ignored, i.e. '|| | | ||' is the same as '|'.\n\n  {           Starts an array. Characters following are always treated as text\n              (never a command), but whitespace is still ignored. Example:\n                {array item 1| array item 2}\n\n  }           Ends an array. Characters following are always treated as text\n              (never a command), but whitespace is still ignored. Example above.\n\n              Note: {{}} will create an empty array inside a array. However,\n              adding '|' does nothing. i.e. '|{| {||} |}|' is exactly the same.\n              This allows for trailing pipes. If you've ever used JSON you know\n              the pain of not being able to use trailing commas, this avoids\n              that pain.\n\n  | \u003cdata\u003e    A set of data until the next '|', '{' or '}'. There _must_ be\n              whitespace (newline, tab or space) after | otherwise the character\n              will be treated as a command. Whitespace will be ignored until the\n              first non-whitespace character.\n\n  |\\          Starts text immediately using the escape. Example:\n                  |\\nThis text started on a new line.|\n\n  |*          Line comment. Text is ignored until a newline.\n\n  {*          Block comment {*nesting allowed*}, it must be closed with: *}\n              After a closing block comment a new command can start immediately.\n              Example:\n                  {* comment *} |h1 header\n\n  |'''        Plain-text/code block. Any number of ' can be used, and must be\n              escaped with the same number of ', which ends the item (but will\n              not execute a command. As a general rule only '|' executes\n              commands). The raw text will ignore escapes (i.e. \\n) and preserve\n              newlines, _except_ for the first newline if it immediately follows\n              the opening ' block or the last newline if it immediately preceeds\n              the ' block. Also, the text will be de-indented by the indent at\n              the start of the command. Example:\n                |'''\n                This is some raw text.\n                It will have no indent.\n                I can use { | } \\n etc and they are interpreted literally.\n                I can even end with a space. '''\n                This is a new item, as if the previous had ended with pipe|\n\n  |+          Concatenate (join) this item to the last item of the same type.\n              Note that text starts immediately.\n              Example:\n                all one |+single item. ==\u003e all one single item.|\n                {1| 2} |+ {3| 4}       ==\u003e {1| 2| 3| 4}\n\n  |.          Begins the variable compiler, similar to fngi's. When assigning\n              variables, values are not added until referenced. Examples:\n\n                |.foo = fngi |              |* set foo equal to \"fngi \"\n                { I think |.foo|is great } |* { I think |fngi |is great }\n\n              An array can be accessed by index using @\u003cindex\u003e\n\n                |.myArray @0 @3|            |* C equivalent: myArray[0][3]\n\n              If the data is an array of two-length arrays with strings as the\n              first item, you can access by key with '.'. Whitespace or special\n              characters in the keys are not supported.\n\n                |.myDict.foo.bar|\n\n  |.+         Joins the variable to the last item. Example:\n                { I think |.+foo|+is great } |* { I think fngi is great }\n\n  |[0-9]      An integer (big-endian). 0x (hex), 0b (binary) and 0c (char) are\n              supported. (see Type Specification Dyn)\n\n  |-[0-9]     A negative integer (big-endian, see Type Specification Dyn)\n[###]\n\n[!]##########################[/]\n[h2]zoab: structured array and byte data[/]\n\nThe binary representation of zoa is called zoab. Like zoat it is composed of\nonly two types: data and array. Both can only have sections of length 63, but a\njoin bit may be used to make longer values.\n\nThe start of a zoab item has a single byte which specifies it's type, join\nstatus and length. it looks like:\n\n[###]\n  Bitmap      Description\n  JTLL LLLL : J=join bit   T=type bit   L=length bits (0-63)\n[###]\n\n[+]\n  * `J` can be 1, which will cause it to be \"joined\" to the next item (which\n      must be the same type). This allows the true length to be longer than 63\n      (any length is possible with multiple joins).\n  * `T` can be 0 for \"data\" or 1 for \"array\".\n  * `L` contains a length (0-63) of the data or array.\n[/]\n\n\n[!]##########################[/]\n[h2]Binary Runtime Type Selection[/]\n\nThere is one byte that is invalid for both zoab and utf8 (and ascii):\n`J=1 T=0 L=0`, which in binary is `1000 0000` or in hex is 0x80.\n\nInstead of making this byte illegal, we will _require_ it in the `zoa(...)`\nfunction, which is intended to deserialize either zoab or zoat from user\ninputs, data stored in file systems or data transferred over the wire. If 0x80\nis the first character then it is assumed this is zoab data. If 0x80\nis [i]not[i] the first byte then it should be assumed that the file/etc is\nhuman written zoat.\n\n[\"] 0x80 will still be illegal for pure-zoab related functions, which expect\n    the data to already be deserialzed.[/]\n\n0x80 will signal that the next byte \"steam type\". The supported ones are:[+]\n  1. pure unstructured data with no special meaning.\n  2. _mostly_ human readable data. i.e. struct fields are names, integers are in\n    text format, etc. This is typically what should be passed to dbg logs, human\n    readable compressed configs, etc.\n  3. protozoa data, which is similar to protobuf. Fields are represented by\n    user-assigned integers, data is in binary format, etc. The deserializer\n    obviously needs access to a backwards-compatible reference (i.e. a struct\n    definition) to know how to deserialize this data.\n  4. log event, used for zoa's own core logging.\n[/]\n\nWe also reserve `J=1 T=1 L=0` to mean that the next data value is actually a\npointer to more data. This allows fngi (or other languages) to use a maximum\nblock size of 4kB while still permitting arbitrary length zoab data.\n\n[h2]Type Specification[/]\n[\"][b]WARNING:[b] This section is currently being worked on so is not yet\npolished.[/]\n\nZoa also has a default type protocol and text specification for generating\nserializers and deserializers for zoab data. These are typically contained in\n`.zt` files.\n\nZoa types are specified in a language similar to protobufs, with a few key\ndifferences. For an overview:\n\nNative types include:[+]\n * Data, Int, Num, Time (integers: seconds, nanoseconds), Path\n   (array of components)\n * Arr(generic)\n * IntMap(Int key, generic value), DataMap(Data key, generic value)\n * Dyn: dynamic (runtime) type\n[/]\n\nUsers can create their own struct types with the format below. Struct fields can\nbe positional or have an id. Default values can be provided, otherwise all\narguments are required.\n\n[###]\n\\ Comments use '\\' character\nstruct Foo [\n  a: Int; \\ positional required argument.\n  b: Int = 0; \\ positional argument with default.\n  c: Data = ||; \\ data argument with default.\n  d:0 Int = 0; \\ indexed argument with default.\n  d:1 Arr[Int] = 0;\n]\n[###]\n\nThe struct protocol is:[+]\n * The first items contains the number of positional args P.\n * The next P items contains the first P specified positional arguments.\n   Any unspecified positional args must have a default value.\n * Indexed args can be specified in any order.\n * Any unspecified args must have a default.\n[/]\n\nSimilarily, enumerated values can be defined. Enums must always specify their\nid. All ids from [[0 - max specified]] must be declared (in any order).\n\n[#]\nstruct Cheese [ isCheddar: Int ]\nstruct Cake [ isChocolate: Int ]\nstruct Pizza [\n  cheese: Cheese = {\n    isCheddar|1|\n  };\n  hasPeperoni: Int = 0;\n]\n\nenum Food [\n  none:0; \\ note: empty type, no data\n  someNuts:3 Int; \\ note: declared out of order (OK)\n  pizza:1 Pizza;\n  cake:2 Cake;\n  cheeses:3 Arr[Cheese];\n]\n[#]\n\nThe enum protocol is:[+]\n * The first item contains the enum variant, i.e. noaccount, user, admin\n * If the variant is not an `Arr` or `*Map`, the next item is the value.\n * else, the next N values are the data for the array.\n[/]\n\n[!]##########################[/]\n[h2]Contributing[/]\n\nTo build the README.md and run the tests, simply run `make`.\n\nWhen opening a PR to submit code to this repository you must include the\nfollowing disclaimer in your first commit message:\n\n[###]\nI \u003cauthor\u003e assent to license this and all future contributions to this project\nunder the dual licenses of the UNLICENSE or MIT license listed in the\n`UNLICENSE` and `README.md` files of this repository.\n[###]\n\n[!]##########################[/]\n[h2]LICENSING[/]\n\nThis work is part of the @Civboot project and therefore primarily exists for\neducational purposes. Attribution to the authors and project is appreciated but\nnot necessary.\n\nTherefore this body of work is licensed using the UNLICENSE unless otherwise\nspecified at the beginning of the source file.\n\nIf for any reason the UNLICENSE is not valid in your jurisdiction or project,\nthis work can be singly or dual licensed at your discression with the MIT\nlicense below.\n\n[###]\nCopyright 2022 Garrett Berg\n\nPermission is hereby granted, free of charge, to any person obtaining a copy of\nthis software and associated documentation files (the \"Software\"), to deal in\nthe Software without restriction, including without limitation the rights to\nuse, copy, modify, merge, publish, distribute, sublicense, and/or sell copies\nof the Software, and to permit persons to whom the Software is furnished to do\nso, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all\ncopies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\nSOFTWARE.\n[###]\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvitiral%2Fzoa","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvitiral%2Fzoa","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvitiral%2Fzoa/lists"}