{"id":27311767,"url":"https://github.com/mark-summerfield/uxf","last_synced_at":"2025-10-08T03:13:21.025Z","repository":{"id":293266731,"uuid":"479717390","full_name":"mark-summerfield/uxf","owner":"mark-summerfield","description":"Uniform eXchange Format (uxf) is a plain text human readable optionally typed storage format that supports custom types. It may serve as a convenient alternative to csv, ini, json, sqlite, toml, xml, or yaml.","archived":false,"fork":false,"pushed_at":"2023-12-13T01:16:54.000Z","size":4804,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-09-25T14:13:38.875Z","etag":null,"topics":["data","ini","json","parser","pretty-printer","sqlite","storage-engine","toml","xml","yaml"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mark-summerfield.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2022-04-09T12:18:38.000Z","updated_at":"2025-06-17T17:06:48.000Z","dependencies_parsed_at":"2025-05-14T14:04:44.963Z","dependency_job_id":"dee42cb7-b5dc-4250-b077-93596a9ad789","html_url":"https://github.com/mark-summerfield/uxf","commit_stats":null,"previous_names":["mark-summerfield/uxf"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/mark-summerfield/uxf","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mark-summerfield%2Fuxf","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mark-summerfield%2Fuxf/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mark-summerfield%2Fuxf/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mark-summerfield%2Fuxf/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mark-summerfield","download_url":"https://codeload.github.com/mark-summerfield/uxf/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mark-summerfield%2Fuxf/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":278882455,"owners_count":26062287,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-08T02:00:06.501Z","response_time":56,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data","ini","json","parser","pretty-printer","sqlite","storage-engine","toml","xml","yaml"],"created_at":"2025-04-12T06:35:57.737Z","updated_at":"2025-10-08T03:13:21.018Z","avatar_url":"https://github.com/mark-summerfield.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# UXF Overview\n\nUniform eXchange Format (UXF) is a plain text human readable optionally\ntyped storage format that supports custom types.\n\nUXF is designed to make life easier for software developers and data\ndesigners. It directly competes with csv, ini, json, toml, and yaml formats.\nA key advantage of UXF is its support for custom (i.e., user-defined) types.\nThis can result in more compact, more readable, and easier to parse data.\nAnd in some contexts it may prove to be a convenient alternative to sqlite\nor xml.\n\n- [Datatypes](#datatypes)\n    - [Table of Built-in Types](#table-of-built-in-types)\n    - [Terminology](#terminology)\n    - [Minimal empty UXF](#minimal-empty-uxf)\n    - [Built-in Types](#built-in-types)\n    - [Custom Types](#custom-types)\n    - [Formatting](#formatting)\n- [Examples](#examples)\n    - [JSON](#json)\n    - [CSV](#csv)\n    - [TOML](#toml)\n    - [Configuration Files](#configuration-files)\n    - [Database](#database)\n- [Libraries](#libraries) [[Python](py/README.md)] [[Rust](rs/README.md)]\n    - [Implementation Notes](#implementation-notes)\n- [Imports](#imports)\n- [BNF](#bnf)\n- [Supplementary](#supplementary)\n    - [Vim Support](#vim-support)\n    - [UXF Logo](#uxf-logo)\n\n## Datatypes\n\nUXF supports the following eleven built-in datatypes.\n\n\n|**Type**\u003ca name=\"table-of-built-in-types\"\u003e\u003c/a\u003e|**Example(s)**|**Notes**|\n|-----------|----------------------|--|\n|`null`     |`?`|`?` is the UXF _null_ type's literal representation.|\n|`bool`     |`no` `yes`|Use `no` for false and `yes` for true.|\n|`bytes`    |`(:20AC 65 66 48:)`|There must be an even number of case-insensitive hex digits; whitespace (spaces, newlines, etc.) optional.|\n|`date`     |`2022-04-01`|Basic ISO8601 YYYY-MM-DD format.|\n|`datetime` |`2022-04-01T16:11:51`|ISO8601 YYYY-MM-DDTHH[:MM[:SS]] format; 1-sec resolution no timezone support (see also [Custom Types](#custom-types)).|\n|`int`      |`-192` `+234` `7891409`|Standard integers with optional sign.|\n|`real`     |`0.15` `0.7e-9` `2245.389`|Standard and scientific notation.|\n|`str`      |`\u003cSome text which may include newlines\u003e`|For \u0026, \u003c, \u003e, use \\\u0026amp;, \\\u0026lt;, \\\u0026gt; respectively.|\n|`list`     |`[value1 value2 ... valueN]`|A list of values of any type.|\n|`list`     |`[vtype value1 value2 ... valueN]`|A list of values of type _vtype_.|\n|`map`      |`{key1 value1 key2 value2 ... keyN valueN}`|A map with keys of any valid key type and values of any type.|\n|`map`      |`{ktype key1 value1 key2 value2 ... keyN valueN}`|A map with keys of type _ktype_ and values of any type.|\n|`map`      |`{ktype vtype key1 value1 key2 value2 ... keyN valueN}`|A map with keys of type _ktype_ and values of type _vtype_.|\n|`table`    |`(ttype \u003cvalue0_0\u003e ... \u003cvalue0_N\u003e ... \u003cvalueM_0\u003e ... \u003cvalueM_N\u003e)`|A table of values. Each value's type must be of the corresponding type specified in the _ttype_, or any value type where no type has been specified.|\n\nNote that it is also possible to represent [Custom Types](#custom-types).\n\n### Terminology\n\n- A `map` _key-value_ is collectively called an _item_.\n- A “single” valued type (`bool`, `bytes`, `date`, `datetime`, `int`,\n  `str`), is called a _scalar_.\n- A “multi-” valued type (`list`, `map`, `table`) is called a _collection_.\n- A `list`, `map`, or `table` which contains only scalar values is called a\n  scalar `list`, scalar `map`, or scalar `table`, respectively.\n- A _`ttype`_ is the name of a user-defined table type.\n\n### Minimal empty UXF\n\n    uxf 1\n    []\n\nEvery UXF file consists of a single header line (starting `uxf 1`,\noptionally followed by custom text). This may be followed by an optional\nfile-level comment, then any _ttype_ (table type) imports, then any _ttype_\ndefinitions. After this comes the data in the form of a single `list`,\n`map`, or `table` in which all the values are stored. The data must be\npresent even if it is merely an empty list (as here), an empty map (e.g.,\n`{}`), or an empty table. Since ``list``s, ``map``s, and ``table``s can be\nnested inside each other, the UXF format is extremely flexible.\n\n### Built-in Types\n\nMap keys (i.e., _ktype_) may only be of types `bytes`, `date`, `datetime`,\n`int`, and `str` and may not be null (`?`).\n\nList, map, and table values may be of _any_ type (including nested ``map``s,\n``list``s, and ``table``s), unless constrained to a specific type. If\nconstrained to a specific _vtype_, the _vtype_ may be any built-in type (as\nlisted above, except `null`), or any user-defined _ttype_, and the\ncorresponding value or values must be any valid value for the specified\ntype, or `?` (null).\n\nLists and tables preserve the order in which values appear. So the first\nvalue is at index/row 0, the second at index/row 1, etc. Maps are\nkey-ordered. In particular when two keys are of different types they are\nordered `bytes` `\u003c` `date` `\u003c` `datetime` `\u003c` `int` `\u003c` `str`, and when two\nkeys have the same types they are ordered using `\u003c` except for ``str``s\nwhich use case-insensitive `\u003c`.\n\nA `table` starts with a _ttype_. Next comes the table's values. The number\nof values in any given row is equal to the number of field names in the\n_ttype_.\n\nLists, maps, tables, and _ttype_ definitions may begin with a comment. And\nlists, maps, and tables may optionally by typed as indicated above. (See\nalso the examples below and the BNF near the end).\n\nStrings may not include `\u0026`, `\u003c` or `\u003e`, so if they are needed, they must be\nreplaced by the XML/HTML escapes `\u0026amp;`, `\u0026lt;`, and `\u0026gt;` respectively.\nStrings respect any whitespace they contain, including newlines.\n\nWhere whitespace is allowed (or required) it may consist of one or more\nspaces, tabs, or newlines in any combination.\n\nIf you don't want to be committed to a particular UXF type, just use a `str`\nand do whatever conversion you want, or use a [Custom Type](#custom-types).\n\n### Custom Types\n\nThere are two common approaches to handling custom types in UXF. Both\nallow for UXFs to remain round-trip readable and writeable even by UXF\nprocessors that aren't aware of the use of custom types as such.\n\nHere, we'll look at both approaches for three different custom types, a\npoint and some constants which we'll treat as enumerations.\n\n    uxf 1\n    [\n      {\u003cPoint\u003e [1.4 9.8]} {\u003cPoint\u003e [-0.7 3.0]} {\u003cPoint\u003e [2.1 -6.3]}\n      \u003cTrafficLightGreen\u003e \u003cTrafficLightAmber\u003e \u003cTrafficLightRed\u003e\n    ]\n\nThis first approach shows three points, each represented by a `map` with a\n`str` indicating the custom type (“Point”), and using ``list``s of two\n``real``s for the _x_ and _y_ coordinates. The example also shows traffic\nlight constants each represented by a `str`.\n\n    uxf 1\n    [\n      {\u003cPoint\u003e [1.4 9.8 -0.7 3.0 2.1 -6.3]}\n      \u003cTrafficLightGreen\u003e \u003cTrafficLightAmber\u003e \u003cTrafficLightRed\u003e\n    ]\n\nSince we have multiple points we've changed to a single `map` with a `list`\nof point values. This is more compact but assumes that the reading\napplication knows that points come in pairs.\n\nA UXF processor has no knowledge of these representations of points or\nconstants (or constants used as enumerations), but will handle both\nseamlessly since they are both represented in terms of built-in UXF types.\nNonetheless, an application that reads such UXF data can recognize and\nconvert to and from these representations to and from the actual types.\n\n    uxf 1\n    =Point x:real y:real\n    =TrafficLightGreen\n    =TrafficLightAmber\n    =TrafficLightRed\n    [\n      (Point 1.4 9.8 -0.7 3.0 2.1 -6.3)\n      (TrafficLightGreen) (TrafficLightAmber) (TrafficLightRed)\n    ]\n\nThis second approach uses four _ttypes_ (custom table types). For the Point\nwe specify it as having two real fields (so the processor now knows that\nPoints have two `real` values). And for the enumeration we used three\nseparate fieldless tables, i.e., three constants.\n\nUsing tables has the advantage that we can represent any number of values of\na particular _ttype_ in a single table (including just one, or even none),\nthus cutting down on repetitive text. Here, the Point table has three Points\n(rows). And some UXF processor libraries will be able to return table values\nas custom types. (For example, the [Python UXF library](py/README.md) would\nreturn these as custom class instances—as “editable tuples”.)\n\nIf many applications need to use the same _ttypes_, it _may_ make sense to\ncreate some shared _ttype_ definitions. See [Imports](#imports) for how to\ndo this.\n\n### Formatting\n\nA UXF file's header must always occupy its own line (i.e., end with a\nnewline). The rest of the file could in theory be a single line no matter\nhow long. In practice and for human readability it is normal to limit the\nwidth of lines, for example, to 76, 80, or the UXF default of 96 characters.\n\nA UXF processor is expected to provide formatting options for pretty\nprinting UXF files with user defined indentation, wrap width, and real\nnumber formatting.\n\nUXF `bytes` and ``str``s can be of any length, but nonetheless they can be\nwidth-limited without changing their semantics.\n\n#### Bytes\n\nAny `bytes` value may be written with any amount of whitespace including\nnewlines—with all the whitespace ignored. For example:\n\n    (:AB DE 01 57:) ≣ (:ABDE0157:)\n\nThis makes it is easy to convert a `bytes` that is too long into chunks,\ne.g.,\n\n    (:20 AC 40 41 ... lots more ... FF FE:)\n\nto, say:\n\n    (:20 AC 40 41\n    ... some more ...\n    ... some more ...\n    FF FE:)\n\n#### Strings\n\nBecause UXF strings respect any whitespace they contain they cannot be split\ninto chunks like `bytes`. However, UXF supports a string concatenation\noperator such that:\n\n    \u003cThis is one string\u003e ≣ \u003cThis \u003e \u0026 \u003cis one \u003e \u0026 \u003cstring\u003e\n\nWhich means, of course, that given a long string that might not contain\nnewlines or whose lines are too long, we can easily split it into chunks,\ne.g.,\n\n    \u003cImagine this is a really long string...\u003e\n\nto, say:\n\n    \u003cImagine \u003e \u0026\n    \u003cthis is a \u003e \u0026\n    \u003creally long \u003e \u0026\n    \u003cstring...\u003e\n\nComments work the same way, but note that the comment marker must only\nprecede the _first_ fragment.\n\n    #\u003cThis is a comment in one or more strings.\u003e ≣ #\u003cThis is a \u003e \u0026 \u003ccomment in \u003e \u0026 \u003cone or more\u003e \u0026 \u003c strings.\u003e\n\n## Examples\n\n### Minimal UXFs\n\n    uxf 1\n    {}\n\nWe saw earlier an example of a minimal UXF file with an empty list; here we\nhave one with an empty map.\n\n    uxf 1\n    =Pair first second\n    (Pair)\n\nHere is a UXF with a _ttype_ specifying a Pair that has two fields each of\nwhich can hold _any_ UXF value (including nested collections). In this case\nthe data is a single _empty_ Pair table.\n\n    uxf 1\n    =Pair first second\n    (Pair (Pair 1 2) (Pair 3 (Pair 4 5)))\n\nAnd here is a UXF with a single Pair table that contains two nested Pair\ntables, the second of which itself contains a nested pair.\n\n## JSON\n\nJSON is a very widely used format, but unlike UXF it lacks user-defined\ntypes. Here's an example of GeoJSON data from Wikipedia:\n\n    {\n    \"type\": \"FeatureCollection\",\n    \"features\": [\n        {\n        \"type\": \"Feature\",\n        \"geometry\": {\n            \"type\": \"Point\",\n            \"coordinates\": [102.0, 0.5]\n        },\n        \"properties\": {\n            \"prop0\": \"value0\"\n        }\n        },\n        {\n        \"type\": \"Feature\",\n        \"geometry\": {\n            \"type\": \"LineString\",\n            \"coordinates\": [\n            [102.0, 0.0], [103.0, 1.0], [104.0, 0.0], [105.0, 1.0]\n            ]\n        },\n        \"properties\": {\n            \"prop0\": \"value0\",\n            \"prop1\": 0.0\n        }\n        },\n        {\n        \"type\": \"Feature\",\n        \"geometry\": {\n            \"type\": \"Polygon\",\n            \"coordinates\": [\n            [\n                [100.0, 0.0], [101.0, 0.0], [101.0, 1.0],\n                [100.0, 1.0], [100.0, 0.0]\n            ]\n            ]\n        },\n        \"properties\": {\n            \"prop0\": \"value0\",\n            \"prop1\": { \"this\": \"that\" }\n        }\n        }\n    ]\n    }\n\nIt would be easy to “translate” this directly into UXF:\n\n    uxf 1\n    {\n    \u003ctype\u003e: \u003cFeatureCollection\u003e,\n    \u003cfeatures\u003e: [\n        {\n        \u003ctype\u003e: \u003cFeature\u003e,\n        \u003cgeometry\u003e: {\n            \u003ctype\u003e: \u003cPoint\u003e,\n            \u003ccoordinates\u003e: [102.0, 0.5]\n        },\n        \u003cproperties\u003e: {\n            \u003cprop0\u003e: \u003cvalue0\u003e\n        }\n        ...\n\nNaturally this works, but doesn't take advantage of any of UXF's benefits.\n\nHere's a more realistic possible UXF alternative:\n\n    uxf 1\n    =Feature geometry properties:map\n    =LineString x:real y:real\n    =Point x:real y:real\n    =Polygon x:real y:real\n    (Feature\n        (Point 102.0 0.5) {\u003cprop0\u003e \u003cvalue0\u003e}\n        (LineString 102.0 0.0 103.0 1.0 104.0 0.0 105.0 1.0)\n                    {\u003cprop0\u003e \u003cvalue0\u003e \u003cprop1\u003e 0.0}\n        (Polygon 100.0 0.0 101.0 0.0 101.0 1.0 100.0 1.0 100.0 0.0)\n                    {\u003cprop0\u003e \u003cvalue0\u003e \u003cprop1\u003e {\u003cthis\u003e \u003cthat\u003e}}\n    )\n\nWe don't need a FeatureCollection because UXF tables can accept zero or more\nvalues, so a Feature table is sufficient.\n\nHere's a last JSON alternative, this time avoiding the duplication of\n`x:real` and `y:real`:\n\n    uxf 1\n    =Feature geometry properties:map\n    =LineString points:Point\n    =Point x:real y:real\n    =Polygon points:Point\n    (Feature\n\t(Point 102.0 0.5) {\u003cprop0\u003e \u003cvalue0\u003e}\n\t(LineString (Point 102.0 0.0 103.0 1.0 104.0 0.0 105.0 1.0))\n\t            {\u003cprop0\u003e \u003cvalue0\u003e \u003cprop1\u003e 0.0}\n\t(Polygon (Point 100.0 0.0 101.0 0.0 101.0 1.0 100.0 1.0 100.0 0.0))\n\t         {\u003cprop0\u003e \u003cvalue0\u003e \u003cprop1\u003e {\u003cthis\u003e \u003cthat\u003e}}\n    )\n\nThis seems like the clearest solution.\n\n### CSV\n\nAlthough widely used, the CSV format is not standardized and has a number of\nproblems. UXF is a standardized alternative that can distinguish fieldnames\nfrom data rows, can handle multiline text (including text with commas and\nquotes) without formality, and can store one—or more—tables in a single UXF\nfile.\n\nHere's a simple CSV file:\n\n    Date,Price,Quantity,ID,Description\n    \"2022-09-21\",3.99,2,\"CH1-A2\",\"Chisels (pair), 1in \u0026 1¼in\"\n    \"2022-10-02\",4.49,1,\"HV2-K9\",\"Hammer, 2lb\"\n    \"2022-10-02\",5.89,1,\"SX4-D1\",\"Eversure Sealant, 13-floz\"\n\nLike with JSON we could simply “translate” this directly into UXF as a list\nof lists. But doing so would leave us with the same problem as `.csv` files:\nis the first row data values or column titles? (For software this isn't\nalways obvious, for example, if all the values are strings.) Even so, this\nis still an improvement, since unlike the `.csv` representation, every value\nwould have a concrete type (all ``str``s for the first row, and `date`,\n`real`, `int`, `str`, `str`, for the subsequent rows).\n\nThe most _appropriate_ UXF equivalent is to use a UXF `table`:\n\n    uxf 1\n    =PriceList Date Price Quantity ID Description\n    (PriceList\n      2022-09-21 3.99 2 \u003cCH1-A2\u003e \u003cChisels (pair), 1in \u0026amp; 1¼in\u003e \n      2022-10-02 4.49 1 \u003cHV2-K9\u003e \u003cHammer, 2lb\u003e \n      2022-10-02 5.89 1 \u003cSX4-D1\u003e \u003cEversure Sealant, 13-floz\u003e \n    )\n\nWhen one or more tables are used each one's _ttype_ (table type) must be\ndefined at the start of the `.uxf` file. A _ttype_ definition begins with an\n`=` sign followed by the _ttype_ (i.e., the table name), followed by zero or\nmore fields. A field consists of a name optionally followed by a `:` and\nthen a type (here only names are given).\n\nBoth table and field names are user chosen and consist of 1-60 letters,\ndigits, or underscores, starting with a letter or underscore. No table or\nfield name may be the same as any built-in type name, so no table or field\ncan be called `bool`, `bytes`, `date`, `datetime`, `int`, `list`, `map`,\n`null`, `real`, `str`, or `table`. (But `Date`, `DateTime`, and `Real` or\n`real_` are fine, since names are case-sensitive and none of the built-in\ntypes contains an underscore or uses uppercase letters.) If whitespace is\nwanted one convention is to use underscores in their place.\n\nOnce we have defined a _ttype_ we can use it.\n\nHere, we've created a single table whose _ttype_ is “PriceList”. There's no\nneed to group rows into lines as we've done here (although doing so is\ncommon and easier for human readability), since the UXF processor knows how\nmany values go into each row based on the number of field names. In this\nexample, the UXF processor will treat every five values as a single record\n(row) since the _ttype_ has five fields.\n\nThis is already an improvement on `.csv`—we know the table's name and field\nnames, and could easily store two or more tables (as we'll see later).\nAlthough the UXF processor will correctly determine the field types, what if\nwe want to constrain each field's value to a particular type?\n\n    uxf 1 Price List\n    =PriceList Date:date Price:real Quantity:int ID:str Description:str\n    (PriceList\n      2022-09-21 3.99 2 \u003cCH1-A2\u003e \u003cChisels (pair), 1in \u0026amp; 1¼in\u003e \n      2022-10-02 4.49 1 \u003cHV2-K9\u003e \u003cHammer, 2lb\u003e \n      2022-10-02 5.89 1 \u003cSX4-D1\u003e \u003cEversure Sealant, 13-floz\u003e \n    )\n\nHere we've added a custom file description in the header, and we've also\nadded field types to the _ttype_ definition. When types are specified, the\nUXF processor is expected to be able to check that each value is of the\ncorrect type. Omit the type altogether (as in the earliler examples) to\nindicate _any_ valid table type.\n\n## TOML\n\nHere is a TOML example from the TOML website and Wikipedia:\n\n    # This is a TOML document.\n\n    title = \"TOML Example\"\n\n    [owner]\n    name = \"Tom Preston-Werner\"\n    dob = 1979-05-27T07:32:00-08:00 # First class dates\n\n    [database]\n    server = \"192.168.1.1\"\n    ports = [ 8000, 8001, 8002 ]\n    connection_max = 5000\n    enabled = true\n\n    [servers]\n\n        # Indentation (tabs and/or spaces) is allowed but not required\n        [servers.alpha]\n        ip = \"10.0.0.1\"\n        dc = \"eqdc10\"\n\n        [servers.beta]\n        ip = \"10.0.0.2\"\n        dc = \"eqdc10\"\n\n    [clients]\n    data = [ [\"gamma\", \"delta\"], [1, 2] ]\n\n    # Line breaks are OK when inside arrays\n    hosts = [\n    \"alpha\",\n    \"omega\"\n    ]\n\nAnd here's a possible UXF alternative:\n\n    uxf 1\n    #\u003cUXF version of TOML Example\u003e\n    =Clients a b\n    =Database server:str ports:list connection_max:int enabled:bool\n    =DateTime when:datetime tz:str\n    =Owner name:str dob:DateTime\n    =Server name:str ip:str dc:str\n    =Hosts name:str\n    [\n      (Owner \u003cTom Preston-Werner\u003e (DateTime 1979-05-27T07:32:00 \u003c-08:00\u003e))\n      (Database \u003c192.168.1.1\u003e [8000 8001 8002] 5000 yes)\n      (Server \u003calpha\u003e \u003c10.0.0.1\u003e \u003ceqdc10\u003e\n              \u003cbeta\u003e \u003c10.0.0.2\u003e \u003ceqdc10\u003e)\n      (Clients \u003cgamma\u003e \u003cdelta\u003e 1 2)\n      (Hosts\n        \u003calpha\u003e\n        \u003comega\u003e)\n    ]\n\nThe main differences from ``.toml`` are that UXF quotes strings using\n``\u003c\u003e``s, and uses ``yes`` and ``no`` for ``bool``s. UXF doesn't require the\nuse of indentation, but UXF processors default to using it for pretty\nprinting.\n\nUnlike TOML, UXF doesn't natively support timezones, so we've created a\nDateTime _ttype_ which has a when datetime and a timezone offset. For\nClients the data will come in pairs because we've specified two fields.\nAlthough written compactly, we could have newlines wherever whitespace is\nrequired—or optional.\n\nThere are many similar formats, including ``.conf``, ``.ini``, and\n``.yaml``, all of which can easily be advantageously translated into UXF.\n\n### Configuration Files\n\nIn general, if you want all available options to be in the UXF file then it\nis best to use _ttypes_. However, if you are happy for the UXF file to\nhave only those options the user chooses to change from the defaults, it is\nprobably easiest to use a ``map``.\n\nIn this example we have a configuration file which uses _ttypes_ so all data\nis always present:\n\n    uxf 1 TLM Config\n    =Config window:Window volume:real historysize:int recentfiles:list\n    =Window x:int y:int width:int height:int\n    (Config\n        (Window 441 67 729 702)\n        1.00\n        35\n        [\n            \u003c/home/mark/data/Suki-Playlists.tlm.gz\u003e\n            \u003c/home/mark/data/playlists-all.tlm.gz\u003e\n            \u003c/home/mark/app/rs/tlm/Test-Playlists.tlm.gz\u003e\n        ]\n    )\n\nIf we are happy to use default values, we could, of course use ``null``s or\nempty tables:\n\n    uxf 1 TLM Config\n    =Config window:Window volume:real historysize:int recentfiles:list\n    =Window x:int y:int width:int height:int\n    (Config\n        (Window)\n        ?\n        ?\n        [\n            \u003c/home/mark/data/Suki-Playlists.tlm.gz\u003e\n            \u003c/home/mark/data/playlists-all.tlm.gz\u003e\n            \u003c/home/mark/app/rs/tlm/Test-Playlists.tlm.gz\u003e\n        ]\n    )\n\nHere, the `Window` table is empty; an equally valid alternative would be to\nmake it null (`?`). Then, when this file is loaded a null or empty `Window`\nwould cause the application to use the default size and position, and\nsimilarly, the nulls for `volume` and `historysize` would lead to defaults\nbeing used.\n\nThe advantage of using _ttypes_ is that every option is always visible in\nthe UXF file (even if only in the _ttype_ definitions).\n\nHowever, if you want the simplest and most minimal configuration files,\nusing a `map` is a sensible alternative:\n\n    uxf 1 TLM Config\n    {\n        \u003cwindow\u003e [441 67 729 702]\n        \u003cvolume\u003e 1.00\n        \u003chistorysize\u003e 35\n        \u003crecentfiles\u003e [\n            \u003c/home/mark/data/Suki-Playlists.tlm.gz\u003e\n            \u003c/home/mark/data/playlists-all.tlm.gz\u003e\n            \u003c/home/mark/app/rs/tlm/Test-Playlists.tlm.gz\u003e\n        ]\n    }\n\nHere we've shown every option in use, but of course, with a `map` we need\nonly store the options actually used:\n\n    uxf 1 TLM Config\n    {\n        \u003crecentfiles\u003e [\n            \u003c/home/mark/data/Suki-Playlists.tlm.gz\u003e\n            \u003c/home/mark/data/playlists-all.tlm.gz\u003e\n            \u003c/home/mark/app/rs/tlm/Test-Playlists.tlm.gz\u003e\n        ]\n    }\n\nAs with using nulls in the _ttype_-based version, here the application is\nassumed to use default values for absent items.\n\n### Database\n\nDatabase files aren't normally human readable and usually require\nspecialized tools to read and modify their contents. Yet many databases are\nrelatively small (both in size and number of tables), and would be more\nconvenient to work with if human readable. For these, UXF format provides a\nviable alternative.\n\nA UXF equivalent to a database of tables can easily be created using a\n`list` of ``table``s:\n\n    uxf 1 MyApp Data\n    =Customers CID Company Address Contact Email\n    =Invoices INUM CID Raised_Date Due_Date Paid Description\n    =Items IID INUM Delivery_Date Unit_Price Quantity Description\n    [#\u003cThere is a 1:M relationship between the Invoices and Items tables\u003e\n      (Customers\n        50 \u003cBest People\u003e \u003c123 Somewhere\u003e \u003cJohn Doe\u003e \u003cj@doe.com\u003e \n        19 \u003cSupersuppliers\u003e ? \u003cJane Doe\u003e \u003cjane@super.com\u003e \n      )\n      (Invoices\n        152 50 2022-01-17 2022-02-17 no \u003cCOD\u003e \n        153 19 2022-01-19 2022-02-19 yes \u003c\u003e \n      )\n      (Items\n        1839 152 2022-01-16 29.99 2 \u003cBales of hay\u003e \n        1840 152 2022-01-16 5.98 3 \u003cStraps\u003e \n        1620 153 2022-01-19 11.5 1 \u003cWashers (1-in)\u003e \n      )\n    ]\n\nHere we have a `list` of ``table``s representing three database tables.\nThe `list` begins with a comment.\n\nNotice that the second customer has a null (`?`) address and the second\ninvoice has an empty description.\n\n    uxf 1 MyApp Data\n    #\u003cIt is also possible to have one overall comment at the beginning,\n    after the uxf header and before any ttype definitions or the data.\u003e\n    =Customers CID:int Company:str Address:str Contact:str Email:str\n    =Invoices INUM:int CID:int Raised_Date:date Due_Date:date Paid:bool Description:str\n    =Items IID:int INUM:int Delivery_Date:date Unit_Price:real Quantity:int Description:str\n    [#\u003cThere is a 1:M relationship between the Invoices and Items tables\u003e\n      (Customers\n        50 \u003cBest People\u003e \u003c123 Somewhere\u003e \u003cJohn Doe\u003e \u003cj@doe.com\u003e \n        19 \u003cSupersuppliers\u003e ? \u003cJane Doe\u003e \u003cjane@super.com\u003e \n      )\n      (Invoices\n        152 50 2022-01-17 2022-02-17 no \u003cCOD\u003e \n        153 19 2022-01-19 2022-02-19 yes \u003c\u003e \n      )\n      (Items\n        1839 152 2022-01-16 29.99 2 \u003cBales of hay\u003e \n        1840 152 2022-01-16 5.98 3 \u003cStraps\u003e \n        1620 153 2022-01-19 11.5 1 \u003cWashers (1-in)\u003e \n      )\n    ]\n\nHere, we've added types to each table's _ttype_.\n\nIt is conventional in a database to have IDs and foreign keys. But these can\noften be avoided by using hierarchical data. For example:\n\n    uxf 1 MyApp Data\n    #\u003cThere is a 1:M relationship between the Invoices and Items tables\u003e\n    =Database customers:Customers invoices:Invoices\n    =Customers CID:int Company:str Address:str Contact:str Email:str\n    =Invoices INUM:int CID:int Raised_Date:date Due_Date:date Paid:bool\n    Description:str Items:Items\n    =Items IID:int Delivery_Date:date Unit_Price:real Quantity:int Description:str\n    (Database\n        (Customers\n        50 \u003cBest People\u003e \u003c123 Somewhere\u003e \u003cJohn Doe\u003e \u003cj@doe.com\u003e \n        19 \u003cSupersuppliers\u003e ? \u003cJane Doe\u003e \u003cjane@super.com\u003e \n        )\n        (Invoices\n        152 50 2022-01-17 2022-02-17 no \u003cCOD\u003e (Items\n            1839 2022-01-16 29.99 2 \u003cBales of hay\u003e \n            1840 2022-01-16 5.98 3 \u003cStraps\u003e \n            )\n        153 19 2022-01-19 2022-02-19 yes \u003c\u003e (Items\n            1620 2022-01-19 11.5 1 \u003cWashers (1-in)\u003e \n            )\n        )\n    )\n\nNotice that Items no longer need an INUM to identify the Invoice they belong\nto because they are nested inside their Invoice. However, the relational\napproach has been retained for Customers since more than one Invoice could\nbe for the same Customer.\n\nIn addition, rather than using a simple `list` of tables, we've created a\n“Database” _ttype_ and specified it as containing two tables.\n\nWhat if we wanted to add some extra configuration data to the database? One\nsolution would be to add a third field to the “Database” _ttype_ (e.g., \n`=Database customers:Customers invoices:Invoices config:map`). Or we could\ngo further and specify a “Config” _ttype_ and specify the third field as\n`config:Config`.\n\n### Additional Examples\n\nSee the `testdata` folder for more examples of `.uxf` files (some with other\nsuffixes). See also the `t` and `eg` folders in each language-specific\nlibrary (e.g., `py/t` and `py/eg`) for additional examples.\n\n## Libraries\n\n|**Library**|**Language**|**Notes**                    |\n|-----------|------------|-----------------------------|\n|uxf        | Python 3   | See the [Python UXF library](py/README.md).|\n|uxf        | Rust       | See the [Rust UXF library](rs/README.md).|\n\n### Implementation Notes\n\nWe very much hope that additional UXF library implementations will be\nwritten in other languages (as well as superior implementations—e.g., faster\nand with better APIs—in Python and Rust).\n\nIf you create a UXF library please let us know so that we can add a link\nhere (providing your library passes the regression tests!).\n\nImplmenting a UXF pretty printer should be doable by a CS major as a final\nyear project. Implementing a UXF parser—without support for imports or\nstring concatenation—should be doable by a CS major as a _big_ final year\nproject.\n\n## Imports\n\nUXF files are normally completely self-contained. However, in some cases it\nmay be desirable to share a set of _ttype_ definitions amongst many UXF\nfiles.\n\nThe _disadvantages_ of doing this are: first, that the relevant UXF files\nbecome dependent on one or more external dependencies; second, it is\npossible to have import conflicts (i.e., two _ttypes_ with the same name but\ndifferent definitions; and third, if URL imports are used, load times will\nbe affected by network availability and latency. (However, the first and\nthird disadvantages don't apply if all the dependencies are provided by the\nUXF processor itself, i.e., are system imports.)\n\nThe _advantage_ of importing _ttype_ definitions is that for UXF's that have\nlots of _ttypes_, only the import(s) and the data need be in the file,\nwithout having to repeat all the _ttype_ definitions.\n\nImports go at the start of the file _after_ the header and _after_ any\nfile-level comment, and _before_ any _ttype_ definitions. Each import must\nbe on its own line and may not span lines, nor have comments.\n\nIf a filename import has no path or a relative path, the import attempt will\nbe made relative to the importing `.uxf` file, and failing that, relative to\nthe current folder, and failing those, relative to each path in the\n`UXF_PATH` environment variable (if it exists and is nonempty).\n\nAny _ttype_ definition that follows an import will redefine any imported\ndefintion of the same name.\n\n|**Import**|**Notes**|\n|----------|---------|\n|`! complex`|System import of _ttype_ `Complex`|\n|`! fraction`|System import of _ttype_ `Fraction`|\n|`! numeric`|System import of _ttypes_ `Complex` and `Fraction`|\n|`! mydefs.uxi`|Import the _ttypes_ from `mydefs.uxi` in the importing `.uxf` file's folder, or from the current folder, or from a folder in the `UXF_PATH`|\n|`! /path/to/shared.uxf`|Import the _ttypes_ from the given file|\n|`! http://www.qtrac.eu/ttype-eg.uxf`|Import from the given URL|\n\nImports with no suffix (e.g., `complex`, `fraction`, `numeric`), are\nprovided by the UXF processor itself.\n\nThe imported file must be a valid UXF file. It need not have a `.uxf` suffix\n(e.g., you might prefer `.uxt` or `.uxi`), but must have _a_ suffix (to\ndistinguish it from a system import), and must have a `.gz` suffix if gzip\ncompressed. Any custom string, comments, or data the imported file may\ncontain are ignored: only the _ttype_ definitions are used.\n\n    uxf 1\n    !complex\n    !fraction\n    [(Complex 5.1 7.2 8e-2 -9.1e6 0.1 -11.2) \u003ca string\u003e (Fraction 22 7 355 113)]\n\nHere we've used the official system ``complex``'s `Complex` and\n``fraction``'s `Fraction` _ttypes_ without having to specify them\nexplicitly. The data represented is a list consisting of three Complex\nnumbers each holding two ``real``s each, a `str`, and two Fractions holding\ntwo ``int``s each.\n\n    uxf 1\n    !numeric\n    [(Complex 5.1 7.2 8e-2 -9.1e6 0.1 -11.2) \u003ca string\u003e (Fraction 22 7 355 113)]\n\nThis is the same as the previous example, but using the system convenience\n`numeric` import to pull in both the `Complex` and `Fraction` _ttypes_.\n\nIf you choose to use imports we recommed that UXF files intended for import\n_either_ contain a single _ttype_ definition _or_ two or more imports.\n\nWe recommend avoiding imports and using stand-alone UXF files wherever\npossible. Some UXF processors can do UXF to UXF conversions that will\nreplace imports with (actually used) _ttype_ definitions. (For example, the\n[Python UXF library](py/README.md)'s `uxf.py` module can do this.)\n\n## BNF\n\nA UXF file consists of a mandatory header followed by an optional file-level\ncomment, optional imports, optional _ttype_ definitions, and then a single\nmandatory `list`, `map`, or `table` (which may be empty).\n\n    UXF          ::= 'uxf' RWS VERSION CUSTOM? '\\n' CONTENT\n    VERSION      ::= /\\d{1,3}/\n    CUSTOM       ::= RWS [^\\n]+ # user-defined data e.g. filetype and version\n    CONTENT      ::= COMMENT? IMPORT* TTYPEDEF* (MAP | LIST | TABLE)\n    IMPORT       ::= '!' /\\s*/ IMPORT_FILE '\\n' # See below for IMPORT_FILE\n    TTYPEDEF     ::= '=' COMMENT? OWS IDENFIFIER (RWS FIELD)* # IDENFIFIER is the ttype (i.e., the table name)\n    FIELD        ::= IDENFIFIER (OWS ':' OWS VALUETYPE)? # IDENFIFIER is the field name (see note below)\n    MAP          ::= '{' COMMENT? MAPTYPES? OWS (KEY RWS VALUE)? (RWS KEY RWS VALUE)* OWS '}'\n    MAPTYPES     ::= OWS KEYTYPE (RWS VALUETYPE)?\n    KEYTYPE      ::=  'bytes' | 'date' | 'datetime' | 'int' | 'str'\n    VALUETYPE    ::= KEYTYPE | 'bool' | 'real' | 'list' | 'map' | 'table' | IDENFIFIER # IDENFIFIER is table name\n    LIST         ::= '[' COMMENT? LISTTYPE? OWS VALUE? (RWS VALUE)* OWS ']'\n    LISTTYPE     ::= OWS VALUETYPE\n    TABLE        ::= '(' COMMENT? OWS IDENFIFIER (RWS VALUE)* ')' # IDENFIFIER is the ttype (i.e., the table name)\n    COMMENT      ::= OWS '#' STR\n    KEY          ::= BYTES | DATE | DATETIME | INT | STR\n    VALUE        ::= KEY | NULL | BOOL | REAL | LIST | MAP | TABLE\n    NULL         ::= '?'\n    BOOL         ::= 'no' | 'yes'\n    INT          ::= /[-+]?\\d+/\n    REAL         ::= # standard or scientific notation\n    DATE         ::= /\\d\\d\\d\\d-\\d\\d-\\d\\d/ # basic ISO8601 YYYY-MM-DD format\n    DATETIME     ::= /\\d\\d\\d\\d-\\d\\d-\\d\\dT\\d\\d(:\\d\\d(:\\d\\d)?)?/ # see note below\n    STR          ::= STR_FRAGMENT (OWS '\u0026' OWS STR_FRAGMENT)*\n    STR_FRAGMENT ::= /[\u003c][^\u003c\u003e]*?[\u003e]/ # newlines allowed, and \u0026amp; \u0026lt; \u0026gt; supported i.e., XML\n    BYTES        ::= '(:' (OWS [A-Fa-f0-9]{2})* OWS ':)'\n    IDENFIFIER   ::= /[_\\p{L}]\\w{0,31}/ # Must start with a letter or underscore; may not be a built-in typename or constant\n    OWS          ::= /[\\s\\n]*/\n    RWS          ::= /[\\s\\n]+/ # in some cases RWS is actually optional\n\nNote that a UXF file _must_ contain a single list, map, or table, even if\nit is empty.\n\nAn `IMPORT_FILE` may be a filename which does _not_  have a file suffix, in\nwhich case it is assumed to be a “system” UXF provided by the UXF processor\nitself. (Currently there are just three system UXFs: `complex`, `fraction`,\nand `numeric`.) Or it may be a filename with an absolute or relative path.\nIn the latter case the import is searched for in the importing `.uxf` file's\nfolder, or the current folder, or a folder in the `UXF_PATH` until it is\nfound—or not). Or it may be a URL referring to an external UXF file. (See\n[Imports](#imports).)\n\nTo indicate any type valid for the context, simply omit the type name.\n\nAs the BNF shows, `list`, `map`, and `table` values may be of _any_ type\nincluding nested ``list``s, ``map``s, and ``table``s.\n\nFor a `table`, after the optional comment, there must be an identifier which\nis the table's _ttype_. This is followed by the table's values. There's no\nneed to distinguish between one row and the next (although it is common to\nstart new rows on new lines) since the number of fields indicate how many\nvalues each row has. It is possible to create tables that have no fields;\nthese might be used for representing constants (or enumerations or states).\n\nNote that for any given table each field name must be unique.\n\nIf a list value, map key, or table value's type is specified, then the UXF\nprocessor is expected to be able to check for (and if requested and\npossible, correct) any mistyped values. UXF writers are expected output\ncollections—``list`` values and  ``table`` records (and values within\nrecords) in order. Similarly `map` items should be output in key-order: when\ntwo keys are of different types they should be ordered `bytes` `\u003c` `date`\n`\u003c` `datetime` `\u003c` `int` `\u003c` `str`, and when two keys have the same types\nthey should be ordered using `\u003c` except for ``str``s which should use\ncase-insensitive `\u003c`.\n\nFor ``datetime``'s, only 1-second resolution is supported and no timezones.\nIf microsecond resolution or timezones are required, consider using custom\n_ttypes_, e.g.,\n\n    =Timestamp when:datetime microseconds:real\n    =DateTime when:datetime tz:str\n\nAlternatively, if all the ``datetime``s in a UXF have the _same_ timezone,\none approach would be to to just set it once, and then use plain\n``datetime``s throughout e.g.,\n\n    =Timezone tz:str\n    [(Timezone \u003c+01:00\u003e) ... 1990-01-15T13:05 ...]\n\nNote that a UXF reader (writer) _must_ be able to read (write) a plain text\n`.uxf` file containing UTF-8 encoded text, and _ought_ to be able to read\nand write gzipped plain text `.uxf.gz` files.\n\nNote also that UXF readers and writers should _not_ care about the actual\nfile extension (apart from the `.gz` needed for gzipped files), since users\nare free to use their own. For example, `data.myapp` and `data.myapp.gz`.\n\n## Supplementary\n\n### Vim Support\n\nIf you use the vim editor, simple color syntax highlighting is available.\nCopy `uxf.vim` into your `$VIM/syntax/` folder and add these lines (or\nsimilar) to your `.vimrc` or `.gvimrc` file:\n\n    au BufRead,BufNewFile,BufEnter * if getline(1) =~ '^uxf ' | setlocal ft=uxf | endif\n    au BufRead,BufNewFile,BufEnter *.uxf set ft=uxf|set expandtab|set tabstop=2|set softtabstop=2|set shiftwidth=2\n\n### UXF Logo\n\n![uxf logo](uxf.svg)\n\n---\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmark-summerfield%2Fuxf","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmark-summerfield%2Fuxf","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmark-summerfield%2Fuxf/lists"}