{"id":15886691,"url":"https://github.com/olebedev/ron","last_synced_at":"2025-04-02T16:23:52.305Z","repository":{"id":142443323,"uuid":"152226347","full_name":"olebedev/ron","owner":"olebedev","description":null,"archived":false,"fork":false,"pushed_at":"2018-10-09T09:53:21.000Z","size":406,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-02-08T07:11:51.286Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/olebedev.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-10-09T09:42:27.000Z","updated_at":"2018-10-09T09:53:23.000Z","dependencies_parsed_at":"2023-03-21T20:32:47.690Z","dependency_job_id":null,"html_url":"https://github.com/olebedev/ron","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/olebedev%2Fron","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/olebedev%2Fron/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/olebedev%2Fron/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/olebedev%2Fron/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/olebedev","download_url":"https://codeload.github.com/olebedev/ron/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246848302,"owners_count":20843681,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-06T05:41:22.715Z","updated_at":"2025-04-02T16:23:52.288Z","avatar_url":"https://github.com/olebedev.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"Table of Contents\n=================\n\n   * [Swarm Replicated Object Notation 2.0.1](#swarm-replicated-object-notation-201)\n      * [Formal model](#formal-model)\n      * [Wire format (text)](#wire-format-text)\n      * [Wire format (binary)](#wire-format-binary)\n         * [Descriptors](#descriptors)\n         * [Op terms](#op-terms)\n         * [Uncompressed UUIDs](#uncompressed-uuids)\n         * [Compressed UUIDs](#compressed-uuids)\n         * [Atoms](#atoms)\n      * [The math](#the-math)\n      * [Acknowledgements](#acknowledgements)\n      * [History](#history)\n\n# Swarm Replicated Object Notation 2.0.1 [![Build Status][badge]][travis]\n\nSwarm Replicated Object Notation is a format for *distributed live data*.  RON's\nfocus is on continuous data synchronization.  Every RON *object* may naturally\nhave an unlimited number of *replicas* that synchronize incrementally, mostly in\nreal-time.  RON data always merges correctly and deterministically.\n\nRON is information-centric: it aims to liberate the data from its location,\nstorage, application or transport.  There is no \"master\" replica, no \"source of\ntruth\". Every event has an *origin*, but every replica is as good as the other\none.  Every single object, event or data type is uniquely identified and\nglobally referenceable.  RON metadata makes objects completely independent of\nthe context.  A program may read RON object versions and/or updates from the\nnetwork,  filesystem, database, message bus and/or local cache, in any order,\nand merge them correctly.\n\nConsider JSON. It expresses relations by element positioning:\n\n    {\n        \"foo\": {\n            \"bar\": 1\n        }\n    }\n\nRON may express that state as:\n\n    *lww #1TUAQ+gritzko @`   :bar = 1;\n         #(R            @`   :foo \u003e (Q;\n\nThose are two RON *ops*:\n\n 1. some last-write-wins object is created with a field `bar` set to `1` (on\n    2017-10-31 10:26:00 UTC, by gritzko),\n 2. another object is created with a field `foo` pointing to the first object\n    (10:27:00, by gritzko).\n\nEach op is a tuple of four globally-unique UUIDs for its data type, object,\nevent and location, plus some number of value *atoms*.  You may not see any\nUUIDs in the above example, initially.  The notation does a lot to compress that\nmetadata away.\n\nThese are the key features of RON:\n\n* RON's basic unit is an immutable *op*. Every change to the data is an\n  *event*; every event produces an op. An op may flow from a replica to a\n  replica, from a database to a database, while fully intact and maintaining its\n  original identity.\n* Each RON op is context-independent. Nothing is implied by the context,\n  everything is specified explicitly and unambiguously in the op itself. An op\n  has four globally unique UUIDs for its data type, object, event and location.\n* An object can be referenced by its UUID (e.g. `\u003e 1TUAQ+gritzko`), thus RON can\n  express object graph structures beyond simple nesting.  Overall, RON relates\n  pieces of data by their UUIDs.  Thanks to that, RON data can be cached\n  locally, updated incrementally and edited while offline.\n* An object's state is a *reduction* of its ops. A data type is a reducer\n  function: `lww(state,change) = new_state`. Reducers tolerate partial order of\n  updates. Hence, all ops are applied immediately, without any linearization by\n  a central server.\n* There is no sharp border between a state snapshot and a state update. State is\n  change and change is state (state-change duality). A transactional unit of\n  data storage/transmission is a *frame*. A frame can contain a single op, a\n  complete object graph or anything inbetween: object state, stale state, patch,\n  otherwise a piece of an object.\n* RON model implies no special \"source of truth\". The event's *origin* is the\n  source of truth, not a server in the cloud. Every event/object is marked with\n  its origin (e.g. `gritzko` in `1TUAQ+gritzko`).\n* A RON frame is not a \"message\": it has an *origin* but it has no\n  \"destination\". RON speaks in terms of data updates and subscriptions.  Once\n  you subscribe to an object, you receive the state and all the future updates,\n  till you unsubscribe.\n* RON is information-centric. Consider git: once you clone a repo, your copy is\n  as good as the original one. Same with RON.\n* RON is a hypermedia format, as data pieces can reference each other globally\n  (imagine a RON-based real-time World-Wide-Web-of-Data).  Although, both\n  replica ids and data routing must work at global scale then (federated, etc).\n* RON is not optimized for human consumption. It is a machine-to-machine\n  language mostly. \"Human\" APIs are produced by mappers (see below).\n* RON employs compression for its metadata. The RON UUID syntax is specifically\n  fine-tuned for easy compression.\n\nConsider the above frame uncompressed:\n\n    *lww #1TUAQ+gritzko @1TUAQ+gritzko :bar = 1;\n    *lww #1TUAR+gritzko @1TUAR+gritzko :foo \u003e 1TUAQ+gritzko;\n\n\nOne may say, what metadata solves is [naming things and cache\ninvalidation][2problems].  What RON solves is compressing that metadata.\n\nRON makes no strong assumptions about consistency guarantees: linearized,\ncausal-order or gossip environments are all fine (certain restrictions apply,\nsee below).  Once all the object's ops are propagated to all the object's\nreplicas, replicas converge to the same state.  RON formal model makes this\nprocess correct.  RON wire format makes this process efficient.\n\n\n## Formal model\n\nSwarm RON formal model has five key components:\n\n1. An UUID is a globally unique 128-bit identifier. An UUID consists of two\n60-bit parts: *value* and *origin*. 4+4 bits are reserved for flags. There are\nfour UUID types:\n    * an event timestamp: logical/hybrid timestamp, e.g. `1TUAQ+gritzko`, value\n      is a monotonous counter `1TUAQ`, origin is a a replica id `gritzko`,\n      roughly corresponds to RFC4122 v1 UUIDs,\n    * a derived timestamp: same as event timestamp, but refers to some derived\n      calculation, not the original event (e.g. `1TUAQ-gritzko`),\n    * a name, either global or scoped to a replica, e.g. `foo`, `lww`, `bar`\n      (global), `MyVariable$gritzko` (scoped),\n    * a hash (e.g. `4Js8lam4LB%kj529sMEsl`, both parts are hash sum bits).\n\n2.  An op is an immutable atomic unit of data change. An op is a tuple of four\n    or more *atoms*. First four atoms of an op are UUIDs forming the op's key.\n\n    These UUIDs are:\n\n    1. data type UUID, e.g. `lww` a last-write-wins object,\n    2. object UUID `1TUAQ+gritzko`,\n    3. event UUID `1TUAQ+gritzko` and\n    4. location/reference UUID, e.g. `bar`.\n\n    Other atoms (any number, any type) form the op's value. Op atoms types are:\n\n    1. UUID,\n    2. integer,\n    3. string, or\n    4. float.\n\n    Importantly, an op goes under one of four *terms*:\n\n    1. raw ops (a single op, before being processed by a reducer),\n    2. reduced ops (an op in a frame, processed by a reducer),\n    3. frame headers (first op of a frame, planted by a reducer),\n    4. queries (part of connection/subscription state machines).\n\n3. A frame is an ordered collection of ops, a transactional unit of data\n\n    * an object's state is a frame\n    * a \"patch\" (aka \"delta\", \"diff\") is also a frame\n    * in general, data is seen as a [partially ordered][po] log of frames\n      or chunks\n    * frame may contain any number of reduced chunks and raw ops in any order;\n      a chunk consists of a header or a query header op followed by reduced ops\n      belonging to the chunk; raw ops form their own one-op chunk.\n\n4.  A reducer is a RON term for a \"data type\"; reducers define how object state\n    is changed by new ops\n\n    *   a reducer is a pure function: `f(state_frame, change_frame) -\u003e\n        new_state_frame`, where frames are either empty frames or single ops or\n        products of past reductions by the same reducer,\n\n    *   reducers are:\n\n        1.  associative, e.g. `f( f(state, op1), op2 ) == f( state, patch )`\n            where `patch == f(op1,op2)`\n        2.  commutative for concurrent ops (can tolerate causally consistent\n            partial orders), e.g. `f(f(state,a),b) == f(f(state,b),a)`, assuming `a`\n            and `b` originated concurrently at different replicas,\n        3.  idempotent, e.g. `f(state, op1) == f(f(state, op1), op1) == f(state,\n            f(op1, op1))`, etc.\n\n    *   optionally, reducers may have stronger guarantees, e.g. full commutativity\n        (tolerates causality violations),\n\n    *   a frame could be an op, a patch or a complete state. Hence, a baseline\n        reducer can \"switch gears\" from pure op-based CRDT mode to state-based\n        CRDT to delta-based, e.g.\n\n        1. `f(state, op)` is op-based\n        2. `f(state1, state2)` is state-based\n        3. `f(state, patch)` is delta-based\n\n4. a mapper translates a replicated object's state frame into other formats\n\n    * mappers turn RON objects into JSON or XML documents, C++, JavaScript or\n      other objects\n    * mappers are one-way: RON metadata may be lost in conversion\n    * mappers can be pipelined, e.g. one can build a full RON-\u003eJSON-\u003eHTML\n      [MVC][mvc] app using just mappers.\n\n\nSingle ops assume [causally consistent][causal] delivery.  RON implies causal\nconsistency by default.  Although, nothing prevents it from running in a\nlinearized [ACIDic][peterb] or gossip environment.  That only relaxes (or\nrestricts) the choice of reducers.\n\n## Wire format (text)\n\nDesign goals for the RON wire format is to be reasonably readable and reasonably\ncompact.  No less human-readable than regular expressions.  No less compact than\n(say) three times plain JSON (and at least three times more compact than JSON\nwith comparable amounts of metadata).\n\nThe syntax outline:\n\n1. atoms follow very predictable conventions:\n    * integers: `1`\n    * e-notation floats: `3.1415`, `1.0e+6`\n    * UTF-8 JSON-escaped strings: `строка\\n线\\t\\u7ebf\\n라인`,\n      except that `'` (U+0027 APOSTROPHE) must be encoded as `\\u0027` or `\\'`\n    * RON UUIDs `1D4ICC-XU5eRJ`, `1TUAQ+gritzko`\n2. UUIDs use a compact custom serialization\n    * RON UUIDs are Base64 to save space (compare [RFC4122][rfc4122]\n      `123e4567-e89b-12d3-a456-426655440000` and RON `1D4ICC-XU5eRJ`)\n    * also, RON timestamp UUIDs may vary in precision, like floats (no need to\n      mention nanoseconds everywhere) -- trailing zeroes are skipped\n    * UUIDs are lexically/numerically comparable (same order), the Base64\n      variant is\n      `0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ_abcdefghijklmnopqrstuvwxyz~`\n3. serialized ops use some punctuation, e.g. `*lww #1D4ICC-XU5eRJ\n   @1D4ICC2-XU5eRJ :keyA 'valueA'`\n    * `*` starts a data type UUID\n    * `#` starts an object UUID\n    * `@` starts an op's own event UUID\n    * `:` starts a location UUID\n    * `=` starts an integer\n    * `'` starts and ends a string;\n          may occur inside a string if prefixed by backslash — `\\'`\n    * `^` starts a float (e-notation)\n    * `\u003e` starts an UUID\n    * `!` ends a frame header op (a reduced chunk has one header op)\n    * `?` ends a query header op (a subscription frame has a header)\n    * `,` ends a reduced op (optional)\n    * `;` ends a raw op\n    * `.` ends a frame (required for streaming transports, e.g. TCP)\n4. frame format employs cross-columnar compression\n    * repeated key UUIDs can be skipped altogether (\"same as in the last op\");\n      in the first op all key UUIDs are mandatory;\n    * RON abbreviates similar UUIDs using prefix compression, e.g.\n      `1D4ICCE+XU5eRJ` gets compressed to `{E` if preceded by `1D4ICC+XU5eRJ`\n      (symbols `([{}])` corespond to 4,5,..9 symbols of shared prefix)\n    * by default, a key UUID is compressed against the same UUID in the previous\n      op (e.g. event id against the previous event id);\n    * backtick \\` changes the default UUID to the previous UUID of the same op\n      (e.g. event id against same op's object id)\n    * the first value UUID is compressed against the object UUID of the op,\n      each other is compressed against the previous one.\n\nConsider a simple JSON object:\n\n    {\"keyA\":\"valueA\", \"keyB\":\"valueB\"}\n\nA RON frame for that object will have three ops: one frame header op and two\nkey-value ops.  In the tabular form, that frame may look like:\n\n    type object         event           location value\n    -----------------------------------------------------\n    *lww #1D4ICC+XU5eRJ @1D4ICCE+XU5eRJ :0       !\n    *lww #1D4ICC+XU5eRJ @1D4ICCE+XU5eRJ :keyA    'valueA'\n    *lww #1D4ICC+XU5eRJ @1D4ICC1+XU5eRJ :keyB    'valueB'\n\nThere are lots of repeating bits here.\nWe may skip repeating UUIDs and prefix-compress close UUIDs.\nThe compressed frame will be just a bit longer than bare JSON:\n\n    *lww#1D4ICC+XU5eRJ@`{E! :keyA'valueA' @{1:keyB'valueB'\n\nThe frame contains *twelve* UUIDs (6 distinct UUIDs, 3 distinct\ntimestamps) and also the data.\nDespite the impressive amount of metadata, it takes less space than *two* [RFC4122\nUUIDs][rfc4122].  The point becomes even clearer if we add the\nobject UUID to JSON using the RFC4122 notation:\n\n    {\"_id\": \"0651a600-2b49-11e6-8000-1696d3000000\", \"keyA\":\"valueA\",\n    \"keyB\":\"valueB\"}\n\n\nWe may take this to the extreme if we consider the case of a CRDT-based\ncollaborative real-time editor.  Then, every letter in the text has its own\nUUID.  With RFC4122 UUIDs and JSON, that is simply ridiculous.  With RON, that\nis perfectly OK.\n\nConsider \"Hello world!\" collaboratively written by two users, `bart` and `lisa`\non 27 Nov 2017 around 9am GMT.  A compressed RGA (Replicated Growable Array)\nframe would look like:\n\n    *rga#1UQ8p+bart@1UQ8yk+lisa:0!\n        @(s+bart'H'@[r'e'@(t'l'@[T'l'@[i'o'\n        @(w+lisa' '@(x'w'@(y'o'@[1'r'@{a'l'@[2'd'@[k'!'\n\nThe `txt` mapper may convert the RGA frame into text:\n\n    *txt #1UQ8p+bart @1UQ8yk+lisa 'Hello world!'\n\nIf nicely indented, the compressed frame is easier to read:\n\n    *rga #1UQ8p+bart @1UQ8yk+lisa :0  !\n                     @(s+bart        'H'\n                     @[r             'e'\n                     @(t             'l'\n                     @[T             'l'\n                     @[i             'o'\n                     @(w+lisa        ' '\n                     @(x             'w'\n                     @(y             'o'\n                     @[1             'r'\n                     @{a             'l'\n                     @[2             'd'\n                     @[k             '!'\n\nIf fully uncompressed, the frame takes more space:\n\n    *rga   #1UQ8p+bart   @1UQ8yk+lisa     :0      !\n    *rga   #1UQ8p+bart   @1UQ8s+bart      :0     'H'\n    *rga   #1UQ8p+bart   @1UQ8sr+bart     :0     'e'\n    *rga   #1UQ8p+bart   @1UQ8t+bart      :0     'l'\n    *rga   #1UQ8p+bart   @1UQ8tT+bart     :0     'l'\n    *rga   #1UQ8p+bart   @1UQ8ti+bart     :0     'o'\n    *rga   #1UQ8p+bart   @1UQ8w+lisa      :0     ' '\n    *rga   #1UQ8p+bart   @1UQ8x+lisa      :0     'w'\n    *rga   #1UQ8p+bart   @1UQ8y+lisa      :0     'o'\n    *rga   #1UQ8p+bart   @1UQ8y1+lisa     :0     'r'\n    *rga   #1UQ8p+bart   @1UQ8y1a+lisa    :0     'l'\n    *rga   #1UQ8p+bart   @1UQ8y2+lisa     :0     'd'\n    *rga   #1UQ8p+bart   @1UQ8yk+lisa     :0     '!'\n\n\nIf rendered in JSON, the same document would probably start as\n\n    {\n        \"_id\": \"3b127800-d350-11e7-8000-9a5db8000000\",\n        \"_version\": \"98f38f80-d351-11e7-8000-c2dde5000000\",\n        ...\n\n...which is already 90% of the size of the entire compressed frame above.\nWith idiomatic JSON, per-symbol metadata is both difficult and expensive.\n\nSo, let's be precise. Let's put UUIDs on everything. RON makes it possible.\n\n## Wire format (binary)\n\nThe binary format is more efficient because of higher bit density; it is also\nsimpler and safer to parse because of explicit field lengths.  Obviously, it is\nnot human-readable.\n\nLike the text format, the binary one is only optimized for iteration.  Because of\ncompression, records are inevitably of variable length, so random access is not\npossible.  Also, compression depends on iteration, as UUIDs get abbreviated\nrelative to similar preceding UUIDs.\n\nA binary RON frame starts with magic bytes `RON2` and frame length.  The rest\nof the frame is a sequence of *fields*.  Each field starts with a *descriptor*\nspecifying the type of the field and its length.\n\nFrame length is serialized as a 32-bit big-endian integer.  The maximum length\nof a frame is 2^31-1 bytes.  If the length value has its most significant bit\nset to 1, then the frame is *chunked*.  A chunked frame is followed by a\ncontinuation frame.  A continuation frame has no magic bytes, just a 4-byte\nlength field.  The last continuation frame must have the m.s.b. of its length\nset to 0.\n\n### Descriptors\n\nA descriptor's first byte spends four most significant (m.s.) bits to describe\nthe type of the field, other four bits describe its length.\n\n```\n   7    6    5    4    3    2    1    0\n+----+----+----+----+----+----+----+----+\n| major   | minor   |     field         |\n|    type |    type |        length     |\n+----+----+----+----+----+----+----+----+\n  128  64   32   16    8    4    2    1\n   80  40   20   10    8    4    2    1\n```\n\nField descriptor major/minor type bits are set as follows:\n\n0. `00` RON op term,\n    * `0000` raw op,\n    * `0001` reduced op,\n    * `0010` header op,\n    * `0011` query header op.\n1. `01` UUID, uncompressed\n    * `0100` type (reducer) id,\n    * `0101` object id,\n    * `0110` event id,\n    * `0111` ref/location id\n2. `10` UUID, compressed (zipped)\n    * `1000` value UUID, zipped (note: not type id)\n    * `1001` object id,\n    * `1010` event id,\n    * `1011` ref/location id\n3. `11` Atom\n    * `1100` value UUID, uncompressed (lengths 1..16)\n    * `1101` integer (big-endian, [zigzag-coded][zigzag], lengths 1, 2, 4, 8)\n    * `1110` string (UTF-8, length 0..2^31-1)\n    * `1111` float (IEEE 754-2008, binary 16, 32 or 64, lengths 2, 4, 8 resp)\n\nA descriptor's four least significant bits encode the length of the field in\nquestion.  The length value given by a descriptor does not include the length\nof the descriptor itself.\n\nIf a field or a frame is 1 to 16 bytes long then it has its length coded\ndirectly in the four l.s. bits of the descriptor. Zero stands for the length of\n16 because most field types are limited to that length.  Op terms specify no\nlength.  With string atoms, zero denotes the presence of an extended length\nfield which is either 1 or 4 bytes long. The maximum allowed string length is\n2Gb (31 bits).  In case the descriptor byte is exactly `1110 0000`, the m.s.\nbit of the next byte denotes the length of the extended length field (`0` for\none, `1` for four bytes).  The rest of the next byte (and possibly other three)\nis a big-endian integer denoting the byte length of the string.\n\nConsider a time value query frame: `*now?.`\n\n* 4 bytes are magic bytes (RON, `0101 0010  0100 1111  0100 1110  0011 0010`)\n* frame length: 4 bytes (length 5, `0000 0000  0000 0000  0000 0000  0000 0101`)\n* op term descriptor: 1 byte (`0011 0000`)\n* uncompressed UUID descriptor: 1 byte (cited length 3, `0100 0011`)\n* `now` RON UUID: 3 bytes (`0000 1100  1011 0011  1110 1100`,\n  the \"uncompressed\" coding still trims a lot of zeroes, see below).\n\nAs UUID length is up to 16 bytes, UUID fields never use a separate length\nnumber. UUID descriptors are always 1 byte long. The length of 0 stands for 16.\n\nLength bits `0000` stand for:\n\n* zero length for op terms,\n* 16 for integer/float atoms, zipped/unzipped UUIDs,\n* for strings, that signals an extended length record (1 or 4 bytes).\n\nAn extended length record is used for strings cause those can be up to 2GB\nlong. An extended length record is either 1 or four bytes. Four-byte record is\na big-endian 32-bit int having its m.s. bit set to 1. Thus, strings of 127\nbytes and shorter may use 1 byte long length record.\n\n### Op terms\n\nOp term fields may have cited length of `0000` or be skipped if they match the\nprevious op's term.  Still, sometimes we want to introduce redundancy,\nCRC/checksumming, hashing, etc.  Exactly for this purpose we may use non-empty\nterms.  The checksumming method is specified by the field length (TODO).\n\n### Uncompressed UUIDs\n\nUncompressed UUIDs are not compressed relative to preceding UUIDs (not *zipped*).\nStill, zero bytes are skipped to optimize for some often-used cases.\nThe skip pattern is determined based on the cited field length.\n\nNamely, UUIDs 1..8 bytes long have the *origin* part set to zeros (all 8 bytes)\nand the least significant bytes of the value also set to zeroes.\nThese are often-used \"transcendent\" name UUIDs (`lww`, `rga`, `db`, `now`, etc).\nFor example, `lww` is the data type UUID for last-write-wins objects.\nIn the unabbreviated RON Base64 form, `lww` is `0/lww0000000 00000000000`\n(see the [UUID spec](uuid.md) for the details).\n\nUUIDs 9 to 15 bytes long have their l.s. value bytes set to zero.\nThis case is optimized for arbitrary-precision timestamps.\n\nUUIDs 16 bytes long are full 128-bit RON UUIDs.\n\n### Compressed UUIDs\n\nZipped UUIDs are serialized as deltas to similar past UUIDs.  That provides\nsignificant savings when UUIDs come from the same source (same origin bytes) or\nhave close timestamp values.  Repeated UUIDs are simply skipped, same as in the\nBase64 notation.\n\nThe origin part is either reused in full or rewritten in full. That is decided\nby the field length (\u003c9 reuse, \u003e=9 rewrite). Implicitly, origin ids are\nconsidered uncompressible.\n\nThere are two zip modes: *short* and *long*.  In the short mode, an UUID is\ncompressed relative to the same kind of UUID in the previous op (e.g. event id\nrelative to the previous event id).  In the long mode, an UUID is compressed\nrelative to a past uncompressed UUID. A decoder must remember 16 last\nuncompressed timestamp-based UUIDs (no names, no hashes), to perform\nuncompression. For encoders, that is optional.\n\nA zipped UUID starts with a *zip byte* referencing the compression details.\n\nShort zip byte:\n\n```\n   7    6    5    4    3    2    1    0\n+----+----+----+----+----+----+----+----+\n|  0 | zero tail len|                   |\n|    | (half-bytes) |  m.s. half-byte   |\n+----+----+----+----+----+----+----+----+\n  128  64   32   16    8    4    2    1\n```\n\nIn this mode, the zip byte specifies how many l.s. half-bytes of the value are\nzeroes. Based on the field length, we decide how many \"middle\" half-bytes need\nto be changed, relative to the past UUID. M.s. half-bytes stay the same as in\nthe past UUID.\n\nLong zip byte:\n\n```\n   7    6    5    4    3    2    1    0\n+----+----+----+----+----+----+----+----+\n|  1 |zero tail len | past uncompressed |\n|    |  (half-bytes)|   UUID index      |\n+----+----+----+----+----+----+----+----+\n  128  64   32   16    8    4    2    1\n```\n\nIn this mode, the zip byte specifies the past uncompressed UUID we use as a\nreference. Index 0 points at the recentmost uncompressed UUID, 1 to the\nprevious one, etc.\nSimilarly to the short mode, we set a number of l.s. half-bytes to\nzeroes, replace middle half-bytes with new values and keep the m.s. half-bytes\nthe same.\n\n### Atoms\n\nStrings are serialized as UTF-8.\n\nIntegers are serialized using the zig-zag coding (the l.s. bit conveys the sign).\n\nFloats are serialized as IEEE 754 floats (4-byte and 8-byte support is required,\nother lengths are optional).\n\n## The math\n\nRON is [log-structured][log]: it stores data as a stream of changes first,\neverything else second.  Algorithmically, RON is LSMT-friendly (think [BigTable\nand friends][lsmt]).  RON is [information-centric][icn]: the data is addressed\nindependently of its place of storage (think [git][git]).  RON is CRDT-friendly;\n[Conflict-free Replicated Data Types][crdt] enable real-time data sync (think\nGoogle Docs).\n\nSwarm RON employs a variety of well-studied computer science models.  The\ngeneral flow of RON data synchronization follows the state machine replication\nmodel.  Offline writability, real-time sync and conflict resolution are all\npossible thanks to [Commutative Replicated Data Types][crdt] and [partially\nordered][po] op logs.  UUIDs are essentially [Lamport logical\ntimestamps][lamport], although they borrow a lot from RFC4122 UUIDs.  RON wire\nformat is a [regular language][regular].  That makes it (formally) simpler than\neither JSON or XML.\n\nThe core contribution of the RON format is *practicality*.  RON arranges\nprimitives in a way to make metadata overhead acceptable.  Metadata was a known\nhurdle in CRDT-based solutions, as compared to e.g. [OT-family][ot] algorithms.\nSmall overhead enables such real-time apps as collaborative text editors where\none op is one keystroke.  Hopefully, it will enable some yet-unknown\napplications as well.\n\nUse Swarm RON!\n\n## Acknowledgements\n\n* Russell Sullivan\n* Yuriy Syrovetskiy\n\n## History\n\n* 2012-2013: project started (initially, as a part of the Yandex Live Letters\n  project)\n* 2014 Feb: becomes a separate project\n* 2014 Oct: version 0.3 is demoed (per-object logs and version vectors, not\n  really scalable)\n* 2015 Sep: version 0.4 is scrapped, the math is changed to avoid any version\n  vector use\n* 2016 Feb: version 1.0 stabilizes (no v.vectors, new asymmetric client\n  protocol)\n* 2016 May: version 1.1 gets peer-to-peer (server-to-server) sync\n* 2016 Jun: version 1.2 gets crypto (Merkle, entanglement)\n* 2016 Oct: functional generalizations (map/reduce)\n* 2016 Dec: cross-columnar compression\n* 2017 Jun: Swarm RON 2.0.0\n* 2017 Jul: new frame-based Causal Tree / Replicated Growable Array\n  implementation\n* 2017 Jul: Ragel parser\n* 2017 Aug: punctuation tweaks\n* 2017 Oct: streaming parser\n* 2017 Oct: binary encoding\n\n[2sided]: http://lexicon.ft.com/Term?term=two_sided-markets\n[super]: http://ilpubs.stanford.edu:8090/594/1/2003-33.pdf\n[opbased]: http://haslab.uminho.pt/sites/default/files/ashoker/files/opbaseddais14.pdf\n[cap]: https://www.infoq.com/articles/cap-twelve-years-later-how-the-rules-have-changed\n[swarm]: https://gritzko.gitbooks.io/swarm-the-protocol/content/\n[po]: https://en.wikipedia.org/wiki/Partially_ordered_set#Formal_definition\n[crdt]: https://en.wikipedia.org/wiki/Conflict-free_replicated_data_type\n[icn]: http://www.networkworld.com/article/3060243/internet/demystifying-the-information-centric-network.html\n[kafka]: http://kafka.apache.org\n[git]: https://git-scm.com\n[log]: http://blog.notdot.net/2009/12/Damn-Cool-Algorithms-Log-structured-storage\n[re]: https://blogs.msdn.microsoft.com/csliu/2009/11/10/mapreduce-in-functional-programming-parallel-processing-perspectives/\n[rfc4122]: https://tools.ietf.org/html/rfc4122\n[causal]: https://en.wikipedia.org/wiki/Causal_consistency\n[UUID]: https://en.wikipedia.org/wiki/Universally_unique_identifier\n[peterb]: https://martin.kleppmann.com/2014/11/isolation-levels.png\n[regular]: https://en.wikipedia.org/wiki/Regular_language\n[mvc]: https://en.wikipedia.org/wiki/Model–view–controller\n[ot]: https://en.wikipedia.org/wiki/Operational_transformation\n[lamport]: http://lamport.azurewebsites.net/pubs/time-clocks.pdf\n[2problems]: https://martinfowler.com/bliki/TwoHardThings.html\n[lsmt]: https://en.wikipedia.org/wiki/Log-structured_merge-tree\n[zigzag]: https://developers.google.com/protocol-buffers/docs/encoding#signed-integers\n[badge]: https://travis-ci.com/gritzko/ron.svg?branch=master\n[travis]: https://travis-ci.com/gritzko/ron","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Folebedev%2Fron","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Folebedev%2Fron","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Folebedev%2Fron/lists"}