{"id":13724901,"url":"https://github.com/AWinterman/simple-scuttle","last_synced_at":"2025-05-07T19:31:14.442Z","repository":{"id":11896904,"uuid":"14460087","full_name":"AWinterman/simple-scuttle","owner":"AWinterman","description":"Node Scuttlebutt like the paper describes it.","archived":false,"fork":false,"pushed_at":"2022-12-10T10:53:25.000Z","size":1070,"stargazers_count":38,"open_issues_count":8,"forks_count":3,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-04-03T18:41:09.806Z","etag":null,"topics":["gossip-instances","scuttlebutt"],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/AWinterman.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null},"funding":{"github":"AWinterman"}},"created_at":"2013-11-17T02:47:27.000Z","updated_at":"2024-05-21T18:40:12.000Z","dependencies_parsed_at":"2023-01-13T16:42:51.139Z","dependency_job_id":null,"html_url":"https://github.com/AWinterman/simple-scuttle","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AWinterman%2Fsimple-scuttle","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AWinterman%2Fsimple-scuttle/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AWinterman%2Fsimple-scuttle/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AWinterman%2Fsimple-scuttle/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/AWinterman","download_url":"https://codeload.github.com/AWinterman/simple-scuttle/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252943700,"owners_count":21829291,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["gossip-instances","scuttlebutt"],"created_at":"2024-08-03T01:02:06.068Z","updated_at":"2025-05-07T19:31:09.426Z","avatar_url":"https://github.com/AWinterman.png","language":"JavaScript","funding_links":["https://github.com/sponsors/AWinterman"],"categories":["JavaScript"],"sub_categories":[],"readme":"# Simple-Scuttle #\n[![Coverage Status](https://coveralls.io/repos/github/AWinterman/simple-scuttle/badge.svg?branch=master)](https://coveralls.io/github/AWinterman/simple-scuttle?branch=master)\n\n[![Scuttlebutt](https://f.cloud.github.com/assets/37303/1836776/ce8b825c-740e-11e3-8a89-70861d073c03.png)](https://github.com/chrisdickinson/)\n\nReplicate state across a network with the scuttlebutt protocol.\n\nI recommend you at least skim the [paper][] which describes the\nprotocol before continuing too much further.\n\n## Overview ##\n\nSimple-Scuttle exports a class - soit `Gossip` - whose instances are transform\nstreams in objectMode. `Gossip` instances maintain several data structures with\nwhich they manages state (see [`Gossip.state`](#gossipstate)) and the\npropagation of state changes (see [`Gossip.history`](#gossiphistory)).\n\nRather than implementing several parallel streams, `Gossip` instances choose\nlogical branches based on the semantics of the objects written to them--  the\nshape of inputs to the stream determine the resulting action.  These are\ndocumented in the [Expected Objects](#expected-objects) section.\n\n# API #\n\n```js\nvar deserializer = require('desrializer')\n  , scuttle = require('simple-scuttle')\n  , serializer = require('serializer')\n  , fs = require('fs')\n\npersist = fs.createWriteStream('/path/to/logs', {flag: 'a'})\n\nvar peer_io = // a stream which sends and receives messages from the peer.\n  , config  = scuttle.base.config\n\nconfig.resolve = base.resolution.strictly_order_values\n\nvar gossip = new scuttle.Gossip('id', config)\n\ngossip.pipe(serializer).pipe(peer_io).pipe(deserializer).pipe(gossip)\n\ngossip.history.on('update', function(update) {\n  persist.write(serializer.serialize(update))\n})\n\ngossip.history.on('compaction', compact)\n\nfunction compact(memory, history_instance)  {\n  /* Resolve history to be more compact somehow */\n}\n```\n\n## Exports ##\n\n### `require('simple-scuttle').base` ###\n\n- `base.config`: The default config object, described in detail below.\n- `base.resolution`: Some sample conflict resolution functions.\n\nThis is a module,\n[lib/base.js](https://github.com/AWinterman/simple-scuttle/blob/master/lib/base.js),\nwith some sample defaults.\n\n### `require('simple-scuttle').Gossip` ###\n\n```js\nGossip(String id, Object config, Object state) -\u003e gossip\n```\n \n- `id`: The unique identifier for each `Gossip` instance.  \n- `config`: an Object which must have the following properties:\n  - `config.mtu`: Stands for Maximum Transmission Unit. Determines how many\n    messages the network can handle at once-- this is used to set\n    [opts.highWaterMark](http://nodejs.org/api/stream.html#stream_new_stream_readable_options). \n  - `config.max_history`: How many updates to store before we begin to forget\n    old ones. Such concerns are absent from the paper, but they seem important\n    to me. Defaults to 10 if falsey.\n  - `config.resolve` `(gossip, update)` -\u003e `Boolean`: A function which\n    determines whether or not a given update should be applied.\n  - `config.sort`: A function which describes how to order updates. Has the\n    same signature as javascripts\n    [Array.sort](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/sort),\n    and will be called by Array.sort under the hood. This function is used to\n    order updates when another Gossip instance requests updates more recent\n    than a given version number.\n- `state`: if you would like to pass a specific object to the Gossip to use as its key value store, perhaps a [proxy object](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Meta_programming) which you have overrided to be backed by a database.\n\n# Admonition #\n\nThe `config.resolve` function is one of the most consequential decisions you will\nmake when constructing your distributed system. Please make an informed\ndecision. Investigate:\n\n- http://aphyr.com/posts/299-the-trouble-with-timestamps,\n- http://aphyr.com/posts/286-call-me-maybe-final-thoughts,\n- http://pagesperso-systeme.lip6.fr/Marc.Shapiro/papers/RR-6956.pdf.\n\nYou will encounter concurrent updates across your system, and how you manage\nthem will determine the reliability and persistence of your data.\n\nNote that regardless of this function, every valid update (updates to any value\nthat is not on Object.prototype) will be written to history.\n\n## Expected Objects ##\n\nGossip instances expect objects written to them (with `gossip.write`) to either\nbe `digest`s or `updates`s.\n\n#### digests ####\n\n```js\nvar digest\n\nif(more_digest) {\n   digest = {\n        digest: true\n      , source_id: source_id\n      , version: last_seen_version_for_source_id\n   }\n } else {\n  digest = { \n      digest: true \n    , done: true\n  }\n}\n```\n\nThe assumption is that a digest object is sent from one node  to another, and\nspecifies what information the receiver should send back to the sender (all\nupdates the receiver has seen for the specified source node, with version\nnumber greater than `digest.version`). Upon receiving a digest, the\nreceiver queues all such updates into its Readable buffer.\n\nIf `!!digest.done` is true, then the receiver will also send back updates on any\npeers it knows about that have version number greater than the version in the\ndigest. They will be ordered according to `config.sort`, (updated each time\nhistory is updated).\n\n#### updates ####\nThe other kind of object, the update, is an object that appears like the\nfollowing:\n\n```js\n\nvar update = {\n    key: 'age'\n  , value: 100\n  , source_id: '#A'\n  , version: 10\n}\n```\n\nThis says: \"source `update.source_id` thought `update.key` mapped to\n`update.value` at version `update.version`.\" The `config.resolve` is a function\nthat takes an update and the gossip instance and determines whether the gossip\ninstance should include the update.\n\n## Methods ##\n\n`Gossip` instances are [Transform\nStreams](http://nodejs.org/api/stream.html#stream_class_stream_transform_1), so\nthey implement all of the methods and events as described in the node core\ndocumentation. In addition, there are a few methods specific to this purpose:\n\n### `Gossip.set(key, value) -\u003e Boolean` ###\n\nThis method applies a local update, simply setting the given key to the given\nvalue in the local instance, and tacking on the appropriate `version` number and\n`source_id`. It's return value indicates whether the underlying stream has hit\nits high water mark. If the return value is `false`, do not write until\n`Gossip` has emitted a `\"drain\"` event. \n\n### ` Gossip.get(key) -\u003e {version: \u003cversion\u003e, value: \u003cobj\u003e}` ###\n\nA method  which provides convenient lookup for arbitrary keys. If\n`Gossip.state` has no entry for a given key, this method returns \n`{version:  -Infinity, value: null}`. Otherwise it returns `{version: version,\n  value: value}`\n\n### `Gossip.gossip() -\u003e null` ###\n\nCauses `Gossip` to queue a randomly sorted set of `digest` objects into its\nReadable buffer. If another `Gossip` stream reads these, it will respond\nwith a series of `update` objects. See [Expected Object](#expected-objects) for\ninformation on the shape of the objects. \n\n`.gossip` will not write to the underlying stream past the highWaterMark, i.e.\nafter\n[gossip.push](http://nodejs.org/api/stream.html#stream_readable_push_chunk_encoding)\nreturns false. \n\n## Attributes ##\n\n### `Gossip.state` ###\nAs specified in the [paper][], state is a\nkey-value map (modeled as a native javascript object), available in the\n`.state` attribute of a `Gossip` instance. Each key maps to a value and a\nversion number, so `state[key]` -\u003e `{version: version, value: value}`\n\n### `Gossip.version` ###\n\nThe highest version number the `Gossip` instance has seen (both locally and\nfrom other instances)\n\n### `Gossip.history` ###\n\nAn object for keeping track of updates, and replaying updates from a given peer\non demand. `update` objects are transmitted individually via the `Gossip`'s\nstreaming methods. \n\n#### Events ####\n`Gossip.history` is an event emitter, which emits two events:\n\n- `\"update\"`: Any time an update is applied, the `\"update\"` event is emitted,\nwith the [`update`](#update) to be applied.\n\n- `\"compaction\"`: If the number of updates recorded in the history exceeds the `max_history` parameter, the `\"compaction\"` event is emitted prior to removing old updates from the history. This way the client can implement more dramatic compaction, making their own tradeoffs between performance, replayability, and speed.\n\n#### `Gossip.history.write(key, value, source_id, version)` ####\n\nWrite a new update to the history. The update is recorded into\n`Gossip.history.memory`, an array of updates, which is then sorted via `sort`\nargument to the `Gossip` constructor. Next `Gossip.history` emits an `\"update\"`\nevent is emitted with the update as its argument. This event is emitted to allow\nthe client to take action prior to pruning the `memory` array to\n`max_history`'s length.\n\n#### `Gossip.history.news(id, version)` -\u003e [`Array updates`](#updates) ####\n\nReturns an array of `update`s which came from a source with unique identifier\nmatching `id`, and which occurred after `version`.\n\n# TODO: #\n\n- Investigate whether history's memory attribute should be an array that is\n`.sort(fn)`-ed, or a custom implementation, such as [this\none][cross-filter-sort].\n- Find a way to safely delete keys from the state.\n\n[npm.im/scuttlebutt]: https://npmjs.org/package/scuttlebutt\n[paper]: http://www.cs.cornell.edu/home/rvr/papers/flowgossip.pdf\n[vector-clocks-hard]: http://basho.com/why-vector-clocks-are-hard/\n[cross-filter-sort]: https://github.com/square/crossfilter/blob/master/src/quicksort.js\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FAWinterman%2Fsimple-scuttle","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FAWinterman%2Fsimple-scuttle","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FAWinterman%2Fsimple-scuttle/lists"}