{"id":13630167,"url":"https://github.com/ssbc/epidemic-broadcast-trees","last_synced_at":"2025-07-15T17:46:30.834Z","repository":{"id":40290556,"uuid":"87158621","full_name":"ssbc/epidemic-broadcast-trees","owner":"ssbc","description":"bandwidth efficient broadcast gossip","archived":false,"fork":false,"pushed_at":"2023-02-24T10:43:36.000Z","size":322,"stargazers_count":125,"open_issues_count":11,"forks_count":20,"subscribers_count":12,"default_branch":"master","last_synced_at":"2025-06-09T15:55:17.060Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ssbc.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2017-04-04T07:20:42.000Z","updated_at":"2025-02-02T15:10:48.000Z","dependencies_parsed_at":"2022-08-09T16:39:41.568Z","dependency_job_id":"6178d6c8-d004-47ae-95db-6d39bc2e2dd4","html_url":"https://github.com/ssbc/epidemic-broadcast-trees","commit_stats":null,"previous_names":[],"tags_count":16,"template":false,"template_full_name":null,"purl":"pkg:github/ssbc/epidemic-broadcast-trees","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ssbc%2Fepidemic-broadcast-trees","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ssbc%2Fepidemic-broadcast-trees/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ssbc%2Fepidemic-broadcast-trees/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ssbc%2Fepidemic-broadcast-trees/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ssbc","download_url":"https://codeload.github.com/ssbc/epidemic-broadcast-trees/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ssbc%2Fepidemic-broadcast-trees/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":260090117,"owners_count":22957185,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-01T22:01:32.474Z","updated_at":"2025-07-15T17:46:30.780Z","avatar_url":"https://github.com/ssbc.png","language":"JavaScript","funding_links":[],"categories":["JavaScript"],"sub_categories":[],"readme":"# Epidemic Broadcast Trees\n\nThis module is loosely based on plumtree Epidemic Broadcast Trees\n[EBT paper], but adapted to also replicate logs, and optimized\nto achive a minimal overhead (the cost of the protocol is linear with\nthe number of messages to be sent)\n\nIt's a algorithm that combines the robustness of a flooding epidemic\ngossip broadcast, with the efficiency of a tree model. It's intended\nfor implementing realtime protocols (such as chat, scuttlebutt, also\nradio/video) over networks with random topology - or networks where\notherwise peers may be unable to all connect to each other or to a\ncentral hub.\n\nAlthough the primary motivation for this module is to use it in secure\nscuttlebutt, it's intended to be decoupled sufficiently to use for\nother applications.\n\n## Example\n\nimplement a simple in memory log replicator.\n\n``` js\nvar clocks = {}\nvar logs = {}\n\nfunction append (msg, cb) {\n  var log = logs[msg.author] || {}\n  //check that this is the next expected message.\n  if(msg.sequence != Object.keys(log).length + 1)\n    cb(new Error('out of order, found:'+msg.sequence+', expected:'+log.length))\n  else {\n    log[msg.sequence] = msg\n    ebt.onAppend(msg)\n    cb()\n  }\n}\n\nvar ebt = EBT({\n  //NOTE: in this example, we are using readable strings for clarity\n  //but ideally you'd use cryptographic ids, like public keys.\n  id: 'alice',\n  getClock: function (id, cb) {\n    //load the peer clock for id.\n    cb(null, clocks[id] || {})\n  },\n  setClock: function (id, clock) {\n    //set clock doesn't have take a cb, but it's okay to be async.\n    clocks[id] = clock\n  },\n  getAt: function (pair, cb) {\n    //load a message particular message, by id:sequence\n    if(!logs[pair.id] || !logs[pair.id][pair.sequence])\n      cb(new Error('not found'))\n    else\n      cb(null, logs[pair.id][pair.sequence])\n  },\n  append: append\n})\n\nebt.append({\n  author: 'alice', sequence: 1, content: {}\n}, function () {})\n\n//must explicitly say we are replicating which peers.\nebt.request('alice', true)\nebt.request('bob', true)\n\n//create a stream and pipe it to another instance\n//isClient and version are required.\nvar stream = ebt.createStream('bob', version=3, isClient = true)\nstream.pipe(remote_stream).pipe(stream)\n```\n\n\u003e note about push-stream: push-stream is only new, so you'll probably\n  need to convert this to a pull-stream to connect stream to a network\n  io stream and serialization\n\n``` js\nvar pushToPull = require('push-stream-to-pull-stream')\nvar stream = pushToPull(ebt.createStream(remote_id, 3, isCient = true))\npull(stream, remote_pull_stream, stream)\n```\n\n## API\n\n### EBT(opts) =\u003e ebt\n\nwhere opts provides the necessary things to connect ebt\nto your system.\n\n```\nopts = {\n  id: string,\n  timeout: 3000, //default,\n  getClock: function (id, cb),\n  setClock: function (id, clock),\n  getAt: function ({id:string, sequence:number}, cb),\n  append: function (msg, cb),\n  isFeed: function (id),\n  isMsg: function(data),\n  getMsgAuthor: function(msg),\n  getMsgSequence: function(msg)\n}\n```\n\nCreate a new EBT instance. `id` is a unique identifier of the current\npeer.  In [secure-scuttlebutt](https://scuttlebutt.nz) this is a\ned25519 public key.\n\n`getClock(id, cb)` and `setClock(id, clock)` save a peer's clock\nobject.  This is used to save bandwidth when reconnecting to a peer\nagain.\n\n`getAt({id, sequence}, cb)` retrives a message in a feed and an\nsequence.  messages must have `{author, sequence, content}` fields.\n\n`append(msg, cb)` append a particular message to the log.\n\n`timeout` is used to decide when to switch a feed to another peer.\nThis is essential to detecting when a peer may have stalled.\n\n`isFeed(id)` is a validation function that returns true if `id` is a\nvalid feed identifier. If not, it is ignored'\n\n### optional for backwards compatibility\n\n`isMsg(data)` is a validation function used to distinguish between data\nmessages and status messages. A message must contain an `author` field\nthat corresponds to the feed identifier and a `sequence` field.\n\n`getMsgAuthor(msg)` is a function that given a message returns the\nauthor.\n\n`getMsgSequence(msg)` is a function that given a message returns the\nsequence.\n\n### ebt.onAppend (msg)\n\nWhen a message is appended to the database, tell ebt about it.  this\nmust be called whenever a message is successfully appended to the\ndatabase.\n\n### ebt.createStream(id, version, isClient) =\u003e PushStream\n\nCreate a stream for replication. returns a [push-stream]. The current\nversion is 3, and `isClient` must be either true or false.  On the\nclient side stream, it will wait for the server to send their vector\nclock, before replying. This means that if the server doesn't actually\nsupport this api, you give them a change to send back an error before\nsending a potentially large vector clock.\n\n### ebt.request(id, follow)\n\nTell ebt to replicate a particular feed. `id` is a feed id, and\n`follow` is a `boolean`.  If `follow` is `false`, but previously was\ncalled with true, ebt will stop replicating that feed.\n\n#### `ebt.progress()`\n\nreturns an object which represents the current replication progress.\n\nan example object output looks like this, all values are integers \u003e= 0.\n\n``` js\n{\n  start: S, //where we where at when we started\n  current: C, //operations done\n  total: T //operations expected\n}\n```\n\nthis follows a common pattern used across ssbc modules for\nrepresenting progress, used for example here:\n`https://github.com/ssbc/scuttlebot/blob/master/lib/progress.js`\n\n#### ebt.state\n\nThe state of the replication is available at `ebt.state`.  Read only\naccess is okay, but updating should only be done via ebt methods.\n\n```\n{\n  id: \u003cid\u003e, //our id,\n  clock: {\u003cid\u003e: \u003cseq\u003e}, //our local clock,\n  follows: {\u003cid\u003e: \u003cboolean\u003e}, //who we replicate, true if we replicate.\n  blocks: {\u003cid\u003e: {\u003cid\u003e: \u003cboolean\u003e}}, //who blocks who, true if they are blocked.\n  peers: { //currently connected peers\n    \u003cid\u003e: {\n      clock: {\u003cid\u003e: \u003cseq|-1\u003e}, //feeds that we KNOW the peer is up to. -1 if they do not replicate that feed.\n      msgs: [\u003cmsg\u003e], //queue of messages waiting to be sent.\n      retrive: [\u003cid\u003e], //ids of feeds ready for the next message to be retrived.\n      notes: null || {\u003cid\u003e: \u003cencoded_seq\u003e}, //notes object (encoded vector clock to be sent)\n      replicating: { //feeds being replicated to peer.\n        \u003cid\u003e: {\n          rx: \u003cboolean\u003e, //true if we have asked to recieve this feed\n          tx: \u003cboolean\u003e, //true if we have been asked to send this feed\n          sent: \u003cseq|-1|null\u003e, //sequence number of message we have sent.\n          requested: \u003cseq|-1|null\u003e //sequence number the remote peer asked for, and thus we know they have.\n        }\n      }\n    }\n  },\n  receive: [\u003cmsg\u003e] //queue of incoming messages\n}\n```\n\nnotes: `\u003cX\u003e` is a value type.\n\n`\u003cid\u003e` is a \"feed id\" value that `opts.isFeed(id) === true`.  (note,\nthis doesn't actually need to be an ssb feed id, this module can be\nused for other things too)\n\n`\u003cseq\u003e` is an positive integer or zero. -1 is used to represent if the\nare explicitly not replicating that feed.\n\n`\u003cmsg\u003e` is a message where `opts.isMsg(id) === true`.\n\n## Replication overview\n\nThe state of other peers are stored outside this module in the SSB-EBT\nmodule. See `getClock` \u0026 `setClock`.\n\nNotes (aka the vector clock) are stored as { feed: (seq === -1 ? -1 :\nseq \u003c\u003c 1 | !rx) } (= * 2 + 1?). The sequence can be extracted using\n`getSequence` and rx/tx using `getReceive` (is even). -1 means do not\nreplicate.\n\ntwo peers connect, one is the client (who initiated the connection),\nand the other is the server (that received the request).\nThe protocol is mostly the same for clients and servers, but one exception is that\nthe server starts by sending their vector clock (notes first).\n\nIt's assumed that data changes following a long tail pattern,\na small number of peers are highly active, but many peers only add data slowly.\nConnections between peers may be short lived, and data may change more slowly than subsequent connections.\nIf we sent the whole vector clock on each connection that would add up to a significant overhead.\n_Request Skipping_ is a way to avoid a great deal of bandwidth overhead.\nEach peer remembers the vector clock _sent_ by the other peer.\nAnd, on a new connection, when they send their vector clock they first check it against the vector clock that the remote peer sent last time.\nIf the sequence in the peer's clock is the same as the one in the saved record of the remote peer,\nthen that feed is left out of the request (hence the name \"request skipping\")\nThe saved record of the remote peer's vector clock may be different to their actual vector clock,\nbut if they have a new sequence for that feed, they will include that feed in their request,\nand the local peer will respond by sending an additional partial vector clock including their sequence for that feed,\nonce both sides have exchanged their sequence for a particular feed, replication of messages in that feed may occur.\n\nWhen connecting to multiple peers, only request new messages using rx\nfor a feed from one of the nodes. See `test/multiple.js`.\n\nFollowing and blocking are handled in EBT. Following acts as the\nsignal of what feeds to replicate. EBT won't connect to someone that\nhas been blocked. It will not send messages of a peer (including self)\nto another peer if the first peer blocks the second.\n\nThe tests are very readable because they use a simulator where a trace\nof the run is saved and pretty printed. See `test/two.js` for a good\nexample.\n\n## Comparison to plumtree\n\nI had an idea for a gossip protocol that avoided retransmitting\nmessages by putting unneeded connections into standby mode (which can\nbe brought back into service when necessary) and then was pleasantly\nsurprised to discover it was not a new idea, but had already been\ndescribed in a paper - and there is an [EBT implementation in erlang]\nof that paper.\n\nThere are some small differences, mainly because I want to send\nmessages in order, which makes it easy to represent what messages have\nnot been seen using just a incrementing sequence number per feed.\n\nBut plumbtree is solely a broadcast protocol, not an eventually\nconsistent replication protocol.  Since we are replicating _logs_ it's\nalso necessary to send a handshake to request the feeds from the right\npoints. If you are replicating thousands of feeds the size of the\nhandshake is significant, so we introduce an algorithm for \"request\nskipping\" that avoids sending unnecessary requests, and saves a lot of\nbandwidth compared to just requesting all feeds each connection.\n\n## Related work\n\n[Brisa] also describes a broadcast protocal that at first glace looks\nvery close to the [EBT paper]. It is modelled using two components:\ntree construction/maintenance and peer sampling. Peer sampling is in\nSSB terminology where [SSB conn] is used. Brisa uses [HyParView],\nwritten by the same authors as the [EBT paper] for peer\nsampling. Compared to EBT, Brisa does not depend on lazy mode between\npeers where only the sequence information is maintained, instead it\ndepends on HyParView to detect failures. This has the advantage that\nit does not need a timer, that is highly latency sensitive. It also\nhas a nice property in how messages are disseminated in that they are\npiggybacked with information about the tree that allows the parent\nselection to make better choices as it has a better view of the\nnetwork. The sequence numbers are an important part of the protocol\nimplemented here because they, as described earlier, are used to\nensure that messages are disseminated in a eventually consistent\nmanor.\n\n\n## TODO\n\n* handle models where it's okay to have gaps in a log (as with classic\n  [insecure scuttlebutt]\n\n## License\n\nMIT\n\n[EBT paper]: http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.190.3504\n[Brisa]: http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.360.1724\n[HyParView]: http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.190.3289\n[push-stream]: https://github.com/push-stream/push-stream\n[SSB conn]: https://github.com/staltz/ssb-conn\n[secure-scuttlebutt]: https://scuttlebutt.nz\n[insecure scuttlebutt]: https://github.com/dominictarr/scuttlebutt\n[EBT implementation in erlang]: https://github.com/helium/plumtree\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fssbc%2Fepidemic-broadcast-trees","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fssbc%2Fepidemic-broadcast-trees","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fssbc%2Fepidemic-broadcast-trees/lists"}