{"id":28396306,"url":"https://github.com/ssbc/ssb-subset-replication-spec","last_synced_at":"2026-02-05T15:32:03.604Z","repository":{"id":47162046,"uuid":"302711227","full_name":"ssbc/ssb-subset-replication-spec","owner":"ssbc","description":null,"archived":false,"fork":false,"pushed_at":"2021-10-11T15:30:19.000Z","size":65,"stargazers_count":14,"open_issues_count":3,"forks_count":0,"subscribers_count":7,"default_branch":"main","last_synced_at":"2025-10-14T00:26:14.427Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ssbc.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSES/CC-BY-4.0.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-10-09T17:33:43.000Z","updated_at":"2022-05-13T20:48:58.000Z","dependencies_parsed_at":"2022-08-29T10:10:56.778Z","dependency_job_id":null,"html_url":"https://github.com/ssbc/ssb-subset-replication-spec","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/ssbc/ssb-subset-replication-spec","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ssbc%2Fssb-subset-replication-spec","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ssbc%2Fssb-subset-replication-spec/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ssbc%2Fssb-subset-replication-spec/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ssbc%2Fssb-subset-replication-spec/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ssbc","download_url":"https://codeload.github.com/ssbc/ssb-subset-replication-spec/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ssbc%2Fssb-subset-replication-spec/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29124801,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-05T14:05:12.718Z","status":"ssl_error","status_checked_at":"2026-02-05T14:03:53.078Z","response_time":65,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-05-31T21:37:52.418Z","updated_at":"2026-02-05T15:32:03.587Z","avatar_url":"https://github.com/ssbc.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003c!--\nSPDX-FileCopyrightText: 2021 Anders Rune Jensen\n\nSPDX-License-Identifier: CC-BY-4.0\n--\u003e\n\n# Subset replication for SSB\n\nStatus: Design phase\n\nThis module aims to provide methods for synchronizing a subset of one\nor more SSB feeds. The main motivation is to enable faster initial\nsync by only replicating a certain subset of messages that can then be\nsupplimented with a frontier sync such as [EBT].\n\nFeeds in classical SSB can said to contain multiple subset of\nmessages, this could be messages of a certain type (eg. contact or\nchess moves) or could be private messages pertaining to a [private\ngroup].\n\nIn classical SSB, feeds up to a hops count (normally 2 or 3) are\nreplicated in full. There also exists a method to get a particular\nmessage out of order, the security of this hinges on the fact that you\nget the ooo message by the hash and so is a special case of engtangled\nmessages.\n\nHere we are interested in adding APIs that, given [ssb-meta-feed],\nallows two peers to exchange feeds and subset of feeds to allow for\npartial replication.\n\nWe are mainly interested in two classes of messages: a subset of\nmessages of a classical SSB feed and a tangle of messages given a\ncommon root message.\n\n## getSubset(query, options): source\n\nThis method can be seen as a more general version of\ncreateHistoryStream where the id, or what you are interesting in,\nfirst needs to be defined. Query specifies what data you are\ninterested in.\n\nThe query parameter is an instance of the query language defined\nin `options.querylang`.\n\nTo support pagination, `startFrom`, `pageSize` and `descending` can be\nspecified in the options parameter.\n\nIt is up to the receiving end of the query to determine if the query\nresults in too much load. This could be if someone tries to get data\nfor a large number of feeds without proper pagination, or if the query\nincludes arguments in a form that strains the query planner.\n\nTo get the latest 10 public post messages of a particular feed the\nfollowing `query` and `options` arguments can be used:\n\n```json\n{\n  \"author\": \"@6CAxOI3f+LUOVrbAl0IemqiS7ATpQvr9Mdw9LC4+Uv0=.ed25519\",\n  \"type\": \"post\",\n  \"private\": false\n}\n```\n\n```\n{\n  \"querylang\": \"ssb-ql-0\",\n  \"descending\": true,\n  \"pageSize\": 10\n}\n```\n\nResult:\n\n```\n[msg1, msg2, ..., msg10]\n```\n\n### Query language\n\n#### ssb-ql-0\n\nWe start by supporting a proof-of-concept query language called\n\"ssb-ql-0\" which is a precursor for the more powerful \"ssb-ql-1\"\n(see below).\n\nQueries in ssb-ql-0 are merely JSON objects with three *required*\nfields:\n\n- `author`: a valid feed ID as a string\n- `type`: a message type, can be any non-empty string, or null,\n  which signals that messages of any type are accepted\n- `private`: a boolean \n\nIf `private: true` then `type` **MUST** be `null`. This is to prevent\nleaking metadata about private messages to the public.\n\n#### ssb-ql-1\n\n*Status: under design. We recommend using ssb-ql-0 in the meantime.\nssb-ql-1 has challenges such as determining when two queries are\nequivalent, which may be an NP complete problem. Read more on this\nSSB thread:* `%FjK8SdR2Q8n3XBwFKiI8ebdSr/Tp7bMzhr2R6YMq+b0=.sha256`\n\nHere we define a mini query language that can be used to specify a\nsubset of data. The goal of this format is to be easy to parse and\nalso be restrictive in the number of operations supported to make it\neasier to map to indexes and to limit the attack surface.\n\nThe query uses an object notation for operators that looks very\nsimilar to JSON. The objects contains two keys: `op` the name of the\noperation and a second parameter for the arguments given the\noperation.\n\nBase operators:\n\nop     | args name   | args type\n------ | ---------   | ---------\nand    | args        | [op, ...]\nor     | args        | [op, ...]\ntype   | string      | string\nauthor | feed        | string\n\nNote that `ssb-ql-1` is only for public messages and is incapable of\nquerying private messages, unlike `ssb-ql-0`.\n\nThe spec is open for implementations to add new operators relatively\neasy. This allows experimentation and implementations should simply\nerror if presented with an unsupported operator. A new potential\noperator could be `isPrivate` or `isBox2` but extreme care must be\ntaken not to leak information using these queries around private\ninformation.\n\n## getTangle (TBD)\n\nTangles in SSB behave differently than other types of messages in\nthat they are linked together across multiple feeds using links. This\nallows for a different kind of synchonization than `getSubset`. There\nare two common examples of where this could be useful: for getting all\nthe messages of an old message threads that is outside the frontier of\nmessages and to synchronize a private group (as all messages in those\nare part of a single tangle). While private groups are by definition\nprivate to external observers, the fact that nodes connecting in SSB\nare authenticated by their ID and that rooms allow direct connections\nbetween peer allows synchronization of a private group directly with\nanother group member who also knows which messages belong to the group.\n\nAs private groups can grow quite large it becomes important that there\nis a way to synchronize only subsets but also changes or the latest\npart of a tangle. The membership part of a private group has a tangle\nroot and thus can be replicated on its own.\n\nConsider using [set-reconciliation] for this.\n\n# Prior work\n\nEarlier threads on SSB discussing partial replication:\n\n- %L9m5nHRqpXM4Zkha1ENTk5wNOXQMduve8Hc9+F0RLZI=.sha256\n- %VNCOf3pfP7hjL3lcFvBSSOzRPtR0WHDtvlXvkBcha3I=.sha256\n- %gaYxXAEoSKf3dnw2OlI2EuoduFmkiU4kU+CncbBImPk=.sha256\n- %jM6Dw0BEg9IPVkDH5VaoiVSbt3gCnUfSoO1BhX2Ntpw=.sha256\n- %SEXkvEQCJd2jBrUXDQcFPVix/5eUffoKBURWyy5yYrU=.sha256\n\n[JITDB]: https://github.com/arj03/jitdb\n[ssb-meta-feed]: https://github.com/arj03/ssb-meta-feed\n[set-reconciliation]: https://github.com/AljoschaMeyer/set-reconciliation\n[EBT]: https://github.com/ssbc/epidemic-broadcast-trees/\n[private group]: https://github.com/ssbc/private-group-spec\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fssbc%2Fssb-subset-replication-spec","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fssbc%2Fssb-subset-replication-spec","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fssbc%2Fssb-subset-replication-spec/lists"}