{"id":21365464,"url":"https://github.com/frameable/syncable","last_synced_at":"2025-03-16T07:41:17.223Z","repository":{"id":214531203,"uuid":"736740187","full_name":"frameable/syncable","owner":"frameable","description":"Synchronize JSON data structures across clients and servers over WebSockets and Redis","archived":false,"fork":false,"pushed_at":"2024-03-10T19:58:30.000Z","size":141,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":5,"default_branch":"main","last_synced_at":"2025-01-22T20:09:29.114Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/frameable.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2023-12-28T18:30:58.000Z","updated_at":"2024-07-20T13:52:24.000Z","dependencies_parsed_at":"2024-01-06T21:23:30.926Z","dependency_job_id":null,"html_url":"https://github.com/frameable/syncable","commit_stats":null,"previous_names":["frameable/syncable"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/frameable%2Fsyncable","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/frameable%2Fsyncable/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/frameable%2Fsyncable/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/frameable%2Fsyncable/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/frameable","download_url":"https://codeload.github.com/frameable/syncable/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243841204,"owners_count":20356441,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-22T07:11:19.731Z","updated_at":"2025-03-16T07:41:17.199Z","avatar_url":"https://github.com/frameable.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Syncable\n\nSynchronize JSON data structures across servers and clients over WebSockets and Redis\n\n```javascript\n/* server.js */\n\nconst app = express()\n\n// our dynamic websocket route handled by syncable\napp.get('/documents/:slug', syncable())\n```\n\n```javascript\n/* client.js */\n\n// connect to the live document\nlet doc = await syncable.client({url: `wss://${host}/documents/my-document-10329`})\n\n// make a change the the document, which will be reflected on the server and all other clients\ndoc.sync(d =\u003e d.title = 'New title')\n```\n\n## Introduction\n\nSyncable is a framework for synchronizing JSON data structures across many servers and many clients, to facilitate collaborative real-time web applications.  On the server, install a syncable route handler for each type of document you wish to service.  In the browser, use the syncable client to make a websocket connection and get back a document.  Once you have the document on the client or server, call its `sync` method to make changes, and those changes will propagate to all other servers and clients, via Redis streams, and WebSockets.\n\n## How it works\n\nSyncable documents consist of snapshots and change events.  At any time, the current state of a document can be derived from its latest snapshot and any subsequent changes.  Snapshots are stored in persistent storage, and changes are temporarily queued in Redis streams.  By default, snapshots are taken within 30 seconds after each change.  Change events live in Redis only until they've been incorporated into a snapshot, after which point may be removed.\n\nUnderlying documents are based on [Pigeon](https://github.com/frameable/pigeon), which itself is heavily inspired by [Automerge](https://github.com/automerge/automerge).  When a client changes a document, a JSON Patch style diff is generated, and propagated to all other servers and clients who have that document loaded.  Even when changes arrive in a different order, the result is deterministic.\n\n## Performance and scalability\n\nSyncable scales across many backend servers and up to hundreds or thousands of simultaneous clients per document.  In lower-volume settings, each change is broadcast and applied individually, but as volume increases, changes are batched and applied in bulk.  For example, if changes are happening at a rate of 1 per second, then they will be applied without delay. However, once changes are arriving at 10 per second, then the changes will be queued for 1 second, and then applied together as a batch.\n\n## Client API\n\n#### syncable.client(options)\n\nLoad a live syncable document from the server.  Options include:\n\n- `url` - WebSocket url to a document where a syncable handler is listening.\n\n\n```javascript\nlet doc = await syncable.client({ url: `wss://localhost/documents/my-document` })\n```\n\n### doc.sync(fn)\n\nMake a change to the document and sync that change to all other servers and clients.\n\n```javascript\nlet doc = await syncable.client({ url });\ndoc = await doc.sync(d =\u003e d.title = 'My title');\n\nconsole.log(JSON.stringify(doc));\n// { title: \"My title\" }\n```\n\n### doc.on(eventName, handler)\n\nAdd an event handler function for a given event.  Emitted events include:\n\n- `initialized` - Document has been loaded from the server and is ready for consumption.\n- `changed` - Document has been changed, either by us or by another client.\n- `rejected` - Our change has been rejected by the server by the validator function.\n- `connected` - WebSocket connection has been established.\n- `reconnecting` - WebSocket is reconnecting, possibly after a ping timeout or other network event.\n- `closed` - WebSocket connection has been closed.\n- `error` - WebSocket error has occurred.\n\n\n## Server API\n\n#### syncable.initialize(options)\n\nConfigure and initialize the syncable library.  All properties are optional:\n\n- `redis` - Configuration to be passed to [ioredis](https://github.com/redis/ioredis?tab=readme-ov-file#connect-to-redis).\n\n- `writer` - Function to override writing document snapshots to persistent storage.  By default, snapshots are written to Redis, but use this function if you prefer to write somewhere else such as S3, Postgres, local disk, etc.  Snapshot writes are debounced, occurring as often as every 30 seconds by default following a change. See the `window` option to configure the timing.  Function takes `key` and `data` parameters.\n\n  ```javascript\n  function writer(key, data) {\n    redis.set(key, data);\n  }\n\n- `reader` - Function to override reading document snapshots from persistent storage.  This is the reciprocal of the `writer` function above.  Takes a `key` parameter and returns data that was written by `writer`.\n\n  ```javascript\n  function reader(key) {\n    return redis.get(key);\n  }\n  ```\n\n- `validator` - Function to validate incoming changes.  Useful for example to ensure the user has permissions to make the specified modification, or that the change is to an appropriate part of the document.\n\n  ```javascript\n  function validator(ws, req, { changes }) {\n    if (!req.session.isAdmin \u0026\u0026 changes.diff.filter(d =\u003e d.path.match('/settings')).length) {\n      return false;\n    } else {\n      return true;\n    }\n  }\n  ```\n\n- `window` - Minimum number of milliseconds between subsequent writes to persistent storage.  Intermediate document changes will be queued in Redis streams at least until the next write.  Defaults to `30_000` (30 seconds).\n\n\n#### syncable.load(key)\n\nLoad the document with the given key.  Document will be retrieved from memory if it has already been loaded.  Otherwise, it will be fetched from persistent storage with `reader`, and have any subsequent queued changes applied.  Returns the loaded document.\n\n```javascript\nlet doc = await syncable.load('/documents/my-document');\n```\n\n#### syncable.unload(key)\n\nUnload the document from memory.  Any next call to `load` will fetch from persistent storage.\n\n#### doc.sync(fn)\n\nMake a change to the document and sync that change to all other servers and clients.\n\n```javascript\nlet doc = await syncable.load('/documents/my-document');\ndoc = await doc.sync(d =\u003e d.title = 'My title');\n\nconsole.log(doc);\n// { title: \"My title\" }\n```\n\n## License\n\nThe MIT License\n\nCopyright (c) 2023 Frameable Inc, David Chester, Doug Brunton, Logan Bell, Daniel Dyssegaard Kallick\n\nPermission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \"Software\"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fframeable%2Fsyncable","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fframeable%2Fsyncable","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fframeable%2Fsyncable/lists"}