{"id":18718286,"url":"https://github.com/worldbrain/storex-sync","last_synced_at":"2025-04-12T13:33:38.244Z","repository":{"id":97337666,"uuid":"169285131","full_name":"WorldBrain/storex-sync","owner":"WorldBrain","description":"Sync \u0026 conflict resolution for peer-to-peer and offline-first applcations using Storex","archived":false,"fork":false,"pushed_at":"2025-02-27T08:06:14.000Z","size":369,"stargazers_count":16,"open_issues_count":4,"forks_count":3,"subscribers_count":8,"default_branch":"master","last_synced_at":"2025-04-09T22:21:31.996Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/WorldBrain.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"License","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2019-02-05T17:51:42.000Z","updated_at":"2025-02-27T08:06:35.000Z","dependencies_parsed_at":"2024-01-23T21:29:21.229Z","dependency_job_id":"b99c010c-1067-4234-a438-d6d80c8c5aef","html_url":"https://github.com/WorldBrain/storex-sync","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/WorldBrain%2Fstorex-sync","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/WorldBrain%2Fstorex-sync/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/WorldBrain%2Fstorex-sync/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/WorldBrain%2Fstorex-sync/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/WorldBrain","download_url":"https://codeload.github.com/WorldBrain/storex-sync/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248573639,"owners_count":21126876,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-07T13:20:26.169Z","updated_at":"2025-04-12T13:33:38.220Z","avatar_url":"https://github.com/WorldBrain.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"This package provides Sync functionality between multiple instances of any applications built on [Storex](https://github.com/WorldBrain/storex). This includes:\n\n-   Multiple devices that run an application storing all of it data data in IndexedDB\n-   Multiple SQL databases asynchronously sync'ed, like product catelogues in different physical shops\n-   In the future, offline-first single-user applications storing their data both on devices and in the cloud\n\nBy itself, right now this package cannot provide offline-first for multi-user application due to the need for access right management. That being said, its code is modular enough to be able to evolve to support such a scenario, opening up the possibility for permission decentralized applications.\n\n# How it works\n\n1. When you set up Storex as the storage layer for your application (with IndexedDB as the backend for example) you set it up with the Custom PK and Sync Log [middleware](https://github.com/WorldBrain/storex/blob/master/docs/middleware.md).\n    - The Custom PK middleware generates a random ID for each new object instead of an auto-incremented ID to prevent ID conflicts between devices\n    - The Sync Log middleware intercepts all modifications to the database and also writes them to the Client Sync Log\n2. Once in a while you sync the Client Log with the Shared Log, sending and receiving changes\n3. When new changes are received, the Reconciliation Algorithm is ran to determine which changes have to be made to the client database, and execute them\n\n# Usage\n\n```\nimport uuid from 'uuid/v1'\nimport StorageManager, { StorageBackend, StorageRegistry } from \"@worldbrain/storex\"\nimport { registerModuleMapCollections, StorageModule } from \"@worldbrain/storex-pattern-modules\"\n\nimport { CustomAutoPkMiddleware } from '@worldbrain/storex-sync/lib/custom-auto-pk'\nimport { SyncLoggingMiddleware } from '@worldbrain/storex-sync/lib/logging-middleware'\nimport { ClientSyncLogStorage } from '@worldbrain/storex-sync/lib/client-sync-log'\nimport { SharedSyncLog } from '@worldbrain/storex-sync/lib/shared-sync-log'\nimport { SharedSyncLogStorage } from '@worldbrain/storex-sync/lib/shared-sync-log/storex'\nimport { reconcileSyncLog } from '@worldbrain/storex-sync/lib/reconciliation'\nimport { doSync } from '@worldbrain/storex-sync'\n\nexport async function setupClientStorage() {\n    const storageManager = ... // Set up your storage backend, manager, modules and collections here\n    const clientSyncLog = new ClientSyncLogStorage({storageManager})\n    registerModuleMapCollections({ clientSyncLog })\n    await storageManager.finishInitialization()\n\n    // Prevent auto-incremented ID clashes by generating UUIDs instead\n    const pkMiddleware = new CustomAutoPkMiddleware({ pkGenerator: () =\u003e uuid() })\n    pkMiddleware.setup({ storageRegistry: storageManager.registry, collections: includeCollections })\n\n    const syncLoggingMiddleware = new SyncLoggingMiddleware({ storageManager, clientSyncLog: modules.clientSyncLog, includeCollections })\n    syncLoggingMiddleware._getNow = options.getNow\n\n    storageManager.setMiddleware([\n        pkMiddleware,\n        syncLoggingMiddleware\n    ])\n\n    // From now on, all write operations will be logged to the Sync log\n    await storageManager.collection('user').createObject({ displayName: 'Joe' })\n}\n\nexport async function sync(options : { storageManager : StorageManager, clientSyncLog : ClientSyncLog, sharedSyncLog : SharedSyncLog }) {\n    await doSync({\n        storageManager, clientSyncLog, sharedSyncLog,\n\n        // The default reconciliation algorithm, swappable\n        reconciler: reconcileSyncLog,\n\n        // For unit test, it may be useful to specify a custom timestamp here\n        now: '$now',\n\n        // This depends on the user management of your application\n        userId,\n\n        // This can be created with `sharedSyncLog.createDeviceId({ ... })`\n        deviceId\n    })\n}\n```\n\n# The shared sync log\n\nAs mentioned above, Sync works by sending and receiving changes from a shared log. Currently, we have working PoCs of doing this through GraphQL to a custom back-end, through Firestore, the local Filesystem and entirely within the same browser for testing purposes. However, all that's neded for a different kind of shared log is implementing the [SharedSyncLog interface](./ts/shared-sync-log/types.ts) and passing that into the `doSync()` function as shown above. Example implementations can be found [here](./ts/shared-sync-log/storex.ts) and [here](./ts/shared-sync-log/fs.ts).\n\n# Deeper understanding\n\nSince this a complex piece of software that with the risk that it brings with it, it's highly recommended to dive into the code and get a thorough understand of it before implementing any of this in your own application. The best point to start would be the [integration tests](./ts/index.test.ts) and drilling down from there.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fworldbrain%2Fstorex-sync","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fworldbrain%2Fstorex-sync","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fworldbrain%2Fstorex-sync/lists"}