{"id":21368288,"url":"https://github.com/chrisvilches/collection-sync","last_synced_at":"2026-02-11T19:04:53.148Z","repository":{"id":94022558,"uuid":"404468670","full_name":"ChrisVilches/Collection-Sync","owner":"ChrisVilches","description":"Javascript Library for bi-directional database synchronization between multiple devices or servers. Customizable and completely database agnostic.","archived":false,"fork":false,"pushed_at":"2021-09-16T10:35:25.000Z","size":469,"stargazers_count":5,"open_issues_count":3,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-08-03T23:57:03.835Z","etag":null,"topics":["collection","database","database-sync","database-synchronization","javascript","node","nodejs","npm","sync","synchronization","typescript"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ChrisVilches.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-09-08T19:20:30.000Z","updated_at":"2025-03-17T23:44:49.000Z","dependencies_parsed_at":"2023-03-13T17:08:14.577Z","dependency_job_id":null,"html_url":"https://github.com/ChrisVilches/Collection-Sync","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/ChrisVilches/Collection-Sync","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ChrisVilches%2FCollection-Sync","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ChrisVilches%2FCollection-Sync/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ChrisVilches%2FCollection-Sync/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ChrisVilches%2FCollection-Sync/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ChrisVilches","download_url":"https://codeload.github.com/ChrisVilches/Collection-Sync/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ChrisVilches%2FCollection-Sync/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29341717,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-11T18:58:20.535Z","status":"ssl_error","status_checked_at":"2026-02-11T18:56:44.814Z","response_time":97,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["collection","database","database-sync","database-synchronization","javascript","node","nodejs","npm","sync","synchronization","typescript"],"created_at":"2024-11-22T07:23:39.488Z","updated_at":"2026-02-11T19:04:53.134Z","avatar_url":"https://github.com/ChrisVilches.png","language":"TypeScript","readme":"[![npm version](https://badge.fury.io/js/collection-sync.svg)](https://badge.fury.io/js/collection-sync)\n\n# Collection Sync\n\nJavascript Library for bi-directional database synchronization between multiple devices or servers. Customizable and completely database agnostic.\n\nSee [Documentation](/docs/modules.md).\n\n## Use cases\n\nSome examples where this mechanism would be useful:\n\n* Memo app that works offline, and updates remote database when it goes online.\n* Multiplatform app (desktop, mobile, web app) which works offline (in mobile and desktop), and updates remote database when going online. When using the app in a different device, the new data is downloaded, making all devices up to date.\n\n## Install\n\n```bash\nnpm install collection-sync\n```\n\n## How to use\n\nMake sure your project is using TypeScript.\n\nImport dependencies:\n\n```ts\nimport { SyncItem } from \"collection-sync\";\nimport { SynchronizableCollection } from \"collection-sync\";\nimport { CollectionSyncMetadata } from \"collection-sync\";\nimport { SyncOperation, SyncConflictStrategy } from \"collection-sync\";\nimport DocId from \"collection-sync/dist/types/DocId\";\n```\n\nCreate a class that extends [SyncItem](/docs/classes/SyncItem.md), and populates its data starting from a document from your database. You must customize the way to extract the ID and date it was last updated.\n\nIf we use Mongo's `_id`, when synchronizing from one database to another it might be impossible to set it, since some database engines auto generate the `_id` value, hence you must choose how to identify documents using a custom way.\n\nThe same applies for `updatedAt`. If your database engine automatically sets `updatedAt` when saving a record, then you'll need to have a custom attribute you can set freely.\n\nSynchronization will fail if you try to set the ID and/or `updatedAt` but your database refuses to leave you set a specific value.\n\n```ts\nclass CustomItem extends SyncItem {\n  constructor(doc: any /* e.g. Mongo Document */){\n    super(doc.documentId, doc, doc.updatedAt);\n  }\n}\n```\n\nThen, create a class that extends [SynchronizableCollection](/docs/classes/SynchronizableCollection.md) and implement its abstract methods:\n\n```ts\nclass LocalCollection extends SynchronizableCollection {\n  countAll(): number | Promise\u003cnumber\u003e {\n    // Count collection documents.\n    return 100;\n  }\n  async initialize(): Promise\u003cvoid\u003e {\n    // Executes async logic to initialize collection or datastore (open file, create database connection, etc).\n  }\n  findByIds(ids: DocId[]): SyncItem[] | Promise\u003cSyncItem[]\u003e {\n    const docs = [\n      { /* Doc from DB */ },\n      { /* Doc from DB */ },\n      { /* Doc from DB */ }\n    ];\n    return docs.map(d =\u003e new CustomItem(d)); // Convert to CustomItem.\n  }\n  syncBatch(items: SyncItem[]): SyncItem[] | Promise\u003cSyncItem[]\u003e {\n    // Implement batch upsert/delete of records.\n  }\n  itemsNewerThan(date: Date | undefined, limit: number): SyncItem[] | Promise\u003cSyncItem[]\u003e {\n    // Returns a list of items that have updatedAt greater than argument provided.\n    // The list MUST be ordered by updatedAt ASC, otherwise an exception will be\n    // thrown (no syncing will be executed).\n  }\n  latestUpdatedItem(): SyncItem | Promise\u003cSyncItem | undefined\u003e | undefined {\n    // Gets the highest updateAt date in the collection.\n  }\n}\n```\n\nInstall `npm install --save @types/node` if you get errors related to missing Node types.\n\nYou can also implement several lifecycle hooks for granular control over syncing. See [which methods can be overriden from SynchronizableCollection class](/docs/classes/SynchronizableCollection.md) for details.\n\nAll methods allow the use of `async/await` if needed.\n\nNext, implement a class that communicates with the remote collection (datastore).\n\nIn cases where the local collection is a database in a mobile app, you don't want to connect directly to a remote database, but instead you'd have to prepare a backend API to connect to, which provides the operations needed (syncBatch, findByIds, etc). This class must work as a communication layer between the client and that API.\n\nIf both collections are inside a private/secure network, then connecting directly to another database would be fine.\n\n```ts\nclass RemoteCollection extends SynchronizableCollection {\n  countAll(): number | Promise\u003cnumber\u003e {\n    // Execute some API call to\n    // https://your_server.com/api/users/count_all\n    // and return its value here.\n  }\n  async initialize(): Promise\u003cvoid\u003e {\n    // ...\n  }\n  findByIds(ids: DocId[]): SyncItem[] | Promise\u003cSyncItem[]\u003e {\n    // ...\n  }\n  syncBatch(items: SyncItem[]): SyncItem[] | Promise\u003cSyncItem[]\u003e {\n    // ...\n  }\n  itemsNewerThan(date: Date | undefined, limit: number): SyncItem[] | Promise\u003cSyncItem[]\u003e {\n    // ...\n  }\n  latestUpdatedItem(): SyncItem | Promise\u003cSyncItem | undefined\u003e | undefined {\n    // ...\n  }\n}\n```\n\nFinally, implement a mechanism to store and retrieve two dates (last fetch and post dates).\n\nA persistent storage is recommended.\n\n```ts\nclass MySyncMetadata extends CollectionSyncMetadata{\n  setLastFetchAt(d: Date): void {\n    // ...\n  }\n  setLastPostAt(d: Date): void {\n    // ...\n  }\n  getLastFetchAt(): Date | Promise\u003cDate | undefined\u003e | undefined {\n    // ...\n  }\n  getLastPostAt(): Date | Promise\u003cDate | undefined\u003e | undefined {\n    // ...\n  }\n  async initialize(): Promise\u003cvoid\u003e {\n    // ...\n  }\n}\n```\n\nNote that both classes have a `initialize` method. Some storage mechanisms require to open a file, create a DB connection, or do some asynchronous logic before beginning to use them. You can put that logic there.\n\nIn this example, however, we'll import and use [BasicSyncMetadata](/docs/classes/BasicSyncMetadata.md), which provides an in-memory storage for synchronization metadata. This is the simplest way to get started.\n\nAdd a new import to the top of the file:\n\n```ts\nimport { BasicSyncMetadata } from \"collection-sync\";\n```\n\nThen, create two synchronization metadata managers:\n\n```ts\nconst syncMetadataSlave: CollectionSyncMetadata = new BasicSyncMetadata();\nconst syncMetadataMaster: CollectionSyncMetadata = new BasicSyncMetadata();\n```\n\nNow, create two collections:\n\n```ts\nconst collectionSlave = new LocalCollection(syncMetadataSlave);\nconst collectionMaster = new RemoteCollection(syncMetadataMaster);\n```\n\nSince only the slave keeps track of sync metadata, the master doesn't need a `CollectionSyncMetadata` object. However, since all collections could potentially have a master, it is a required argument.\n\nAttach the parent as master:\n\n```ts\ncollectionSlave.parent = collectionMaster;\n```\n\nNote that both `collectionSlave` and `collectionMaster` simply model how your data stores are arranged. The machine where this code is running doesn't actually need to host `collectionMaster`'s data, but since the way to communicate with it was implemented (i.e. `RemoteCollection`'s methods for API communication), we still can modify its data.\n\nIf we assume that some data exists in the datastore `collectionMaster` is pointing at, and the database pointed by `collectionSlave` is empty, then we can perform a fetch to update `collectionSlave`:\n\n```ts\ncollectionSlave.sync(SyncOperation.Fetch, 100, { conflictStrategy: SyncConflictStrategy.Force });\n```\n\nSee [sync](/docs/classes/SynchronizableCollection.md#sync) method documentation.\n\nSee also [other specifications](sync_specifications.md) related to `sync`.\n\nWhen syncing, conflicts might occur, and there are a few strategies to overcome them. A conflict occurs when trying to update a record using a record with an older `updatedAt`. In general, when synchronizing data collections, older data should be overwritten by newer data, but sometimes this is not the case, and that's when a conflict is generated. See [SyncConflictStrategy](/docs/enums/SyncConflictStrategy.md) for details.\n\n## Another example\n\nNote: Omitting some steps from the previous example.\n\n```ts\nconst android = new LocalCollection(new MySyncMetadata());\nconst pc = new LocalCollection(new MySyncMetadata());\nconst backend = new LocalCollection(new MySyncMetadata());\n\nandroid.parent = backend;\npc.parent = backend;\n\n// Data that only exists in Android devide is being pushed...\nandroid.sync(SyncOperation.Post, 1000);\n\n// PC device now has data that previously only the Android device had.\npc.sync(SyncOperation.Fetch, 1000);\n```\n\nIn practice, you'd want to make your slave collection perform both post and fetch operations during a full sync.\n\nWhen a conflict is encountered, a suggestion is to ask the user to manually select how to solve them, and then trigger a new synchronization but using a different configuration (e.g. forcing data from the master collection to overwrite slave data).\n\n## Current limitations and future work\n\n### Locking mechanism\n\nLocking mechanism (to prevent multiple devices from synchronizing at the same time) must be implemented by the user. The addition of `acquireLock` and `releaseLock` abstract methods to `SynchronizableCollection` or `Collection` have been proposed.\n\nIt's possible to implement this logic in the `preExecuteSync` (acquire lock, and return `false` if it failed) and `cleanUp` (release lock) hooks.\n\n### Using it with Vanilla Javascript\n\nUse with vanilla Javascript is not tested. It may not be convenient for development. Typescript is recommended.\n\n## Develop\n\n```bash\nnpm run test:watch\n```\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchrisvilches%2Fcollection-sync","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fchrisvilches%2Fcollection-sync","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchrisvilches%2Fcollection-sync/lists"}