{"id":19359957,"url":"https://github.com/darrachequesne/synceddb","last_synced_at":"2025-04-07T16:17:57.691Z","repository":{"id":57689497,"uuid":"473905509","full_name":"darrachequesne/synceddb","owner":"darrachequesne","description":"This is a fork of the awesome idb library, which adds the ability to sync an IndexedDB database with a remote REST API.","archived":false,"fork":false,"pushed_at":"2024-10-08T06:26:56.000Z","size":758,"stargazers_count":44,"open_issues_count":4,"forks_count":7,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-03-31T15:19:31.275Z","etag":null,"topics":["indexeddb","offline-first","rest-api"],"latest_commit_sha":null,"homepage":"https://www.npmjs.com/package/synceddb","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"isc","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/darrachequesne.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-03-25T07:11:33.000Z","updated_at":"2025-03-23T20:21:21.000Z","dependencies_parsed_at":"2025-02-26T00:00:30.212Z","dependency_job_id":"6ef6e64e-70f5-4176-a0ac-0f8e02e2ab29","html_url":"https://github.com/darrachequesne/synceddb","commit_stats":null,"previous_names":[],"tags_count":5,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/darrachequesne%2Fsynceddb","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/darrachequesne%2Fsynceddb/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/darrachequesne%2Fsynceddb/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/darrachequesne%2Fsynceddb/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/darrachequesne","download_url":"https://codeload.github.com/darrachequesne/synceddb/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247685628,"owners_count":20979085,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["indexeddb","offline-first","rest-api"],"created_at":"2024-11-10T07:16:40.059Z","updated_at":"2025-04-07T16:17:57.677Z","avatar_url":"https://github.com/darrachequesne.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# IndexedDB with usability and remote syncing\n\nThis is a fork of the awesome [`idb`](https://github.com/jakearchibald/idb) library, which adds the ability to sync an IndexedDB database with a remote REST API.\n\n![Video of two clients syncing their IndexedDB database](assets/demo.gif)\n\nThe source code for the example above can be found [here](https://github.com/darrachequesne/synceddb-todo-example).\n\nBundle size: ~3.26 kB brotli'd\n\n**Table of content**\n\n\u003c!-- TOC --\u003e\n* [Features](#features)\n  * [All the usability improvements from the `idb` library](#all-the-usability-improvements-from-the-idb-library)\n  * [Sync with a remote REST API](#sync-with-a-remote-rest-api)\n  * [Auto-reloading queries](#auto-reloading-queries)\n* [Disclaimer](#disclaimer)\n* [Installation](#installation)\n* [API](#api)\n  * [SyncManager](#syncmanager)\n    * [Options](#options)\n      * [`fetchOptions`](#fetchoptions)\n      * [`fetchInterval`](#fetchinterval)\n      * [`buildPath`](#buildpath)\n      * [`buildFetchParams`](#buildfetchparams)\n      * [`updatedAtAttribute`](#updatedatattribute)\n      * [`withoutKeyPath`](#withoutkeypath)\n    * [Methods](#methods)\n      * [`start()`](#start)\n      * [`stop()`](#stop)\n      * [`clear()`](#clear)\n      * [`hasLocalChanges()`](#haslocalchanges)\n      * [`onfetchsuccess`](#onfetchsuccess)\n      * [`onfetcherror`](#onfetcherror)\n      * [`onpushsuccess`](#onpushsuccess)\n      * [`onpusherror`](#onpusherror)\n  * [LiveQuery](#livequery)\n    * [Example with React](#example-with-react)\n    * [Example with Vue.js](#example-with-vuejs)\n* [Expectations for the REST API](#expectations-for-the-rest-api)\n  * [Fetching changes](#fetching-changes)\n  * [Pushing changes](#pushing-changes)\n* [Alternatives](#alternatives)\n* [Miscellaneous](#miscellaneous)\n\u003c!-- TOC --\u003e\n\n## Features\n\n### All the usability improvements from the `idb` library\n\nSince it is a fork of the [`idb`](https://github.com/jakearchibald/idb) library, `synceddb` shares the same Promise-based API:\n\n```js\nimport { openDB, SyncManager } from 'synceddb';\n\nconst db = await openDB('my-awesome-database');\n\nconst transaction = db.transaction('items', 'readwrite');\nawait transaction.store.add({ id: 1, label: 'Dagger' });\n\n// short version\nawait db.add('items', { id: 1, label: 'Dagger' });\n```\n\nMore information [here](https://github.com/jakearchibald/idb#api).\n\n### Sync with a remote REST API\n\nEvery change is tracked in a store. The [SyncManager](#syncmanager) then sync these changes with the remote REST API when the connection is available, making it easier to build offline-first applications.\n\n```js\nimport { openDB, SyncManager } from 'synceddb';\n\nconst db = await openDB('my-awesome-database');\nconst manager = new SyncManager(db, 'https://example.com');\n\nmanager.start();\n\n// will result in the following HTTP request: POST /items\nawait db.add('items', { id: 1, label: 'Dagger' });\n\n// will result in the following HTTP request: DELETE /items/2\nawait db.delete('items', 2);\n```\n\nSee also: [Expectations for the REST API](#expectations-for-the-rest-api)\n\n### Auto-reloading queries\n\nThe [LiveQuery](#livequery) provides a way to run a query every time the underlying stores are updated:\n\n```js\nimport { openDB, LiveQuery } from 'synceddb';\n\nconst db = await openDB('my awesome database');\n\nlet result;\n\nconst query = new LiveQuery(['items'], async () =\u003e {\n  // result will be updated every time the 'items' store is modified\n  result = await db.getAll('items');\n});\n\n// trigger the liveQuery\nawait db.put('items', { id: 2, label: 'Long sword' });\n\n// or manually run it\nawait query.run();\n```\n\nInspired from [Dexie.js liveQuery](https://dexie.org/docs/liveQuery()).\n\n## Disclaimer\n\n- no version history\n\nOnly the last version of each entity is kept on the client side.\n\n- basic conflict management\n\nThe last write wins (though you can customize the behavior in the [`onpusherror`](#onpusherror) handler).\n\n## Installation\n\n```sh\nnpm install synceddb\n```\n\nThen:\n\n```js\nimport { openDB, SyncManager, LiveQuery } from 'synceddb';\n\nasync function doDatabaseStuff() {\n  const db = await openDB('my awesome database');\n\n  // sync your database with a remote server\n  const manager = new SyncManager(db, 'https://example.com');\n\n  manager.start();\n  \n  // create an auto-reloading query\n  let result;\n  const query = new LiveQuery(['items'], async () =\u003e {\n    // result will be updated every time the 'items' store is modified\n    result = await db.getAll('items');\n  });\n}\n```\n\n## API\n\nFor database-related operations, please see the `idb` [documentation](https://github.com/jakearchibald/idb#api).\n\n### SyncManager\n\n```js\nimport { openDB, SyncManager } from 'synceddb';\n\nconst db = await openDB('my-awesome-database');\nconst manager = new SyncManager(db, 'https://example.com');\n\nmanager.start();\n```\n\n#### Options\n\n##### `fetchOptions`\n\nAdditional options for all HTTP requests.\n\n```js\nimport { openDB, SyncManager } from 'synceddb';\n\nconst db = await openDB('my-awesome-database');\nconst manager = new SyncManager(db, 'https://example.com', {\n  fetchOptions: {\n    headers: {\n      'accept': 'application/json'\n    },\n    credentials: 'include'\n  }\n});\n\nmanager.start();\n```\n\nReference: https://developer.mozilla.org/en-US/docs/Web/API/fetch\n\n##### `fetchInterval`\n\nThe number of ms between two fetch requests for a given store.\n\nDefault value: `30000`\n\n```js\nimport { openDB, SyncManager } from 'synceddb';\n\nconst db = await openDB('my-awesome-database');\nconst manager = new SyncManager(db, 'https://example.com', {\n  fetchInterval: 10000\n});\n\nmanager.start();\n```\n\n##### `buildPath`\n\nA function that allows to override the request path for a given request.\n\n```js\nimport { openDB, SyncManager } from 'synceddb';\n\nconst db = await openDB('my-awesome-database');\nconst manager = new SyncManager(db, 'https://example.com', {\n  buildPath: (operation, storeName, key) =\u003e {\n    if (storeName === 'my-local-store') {\n      if (key) {\n        return `/the-remote-store/${key[1]}`;\n      } else {\n        return '/the-remote-store/';\n      }\n    }\n    // defaults to `/${storeName}/${key}`\n  }\n});\n\nmanager.start();\n```\n\n##### `buildFetchParams`\n\nA function that allows to override the query params of the fetch requests.\n\nDefaults to `?sort=updated_at:asc\u0026size=100\u0026after=2000-01-01T00:00:00.000Z,123`.\n\n```js\nimport { openDB, SyncManager } from 'synceddb';\n\nconst db = await openDB('my-awesome-database');\nconst manager = new SyncManager(db, 'https://example.com', {\n  buildFetchParams: (storeName, offset) =\u003e {\n    const searchParams = new URLSearchParams({\n      sort: '+updatedAt',\n      size: '10',\n    });\n    if (offset) {\n      searchParams.append('after', `${offset.updatedAt}+${offset.id}`);\n    }\n    return searchParams;\n  }\n});\n\nmanager.start();\n```\n\n##### `updatedAtAttribute`\n\nThe name of the attribute that indicates the last updated date of the entity.\n\nDefault value: `updatedAt`\n\n```js\nimport { openDB, SyncManager } from 'synceddb';\n\nconst db = await openDB('my-awesome-database');\nconst manager = new SyncManager(db, 'https://example.com', {\n  updatedAtAttribute: 'lastUpdateDate'\n});\n\nmanager.start();\n```\n\n##### `withoutKeyPath`\n\nList entities from object stores without `keyPath`.\n\n```js\nimport { openDB, SyncManager } from 'synceddb';\n\nconst db = await openDB('my-awesome-database');\nconst manager = new SyncManager(db, 'https://example.com', {\n  withoutKeyPath: {\n    common: [\n      'user',\n      'settings'\n    ]\n  },\n  buildPath: (_operation, storeName, key) =\u003e {\n    if (storeName === 'common') {\n      if (key === 'user') {\n        return '/me';\n      } else if (key === 'settings') {\n        return '/settings';\n      }\n    }\n  }\n});\n\nmanager.start();\n\nawait db.put('common', { firstName: 'john' }, 'user');\n```\n\n#### Methods\n\n##### `start()`\n\nStarts the sync process with the remote server.\n\n```js\nimport { openDB, SyncManager } from 'synceddb';\n\nconst db = await openDB('my-awesome-database');\nconst manager = new SyncManager(db, 'https://example.com');\n\nmanager.start();\n```\n\n##### `stop()`\n\nStops the sync process.\n\n```js\nimport { openDB, SyncManager } from 'synceddb';\n\nconst db = await openDB('my-awesome-database');\nconst manager = new SyncManager(db, 'https://example.com');\n\nmanager.stop();\n```\n\n##### `clear()`\n\nClears the local stores.\n\n```js\nimport { openDB, SyncManager } from 'synceddb';\n\nconst db = await openDB('my-awesome-database');\nconst manager = new SyncManager(db, 'https://example.com');\n\nmanager.clear();\n```\n\n##### `hasLocalChanges()`\n\nReturns whether a given entity currently has local changes that are not synced yet.\n\n```js\nimport { openDB, SyncManager } from 'synceddb';\n\nconst db = await openDB('my-awesome-database');\nconst manager = new SyncManager(db, 'https://example.com');\n\nawait db.put('items', { id: 1 });\n\nconst hasLocalChanges = await manager.hasLocalChanges('items', 1); // true\n```\n\n##### `onfetchsuccess`\n\nCalled after some entities are successfully fetched from the remote server.\n\n```js\nimport { openDB, SyncManager } from 'synceddb';\n\nconst db = await openDB('my-awesome-database');\nconst manager = new SyncManager(db, 'https://example.com');\n\nmanager.onfetchsuccess = (storeName, entities, hasMore) =\u003e {\n  // ...\n}\n```\n\n##### `onfetcherror`\n\nCalled when something goes wrong when fetching the changes from the remote server.\n\n```js\nimport { openDB, SyncManager } from 'synceddb';\n\nconst db = await openDB('my-awesome-database');\nconst manager = new SyncManager(db, 'https://example.com');\n\nmanager.onfetcherror = (err) =\u003e {\n  // ...\n}\n```\n\n##### `onpushsuccess`\n\nCalled after a change is successfully pushed to the remote server.\n\n```js\nimport { openDB, SyncManager } from 'synceddb';\n\nconst db = await openDB('my-awesome-database');\nconst manager = new SyncManager(db, 'https://example.com');\n\nmanager.onpushsuccess = ({ operation, storeName, key, value }) =\u003e {\n  // ...\n}\n```\n\n##### `onpusherror`\n\nCalled when something goes wrong when pushing a change to the remote server.\n\n```js\nimport { openDB, SyncManager } from 'synceddb';\n\nconst db = await openDB('my-awesome-database');\nconst manager = new SyncManager(db, 'https://example.com');\n\nmanager.onpusherror = (change, response, retryAfter, discardLocalChange, overrideRemoteChange) =\u003e {\n  // this is the default implementation\n  switch (response.status) {\n    case 403:\n    case 404:\n      return discardLocalChange();\n    case 409:\n      // last write wins by default\n      response.json().then((content) =\u003e {\n        const version = content[VERSION_ATTRIBUTE];\n        change.value[VERSION_ATTRIBUTE] = version + 1;\n        overrideRemoteChange(change.value);\n      });\n      break;\n    default:\n      return retryAfter(DEFAULT_RETRY_DELAY);\n  }\n}\n```\n\n### LiveQuery\n\nThe first argument is an array of stores. Every time one of these stores is updated, the function provided in the 2nd argument will be called.\n\n```js\nimport { openDB, LiveQuery } from 'synceddb';\n\nconst db = await openDB('my awesome database');\n\nlet result;\n\nconst query = new LiveQuery(['items'], async () =\u003e {\n  // result will be updated every time the 'items' store is modified\n  result = await db.getAll('items');\n});\n```\n\n#### Example with React\n\n```js\nimport { openDB, LiveQuery } from 'synceddb';\nimport { useEffect, useState } from 'react';\n\nexport default function MyComponent() {\n  const [items, setItems] = useState([]);\n\n  useEffect(() =\u003e {\n    let query;\n\n    openDB('test', 1, {\n      upgrade(db) {\n        db.createObjectStore('items', { keyPath: 'id' });\n      },\n    }).then(db =\u003e {\n      query = new LiveQuery(['items'], async () =\u003e {\n        setItems(await db.getAll('items'));\n      });\n\n      query.run();\n    });\n\n    return () =\u003e {\n      // !!! IMPORTANT !!! This ensures the query stops listening to the database updates and does not leak memory.\n      query?.close();\n    }\n  }, []);\n\n  return (\n    \u003cdiv\u003e\n      \u003c!-- ... --\u003e\n    \u003c/div\u003e\n  );\n}\n```\n\n#### Example with Vue.js\n\n```vue\n\u003cscript setup\u003e\nimport { openDB, LiveQuery } from 'synceddb';\nimport { ref, onBeforeUnmount } from 'vue';\n\nconst items = ref([]);\n\nconst db = await openDB('test', 1, {\n  upgrade(db) {\n    db.createObjectStore('items', { keyPath: 'id' });\n  },\n})\n\nconst query = new LiveQuery(['items'], async () =\u003e {\n  items.value = await db.getAll('items');\n});\n\nawait query.run();\n\nonBeforeUnmount(() =\u003e {\n  // !!! IMPORTANT !!! This ensures the query stops listening to the database updates and does not leak memory.\n  query.close();\n});\n\u003c/script\u003e\n```\n\n## Expectations for the REST API\n\n### Fetching changes\n\nChanges are fetched from the REST API with `GET` requests:\n\n```\nGET /\u003cstoreName\u003e?sort=updated_at:asc\u0026size=100\u0026after=2000-01-01T00:00:00.000Z\u0026after_id=123\n```\n\nExplanations:\n\n- `sort=updated_at:asc` indicates that we want to sort the entities based on the date of last update\n- `size=100` indicates that we want 100 entities max\n- `after=2000-01-01T00:00:00.000Z\u0026after_id=123` indicates the offset (with an update date above `2000-01-01T00:00:00.000Z`, excluding the entity `123`)\n\nThe query parameters can be customized with the [`buildFetchParams`](#buildfetchparams) option.\n\nExpected response:\n\n```js\n{\n  data: [\n    {\n      id: 1,\n      version: 1,\n      updatedAt: '2000-01-01T00:00:00.000Z',\n      label: 'Dagger'\n    },\n    {\n      id: 2,\n      version: 12,\n      updatedAt: '2000-01-02T00:00:00.000Z',\n      label: 'Long sword'\n    },\n    {\n      id: 3,\n      version: -1, // tombstone\n      updatedAt: '2000-01-03T00:00:00.000Z',\n    }\n  ],\n  hasMore: true\n}\n```\n\nA fetch request will be sent for each store of the database, every X seconds (see the [fetchInterval](#fetchinterval) option).\n\n### Pushing changes\n\nEach successful readwrite transaction will be translated into an HTTP request, when the connection is available:\n\n| Operation                                                     | HTTP request                  | Body                                         |\n|---------------------------------------------------------------|-------------------------------|----------------------------------------------|\n| `db.add('items', { id: 1, label: 'Dagger' })`                 | `POST /items`                 | `{ id: 1, version: 1, label: 'Dagger' }`     |\n| `db.put('items', { id: 2, version: 2, label: 'Long sword' })` | `PUT /items/2`                | `{ id: 2, version: 3, label: 'Long sword' }` |\n| `db.delete('items', 3)`                                       | `DELETE /items/3`             |                                              |\n| `db.clear('items')`                                           | one `DELETE` request per item |                                              |\n\nSuccess must be indicated by an HTTP 2xx response. Any other response status means the change was not properly synced. You can customize the error handling behavior with the [`onpusherror`](#onpusherror) method.\n\nPlease see the Express server [there](https://github.com/darrachequesne/synceddb-todo-example/blob/main/express-server/index.js) for reference.\n\n## Alternatives\n\nHere are some alternatives that you might find interesting:\n\n- idb: https://github.com/jakearchibald/idb\n- Dexie.js: https://dexie.org/ (and its [ISyncProtocol](https://dexie.org/docs/Syncable/Dexie.Syncable.ISyncProtocol) part)\n- pouchdb: https://pouchdb.com/\n- Automerge: https://github.com/automerge/automerge\n- Yjs: https://github.com/yjs/yjs\n- Electric: https://electric-sql.com/\n\n## Miscellaneous\n\n- [Pagination with IndexedDB](https://github.com/darrachequesne/indexeddb-pagination)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdarrachequesne%2Fsynceddb","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdarrachequesne%2Fsynceddb","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdarrachequesne%2Fsynceddb/lists"}