{"id":15066703,"url":"https://github.com/perry-mitchell/data-interchange","last_synced_at":"2026-01-04T00:33:26.175Z","repository":{"id":74696225,"uuid":"302002374","full_name":"perry-mitchell/data-interchange","owner":"perry-mitchell","description":"Read/Write fallback and multi-IO handler for databases/datasources","archived":false,"fork":false,"pushed_at":"2020-10-09T07:26:21.000Z","size":195,"stargazers_count":3,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-03-01T16:23:29.304Z","etag":null,"topics":["database","datasource","fallback","mysql","redis"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/perry-mitchell.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-10-07T10:32:46.000Z","updated_at":"2020-10-12T07:57:48.000Z","dependencies_parsed_at":"2023-02-25T09:00:24.725Z","dependency_job_id":null,"html_url":"https://github.com/perry-mitchell/data-interchange","commit_stats":{"total_commits":27,"total_committers":2,"mean_commits":13.5,"dds":0.07407407407407407,"last_synced_commit":"28be8e3f9f570a71c13ad9d88397c4eb310e7ea0"},"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/perry-mitchell%2Fdata-interchange","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/perry-mitchell%2Fdata-interchange/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/perry-mitchell%2Fdata-interchange/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/perry-mitchell%2Fdata-interchange/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/perry-mitchell","download_url":"https://codeload.github.com/perry-mitchell/data-interchange/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244166640,"owners_count":20409177,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["database","datasource","fallback","mysql","redis"],"created_at":"2024-09-25T01:10:56.074Z","updated_at":"2026-01-04T00:33:26.150Z","avatar_url":"https://github.com/perry-mitchell.png","language":"JavaScript","readme":"# Data Interchange\n\u003e Read/Write fallback and multi-IO handler for several databases/datasources\n\n![Multi-source R/W handler](diagram.png)\n\nA data-interchange handler (multi-read and multi-write) for NodeJS. Designed to help work with multiple databases, such as Redis (for fast storage) with MySQL (slow storage). Interchange adapters produced by this library provide `read` and `write` methods to allow to read from and write to all provided _sources_. You can also specify converters to convert subtle differences between formats.\n\n## Usage\n\nGrab the `createInterchange` method from the library to create an adapter. An adapter is basically an array of sources - each source being a R/W call to some data source. Read/Write calls can be synchronous or asynchronous, but the output of the interchange `read` and `write` methods are **always asynchronous**.\n\nAn interchanger can take care of a number of data-synchronising tasks in the process of reading and writing. When reading a value, if it is not found in a source, it'll try the next one, up-converting the format until it matches the expected output. It can also write this found value to other sources as it returns it. This is very useful for databases with ephemeral storage - Their datasets can be updated only when required - when a read or write is performed. No extra logic is required to handle this.\n\n### Example\n\nTake the following example - Let's read and write \"messages\" to multiple sources, each source having a subtle differences in format:\n\n```typescript\nimport { createInterchange } from \"data-interchange\";\n\ninterface MessageA {\n    id: string;\n    time: number;\n}\ninterface MessageB {\n    id: string;\n    ts: number;\n}\n\nconst readMessageA: (id: string) =\u003e MessageA = { /* ... */ };\nconst readMessageB: (id: string) =\u003e MessageB = { /* ... */ };\nconst writeMessageA: (msg: MessageA) =\u003e MessageA = { /* ... */ };\nconst writeMessageB: (msg: MessageB) =\u003e MessageB = { /* ... */ };\n\nconst { read, write } = createInterchange([\n    {\n        read: (id: string) =\u003e readMessageA(id),\n        write: (msg: MessageA) =\u003e writeMessageA(msg),\n        readError: (err) =\u003e ReadAction.Fallback\n    },\n    {\n        read: (id: string) =\u003e readMessageB(id),\n        write: (msg: MessageB) =\u003e writeMessageB(msg),\n        convert: {\n            read: (msg: MessageB): MessageA =\u003e ({\n                id: msg.id,\n                time: msg.ts\n            }),\n            write: (msg: MessageA): MessageB =\u003e ({\n                id: msg.id,\n                ts: msg.time\n            })\n        }\n    }\n]);\n\n// Read a message using ID 1\nconst readMessage: MessageA = await read(1);\n\n// Write a message\nconst messageToWrite: MessageA = {\n    id: 123,\n    time: Date.now()\n};\nawait write(messageToWrite);\n```\n\nOr if all types are the same:\n\n```typescript\nimport { WriteMode, createInterchange } from \"data-interchange\";\n\nconst readMessageA: (id: string) =\u003e MessageA = { /* ... */ };\nconst readMessageB: (id: string) =\u003e MessageA = { /* ... */ };\nconst writeMessageA: (msg: MessageA) =\u003e MessageA = { /* ... */ };\nconst writeMessageB: (msg: MessageA) =\u003e MessageA = { /* ... */ };\n\ncreateInterchange([\n    {\n        read: (id: string) =\u003e readMessageA(id),\n        write: (msg: MessageA) =\u003e writeMessageA(msg),\n        readError: (err) =\u003e ReadAction.Fallback\n    },\n    {\n        read: (id: string) =\u003e readMessageB(id),\n        write: (msg: MessageA) =\u003e writeMessageB(MessageA)\n    }\n], { writeMode: WriteMode.Parallel });\n```\n\n### Delayed writes\n\nYou can delay writes in certain sources by using the `writeWait` property. Setting it to `true` on a source will result in each write to a source to return immediately, with the writing occuring in the background.\n\n_If you expect reads to happen on the same source, you should use queues to prevent race conditions._\n\n### Queuing\n\nReads and writes can be queued so that, even if a read and a write occur almost simultaneously, data is read and written in order, even asynchronously.\n\nFor queuing to work on a source, specify either `queueReadKey` or `queueWriteKey`, or both. Although you can set them separately, you will mostly need to set them to the **same key** per resource (unique item). Reads use the `queueReadKey` while writes use the `queueWriteKey`, so if a queue is to block reads on a resource that's currently being written to, they should match.\n\nBoth of these keys can either be a `string`, or a function that returns a `string`. The function must be synchronous. The function will be provided the item being written or the ID being read.\n\nExample:\n\n```typescript\nconst source = {\n    read: () =\u003e { /* ... */ },\n    write: () =\u003e { /* ... */ },\n    queueReadKey: (id: string) =\u003e `item:${id}`,\n    queueWriteKey: (item: { id: string }) =\u003e `item:${item.id}`\n}\n```\n\nYou can also provide your own queue using an option:\n\n```typescript\ncreateInterchange([], { queue });\n```\n\nQueues can be created by using `createQueue`, but are essentially `ChannelQueue` instances from [`@buttercup/channel-queue`](https://github.com/buttercup/channel-queue).\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fperry-mitchell%2Fdata-interchange","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fperry-mitchell%2Fdata-interchange","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fperry-mitchell%2Fdata-interchange/lists"}