{"id":28412216,"url":"https://github.com/electric-sql/d2ts","last_synced_at":"2025-06-24T12:31:42.001Z","repository":{"id":269949397,"uuid":"884778513","full_name":"electric-sql/d2ts","owner":"electric-sql","description":"Differential Dataflow in TypeScript","archived":false,"fork":false,"pushed_at":"2025-06-24T09:21:48.000Z","size":704,"stargazers_count":182,"open_issues_count":7,"forks_count":3,"subscribers_count":7,"default_branch":"main","last_synced_at":"2025-06-24T09:27:14.376Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/electric-sql.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-11-07T11:25:45.000Z","updated_at":"2025-06-22T21:35:47.000Z","dependencies_parsed_at":"2025-05-29T15:33:06.004Z","dependency_job_id":"ba0362b7-3161-4920-84dc-4699b95864b1","html_url":"https://github.com/electric-sql/d2ts","commit_stats":null,"previous_names":["electric-sql/d2ts"],"tags_count":11,"template":false,"template_full_name":null,"purl":"pkg:github/electric-sql/d2ts","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/electric-sql%2Fd2ts","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/electric-sql%2Fd2ts/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/electric-sql%2Fd2ts/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/electric-sql%2Fd2ts/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/electric-sql","download_url":"https://codeload.github.com/electric-sql/d2ts/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/electric-sql%2Fd2ts/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":261644519,"owners_count":23189069,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-06-02T20:15:30.232Z","updated_at":"2025-06-24T12:31:41.970Z","avatar_url":"https://github.com/electric-sql.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://electric-sql.com\" target=\"_blank\"\u003e\n    \u003cpicture\u003e\n      \u003csource media=\"(prefers-color-scheme: dark)\"\n          srcset=\"https://raw.githubusercontent.com/electric-sql/meta/main/identity/ElectricSQL-logo-next.svg\"\n      /\u003e\n      \u003csource media=\"(prefers-color-scheme: light)\"\n          srcset=\"https://raw.githubusercontent.com/electric-sql/meta/main/identity/ElectricSQL-logo-black.svg\"\n      /\u003e\n      \u003cimg alt=\"ElectricSQL logo\"\n          src=\"https://raw.githubusercontent.com/electric-sql/meta/main/identity/ElectricSQL-logo-black.svg\"\n      /\u003e\n    \u003c/picture\u003e\n  \u003c/a\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://github.com/electric-sql/d2ts/actions\"\u003e\u003cimg src=\"https://github.com/electric-sql/d2ts/actions/workflows/ci.yml/badge.svg\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://github.com/electric-sql/d2ts/blob/main/LICENSE\"\u003e\u003cimg src=\"https://img.shields.io/badge/license-Apache_2.0-green\" alt=\"License - Apache 2.0\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://github.com/electric-sql/d2ts/\n  ext/milestones\"\u003e\u003cimg src=\"https://img.shields.io/badge/status-alpha-orange\" alt=\"Status - Alpha\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://discord.electric-sql.com\"\u003e\u003cimg src=\"https://img.shields.io/discord/933657521581858818?color=5969EA\u0026label=discord\" alt=\"Chat - Discord\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://x.com/ElectricSQL\" target=\"_blank\"\u003e\u003cimg src=\"https://img.shields.io/twitter/follow/ElectricSQL.svg?style=social\u0026label=Follow @ElectricSQL\"\u003e\u003c/a\u003e\n\u003c/p\u003e\n\n# D2TS - Differential Dataflow in TypeScript \u003c!-- omit in toc --\u003e\n\nD2TS is a TypeScript implementation of [differential dataflow](https://github.com/TimelyDataflow/differential-dataflow) - a powerful data-parallel programming framework that enables incremental computations over changing input data.\n\nYou can use D2TS to build data pipelines that can be executed incrementally, meaning you can process data as it comes in, and only recompute the parts that have changed. This could be as simple as remapping data, or as complex as performing a full join combining two datasources where one is a computed aggregate.\n\nD2TS can be used in conjunction with [ElectricSQL](https://electric-sql.com) to build data pipelines on top of [ShapeStreams](https://electric-sql.com/docs/api/clients/typescript#shapestream) that can be executed incrementally.\n\nA D2TS pipeline is also fully type safe, inferring the types at each step of the pipeline, and supports auto-complete in your IDE.\n\n## Table of Contents\n\n- [Key Features](#key-features)\n- [Quick Start](#quick-start)\n- [Examples](#examples)\n- [API](#api)\n  - [D2 Graph Construction](#d2-graph-construction)\n  - [Input Streams](#input-streams)\n  - [Versions and Frontiers](#versions-and-frontiers)\n  - [MultiSet as a changeset](#multiset-as-a-changeset)\n  - [Operators](#operators)\n  - [Using SQLite Backend](#using-sqlite-backend)\n- [Implementation Details](#implementation-details)\n- [References](#references)\n\n## Key Features\n\n- **Incremental Processing**: Efficiently process changes to input data without recomputing everything\n- **Rich Operators**: Supports common operations with a pipeline API:\n  - [`buffer`](#buffer): Buffer and emit versions when they are complete\n  - [`concat`](#concat): Concatenate two streams\n  - [`consolidate`](#consolidate): Consolidates the elements in the stream at each version\n  - [`count`](#count): Count elements by key\n  - [`distinct`](#distinct): Remove duplicates\n  - [`filter`](#filter): Filter elements based on predicates\n  - [`iterate`](#iterate): Perform iterative computations\n  - [`join`](#join): Join two streams\n  - [`keyBy`](#keyBy): Key a stream by a property\n  - [`map`](#map): Transform elements in a stream\n  - [`reduce`](#reduce): Aggregate values by key\n  - [`rekey`](#rekey): Change the key of a keyed stream\n  - [`unkey`](#unkey): Remove keys from a keyed stream\n  - [`output`](#output): Output the messages of the stream\n  - [`pipe`](#pipe): Build a pipeline of operators enabling reuse of combinations of operators\n- **SQLite Integration**: Optional SQLite backend for persisting operator state allowing for larger datasets and resumable pipelines\n- **Type Safety**: Full TypeScript type safety and inference through the pipeline API\n\n## Quick Start\n\n### Installation\n\n```bash\nnpm install @electric-sql/d2ts\n```\n\n### Basic Usage\n\nHere's a simple example that demonstrates the core concepts:\n\n```typescript\nimport { D2, map, filter, debug, MultiSet, v } from '@electric-sql/d2ts'\n\n// Create a new D2 graph with initial frontier\n// The initial frontier is the lower bound of the version of the data that may\n// come in future.\nconst graph = new D2({ initialFrontier: 0 })\n\n// Create an input stream\n// We can specify the type of the input stream, here we are using number.\nconst input = graph.newInput\u003cnumber\u003e()\n\n// Build a simple pipeline that:\n// 1. Takes numbers as input\n// 2. Adds 5 to each number\n// 3. Filters to keep only even numbers\n// Pipelines can have multiple inputs and outputs.\nconst output = input.pipe(\n  map((x) =\u003e x + 5),\n  filter((x) =\u003e x % 2 === 0),\n  debug('output'),\n)\n\n// Finalize the pipeline, after this point we can no longer add operators or\n// inputs\ngraph.finalize()\n\n// Send some data\n// Data is sent as a MultiSet, which is a map of values to their multiplicity\n// Here we are sending 3 numbers (1-3), each with a multiplicity of 1\n// When you send data, you set the version number, here we are using 0\n// The key thing to understand is that the MultiSet represents a *change* to\n// the data, not the data itself. \"Inserts\" and \"Deletes\" are represented as\n// an element with a multiplicity of 1 or -1 respectively.\ninput.sendData(\n  0, // The version of the data\n  new MultiSet([\n    [1, 1],\n    [2, 1],\n    [3, 1],\n  ]),\n)\n\n// Set the frontier to version 1\n// The \"frontier\" is the lower bound of the version of the data that may come in future.\n// By sending a frontier, you are indicating that you are done sending data for any version less than the frontier and therefor D2TS operators that require them can process that data and output the results.\ninput.sendFrontier(1)\n\n// Process the data\ngraph.run()\n\n// Output will show:\n// 6 (from 1 + 5)\n// 8 (from 3 + 5)\n```\n\n### Using with ElectricSQL\n\nD2TS can be used in conjunction with [ElectricSQL](https://electric-sql.com) to build data pipelines on top of [ShapeStreams](https://electric-sql.com/docs/api/clients/typescript#shapestream) that can be executed incrementally.\n\nHere's an example of how to use D2TS with ElectricSQL:\n\n```typescript\nimport { D2, map, filter, output } from '@electric-sql/d2ts'\nimport { electricStreamToD2Input } from '@electric-sql/d2ts/electric'\nimport { ShapeStream } from '@electric-sql/client'\n\n// Create D2 graph\nconst graph = new D2({ initialFrontier: 0 })\n\n// Create D2 input\nconst input = graph.newInput\u003cany\u003e()\n\n// Configure the pipeline\ninput\n  .pipe(\n    map(([key, data]) =\u003e data.value),\n    filter(value =\u003e value \u003e 10),\n    // ... any other processing / joining\n    output((msg) =\u003e doSomething(msg))\n  )\n\n// Finalize graph\ngraph.finalize()\n\n// Create Electric stream (example)\nconst electricStream = new ShapeStream({\n  url: 'http://localhost:3000/v1/shape',\n  params: {\n    table: 'items',\n    replica: 'full',  // \u003c-- IMPORTANT!\n  }\n})\n\n// Connect Electric stream to D2 input\nelectricStreamToD2Input(electricStream, input)\n```\n\nThere is a complete example in the [./examples/electric](./examples/electric) directory.\n\n## Examples\n\nThere are a number of examples in the [./examples](./examples) directory, covering:\n\n- [Basic usage](./examples/basic.ts) (map and filter)\n- [\"Fruit processed\"](./examples/fruit-processed.ts) (reduce and consolidate)\n- [Joins between two streams](./examples/join.ts)\n- [Iterative computations](./examples/iterate.ts)\n- [Modeling \"includes\" using joins](./examples/includes.ts)\n- [ElectricSQL example](./examples/electric/) (using D2TS with ElectricSQL)\n\n## API\n\n### D2 graph construction\n\n```typescript\nconst graph = new D2({ initialFrontier: 0 })\n```\n\nThe `D2` constructor takes an optional `options` object with the following properties:\n\n- `initialFrontier`: The initial frontier of the graph, defaults to `0`\n\nAn instance of a D2 graph is used to build a dataflow graph, and has the following main methods:\n\n- `newInput\u003cT\u003e(): IStreamBuilder\u003cT\u003e`: Create a new input stream\n- `finalize(): void`: Finalize the graph, after this point no more operators or inputs can be added\n- `run(): void`: Process all pending versions of the dataflow graph\n\n### Input Streams\n\nInput streams are created using the `newInput\u003cT\u003e()` method, and have the following methods:\n\n- `sendData(version: Version | number | number[], data: MultiSet\u003cT\u003e): void`: Send data to the input stream\n- `sendFrontier(version: Antichain | Version | number | number[]): void`: Send a frontier to the input stream\n\n### Versions and Frontiers\n\nVersions are used to represent the version of the data, and are a lattice of integers. For most use cases you will only need to provide a single integer version, and all apis that take a version will work with a single integer. More advanced use cases may require the use of the latice to track multidimensional versions.\n\nFrontiers are used to represent the lower bound of the version of the data that may come in future, and are an antichain of versions. Again in most cases you can just use a single integer version to represent the frontier.\n\n#### Version\n\nThere is a `Version` class that represents a version, the prefered way to create a version is using the `v` helper function as this ensures that you reuse the same object for the same version making equality checks and comparisons more efficient:\n\n```typescript\nconst version = v(1)\n```\n\nMultidimensional versions are also supported, and are created using the `v` helper function:\n\n```typescript\nconst version = v([1, 2])\n```\n\nIn most cases you will only need to use a single integer version to represent the version which can be passed directly to the `sendData` and `sendFrontier` methods:\n\n```typescript\ninput.sendData(1, new MultiSet([[1, 1]]))\n```\n\n#### Antichain (frontier)\n\nAn `Antichain` is a set of versions that are disjoint, it is used to represent the frontier of the data. An antichain can be created using the `Antichain` constructor:\n\n```typescript\nconst frontier = new Antichain([v(1), v([2])])\n```\n\nIn most cases you will only need to use a single integer version to represent the frontier and can be passed directly to the `sendFrontier` method:\n\n```typescript\ninput.sendFrontier(1)\n```\n\n### MultiSet as a changeset\n\nA `MultiSet` is a map of values to their multiplicity. It is used to represent the changes to a collection.\n\nA MultiSet is created by passing an array of `[value, multiplicity]` pairs. Here we are creating a MultiSet with the values 1, 2, and 3, each with a multiplicity of 1:\n\n```typescript\nconst multiSet = new MultiSet([\n  [1, 1],\n  [2, 1],\n  [3, 1],\n])\n```\n\nMultiSets could be used to represent any object:\n\n```typescript\n// Here we have a MultiSet of new \"comments\" with the interface `Comment`\nconst multiSet = new MultiSet\u003cComment\u003e([\n  [{ id: '1', text: 'Hello, world!', userId: '321' }, 1],\n  [{ id: '2', text: 'Hello, world!', userId: '123' }, 1],\n])\n```\n\nAn important principle of D2TS is \"keyed\" MultiSets, where the `value` is a tuple of `[key, value]`.\n\n```typescript\n// Here we have a MultiSet of new \"comments\" but we have keyed them by the\n// `userId`\nconst multiSet = new MultiSet\u003c[string, Comment]\u003e([\n  [['321', { id: '1', text: 'Hello, world!', userId: '321' }], 1],\n  [['123', { id: '2', text: 'Hello, world!', userId: '123' }], 1],\n])\n```\n\nInserts and deletes are represented as an element with a multiplicity of 1 or -1 respectively.\n\n```typescript\n// Here we are inserting a one new comment and deleting one comment\nconst multiSet = new MultiSet\u003c[string, Comment]\u003e([\n  [['321', { id: '1', text: 'Hello, world!', userId: '321' }], 1],\n  [['123', { id: '2', text: 'Hello, world!', userId: '123' }], -1],\n])\n```\n\n### Operators\n\n#### buffer\n\n`buffer()`\n\nBuffers the elements of the stream, emitting a version when the buffer is complete.\n\n```typescript\nconst output = input.pipe(buffer())\n```\n\n#### concat\n\n`concat(other: IStreamBuilder\u003cT\u003e)`\n\nConcatenates two input streams - the output stream will contain the elements of both streams.\n\n```typescript\nconst output = input.pipe(concat(other))\n```\n\n#### consolidate\n\n`consolidate()`\n\nConsolidates the elements in the stream at each version, essentially it ensures the output stream is at the latest known _complete_ version.\n\n```typescript\nconst output = input.pipe(consolidate())\n```\n\n#### count\n\n`count()`\n\nCounts the number of elements in the stream by key\n\n```typescript\nconst output = input.pipe(\n  map((data) =\u003e [data.somethingToKeyOn, data]),\n  count(),\n)\n```\n\n#### debug\n\n`debug(name: string)`\n\nLogs the messages of the stream to the console, the name is used to identify the stream in the logs.\n\n```typescript\nconst output = input.pipe(debug('output'))\n```\n\n#### distinct\n\n`distinct()`\n\nRemoves duplicate values from the stream by key\n\n```typescript\nconst output = input.pipe(distinct())\n```\n\n#### filter\n\n`filter(predicate: (data: T) =\u003e boolean)`\n\nFilters the stream based on a predicate\n\n```typescript\nconst output = input.pipe(filter((x) =\u003e x % 2 === 0))\n```\n\n#### iterate\n\n`iterate\u003cT\u003e(f: (stream: IStreamBuilder\u003cT\u003e) =\u003e IStreamBuilder\u003cT\u003e)`\n\nPerforms iterative computations on a stream by creating a feedback loop. This allows you to repeatedly process data until it reaches a fixed point or meets specific conditions.\n\nThe `iterate` operator takes a function that defines the iteration step. Inside this function, you can apply any series of transformations to the stream, and the results will be fed back into the loop for further iterations.\n\n```typescript\n// This example repeatedly doubles numbers and includes previous values,\n// filtering out any values \u003e 50\nconst output = input.pipe(\n  iterate((stream) =\u003e\n    stream.pipe(\n      map((x) =\u003e x * 2),            // Double each value\n      concat(stream),               // Include original values\n      filter((x) =\u003e x \u003c= 50),       // Keep only values \u003c= 50\n      map((x) =\u003e [x, []]),          // Convert to keyable format\n      distinct(),                   // Remove duplicates \n      map((x) =\u003e x[0]),             // Convert back to simple values\n      consolidate(),                // Ensure consistent version updates\n    )\n  ),\n  debug('results')\n)\n```\n\nIn this example:\n1. The `iterate` function creates a feedback loop on the input stream\n2. Each value is doubled, then combined with all previous values\n3. Values greater than 50 are filtered out\n4. The remaining values are deduplicated and consolidated before the next iteration\n\nThe iteration will continue until no new values are produced (reaching a fixed point) or until the frontier advances beyond the iteration scope.\n\nCommon use cases for the `iterate` operator include:\n- Computing transitive closures in graph algorithms\n- Propagating values until convergence\n- Implementing fixed-point algorithms\n- Simulating recursive processes with bounded results\n\nThis powerful operator enables complex recursive computations while maintaining the incremental nature of differential dataflow.\n\n#### join\n\n`join(other: IStreamBuilder\u003cT\u003e, joinType?: JoinType)`\n\nJoins two keyed streams based on matching keys. The `joinType` parameter controls how the join behaves:\n\n- `'inner'` (default): Returns only records that have matching keys in both streams\n- `'left'`: Returns all records from the left stream, plus matching records from the right (with nulls for non-matches)\n- `'right'`: Returns all records from the right stream, plus matching records from the left (with nulls for non-matches)\n- `'full'`: Returns all records from both streams, with nulls for non-matches on either side\n\n```typescript\nconst input = graph.newInput\u003c[key: string, value: number]\u003e()\nconst other = graph.newInput\u003c[key: string, value: string]\u003e()\n\n// Inner join - only matching keys\nconst innerJoin = input.pipe(join(other, 'inner'))\n\n// Left join - all records from input, matching from other\nconst leftJoin = input.pipe(join(other, 'left'))\n\n// Right join - all records from other, matching from input\nconst rightJoin = input.pipe(join(other, 'right'))\n\n// Full join - all records from both streams\nconst fullJoin = input.pipe(join(other, 'full'))\n```\n\nThe join operation is type-safe, with appropriate nullable types for the different join types:\n\n```typescript\n// The two streams are initially keyed by the userId and commentId respectively\nconst comments = graph.newInput\u003c[commentId: string, comment: Comment]\u003e()\nconst users = graph.newInput\u003c[userId: string, user: User]\u003e()\n\n// Map the comments to be \"keyed\" by the user id\nconst commentsByUser = comments.pipe(\n  map(([commentId, comment]) =\u003e [comment.userId, comment] as [string, Comment]),\n)\n\n// Left join - keeps all comments, even those without matching users\nconst output = commentsByUser.pipe(\n  join(users, 'left'),\n  map(([userId, [comment, user]]) =\u003e {\n    // user can be null in a left join if there's no matching user\n    return [\n      comment.id,\n      {\n        ...comment,\n        userName: user?.name ?? 'Unknown User',\n      },\n    ]\n  }),\n)\n```\n\nWhen using SQLite persistence, you can supply the database as an additional parameter:\n\n```typescript\n// Using SQLite persistence\nconst db = new BetterSQLite3Wrapper(sqlite)\nconst persistedJoin = input.pipe(join(other, 'inner', db))\n```\n\n#### map\n\n`map\u003cU\u003e(f: (data: T) =\u003e U)`\n\nTransforms the elements of the stream using a function\n\n```typescript\nconst output = input.pipe(map((x) =\u003e x + 5))\n```\n\n#### output\n\n`output(messageHandler: (message: Message\u003cT\u003e) =\u003e void)`\n\nOutputs the messages of the stream\n\n```typescript\ninput.pipe(\n  output((message) =\u003e {\n    if (message.type === MessageType.DATA) {\n      console.log('Data message', message.data)\n    } else if (message.type === MessageType.FRONTIER) {\n      console.log('Frontier message', message.data)\n    }\n  }),\n)\n```\n\nThe message is a `Message\u003cT\u003e` object, with the structure:\n\n```typescript\ntype Message\u003cT\u003e =\n  | {\n      type: typeof MessageType.DATA\n      data: DataMessage\u003cT\u003e\n    }\n  | {\n      type: typeof MessageType.FRONTIER\n      data: FrontierMessage\n    }\n```\n\nA data messages represents a change to the output data, and has the following data payload:\n\n```typescript\ntype DataMessage\u003cT\u003e = {\n  version: Version\n  collection: MultiSet\u003cT\u003e\n}\n```\n\nA frontier message represents a new frontier, and has the following data payload:\n\n```typescript\ntype FrontierMessage = Version | Antichain\n```\n\n#### pipe\n\n`pipe(operator: (stream: IStreamBuilder\u003cT\u003e) =\u003e IStreamBuilder\u003cT\u003e)`\n\nPipes the stream through a series of operators\n\n```typescript\n// You can specify the input and output types for the pipeline.\n// Here we are specifying the input type as number and the output type as\n// string.\nconst composedPipeline = pipe\u003cnumber, string\u003e(\n  map((x) =\u003e x + 5),\n  filter((x) =\u003e x % 2 === 0),\n  map((x) =\u003e x.toString()),\n  debug('output'),\n)\n\nconst output = input.pipe(\n  map((x) =\u003e x + 1),\n  composedPipeline,\n)\n\n// Or as a function\n\nconst myPipe = (a: number, b: number) =\u003e\n  pipe\u003cnumber, number\u003e(\n    map((x) =\u003e x + a),\n    filter((x) =\u003e x % b === 0),\n    debug('output'),\n  )\n\nconst output = input.pipe(myPipe(5, 2))\n```\n\n#### reduce\n\n`reduce(f: (values: [T, multiplicity: number][]) =\u003e [R, multiplicity: number][])`\n\nPerforms a reduce operation on the stream grouped by key.\n\nThe function `f` takes an array of values and their multiplicities and returns an array of the result and their multiplicities.\n\n```typescript\n// Sum a values by key from the input stream\nconst output = input.pipe(\n  map((data) =\u003e [data.somethingToKeyOn, data.aValueToSum]),\n  reduce((values) =\u003e {\n    // `values` is an array of [value, multiplicity] pairs for a specific key\n    let sum = 0\n    for (const [value, multiplicity] of values) {\n      sum += value * multiplicity\n    }\n    return [[sum, 1]]\n  }),\n  output((message) =\u003e {\n    if (message.type === MessageType.DATA) {\n      // `message.data` is a MultiSet representing the changes to the output\n      // data\n      // In this example, the output stream will contain the change to the\n      // sum of the values for each key.\n      console.log(message.data)\n    }\n  }),\n)\n```\n\n#### Keying Operators\n\nD2TS provides a set of operators for working with keyed streams, which are useful for operations like joins and grouping.\n\n##### `keyBy`\n\nKeys a stream by a property of each element. This is useful for preparing data for joins or grouping operations.\n\n```typescript\nconst keyedStream = input.pipe(keyBy(item =\u003e item.id))\n```\n\n##### `unkey`\n\nRemoves the keys from a keyed stream, returning just the values.\n\n```typescript\nconst unkeyedStream = keyedStream.pipe(unkey())\n```\n\n##### `rekey`\n\nChanges the key of a keyed stream to a new key based on a property of the value.\n\n```typescript\nconst rekeyedStream = keyedStream.pipe(rekey(item =\u003e item.newKey))\n```\n\nExample usage with joins:\n\n```typescript\n// Transform comments into [issue_id, comment] pairs for joining\nconst commentsByIssue = inputComments.pipe(\n  rekey(comment =\u003e comment.issue_id)\n)\n\n// Join comments with issues\nconst issuesWithComments = issuesForProject.pipe(\n  join(commentsByIssue)\n)\n```\n\n### Using SQLite Backend\n\nFor persistence and larger datasets, a number of operators are provided that persist to SQLite:\n\n- `consolidate()`: Consolidates data into a single version\n- `count()`: Counts the number of elements in a collection\n- `distinct()`: Removes duplicates from a collection\n- `join()`: Joins two collections\n- `map()`: Transforms elements\n- `reduce()`: Aggregates values by key\n\nEach take a SQLite database as the final argument, for example:\n\n```typescript\n// Using better-sqlite3\nconst sqlite = new Database('./my_database.db')\nconst db = new BetterSQLite3Wrapper(sqlite)\n\nconst output = input.pipe(consolidate(db))\n```\n\nThe operators will automatically create the necessary tables and indexes to store the state of the operators. It is advised to use the same database for all operators to ensure that the state is stored in a single location.\n\nThe `BetterSQLite3Wrapper` is a wrapper around the `better-sqlite3` library that provides a unified interface for the operators. Other SQLite database drivers can be supported by implementing the `SQLiteDb` interface.\n\n## Implementation Details\n\nThis implementation started out as a TypeScript port of the [Materialize blog post](https://materialize.com/blog/differential-from-scratch/), but has diverged quite a bit, adopting a pipeline api pattern, persistence to SQLite, and a few other changes to make the DX better.\n\n1. Core data structures:\n\n   - `MultiSet`: Represents collections with multiplicities\n   - `Version`: Handles partially ordered versions\n   - `Antichain`: Manages frontiers\n   - `Index`: Stores versioned operator state\n\n2. Operators:\n\n   - Base operator classes in `src/operators/`\n   - SQLite variants in `src/sqlite/operators/`\n\n3. Graph execution:\n   - Dataflow graph management in `src/graph.ts` and `src/D2.ts`\n   - Message passing between operators\n   - Frontier tracking and advancement\n\n## References\n\n- [Differential Dataflow](https://github.com/MaterializeInc/differential)\n- [Differential Dataflow from Scratch](https://materialize.com/blog/differential-from-scratch/)\n- [Python Implementation](https://github.com/ruchirK/python-differential)\n- [DBSP](https://arxiv.org/abs/2203.16684) (very similar to Differential Dataflow)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Felectric-sql%2Fd2ts","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Felectric-sql%2Fd2ts","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Felectric-sql%2Fd2ts/lists"}