{"id":19635853,"url":"https://github.com/tozd/node-reactive-postgres","last_synced_at":"2025-04-28T08:31:23.028Z","repository":{"id":33967557,"uuid":"163904303","full_name":"tozd/node-reactive-postgres","owner":"tozd","description":"Reactive queries for PostgreSQL","archived":false,"fork":false,"pushed_at":"2022-04-07T08:18:44.000Z","size":107,"stargazers_count":32,"open_issues_count":8,"forks_count":4,"subscribers_count":8,"default_branch":"master","last_synced_at":"2024-09-17T04:19:16.691Z","etag":null,"topics":["live","nodejs","npm","npm-package","observer","postgresql","query","reactivity","select","sql"],"latest_commit_sha":null,"homepage":"https://www.npmjs.com/package/reactive-postgres","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tozd.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null},"funding":{"tidelift":"npm/reactive-postgres","github":"mitar"}},"created_at":"2019-01-03T00:39:01.000Z","updated_at":"2024-03-10T11:40:01.000Z","dependencies_parsed_at":"2022-08-07T23:31:03.842Z","dependency_job_id":null,"html_url":"https://github.com/tozd/node-reactive-postgres","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tozd%2Fnode-reactive-postgres","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tozd%2Fnode-reactive-postgres/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tozd%2Fnode-reactive-postgres/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tozd%2Fnode-reactive-postgres/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tozd","download_url":"https://codeload.github.com/tozd/node-reactive-postgres/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224056571,"owners_count":17248329,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["live","nodejs","npm","npm-package","observer","postgresql","query","reactivity","select","sql"],"created_at":"2024-11-11T12:27:16.139Z","updated_at":"2024-11-11T12:27:17.004Z","avatar_url":"https://github.com/tozd.png","language":"JavaScript","funding_links":["https://tidelift.com/funding/github/npm/reactive-postgres","https://github.com/sponsors/mitar","https://tidelift.com/subscription/pkg/npm-reactive-postgres?utm_source=npm-reactive-postgres\u0026utm_medium=referral\u0026utm_campaign=enterprise\u0026utm_term=repo"],"categories":[],"sub_categories":[],"readme":"# reactive-postgres\n\nThis node.js package brings reactive (or live) queries to PostgreSQL. You can take an arbitrary `SELECT` query,\nusing multiple joins, data transformations, and even custom functions, and besides the initial set of\nresults also get real-time updates about any changes to those results. This can enable you to keep UI in sync\nwith the database in a reactive manner.\n\n## Installation\n\nThis is a node.js package. You can install it using NPM:\n\n```bash\n$ npm install reactive-postgres\n```\n\nThis package uses PostgreSQL. See [documentation](https://www.postgresql.org/docs/devel/tutorial-start.html)\nfor more information how to install and use it.\n\nRequires PostgreSQL 11 or newer.\n\n## reactive-postgres for enterprise\n\nAvailable as part of the Tidelift Subscription.\n\nThe maintainers of reactive-postgres and thousands of other packages are working with Tidelift to deliver commercial support and maintenance for the open source dependencies you use to build your applications. Save time, reduce risk, and improve code health, while paying the maintainers of the exact dependencies you use. [Learn more.](https://tidelift.com/subscription/pkg/npm-reactive-postgres?utm_source=npm-reactive-postgres\u0026utm_medium=referral\u0026utm_campaign=enterprise\u0026utm_term=repo)\n\n## Usage\n\nAs an event emitter:\n\n```js\nconst {Manager} = require('reactive-postgres');\n\nconst manager = new Manager({\n  connectionConfig: {\n    user: 'dbuser',\n    host: 'database.server.com',\n    database: 'mydb',\n    password: 'secretpassword',\n    port: 3211,\n  },\n});\n\n(async () =\u003e {\n  await manager.start();\n\n  const handle = await manager.query(`SELECT * FROM posts`, {\n    uniqueColumn: 'id',\n    mode: 'changed',\n  });\n\n  handle.on('start', () =\u003e {\n    console.log(\"query has started\");\n  });\n\n  handle.on('ready', () =\u003e {\n    console.log(\"initial data have been provided\");\n  });\n\n  handle.on('refresh', () =\u003e {\n    console.log(\"all query changes have been provided\");\n  });\n\n  handle.on('insert', (row) =\u003e {\n    console.log(\"row inserted\", row);\n  });\n\n  handle.on('update', (row, columns) =\u003e {\n    console.log(\"row updated\", row, columns);\n  });\n\n  handle.on('delete', (row) =\u003e {\n    console.log(\"row deleted\", row);\n  });\n\n  handle.on('error', (error) =\u003e {\n    console.error(\"query error\", error);\n  });\n\n  handle.on('stop', (error) =\u003e {\n    console.log(\"query has stopped\", error);\n  });\n\n  handle.start();\n})().catch((error) =\u003e {\n  console.error(\"async error\", error);\n});\n\nprocess.on('SIGINT', () =\u003e {\n  manager.stop().catch((error) =\u003e {\n    console.error(\"async error\", error);\n  }).finally(() =\u003e {\n    process.exit();\n  });\n});\n```\n\nAs a stream:\n\n```js\nconst {Manager} = require('reactive-postgres');\nconst through2 = require('through2');\n\nconst manager = new Manager({\n  connectionConfig: {\n    user: 'dbuser',\n    host: 'database.server.com',\n    database: 'mydb',\n    password: 'secretpassword',\n    port: 3211,\n  },\n});\n\nconst jsonStream = through2.obj(function (chunk, encoding, callback) {\n  this.push(JSON.stringify(chunk, null, 2) + '\\n');\n  callback();\n});\n\n(async () =\u003e {\n  await manager.start();\n\n  const handle = await manager.query(`SELECT * FROM posts`, {\n    uniqueColumn: 'id',\n    mode: 'changed',\n  });\n\n  handle.on('error', (error) =\u003e {\n    console.error(\"stream error\", error);\n  });\n\n  handle.on('close', () =\u003e {\n    console.log(\"stream has closed\");\n  });\n\n  handle.pipe(jsonStream).pipe(process.stdout);\n})().catch((error) =\u003e {\n  console.error(\"async error\", error);\n});\n\nprocess.on('SIGINT', () =\u003e {\n  manager.stop().catch((error) =\u003e {\n    console.error(\"async error\", error);\n  }).finally(() =\u003e {\n    process.exit();\n  });\n});\n```\n\n\n## Design\n\nReactive queries are implemented in the following manner:\n\n* For every reactive query, a `TEMPORARY TABLE` is created in the database\n  which serves as cache for query results.\n* Moreover, the query is `PREPARE`d, so that it does not have to be parsed again\n  and again.\n* Triggers are added to all query sources for the query, so that\n  when any of sources change, this package is notified using\n  `LISTEN`/`NOTIFY` that a source has changed, which can\n  potentially influence the results of the query.\n* Package waits for source changed events, and throttles them based\n  on `refreshThrottleWait` option. Once the delay expires, the package\n  creates a new temporary table with new query results. It compares\n  the old and new table and computes changes using another database query.\n* Changes are returned the package's client and exposed to the user.\n* The old temporary table is dropped and the new one is renamed to take\n  its place.\n\n## Performance\n\n* Memory use of node.js process is\n  [very low, is constant and does not grow with the number of rows in query results](https://mitar.github.io/node-pg-reactivity-benchmark/viewer.html?results/reactive-postgres-id.json).\n  Moreover, node.js process also does no heavy computation\n  and mostly just passes data around. All this is achieved by caching a query using a\n  temporary table in the database instead of the client, and using a database query\n  to compare new and old query results.\n* Computing changes is done through one query, a very similar query to the one used\n  internally by `REFRESH MATERIALIZED VIEW CONCURRENTLY` PostgreSQL command.\n* Based on the design, the time to compute changes and provide them to the client\n  seems to be the lowest when compared with other similar packages. For more\n  information about performance comparisons of this package and related packages,\n  see [this benchmark tool](https://github.com/mitar/node-pg-reactivity-benchmark) and\n  [results at the end](https://github.com/mitar/node-pg-reactivity-benchmark#results).\n* Because this package uses temporary tables, consider increasing\n  [`temp_buffers`](https://www.postgresql.org/docs/devel/runtime-config-resource.html#GUC-TEMP-BUFFERS)\n  PostgreSQL configuration so that there is more space for temporary tables in memory.\n  Consider [creating a dedicated tablespace](https://www.postgresql.org/docs/9.4/sql-createtablespace.html)\n  and configuring [`temp_tablespaces`](https://www.postgresql.org/docs/devel/runtime-config-client.html#GUC-TEMP-TABLESPACES).\n* You might consider increasing `refreshThrottleWait` for reactive queries for\n  which you can tolerate lower refresh rate and higher update latency, to decrease\n  load on the database. Making too many refreshes for complex queries can saturate\n  the database which then leads to even higher delays. Paradoxically, having higher\n  `refreshThrottleWait` could give you lower delay in comparison with a saturated state.\n* Multiple reactive queries share the same connection to the database.\n  So correctly configuring `maxConnections` is important. More connections there are,\n  higher load is on the database, but over more connections reactive queries can spread.\n  But higher load on the database can lead to its saturation.\n* Currently, when any of sources change in any manner, whole query is rerun\n  and results compared with cached results (after a throttling delay).\n  To improve this, refresh could be done only when it is known that a source change\n  is really influencing the results. Ideally, we could even compute changes to\n  results directly based on changes to sources. See\n  [#7](https://github.com/tozd/node-reactive-postgres/issues/7) for more information.\n\n## Limitations\n\n* Queries require an unique column which serves to identify rows and\n  changes to them. Which column this is is configured through\n  `uniqueColumn` query option. By default is `id`.\n* Order of rows in query results are ignored when determining changes.\n  Order still matters when selecting which rows are in query results\n  through `ORDER BY X LIMIT Y` pattern. If you care about order of\n  query results, order rows on the client.\n* Queries cannot contain placeholders or be prepared. You can use\n  `client.escapeLiteral(...)` function to escape values when constructing\n  a query.\n* To be able to determine if an UPDATE query really changed any source tables used in\n  your reactive query, source tables should have only columns of types which have an\n  equality operator defined (e.g., `json` column type does not). If this is not so,\n  it is just assumed that every UPDATE query makes a change. See\n  [#7](https://github.com/tozd/node-reactive-postgres/issues/16) for more information.\n\n## API\n\n### `Manager`\n\nManager manages a pool of connections to the database and organizes reactive queries\nover them. You should initialize one instance of it and use it for your reactive queries\n\n##### `constructor([options])`\n\nAvailable `options`:\n* `maxConnections`, default `10`: the maximum number of connections to the database\n  for reactive queries, the final number of connections is `maxConnections` + 1, for the\n  extra manager's connection\n* `connectionConfig`, default `{}`: PostgreSQL connection configuration,\n  [more information](https://node-postgres.com/api/client#new-client-config-object-)\n* `handleClass`: default `ReactiveQueryHandle`: a class to use for reactive query\n  handles returned by `query` method\n\n##### `async start()`\n\nInitializes the manager, the database, and establishes its connection.\nIt emits `'start'` event.\n\n##### `async stop([error])`\n\nStops all reactive queries and the manager itself.\nIt emits `'stop'` event. Accepts optional `error` argument which is passed\nas payload in `'stop'` event.\n\n##### `async query(query[, options={}])`\n\nConstructs a new reactive query and returns a handle.\n\n`query` can be any arbitrary [`SELECT`](https://www.postgresql.org/docs/devel/sql-select.html),\n[`TABLE`](https://www.postgresql.org/docs/10/sql-select.html#SQL-TABLE), or\n[`VALUES`](https://www.postgresql.org/docs/10/sql-values.html) command, adhering to\n[limitations](#limitations).\n\nAvailable `options`:\n* `uniqueColumn`, default `id`: the name of an unique column in query results, used\nto identify rows and changes to them\n* `refreshThrottleWait`, default 100: this option controls that refresh can happen at most once\nper every `refreshThrottleWait` milliseconds\n  * this introduces a minimal delay between a source change and a refresh, you can\n    control this delay based on requirements for a particular query\n  * lower this value is, higher the load on the database will be, higher it is, lower the load\n    will be\n* `mode`, default `changed`: in which mode to operate, it can be:\n  * `columns`: for every query results change, provide only which row and columns changed\n  * `changed`: for every query results change, provide new values for changed columns, too\n  * `full`: for every query results change, provide full changed rows,\n    both columns which have changed and those which have not\n* `types`, default `null`: [custom type parsers](https://node-postgres.com/features/queries#types)\n\n##### `'start'` event `()`\n\nEvent emitted when manager starts successfully.\n\n##### `'connect'` event `(client)`\n\nEvent emitted when a new [PostgreSQL client](https://node-postgres.com/api/client) is created\nand connected. `client` is provided as an argument.\n\n##### `'disconnect'` event `(client)`\n\nEvent emitted when a PostgreSQL client is disconnected. `client` is provided as an argument.\n\n##### `'error'` event `(error[, client])`\n\nEvent emitted when there is an error at the manager level. `error` is provided as an argument.\nIf the error is associated with a PostgreSQL client, the `client` is provided as well.\n\n##### `'stop'` event `([error])`\n\nEvent emitted when the manager stops. If it stopped because of an error,\nthe `error` is provided as an argument.\n\n### `ReactiveQueryHandle`\n\nReactive query handles can be used as an [`EventEmitter`](https://nodejs.org/api/events.html#events_class_eventemitter)\nor a [`Readable` stream](https://nodejs.org/api/stream.html#stream_readable_streams),\nbut not both.\n\nConstructor is seen as private and you should not create instances of `ReactiveQueryHandle`\nyourself but always through `Manager`'s `query` method. The method also passes all options\nto `ReactiveQueryHandle`.\n\n#### `EventEmitter` API\n\nThere are more methods, properties, and options available through the\n[`EventEmitter` base class](https://nodejs.org/api/events.html#events_class_eventemitter).\n\n##### `async start()`\n\nInitializes the reactive query and starts observing the reactive query, emitting events for\ninitial results data and later changes to results data.\nIt emits `'start'` event.\n\nHaving this method separate from constructing a reactive query allows attaching event handlers\nbefore starting, so that no events are missed.\n\n##### `async stop([error])`\n\nStops the reactive query.\nIt emits `'stop'` event. Accepts optional `error` argument which is passed\nas payload in `'stop'` event.\n\n##### `async refresh()`\n\nForces the refresh of the reactive query, computation of changes, and emitting\nrelevant events. This can override `refreshThrottleWait` option which otherwise\ncontrols the minimal delay between a source change and a refresh.\n\n##### `async flush()`\n\nWhen operating in `changed` or `full` mode, changes are batched together before\na query to fetch data is made. This method forces the query to be made using\ncurrently known changes instead of waiting for a batch as configured by the\n`batchSize` option.\n\n##### `on(eventName, listener)` and `addListener(eventName, listener)`\n\nAdds the `listener` function to the end of the listeners array for the event named `eventName`.\n[More information](https://nodejs.org/api/events.html#events_emitter_on_eventname_listener).\n\n##### `once(eventName, listener)`\n\nAdds a *one-time* `listener` function for the event named `eventName`.\n[More information](https://nodejs.org/api/events.html#events_emitter_once_eventname_listener).\n\n##### `off(eventName, listener)` and `removeListener(eventName, listener)`\n\nRemoves the specified `listener` from the listener array for the event named `eventName`.\n[More information](https://nodejs.org/api/events.html#events_emitter_removelistener_eventname_listener).\n\n##### `'start'` event `()`\n\nEvent emitted when reactive query starts successfully.\n\n##### `'ready'` event `()`\n\nEvent emitted when all events for initial results data have been emitted.\nLater events are about changes to results data.\n\n##### `'refresh'` event `()`\n\nEvent emitted when a refresh has finished and all events for changes to results data\nas part of one refresh have been emitted.\n\n##### `'insert'` event `(row)`\n\nEvent emitted when a new row has been added to the reactive query.\n`row` is provided as an argument. In `columns` mode, `row` contains only the value\nof the unique column of the row which has been inserted. In `changed` and `full` modes,\n`row` contains full row which has been inserted.\n\n##### `'update'` event `(row, columns)`\n\nEvent emitted when a row has been updated.\n`row` is provided as an argument. In `columns` mode, `row` contains only the value\nof the unique column of the row which has been updated. In `changed` mode, `row`\ncontains also data for columns which have changed. In `full` mode, `row`\ncontains the full updated row, both columns which have changed and those which have not.\n`columns` is a list of columns which have changed.\n\n##### `'delete'` event `(row)`\n\nEvent emitted when a row has been deleted. `row` is provided as an argument.\nIn `columns` mode, `row` contains only the value\nof the unique column of the row which has been deleted. In `changed` and `full` modes,\n`row` contains full row which has been deleted.\n\n##### `'error'` event `(error)`\n\nEvent emitted when there is an error at the reactive query level.\n`error` is provided as an argument.\n\n##### `'stop'` event `([error])`\n\nEvent emitted when the reactive query stops. If it stopped because of an error,\nthe `error` is provided as an argument.\n\n#### `Readable` stream API\n\nThere are more methods, properties, and options available through the\n[`Readable` stream base class](https://nodejs.org/api/stream.html#stream_readable_streams).\n\nWhen operating as a stream, the reactive handle is producing objects\ndescribing a change to the reactive query. They are of the following\nstructure:\n\n```js\n{\n  op: \u003cevent name\u003e,\n  ... \u003cpayload\u003e ...\n}\n```\n\n`event name` matches names of events emitted when operating as an event emitter.\nArguments of those events are converted to object's payload.\n\nStream supports backpressure and if consumer of the stream reads slower than\nwhat `refreshThrottleWait` dictates should be the minimal delay between refreshes,\nfurther refreshing is paused until stream is drained.\n\n##### `read([size])`\n\nReads the next object describing a change to the reactive query, if available.\n[More information](https://nodejs.org/api/stream.html#stream_readable_read_size).\n\n##### `pipe(destination[, options])`\n\nThe method attaches the stream to the `destination`, using `options`.\n[More information](https://nodejs.org/api/stream.html#stream_readable_pipe_destination_options).\n\n##### `destroy([error])`\n\nDestroy the stream and stop the reactive query. It emits `'error'` (if `error` argument\nis provided, which is then passed as payload in `'error'` event) and `'close'` events.\n[More information](https://nodejs.org/api/stream.html#stream_readable_destroy_error).\n\n##### 'data' event `(chunk)`\n\nEvent emitted whenever the stream is relinquishing ownership of a chunk\nof data to a consumer.\n`chunk` is provided as an argument.\n[More information](https://nodejs.org/api/stream.html#stream_event_data).\n\n##### 'readable' event `()`\n\nEvent emitted when there is data available to be read from the stream.\n[More information](https://nodejs.org/api/stream.html#stream_event_readable).\n\n##### 'error' event `(error)`\n\nEvent emitted when there is an error at the reactive query level.\n`error` is provided as an argument.\n[More information](https://nodejs.org/api/stream.html#stream_event_error_1).\n\n##### 'close' event `()`\n\nEvent emitted when the stream has been destroyed.\n[More information](https://nodejs.org/api/stream.html#stream_event_close_1).\n\n## Related projects\n\n* [pg-live-select](https://github.com/numtel/pg-live-select) – when the query's sources change, the client reruns the\n  query to obtain updated rows, which are identified through hashes of their full content and client then computes changes,\n  on the other hand, this package maintains cached results of the query in a temporary table in the database\n  and just compares new results with cached results and return updates to the client\n* [pg-live-query](https://github.com/nothingisdead/pg-live-query) – adds revision columns to sources and additional\n  temporary table to store latest revisions for results, moreover, it rewrites queries to expose those additional\n  revision columns, all this then allows simpler determination of what has changed when the query is refreshed,\n  inside the database through a query, but in benchmarks it seems all this additional complexity does not really\n  make things faster in comparison with just comparing new results with cached results directly, which is what this\n  package does\n* [pg-query-observer](https://github.com/Richie765/pg-query-observer) – it seems like a bit cleaned and updated version\n  of `pg-live-select`, but buggy and does not work with multiple parallel queries\n* [pg-reactivity-benchmark](https://github.com/mitar/node-pg-reactivity-benchmark) – a benchmark for this and above\n  mentioned packages\n* [supabase realtime](https://github.com/supabase/realtime) – it broadcasts changes in PostgreSQL over WebSockets\n  by observing write-ahead log, but it does not support arbitrary reactive queries, just changes to tables themselves,\n  so it is more of a database replication over WebSockets\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftozd%2Fnode-reactive-postgres","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftozd%2Fnode-reactive-postgres","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftozd%2Fnode-reactive-postgres/lists"}