{"id":13475604,"url":"https://github.com/imqueue/pg-pubsub","last_synced_at":"2025-04-13T07:52:09.464Z","repository":{"id":36786300,"uuid":"230342576","full_name":"imqueue/pg-pubsub","owner":"imqueue","description":"Reliable PostgreSQL LISTEN/NOTIFY with inter-process lock support","archived":false,"fork":false,"pushed_at":"2024-11-06T19:02:48.000Z","size":754,"stargazers_count":100,"open_issues_count":1,"forks_count":9,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-04-04T06:07:10.391Z","etag":null,"topics":["eventbus","events","inter-process-locking","ip-lock","listen","notifications","notify","postgres","pub","publish","pubsub","sub","subscribe"],"latest_commit_sha":null,"homepage":null,"language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"isc","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/imqueue.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-12-26T23:48:35.000Z","updated_at":"2024-12-28T18:48:59.000Z","dependencies_parsed_at":"2024-01-16T07:23:14.282Z","dependency_job_id":"6e09d0db-bd55-41d8-92ff-7995a30cb8a4","html_url":"https://github.com/imqueue/pg-pubsub","commit_stats":{"total_commits":143,"total_committers":3,"mean_commits":"47.666666666666664","dds":0.07692307692307687,"last_synced_commit":"dad04cb29ed641c4e7ff2d135186d1574e2e6062"},"previous_names":[],"tags_count":55,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/imqueue%2Fpg-pubsub","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/imqueue%2Fpg-pubsub/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/imqueue%2Fpg-pubsub/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/imqueue%2Fpg-pubsub/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/imqueue","download_url":"https://codeload.github.com/imqueue/pg-pubsub/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248681493,"owners_count":21144700,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["eventbus","events","inter-process-locking","ip-lock","listen","notifications","notify","postgres","pub","publish","pubsub","sub","subscribe"],"created_at":"2024-07-31T16:01:21.831Z","updated_at":"2025-04-13T07:52:09.438Z","avatar_url":"https://github.com/imqueue.png","language":"TypeScript","readme":"\u003ch1 align=\"center\"\u003e\n    @imqueue/pg-pubsub\n    \u003ca href=\"https://twitter.com/intent/tweet?text=Reliable%20PostgreSQL%20LISTEN/NOTIFY%20with%20inter-process%20lock%20support\u0026url=https://github.com/imqueue/pg-pubsub\u0026via=github\u0026hashtags=typescript,javascript,nodejs,postgres,developers\"\u003e\n        \u003cimg src=\"https://img.shields.io/twitter/url/http/shields.io.svg?style=social\" alt=\"Tweet\"\u003e\n    \u003c/a\u003e\n\u003c/h1\u003e\n\u003cdiv align=\"center\"\u003e\n    \u003ca href=\"https://travis-ci.com/imqueue/pg-pubsub\"\u003e\n        \u003cimg src=\"https://travis-ci.com/imqueue/pg-pubsub.svg?branch=master\" alt=\"Build Status\"\u003e\n    \u003c/a\u003e\n    \u003ca href=\"https://codebeat.co/projects/github-com-imqueue-pg-pubsub-master\"\u003e\n        \u003cimg src=\"https://codebeat.co/badges/579f6d7c-df61-4bc2-aa2e-d4fa9a3abf5a\" alt=\"Codebeat Grade\"\u003e\n    \u003c/a\u003e\n    \u003ca href=\"https://codeclimate.com/github/imqueue/pg-pubsub/maintainability\"\u003e\n        \u003cimg src=\"https://api.codeclimate.com/v1/badges/22de50ce1d4b1e44c0ad/maintainability\"\u003e\n    \u003c/a\u003e\n    \u003ca href=\"https://codeclimate.com/github/imqueue/pg-pubsub/test_coverage\"\u003e\n        \u003cimg src=\"https://api.codeclimate.com/v1/badges/22de50ce1d4b1e44c0ad/test_coverage\" /\u003e\n    \u003c/a\u003e\n    \u003ca href=\"https://coveralls.io/github/imqueue/pg-pubsub?branch=master\"\u003e\n        \u003cimg src=\"https://coveralls.io/repos/github/imqueue/pg-pubsub/badge.svg?branch=master\" alt=\"Coverage Status\"\u003e\n    \u003c/a\u003e\n    \u003ca href=\"https://rawgit.com/imqueue/core/master/LICENSE\"\u003e\n        \u003cimg src=\"https://img.shields.io/badge/license-ISC-blue.svg\" alt=\"Coverage Status\"\u003e\n    \u003c/a\u003e\n\u003c/div\u003e\n\u003chr\u003e\n\u003cp align=\"center\"\u003e\n    \u003cstrong\u003eReliable PostgreSQL LISTEN/NOTIFY with inter-process lock support\u003c/strong\u003e\n\u003c/p\u003e\n\u003chr\u003e\n\u003cdiv align=\"center\"\u003e\n    \u003cimg src=\"https://raw.githubusercontent.com/Mikhus/blob/master/pg-pubsub-ip.gif\" alt=\"pg-pubsub in action\"\u003e\n\u003c/div\u003e\n\u003chr\u003e\n\n## What Is This?\n\nThis library provides a clean way to use PostgreSQL \n[LISTEN](https://www.postgresql.org/docs/current/sql-listen.html) and \n[NOTIFY](https://www.postgresql.org/docs/current/sql-notify.html) commands\nfor its asynchronous mechanism implementation. It comes as a top-level wrapper\nover [node-postgres](https://www.npmjs.com/package/pg) and provides better,\ncleaner way to work with database notifications engine.\n\nTo make it clear - it solves several major problems you will fall into if \nyou're going to use LISTEN/NOTIFY in your node app:\n\n 1. **Reliable connections**. This library comes with handy reconnect support\n    out-of-the box, so all you need, is, probably to tune several settings if\n    you have special needs, like max retry limit or reconnection delay.\n 2. It provides **clean way working with channels**, so you may subscribe to\n    an exactly required channel with no need to do additional filtering\n    implementation on messages receive. BTW, it does nod hide from you\n    possibility to manage all messages in a single handler. You just choose\n    what you need.\n 3. The most important feature here is that this library comes with the first-class\n    implementation of **inter-process locking mechanism**, allowing avoiding \n    data duplication receive problem in scalable distributed architectures. It\n    means it allows you to define single-listener process across many similar\n    processes (which happens on scales) which would receive notifications\n    and with a guarantee that if it looses connection or dies - another similar\n    process replaces it as listener.\n 4. It comes with support of **graceful shutdown**, so you may don't care about\n    this.\n\n## Install\n\nAs easy as:\n\n~~~bash\nnpm i --save @imqueue/pg-pubsub\n~~~ \n\n## Usage \u0026 API\n\n### Environment\n\nIt supports passing environment variables to configure locker schema name to use\nand shutdown timeout.\n\n - **`PG_PUBSUB_SCHEMA_NAME`** - string, by default is `'pgip_lock'`\n - **`PG_PUBSUB_SHUTDOWN_TIMEOUT`** - number, by default is 1000, \n   in milliseconds\n\n### Importing, instantiation and connecting\n\n~~~typescript\nimport { PgPubSub } from '@imqueue/pg-pubsub';\n\nconst connectionString = 'postgres://user:pass@localhost:5432/dbname';\nconst pubSub = new PgPubSub({ connectionString, singleListener: false });\n\n(async () =\u003e {\n    await pubSub.connect();\n})();\n~~~\n\nWith such instantiation options natural behavior of PgPubSub will be as follows:\n\n[![Natural behavior](https://raw.githubusercontent.com/Mikhus/blob/master/pg-pubsub.gif)]()\n\n[See all options](https://github.com/imqueue/pg-pubsub/wiki/PgPubSubOptions).\n\n### Listening channels\n\nAfter connection established you may decide to listen for any numbers of \nchannels your application may need to utilize:\n\n~~~typescript\nawait pubSub.listen('UserChanged');\nawait pubSub.listen('OrderCreated');\nawait pubSub.listen('ArticleUpdated');\n~~~\n\nBTW, the most reliable way is to initiate listening on `'connect'` event:\n\n~~~typescript\npubSub.on('connect', async () =\u003e {\n    await Promise.all([\n        'UserChanged',\n        'OrderCreated',\n        'ArticleUpdated',\n    ].map(channel =\u003e pubSub.listen(channel)));\n});\n~~~\n\nNow, whenever you need to close/reopen connection, or reconnect occurred for any\nreason you'll be sure nothing broken.\n\n### Handling messages\n\nAll payloads on messages treated as JSON, so when the handler catches a\nmessage it is already parsed as JSON value, so you do not need to manage\nserialization/deserialization yourself.\n\nThere are 2 ways of handling channel messages - by using `'message'` event\nhandler on `pubSub` object, or using `pubSub.channels` event emitter and to\nlisten only particular channel for its messages. On message event fires first,\nchannels events fires afterwards, so this could be a good way if you need to\ninject and transform a particular message in synchronously manner before it\nwill come to a particular channel listeners.\n\nAlso `'message'` listener could be useful during implementation of handling of\ndatabase side events. It is easy imagine that db can send us messages into, so \ncalled, structural channels, e.g. `'user:insert'`, `'company:update'` or \n`'user_company:delete'`, where such names generated by some generic trigger\nwhich handles corresponding database operations and send updates to subscribers \nusing NOTIFY calls. In such case we can treat channel on application side\nas self-describable database operation change, which we can easily manage with\na single piece of code and keep following DRY.\n\n~~~typescript\n// using 'message' handler:\npubSub.on('message', (channel: string, payload: AnyJson) =\u003e {\n    // ... do the job\n    switch (channel) {\n        case 'UserChanged': {\n            // ... do some staff with user change event payload\n            break;\n        }\n        default: {\n            // do something with payload by default\n            break;\n        }\n    }\n});\n~~~\n\n~~~typescript\n// handling using channels\npubSub.channels.on('UserChanged', (payload: AnyJson) =\u003e {\n    // do something with user changed payload\n});\npubSub.channels.on('OrderCreated', (payload: AnyJson) =\u003e {\n    // do something with order created payload\n});\npubSub.channels.on('ArticleUpdated', (payload: AnyJson) =\u003e {\n    // do something with article updated payload\n});\n~~~\n\nOf course, it is better to set up listeners before calling `connect()` that it\nstarts handle payloads right up on connect time.\n\n### Publishing messages\n\nYou can send messages in many ways. For example, you may create\ndatabase triggers which would notify all connected clients with some\nspecific updates. Or you may use a database only as notifications engine\nand generate notifications on application level. Or you may combine both\napproaches - there are no limits!\n\nHere is how you can send notification with `PgPubSub` API (aka application \nlevel of notifications):\n\n~~~typescript\npubSub.notify('UserChanged', {\n    old: { id: 777, name: 'John Doe', phone: '555-55-55' },\n    new: { id: 777, name: 'Sam Peters', phone: '777-77-77' },\n});\n~~~\n\nNow all subscribers, who listening `'UserChanged'` channel will receive a given \npayload JSON object.\n\n## Single Listener (Inter Process Locking)\n\nThere are variety of many possible architectures to come up with when you're\nbuilding scalable distributed system. \n\nWith services on scale in such systems it might be a need to make sure only\nsingle service of much similar running is listening to particular database\nnotifications.\nHere why comes an idea of inter process (IP) locking mechanism, which would\nguarantee that only one process handles notifications and if it dies,\nnext one which is live will immediately handle listening.\n\nThis library comes with this option turned on by default. To make it work in\nsuch manner, you would need to skip passing `singleListener` option to\n`PgPubSub` constructor or set it to `true`:\n\n~~~typescript\nconst pubSub = new PgPubSub({ connectionString });\n// or, equivalently\nconst pubSub = new PgPubSub({ connectionString, singleListener: true });\n~~~ \n\nLocking mechanism utilizes the same connection and LISTEN/NOTIFY commands, so\nit won't consume any additional computing resources.\n\nAlso, if you already work with `pg` library in your application, and you\nhave a need to stay for some reason with that single connection usage, you \ncan bypass it directly as `pgClient` option, but that is not always a good idea.\nNormally, you have to understand what you are doing and why.\n\n~~~typescript\nconst pubSub = new PgPubSub({ pgClient: existingClient });\n~~~\n\n\u003e **NOTE:** With LISTEN connections it is really hard to utilize power of\n\u003e connection pool as long as it will require additional implementation of\n\u003e some connection switching mechanism using listen/unlisten and some specific\n\u003e watchers which may fall into need of re-implementing pools from scratch. So,\n\u003e that is why most of existing listen/notify solutions based on a single\n\u003e connection approach. And this library as well. It is just more simple and \n\u003e reliable.\n\nAlso, PgPubSub supports execution lock. This means all services become listeners\nin single listener mode but only one listener can process a notification. To\nenable this feature, you can bypass `executionLock` as option and set it to\n`true`. By default, this lock type is turned off.\n\n\u003e **NOTE:** Sometimes you might receive the notification with the same payloads\n\u003e in a very short period of time but execution lock will process them as the\n\u003e only notify message. If this important to you and your system will lave data\n\u003e leaks you need to ensure that payloads are unique.\n\n## [Full API Docs](https://github.com/imqueue/pg-pubsub/wiki)\n\nYou may read API docs on [wiki pages](https://github.com/imqueue/pg-pubsub/wiki)\n, read the code of the library itself, use hints in your IDE or generate HTML \ndocs with:\n\n~~~bash\ngit clone git@github.com:imqueue/pg-pubsub.git\ncd pg-pubsub\nnpm i\nnpm run doc\n~~~\n\n## Finally\n\nTry to run the following minimal example code of single listener scenario (do\nnot forget to set proper database connection string):\n\n~~~typescript\nimport { PgPubSub } from '@imqueue/pg-pubsub';\nimport Timer = NodeJS.Timer;\n\nlet timer: Timer;\nconst NOTIFY_DELAY = 2000;\nconst CHANNEL = 'HelloChannel';\n\nconst pubSub = new PgPubSub({\n    connectionString: 'postgres://postgres@localhost:5432/postgres',\n    singleListener: true,\n    // filtered: true,\n});\n\npubSub.on('listen', channel =\u003e console.info(`Listening to ${channel}...`));\npubSub.on('connect', async () =\u003e {\n    console.info('Database connected!');\n    await pubSub.listen(CHANNEL);\n    timer = setInterval(async () =\u003e {\n        await pubSub.notify(CHANNEL, { hello: { from: process.pid } });\n    }, NOTIFY_DELAY);\n});\npubSub.on('notify', channel =\u003e console.log(`${channel} notified`));\npubSub.on('end', () =\u003e console.warn('Connection closed!'));\npubSub.channels.on(CHANNEL, console.log);\npubSub.connect().catch(err =\u003e console.error('Connection error:', err));\n~~~\n\nOr take a look at other minimal code \n[examples](https://github.com/imqueue/pg-pubsub/tree/examples)\n\nPlay with them locally:\n\n~~~bash\ngit clone -b examples git://github.com/imqueue/pg-pubsub.git examples\ncd examples\nnpm i\n~~~\n\nNow you can start any of them, for example:\n\n~~~bash\n./node_modules/.bin/ts-node filtered.ts\n~~~\n\n## Contributing\n\nAny contributions are greatly appreciated. Feel free to fork, propose PRs, open\nissues, do whatever you think may be helpful to this project. PRs which passes\nall tests and do not brake tslint rules are first-class candidates to be\naccepted!\n\n## License\n\n[ISC](https://github.com/imqueue/pg-pubsub/blob/master/LICENSE)\n\nHappy Coding!\n","funding_links":[],"categories":["TypeScript"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fimqueue%2Fpg-pubsub","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fimqueue%2Fpg-pubsub","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fimqueue%2Fpg-pubsub/lists"}