{"id":26833915,"url":"https://github.com/paulgilmartin/django-pgpubsub","last_synced_at":"2025-12-14T12:22:20.823Z","repository":{"id":39592829,"uuid":"449075974","full_name":"PaulGilmartin/django-pgpubsub","owner":"PaulGilmartin","description":"A distributed task processing framework for Django built on top of the Postgres NOTIFY/LISTEN protocol.","archived":false,"fork":false,"pushed_at":"2024-11-17T13:25:48.000Z","size":284,"stargazers_count":263,"open_issues_count":11,"forks_count":15,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-10-26T20:41:32.035Z","etag":null,"topics":["django","message-queue","postgres"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/PaulGilmartin.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"docs/contributing.rst","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-01-17T23:13:58.000Z","updated_at":"2025-08-06T07:21:56.000Z","dependencies_parsed_at":"2023-02-17T22:01:03.282Z","dependency_job_id":"c5277dde-30a6-4e18-8281-6ef7300fc96f","html_url":"https://github.com/PaulGilmartin/django-pgpubsub","commit_stats":null,"previous_names":["opus10/django-pgpubsub"],"tags_count":10,"template":false,"template_full_name":null,"purl":"pkg:github/PaulGilmartin/django-pgpubsub","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PaulGilmartin%2Fdjango-pgpubsub","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PaulGilmartin%2Fdjango-pgpubsub/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PaulGilmartin%2Fdjango-pgpubsub/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PaulGilmartin%2Fdjango-pgpubsub/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/PaulGilmartin","download_url":"https://codeload.github.com/PaulGilmartin/django-pgpubsub/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PaulGilmartin%2Fdjango-pgpubsub/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":27728261,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-12-14T02:00:11.348Z","response_time":56,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["django","message-queue","postgres"],"created_at":"2025-03-30T15:30:29.799Z","updated_at":"2025-12-14T12:22:20.531Z","avatar_url":"https://github.com/PaulGilmartin.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"django-pgpubsub\n===============\n\n``django-pgpubsub`` provides a framework for building an asynchronous\nand distributed message processing network on top of a Django application\nusing a PostgreSQL database. This is achieved by leveraging Postgres'\n[LISTEN/NOTIFY](https://www.postgresql.org/docs/current/sql-notify.html)\nprotocol to build a message queue at the database layer.\nThe simple user-friendly interface,\nminimal infrastructural requirements and the ability to leverage Postgres'\ntransactional behaviour to achieve exactly-once messaging, makes\n``django-pgpubsub`` a solid choice as a lightweight alternative to AMPQ\nmessaging services, such as\n[Celery](https://docs.celeryq.dev/en/stable/search.html?q=ampq)\n\n\nDocumentation\n-------------\n\nhttps://django-pgpubsub.readthedocs.io/en/latest/\n\n\nPrimary Authors\n---------------\n* [Paul Gilmartin](https://github.com/PaulGilmartin)\n* [Wesley Kendall](https://github.com/wesleykendall)\n\n\nHighlights\n==========\n\n- **Minimal Operational Infrastructure**: If you're already running a Django application\n  on top of a Postgres database, the installation of this library is the sum total\n  of the operational work required to implement a framework for a distributed\n  message processing framework. No additional frameworks or technologies\n  are required.\n\n- **Integration with Postgres Triggers (via django-pgtrigger)**:\n  To quote the [official](https://www.postgresql.org/docs/current/sql-notify.html)\n  Postgres docs:\n\n  *\"When NOTIFY is used to signal the occurrence of changes to a particular table,\n  a useful programming technique is to put the NOTIFY in a statement trigger that is triggered\n  by table updates.\n  In this way, notification happens automatically when the table is changed,\n  and the application programmer cannot accidentally forget to do it.\"*\n\n  By making use of the ``django-pgtrigger``\n  [library](https://pypi.org/project/django-pgtrigger/), ``django-pgpubsub``\n  offers a Django application layer abstraction of the trigger-notify Postgres\n  pattern. This allows developers to easily write python-callbacks which will\n  be invoked (asynchronously) whenever a custom ``django-pgtrigger`` is invoked.\n  Utilising a Postgres-trigger as the ground-zero for emitting a\n  message based on a database table event is far more robust than relying\n  on something at the application layer (for example, a ``post_save`` signal,\n  which could easily be missed if the ``bulk_create`` method was used).\n\n- **Lightweight Polling**: we make use of the Postgres ``LISTEN/NOTIFY``\n  protocol to have achieve notification polling which uses\n  [no CPU and no database transactions unless there is a message to read.](https://www.psycopg.org/docs/advanced.html#asynchronous-notifications)\n  (Note: This is true when using the `psycopg2` library, which is the default. `psycopg3` is supported,\n   but polling incurs a small CPU cost to fetch the notifications.\n   We hope to fix this in the future once `psycopg3` makes it easier to poll for notifications client side).\n\n- **Exactly-once notification processing**: ``django-pgpubsub`` can be configured so\n  that notifications are processed exactly once. This is achieved by storing\n  a copy of each new notification in the database and mandating that a notification\n  processor must obtain a postgres lock on that message before processing it.\n  This allows us to have concurrent processes listening to the same message channel\n  with the guarantee that no two channels will act on the same notification. Moreover,\n  the use of Django's ``.select_for_update(skip_locked=True)`` method allows\n  concurrent listeners to continue processing incoming messages without waiting\n  for lock-release events from other listening processes.\n\n- **Durability and Recovery**: ``django-pgpubsub`` can be configured so that\n  notifications are stored in the database before they're sent to be processed.\n  This allows us to replay any notification which may have been missed by listening\n  processes, for example in the event a notification was sent whilst the listening\n  processes were down.\n\n- **Atomicity**: The Postgres ``NOTIFY`` protocol respects the atomicity\n  of the transaction in which it is invoked. The result of this is that\n  any notifications sent using ``django-pgpubsub`` will be sent if and only if\n  the transaction in which it sent is successfully committed to the database.\n\nQuick start\n===========\n\nPrerequisites\n-------------\n\nBefore using this library, you must be running Django 2.2 (or later) on top\nof a (single) PostgreSQL 11 (or later) database.\n\n\nInstalling\n----------\n\n    pip install django-pgpubsub\n\n``django-pgpubsub`` ships with a ``Notification`` model. This table must\nbe added to the app's database via the usual django ``migrate`` command.\nWe should also add `pgpubsub` and `pgtrigger` into `INSTALLED_APPS`.\nAdditionally, if we wish to run the `pgpubsub` tests, we need to add\n`pgpubsub.tests` into `INSTALLED_APPS` too.\n\nMinimal Example\n---------------\n\nLet's get a brief overview of how to use ``pgpubsub`` to asynchronously\ncreate a ``Post`` row whenever an ``Author`` row is inserted into the\ndatabase. For this example, our notifying event will come from a\npostgres trigger, but this is not a requirement for all notifying events.\nA more detailed version of this example, and an example which\ndoes not use a postgres trigger, can be found in the\n**Documentation (by Example)** section below.\n\n**Define a Channel**\n\nChannels are the medium through which we send notifications.\nWe define our channel in our app's ``channels.py`` file as a dataclass\nas follows:\n\n```python\nfrom dataclasses import dataclass\n\nfrom pgpubsub.channel import TriggerChannel\nfrom pgpubsub.tests.models import Author\n\n\n@dataclass\nclass AuthorTriggerChannel(TriggerChannel):\n    model = Author\n```\n\n**Define a Listener**\n\nA *listener* is the function which processes notifications sent through a channel.\nWe define our listener in our app's ``listeners.py`` file as follows:\n\n```python\nimport datetime\n\nimport pgpubsub\nfrom pgpubsub.tests.channels import AuthorTriggerChannel\nfrom pgpubsub.tests.models import Author, Post\n\n\n@pgpubsub.post_insert_listener(AuthorTriggerChannel)\ndef create_first_post_for_author(old: Author, new: Author):\n    print(f'Creating first post for {new.name}')\n    Post.objects.create(\n        author_id=new.pk,\n        content='Welcome! This is your first post',\n        date=datetime.date.today(),\n    )\n```\n\n**Note that since ``AuthorTriggerChannel`` is a trigger-based channel, we need\nto perform a ``migrate`` command after first defining the above listener\nso as to install the underlying trigger in the database.**\n\nFinally, we must also ensure  that this listeners.py module is imported into the app's config\nclass. In this example, our app is calls \"tests\":\n\n```python\n# tests/apps.py\nfrom django.apps import AppConfig\n\n\nclass TestsConfig(AppConfig):\n    name = 'tests'\n\n    def ready(self):\n        import pgpubsub.tests.listeners\n```\n\n\n**Start Listening**\n\nTo have our listener function listen for notifications on the ``AuthorTriggerChannel``,\nwe use the ``listen`` management command:\n\n\n    ./manage.py listen\n\n\nNow whenever an ``Author`` is inserted into our database, our listener process creates\na ``Post`` object referencing that ``Author``:\n\nhttps://user-images.githubusercontent.com/18212082/165683416-b5cbeca1-ea94-4cd4-a5a1-81751e1b0feb.mov\n\nLive Demos\n==========\n\n`bulk_create` over several processes\n------------------------------------\n\nIn the below example we show how `pgpubsub` handles a bulk creation\nof ``Author`` objects when several processes are listening to the\n``AuthorTriggerChannel`` channel. For the sake of the below demonstration,\nwe added a `time.sleep(3)` statement into the `create_first_post_for_author`\nlistener function. Note how only one processes is able to process any given\nnotification:\n\nhttps://user-images.githubusercontent.com/18212082/165823588-df91e84a-47f2-4220-8999-8556665e3de3.mov\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpaulgilmartin%2Fdjango-pgpubsub","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpaulgilmartin%2Fdjango-pgpubsub","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpaulgilmartin%2Fdjango-pgpubsub/lists"}