{"id":27595685,"url":"https://github.com/giddie/elixir-postgresql-message-queue","last_synced_at":"2025-04-22T12:16:49.029Z","repository":{"id":284971604,"uuid":"956668278","full_name":"giddie/elixir-postgresql-message-queue","owner":"giddie","description":"A message queue for Elixir requiring only PostgreSQL","archived":false,"fork":false,"pushed_at":"2025-04-09T20:17:49.000Z","size":52,"stargazers_count":5,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-18T18:37:43.449Z","etag":null,"topics":["broadway","elixir","postgresql"],"latest_commit_sha":null,"homepage":"","language":"Elixir","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"0bsd","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/giddie.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-03-28T16:42:50.000Z","updated_at":"2025-04-09T20:17:52.000Z","dependencies_parsed_at":"2025-03-31T09:45:50.366Z","dependency_job_id":null,"html_url":"https://github.com/giddie/elixir-postgresql-message-queue","commit_stats":null,"previous_names":["giddie/elixir-postgresql-message-broker","giddie/elixir-postgresql-message-queue"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/giddie%2Felixir-postgresql-message-queue","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/giddie%2Felixir-postgresql-message-queue/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/giddie%2Felixir-postgresql-message-queue/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/giddie%2Felixir-postgresql-message-queue/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/giddie","download_url":"https://codeload.github.com/giddie/elixir-postgresql-message-queue/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250237844,"owners_count":21397403,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["broadway","elixir","postgresql"],"created_at":"2025-04-22T12:16:48.581Z","updated_at":"2025-04-22T12:16:49.021Z","avatar_url":"https://github.com/giddie.png","language":"Elixir","funding_links":[],"categories":[],"sub_categories":[],"readme":"# What?\n\nThis is a reference implementation of a **PostgreSQL-based Message Queue for\nElixir**.\n\n# Why?\n\nGreat question! Phoenix already has a [fantastic Pub/Sub\nsystem](https://hexdocs.pm/phoenix_pubsub), right? And Elixir already has great\nbuilt-in messaging primitives. But these options do not offer **durability**. In\nother words, if something fails, the message will be lost. There are no\n**guarantees** that the message will be delivered and processed as intended.\n\nBut we have [Oban](https://hexdocs.pm/oban) for that, right? Well, yes and no.\nOban is primarily a _job framework_, and the \"job\" concept it models is pretty\nheavy, with each type of job requiring its own module. Elixir's messaging is far\nmore lightweight. Can we find something more like Phoenix Pub/Sub, but with\ndelivery guarantees?\n\nWhat you need is a **message queue**. And the obvious choice (especially within\nthe BEAM ecosystem) is [RabbitMQ](https://www.rabbitmq.com/), coupled with\n[Broadway](https://hexdocs.pm/broadway/rabbitmq.html). And this is a genuinely\nfantastic combo, which you can see an example of in [my CQRS example\nrepo](https://github.com/giddie/elixir_cqrs_example). But it requires that you\ndeploy RabbitMQ (or whatever queue/broker you choose) alongside your\napplication, which adds complexity to your deployment.\n\nWouldn't it be great if we could have the benefits of a **simple, lightweight\nmessage queue** like the RabbitMQ/Broadway combo, but use **only PostgreSQL**?\nYep, that's what we have here.\n\n# How?\n\nIn a nutshell, a database table is used to store all messages, with a `queue`\ncolumn allowing us to maintain separate queues. Queue \"processors\" can then pull\nmessages out of these queues and deliver them according to a specified routing\nconfiguration.\n\nThe core functionality is in the [`Messaging`](/lib/messaging.ex) module, which\nis where you'll find functions to **broadcast messages** and **process\nmessages** from a queue:\n\n```elixir\niex\u003e Repo.transaction(fn -\u003e\n...\u003e   [%Message{type: \"Example.Event\", schema_version: 1, payload: %{\"one\" =\u003e 1}}]\n...\u003e   |\u003e Messaging.broadcast_messages!(to_queue: \"my_queue\")\n...\u003e end)\n{:ok, :ok}\n\niex\u003e Messaging.process_message_queue_batch(\"my_queue\")\n[info] Messaging: %PostgresqlMessageQueue.Messaging.Message{type: \"Example.Event\", schema_version: 1, payload: %{}, metadata: %{}}\n1\n```\n\nBut we don't want to **process message batches manually**, of course, so we use\na [`MessageQueueProcessor`](/lib/messaging/message_queue_processor.ex) instead:\n\n```elixir\niex\u003e Messaging.MessageQueueProcessor.start_link(queue: \"my_queue\")\n{:ok, #PID\u003c0.251.0\u003e}\n[info] Messaging: %PostgresqlMessageQueue.Messaging.Message{type: \"Example.Event\", schema_version: 1, payload: %{}, metadata: %{}}\n[info] Messaging.MessageQueueWatcher [#PID\u003c0.326.0\u003e] Subscribing queue: my_queue. Already subscribed: global.\n```\n\nThis server uses [Broadway](https://hexdocs.pm/broadway) under the hood, via a\ncustom [Broadway Producer](/lib/messaging/message_queue_broadway_producer.ex).\n\nWhat's this [`MessageQueueWatcher`](/lib/messaging/message_queue_watcher.ex)?\nIt's a GenServer that leverages [PostgreSQL's LISTEN\ndirective](https://www.postgresql.org/docs/current/sql-listen.html) to **wait\nfor a new message** to arrive in the queue. That way we can avoid constantly\n**polling the database**. (Actually it does this using a generic\n[`NotificationListener`](/lib/persistence/notification_listener.ex) server,\nwhich can subscribe any number of processes to whatever topics they want to\nlisten to, all on a **single database connection**.)\n\nOK, but how do we **consume these messages**? The simplest, default approach is\nvia application config:\n\n```elixir\nconfig :postgresql_message_queue, PostgresqlMessageQueue.Messaging,\n  broadcast_listeners: [\n    {MyContext.MyMessageHandler, [\"MyContext.Commands.*\", \"AnotherContext.Events.*\"]},\n    {MyLogger.EventLogger, [\"*.Events.*\"]}\n  ]\n```\n\nThis configuration allows you to specify a kind of **static routing table** for\nyour messages. Each listed module in the config must implement the\n`Messaging.MessageHandler` behaviour, which requires a single `handle_message/1`\nfunction. And then we list the message \"types\" the module is interested in. As\nyou can see, **wildcards** are supported, which allows you to fire all kinds of\nexotic messages at a message handler with very little boilerplate.\n\nHere's an example from the [ExampleUsage](/lib/example_usage.ex) module:\n\n```elixir\n@impl Messaging.MessageHandler\ndef handle_message(%Messaging.Message{\n      type: \"ExampleUsage.Events.Greeting\",\n      payload: %{\"greeting\" =\u003e greeting}\n    }) do\n  Logger.info(\"ExampleUsage: received greeting: #{greeting}\")\nend\n```\n\n# Handling Failures\n\nIf a message fails, by default it will be retried instantly. You can change this\nbehaviour by passing a `backoff_ms` option to `MessageQueueProcessor`,\ncontaining a function that defines your backoff curve. Here's a simple example:\n\n```elixir\nbackoff_ms = fn attempt when is_integer(attempt) -\u003e\n  base = 2 ** (attempt - 1) * 5 - 5\n  jitter = Enum.random(-base..base) |\u003e Integer.floor_div(20)\n  base + jitter\nend\n```\n\nSee it in context in the\n[Application](/lib/postgresql_message_broker/application.ex) module.\n\n# Can I delay a message for later processing?\n\nYes! Use `process_after` when broadcasting the message:\n\n```elixir\n[%Message{type: \"Example.Event\", schema_version: 1, payload: %{\"one\" =\u003e 1}}]\n|\u003e Messaging.broadcast_messages!(to_queue: \"my_queue\", process_after: {5, :minute})\n{:ok, :ok}\n```\n\nAny unit supported by `DateTime.add/4` is OK. You can also simply specify a\nDateTime struct, if you prefer.\n\n# Can I run more than one `MessageQueueProcessor`\n\nYeah, absolutely. In fact you _should_ do this if you need more than just a\nsimple global queue. Chances are you may need a few queues for, e.g. messages\nthat should be processed in strict order, whereas the global queue can be used\nfor general background processing where ordering doesn't matter. The global\nqueue can have a high `concurrency` value set to help throughput.\n\nYou can also have multiple nodes running `MessageQueueProcessors` for the same\nqueue. This works just fine because PostgreSQL handles row-level locking for the\nmessages. But bear in mind that if you do this, you cannot guarantee that\nmessages will be processed in order.\n\n# Can I use a custom routing table?\n\nYes: you have a couple of options.\n\n1. You can specify a `handler` opt for `MessageQueueProcessor`, which is a\n   function that will receive all messages in that queue. And you can do your\n   own custom routing.\n2. You can use `Messaging.deliver_messages_to_handlers!/2` as the handler, and\n   pass in a custom handler config:\n\n```elixir\nhandler_configs = [\n  {MyContext.MyMessageHandler, [\"MyContext.Commands.*\", \"AnotherContext.Events.*\"]},\n  {MyLogger.EventLogger, [\"*.Events.*\"]}\n]\nhandler = \u0026Messaging.deliver_messages_to_handlers!([\u00261], handler_configs)\nMessageQueueProcessor.start_link(queue: \"my_queue\", handler: handler)\n```\n\n# How do I use this in my project?\n\nRight now, the best way is to copy the modules you're interested in, rename as\nappropriate, and start the relevant GenServers:\n\n* `Persistence.NotificationListener`\n* `Messaging.MessageQueueWatcher`\n* `Messaging.MessageQueueProcessor`\n\n# Shouldn't this be a library?\n\nYeah, maybe. One thing that concerns me is that each project has different\nneeds. It's quite likely that this code will need a little work to adapt it for\nyour project. However, a library hides away the code, making it harder to adapt\nand modify. Essentially, the library starts shaping how the app should work. And\nthat's not necessarily a good thing.\n\nIt might be possible to shape this into a library that is flexible enough for\nany project, and I'm giving that some thought. But for now this is a code\nreference to help anyone who wants to copy what is helpful and adapt as needed.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgiddie%2Felixir-postgresql-message-queue","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgiddie%2Felixir-postgresql-message-queue","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgiddie%2Felixir-postgresql-message-queue/lists"}