{"id":20613228,"url":"https://github.com/robiningelbrecht/drupal-amqp-rabbitmq","last_synced_at":"2026-03-04T16:31:27.303Z","repository":{"id":57307176,"uuid":"479046811","full_name":"robiningelbrecht/drupal-amqp-rabbitmq","owner":"robiningelbrecht","description":"This repository aims to illustrate how to set up AMQP within Drupal","archived":false,"fork":false,"pushed_at":"2022-05-10T18:44:40.000Z","size":3909,"stargazers_count":5,"open_issues_count":1,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-04-15T07:13:39.653Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"PHP","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/robiningelbrecht.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-04-07T15:37:08.000Z","updated_at":"2025-02-19T11:06:15.000Z","dependencies_parsed_at":"2022-08-27T03:08:44.288Z","dependency_job_id":null,"html_url":"https://github.com/robiningelbrecht/drupal-amqp-rabbitmq","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/robiningelbrecht/drupal-amqp-rabbitmq","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/robiningelbrecht%2Fdrupal-amqp-rabbitmq","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/robiningelbrecht%2Fdrupal-amqp-rabbitmq/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/robiningelbrecht%2Fdrupal-amqp-rabbitmq/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/robiningelbrecht%2Fdrupal-amqp-rabbitmq/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/robiningelbrecht","download_url":"https://codeload.github.com/robiningelbrecht/drupal-amqp-rabbitmq/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/robiningelbrecht%2Fdrupal-amqp-rabbitmq/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30086451,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-04T15:40:14.053Z","status":"ssl_error","status_checked_at":"2026-03-04T15:40:13.655Z","response_time":59,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-16T11:09:10.900Z","updated_at":"2026-03-04T16:31:27.280Z","avatar_url":"https://github.com/robiningelbrecht.png","language":"PHP","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003ch1 align=\"center\"\u003eDrupal AMQP example\u003c/h1\u003e\n\n\u003cp align=\"center\"\u003e\n\t\u003cimg src=\"https://github.com/robiningelbrecht/drupal-amqp-rabbitmq/raw/master/readme/rabbitmq.png\" alt=\"RabbitMQ\"\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n\u003ca href=\"https://github.com/robiningelbrecht/drupal-amqp-rabbitmq/actions/workflows/ci.yml\"\u003e\u003cimg src=\"https://github.com/robiningelbrecht/drupal-amqp-rabbitmq/actions/workflows/ci.yml/badge.svg\" alt=\"CI/CD\"\u003e\u003c/a\u003e\n\u003ca href=\"https://codecov.io/gh/robiningelbrecht/drupal-amqp-rabbitmq\"\u003e\u003cimg src=\"https://codecov.io/gh/robiningelbrecht/drupal-amqp-rabbitmq/branch/master/graph/badge.svg?token=QUZxuZ49V4\" alt=\"codecov.io\"\u003e\u003c/a\u003e\n\u003ca href=\"https://github.com/robiningelbrecht/drupal-amqp-rabbitmq/blob/master/LICENSE\"\u003e\u003cimg src=\"https://img.shields.io/github/license/robiningelbrecht/continuous-integration-example?color=428f7e\u0026logo=open%20source%20initiative\u0026logoColor=white\" alt=\"License\"\u003e\u003c/a\u003e\n\u003ca href=\"https://php.net/\"\u003e\u003cimg src=\"https://img.shields.io/packagist/php-v/robiningelbrecht/drupal-amqp-rabbitmq/dev-master?color=777bb3\u0026logo=php\u0026logoColor=white\" alt=\"PHP\"\u003e\u003c/a\u003e\n\u003c/p\u003e\n\n------\n\nThis repository aims to illustrate how to set up AMQP within Drupal. It contains a base structure with some working examples that use CommandHandlers to handle AMQP messages.\n\n## Installation\n\n* Start by installing [Docker](https://docs.docker.com/get-docker/) and [Lando](https://docs.lando.dev/getting-started/)\n* Clone this repository `git clone git@github.com:robiningelbrecht/drupal-amqp-rabbitmq.git`\n* Run `lando start` to build the necessary docker containers\n* Run `lando composer install` to download vendor dependencies\n* Make sure following config is added to `settings.php`\n\n```php\n$databases['default']['default'] = [\n  'database' =\u003e 'drupal9',\n  'username' =\u003e 'drupal9',\n  'password' =\u003e 'drupal9',\n  'prefix' =\u003e '',\n  'host' =\u003e 'database',\n  'port' =\u003e '',\n  'namespace' =\u003e 'Drupal\\\\Core\\\\Database\\\\Driver\\\\mysql',\n  'driver' =\u003e 'mysql',\n];\n$settings['config_sync_directory'] = '../config/sync';\n\n$settings['amqp_credentials'] = [\n  'host' =\u003e '172.21.0.3', // The AMQP host IP address is outputted in your CLI while running `lando start`\n  'port' =\u003e '5672',\n  'username' =\u003e 'guest',\n  'password' =\u003e 'guest',\n  'vhost' =\u003e '/',\n  'api' =\u003e 'http://rabbit.lndo.site/',\n];\n```\n\n* Import the database dump by running `lando drush sql-cli \u003c init.sql`\n\n## The basic idea and setup\n\nThere are basically 3 important terms to keep in mind:\n\n* **Worker**: A specific class that processes a message, also handles failures in case it could not process the message\n* **Queue**: A class that represents a RabbitMQ queue, allows for messages to be pushed to the corresponding queue. Each queue is linked to a worker\n* **Consumer**: Process that consumes a specific queue and its messages. Each queue can have zero or more consumers\n\nThe `amqp` module provides a basic framework that allows you to\n\n* Define queues and workers\n* Push messages to queues\n* Consume queues with a *drush* command\n\n\u003cimg src=\"https://github.com/robiningelbrecht/drupal-amqp-rabbitmq/raw/master/readme/rmq-drupal.svg\" alt=\"RabbitMQ\"\u003e\n\n## Pushing messages and consuming them\n\nThe `amqp` module contains a `SimpleQueue` and a `SimpleQueueWorker`. Let's take a look\nat an example of pushing and consuming messages:\n\n\u003cimg src=\"https://github.com/robiningelbrecht/drupal-amqp-rabbitmq/raw/master/readme/consume-push-example.gif\" alt=\"Consume - Push example\"\u003e\n\n## Adding a new queue\n\nIt's recommended to add a queue for each type of task, for example:\n\n* Sending out notifications: `send-notification-queue`\n* Migrating articles: `migrate-article-queue`\n* Calculate product prices: `calculate-product-price-queue`\n* ...\n\nThis approach ensures that tasks of one type cannot block other ones. It also has the advantage\nthat you can log failed messages on the corresponding failed queues of each queue:\n\n* `send-notification-queue-failed`\n* `migrate-article-queue-failed`\n* `calculate-product-price-queue-failed`\n\n\nTo declare a new queue, just add a new entry to your `services.yml` and tag it with `ampq_queue`:\n\n```yaml\n  Drupal\\your_module\\Queue\\NewQueue:\n    autowire: true\n    tags:\n      - { name: amqp_queue }\n```\n\nMake sure this class extends `BaseQueue`, so you don't have to bother queueing messages yourself.\n\n@TODO: Explain how to push message to Q\n\n### Push a message to it's corresponding failed Q\n\nIf, fore some reason, a message could not be processed, you might want to log it somewhere.\nA \"failed queue\" could be a solution here.\\\nTo push a message to it's corresponding failed queue, you can use the `FailedQueueFactory`:\n\n```php\n  $this-\u003efailedQueueFactory-\u003ebuildFor($queue)-\u003equeue(message);\n```\n\nThis factory can for example be used in the `processFailure` callback of your worker:\n\n```php\n  public function processFailure(Envelope $envelope, AMQPMessage $message, \\Throwable $exception, Queue $queue): void\n  {\n    /** @var Command $command */\n    $command = $envelope;\n    $command-\u003esetMetaData([\n      'exceptionMessage' =\u003e $exception-\u003egetMessage(),\n      'traceAsString' =\u003e $exception-\u003egetTraceAsString(),\n    ]);\n\n    $failedQueue = $this-\u003efailedQueueFactory-\u003ebuildFor($queue)-\u003equeue($command);\n  }\n```\n\n**note**: a failed queue has no worker attached to it, and thus, cannot be consumed. This means\nthat the messages will stay on the queue until they are manually deletd.\n\n### Use a delayed Q to postpone consuming a message\n\nIn some more advanced use cases you might want to delay the consumption of messsages, for example:\n\n* a digist mail that summarizes all content changes occured in the last 30 minutes\n* requeue a failed message automatically after 15 seconds\n* ...\n\nYou can achieve this by pushing the message to it's correspondng delayed queue:\n\n```php\n  $this-\u003edelayedQueueFactory-\u003ebuildWithDelayForQueue(15, $queue)-\u003equeue($message);\n```\n\nFor a delayed queue to work properly you'll have to do two things:\n\n* Add a new exchange with the name `dlx`\n* Make sure the queue is defined as a binding on the `dlx` exchange, where the\nrouting key of the binding is the command queue name to where it has to be routed.\n\n\u003cp align=\"center\"\u003e\n\t\u003cimg src=\"https://github.com/robiningelbrecht/drupal-amqp-rabbitmq/raw/master/readme/dlx-binding-example.png\" width=\"400\" alt=\"Dlx binding example\"\u003e\n\u003c/p\u003e\n\n## Define a new CommandHandler\n\nI like to use Commands and CommandHandlers to persist changes to the database. That is basically what\nthe `cqrs` module is for. It provides a simple framework that\n\n* Allows you to define new commands and their corresponding command handlers\n* Allows you to push messages to command queues\n* Provides a command worker and dispatcher to process the commands comming in from the different queues\n\nTo add a new command (and command handler), just add a new entry to your `services.yml`\nand tag it with `cqrs_command_handler`:\n\n```yaml\n  Drupal\\your_module\\DoSomething\\DoSomethingCommandHandler:\n    autowire: true\n    tags:\n      - { name: cqrs_command_handler }\n```\n\n## Real-time migration example\n\nThe example module contains... an example (deuh) that shows how to implement a \"real-time\" migration for\nthe content type \"Breaking news\".\n\nNavigate to `admin/content/generate-migration-message`. This form allows you to push a migration message to\na queue. It simulates how a third party could push a message to a Drupal migration queue\nwhere it will get picked up by a consumer. The migration framework will then do the heavy lifting.\n\n\u003cimg src=\"https://github.com/robiningelbrecht/drupal-amqp-rabbitmq/raw/master/readme/real-time-migration.gif\" alt=\"Real time migration\"\u003e\n\n## Run consumers as background processes\n\nGenerally you want to run consumers as a background process and keep them \"alive\" for as long\nyour server is up. This can be done using `systemd`, but I choose to use [supervisord](http://supervisord.org/)\n\n\u003e Supervisor is a client/server system that allows its users to monitor and control a number of\n\u003e processes on UNIX-like operating systems.\n\nTo register all consumers as a process, just run `lando consumers-start`. This will spin up supervisord\nand automatically create the necessary consumers for all of you queues.\n\nWhen adding/removing queues or when updating queue config, you need to run `lando consumers-restart`\nfor  your new settings to be picked up.\n\n**Important**: Whenever you make changes to you code, make sure to run the restart command as well,\nas you don't want your consumers to be running with old code.\n\n### Check the status of your consumers\n\nYou can just run `lando consumers-status`, this should output something like this:\n\n```\nampq-consume-queue-one:ampq-consume-queue-one-00   RUNNING   pid 1219, uptime 0:00:06\nampq-consume-queue-one:ampq-consume-queue-one-01   RUNNING   pid 1215, uptime 0:00:07\nampq-consume-queue-one:ampq-consume-queue-two-01   RUNNING   pid 1216, uptime 0:00:07\n```\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frobiningelbrecht%2Fdrupal-amqp-rabbitmq","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frobiningelbrecht%2Fdrupal-amqp-rabbitmq","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frobiningelbrecht%2Fdrupal-amqp-rabbitmq/lists"}