{"id":26234877,"url":"https://github.com/flowpack/jobqueue-common","last_synced_at":"2025-04-09T15:08:09.194Z","repository":{"id":2039121,"uuid":"45603896","full_name":"Flowpack/jobqueue-common","owner":"Flowpack","description":"A base for job queue handling in Flow framework applications","archived":false,"fork":false,"pushed_at":"2024-07-16T08:09:04.000Z","size":215,"stargazers_count":27,"open_issues_count":5,"forks_count":26,"subscribers_count":8,"default_branch":"main","last_synced_at":"2025-04-09T15:08:01.463Z","etag":null,"topics":["flowframework","jobqueue","neoscms"],"latest_commit_sha":null,"homepage":null,"language":"PHP","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Flowpack.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2015-11-05T10:21:00.000Z","updated_at":"2024-08-24T02:56:52.000Z","dependencies_parsed_at":"2024-06-18T13:45:43.668Z","dependency_job_id":"c8667ae0-8f52-402c-88e9-7bc3a0646d8c","html_url":"https://github.com/Flowpack/jobqueue-common","commit_stats":{"total_commits":77,"total_committers":21,"mean_commits":"3.6666666666666665","dds":0.8181818181818181,"last_synced_commit":"439dd588bcfbf6af715728f8e81935b8bc4dd82f"},"previous_names":[],"tags_count":25,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Flowpack%2Fjobqueue-common","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Flowpack%2Fjobqueue-common/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Flowpack%2Fjobqueue-common/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Flowpack%2Fjobqueue-common/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Flowpack","download_url":"https://codeload.github.com/Flowpack/jobqueue-common/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248055284,"owners_count":21040157,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["flowframework","jobqueue","neoscms"],"created_at":"2025-03-13T02:29:43.204Z","updated_at":"2025-04-09T15:08:09.170Z","avatar_url":"https://github.com/Flowpack.png","language":"PHP","readme":"# Flowpack.JobQueue.Common\n\nNeos Flow package that allows for asynchronous and distributed execution of tasks.\n\n### Table of contents\n\n  * [Quickstart](#quickstart-tldr)\n  * [Introduction](#introduction)\n  * [Message Queue](#message-queue)\n  * [Job Queue](#job-queue)\n  * [Command Line Interface](#command-line-interface)\n  * [Signals \u0026 Slots](#signal--slots)\n  * [License](#license)\n  * [Contributions](#contributions)\n\n## Quickstart (TL;DR)\n\n1. **Install this package using composer:**\n\n  ```\n  composer require flowpack/jobqueue-common\n  ```\n  (or by adding the dependency to the composer manifest of an installed package)\n\n2. **Configure a basic queue by adding the following to your `Settings.yaml`:**\n\n  ```yaml\n  Flowpack:\n    JobQueue:\n      Common:\n        queues:\n          'some-queue':\n            className: 'Flowpack\\JobQueue\\Common\\Queue\\FakeQueue'\n  ```\n\n3. **Initialize the queue (if required)**\n\n  With\n\n  ```\n  ./flow queue:setup some-queue\n  ```\n\n  you can setup the queue and/or verify its configuration.\n  In the case of the `FakeQueue` that step is not required.\n\n  *Note:* The `queue:setup` command won't remove any existing messages, there is no harm in calling it multiple times\n\n4. **Annotate any *public* method you want to be executed asynchronously:**\n\n  ```php\n  use Flowpack\\JobQueue\\Common\\Annotations as Job;\n  \n  class SomeClass {\n  \n      /**\n       * @Job\\Defer(queueName=\"some-queue\")\n       */\n      public function sendEmail($emailAddress)\n      {\n          // send some email to $emailAddress\n      }\n  }\n  ```\n\n  or use attributes instead of annotations (PHP 8.0 and later):\n  \n  ```php\n  use Flowpack\\JobQueue\\Common\\Annotations as Job;\n  \n  class SomeClass {\n  \n      #[Job\\Defer(queueName: \"some-queue\")]\n      public function sendEmail($emailAddress)\n      {\n          // send some email to $emailAddress\n      }\n  }\n  ```\n  \n  *Note:* The method needs to be *public* and it must not return anything\n\n5. **Start the worker (if required)**\n\n  With the above code in place, whenever the method `SomeClass::sendEmail()` is about to be called that method call is converted into a job that is executed asynchronously[1].\n\n  Unless you use the `FakeQueue` like in the example, a so called `worker` has to be started, to listen for new jobs and execute them::\n  \n  ```\n  ./flow flowpack.jobqueue.common:job:work some-queue --verbose\n  ```\n\n## Introduction\n\nTo get started let's first define some terms:\n\n\u003cdl\u003e\n  \u003cdt\u003eMessage\u003c/dt\u003e\n  \u003cdd\u003e\n    A piece of information passed between programs or systems, sometimes also referred to as \"Event\".\u003cbr\u003e\n    In the JobQueue packages we use messages to transmit `Jobs`.\n  \u003c/dd\u003e\n  \u003cdt\u003eMessage Queue\u003c/dt\u003e\n  \u003cdd\u003e\n    According to \u003ca href=\"https://en.wikipedia.org/wiki/Message_queue\"\u003eWikipedia\u003c/a\u003e \"message queues [...] are software-engineering components used for inter-process communication (IPC), or for inter-thread communication within the same process\".\u003cbr /\u003e\n    In the context of the JobQueue packages we refer to \"Message Queue\" as a \u003ca href=\"https://en.wikipedia.org/wiki/FIFO_(computing_and_electronics)\"\u003eFIFO\u003c/a\u003e buffer that distributes messages to one or more consumers, so that every message is only processed once.\n  \u003c/dd\u003e\n  \u003cdt\u003eJob\u003c/dt\u003e\n  \u003cdd\u003e\n    A unit of work to be executed (asynchronously).\u003cbr /\u003e\n    In the JobQueue packages we use the Message Queue to store serialized jobs, so it acts as a \"Job stream\".\n  \u003c/dd\u003e\n  \u003cdt\u003eJob Manager\u003c/dt\u003e\n  \u003cdd\u003e\n    Central authority allowing adding and fetching jobs to/from the Message Queue.\n  \u003c/dd\u003e\n  \u003cdt\u003eWorker\u003c/dt\u003e\n  \u003cdd\u003e\n    The worker watches a queue and triggers the job execution.\u003cbr /\u003e\n    This package comes with a `job:work` command that does this (see below)\n  \u003c/dd\u003e\n  \u003cdt\u003esubmit\u003c/dt\u003e\n  \u003cdd\u003e\n    New messages are *submitted* to a queue to be processed by a worker\n  \u003c/dd\u003e\n  \u003cdt\u003ereserve\u003c/dt\u003e\n  \u003cdd\u003e\n    Before a message can be processed it has to be *reserved*.\u003cbr /\u003e\n    The queue guarantees that a single message can never be reserved by two workers (unless it has been released again)\n  \u003c/dd\u003e\n  \u003cdt\u003erelease\u003c/dt\u003e\n  \u003cdd\u003e\n    A reserved message can be *released* to the queue to be processed at a later time.\u003cbr /\u003e\n    The *JobManager* does this if Job execution failed and the `maximumNumberOfReleases` setting for the queue is greater than zero\n  \u003c/dd\u003e\n  \u003cdt\u003eabort\u003c/dt\u003e\n  \u003cdd\u003e\n    If a message could not be processed successfully it is *aborted* marking it *failed* in the respective queue so that it can't be reserved again.\u003cbr /\u003e\n    The *JobManager* aborts a message if Job execution failed and the message can't be released (again)\n  \u003c/dd\u003e\n  \u003cdt\u003efinish\u003c/dt\u003e\n  \u003cdd\u003e\n    If a message was processed successfully it is marked *finished*.\u003cbr /\u003e\n    The *JobManager* finishes a message if Job execution succeeded.\n  \u003c/dd\u003e\n\u003c/dl\u003e\n\n## Message Queue\n\nThe `Flowpack.JobQueue.Common` package comes with a *very basic* Message Queue implementation `Flowpack\\JobQueue\\Common\\Queue\\FakeQueue` that allows for execution of Jobs using sub requests.\nIt doesn't need any 3rd party tools or server loops and works for basic scenarios. But it has a couple of limitations to be aware of:\n\n1. It is not actually a queue, but dispatches jobs immediately as they are queued. So it's not possible to distribute the work to multiple workers\n\n2. The `JobManager` is not involved in processing of jobs so the jobs need to take care of error handling themselves.\n\n3. For the same reason [Signals](#signal--slots) are *not* emitted for the `FakeQueue`.\n\n4. With Flow 3.3+ The `FakeQueue` supports a flag `async`. Without that flag set, executing jobs *block* the main thread!\n\nFor advanced usage it is recommended to use one of the implementing packages like one of the following:\n* [Flowpack.JobQueue.Doctrine](https://github.com/Flowpack/jobqueue-doctrine)\n* [Flowpack.JobQueue.Beanstalkd](https://github.com/Flowpack/jobqueue-beanstalkd)\n* [Flowpack.JobQueue.Redis](https://github.com/Flowpack/jobqueue-redis)\n\n### Configuration\n\nThis is the simplest configuration for a queue:\n\n```yaml\nFlowpack:\n  JobQueue:\n    Common:\n      queues:\n        'test':\n          className: 'Flowpack\\JobQueue\\Common\\Queue\\FakeQueue'\n```\n\nWith this a queue named `test` will be available.\n\n*Note:* For reusable packages you should consider adding a vendor specific prefixes to avoid collisions. We recommend to use a classname or the package name with the function name (e.g. Flowpack.ElasticSearch.ContentRepositoryQueueIndexer. \n\n### Queue parameters\n\nThe following parameters are supported by all queues:\n\n| Parameter               | Type    | Default | Description                                                                                                                     |\n| ----------------------- |---------|--------:|---------------------------------------------------------------------------------------------------------------------------------|\n| className               | string  |       - | FQN of the class implementing the queue                                                                                         |\n| maximumNumberOfReleases | integer |       3 | Max. number of times a message is re-\u003cbr\u003ereleased to the queue if a job failed                                                  |\n| executeIsolated         | boolean |   FALSE | If TRUE jobs for this queue are executed in a separate Thread. This makes sense in order to avoid memory leaks and side-effects |\n| outputResults           | boolean |   FALSE | If TRUE the full output (stdout + stderr) of the respective job is forwarded to the stdout of its \"parent\" (only applicable if `executeIsolated` is `true`)  |\n| queueNamePrefix         | string  |       - | Optional prefix for the internal queue name,\u003cbr\u003eallowing to re-use the same backend over multiple installations                 |\n| options                 | array   |       - | Options for the queue.\u003cbr\u003eImplementation specific (see corresponding package)                                                   |\n| releaseOptions          | array   |       - | Options that will be passed to `release()` when a job failed.\u003cbr\u003eImplementation specific (see corresponding package)            |\n\nA more complex example could look something like:\n\n```yaml\nFlowpack:\n  JobQueue:\n    Common:\n      queues:\n        'email':\n          className: 'Flowpack\\JobQueue\\Beanstalkd\\Queue\\BeanstalkdQueue'\n          maximumNumberOfReleases: 5\n          executeIsolated: true\n          outputResults: true\n          queueNamePrefix: 'staging-'\n          options:\n            client:\n              host: 127.0.0.11\n              port: 11301\n            defaultTimeout: 50\n          releaseOptions:\n            priority: 512\n            delay: 120\n        'log':\n          className: 'Flowpack\\JobQueue\\Redis\\Queue\\RedisQueue'\n          options:\n            defaultTimeout: 10\n```\n\nAs you can see, you can have multiple queues in one installations. That allows you to use different backends/options for queues depending on the requirements.\n\n### Presets\n\nIf multiple queries share common configuration **presets** can be used to ease readability and maintainability:\n\n```yaml\nFlowpack:\n  JobQueue:\n    Common:\n      presets:\n        'staging-default':\n          className: 'Flowpack\\JobQueue\\Doctrine\\Queue\\DoctrineQueue'\n          queueNamePrefix: 'staging-'\n          options:\n            pollInterval: 2\n      queues:\n        'email':\n          preset: 'staging-default'\n          options:\n            tableName: 'queue_email' # default table name would be \"flowpack_jobqueue_messages_email\"\n        'log':\n          preset: 'staging-default'\n          options:\n            pollInterval: 1 # overrides \"pollInterval\" of the preset\n```\n\nThis will configure two `DoctrineQueue`s \"email\" and \"log\" with some common options but different table names and poll intervals.\n\n## Job Queue\n\n\nThe job is an arbitrary class implementing `Flowpack\\JobQueue\\Common\\Job\\JobInterface`.\nThis package comes with one implementation `StaticMethodCallJob` that allows for invoking a public method (see [Quickstart](#quickstart-tldr))\nbut often it makes sense to create a custom Job:\n\n```php\n\u003c?php\nuse Flowpack\\JobQueue\\Common\\Job\\JobInterface;\nuse Flowpack\\JobQueue\\Common\\Queue\\Message;\nuse Flowpack\\JobQueue\\Common\\Queue\\QueueInterface;\n\nclass SendEmailJob implements JobInterface\n{\n    protected $emailAddress;\n\n    public function __construct($emailAddress)\n    {\n        $this-\u003eemailAddress = $emailAddress;\n    }\n\n\n    public function execute(QueueInterface $queue, Message $message)\n    {\n        // TODO: send the email to $this-\u003eemailAddress\n        return true;\n    }\n\n    public function getIdentifier()\n    {\n        return 'SendEmailJob';\n    }\n\n    public function getLabel()\n    {\n        return sprintf('SendEmailJob (email: \"%S\")', $this-\u003eemailAddress);\n    }\n}\n```\n\n*Note:* It's crucial that the `execute()` method returns TRUE on success, otherwise the corresponding message will be released again and/or marked *failed*.\n\n\nWith that in place, the new job can be added to a queue like this:\n\n\n```php\nuse Flowpack\\JobQueue\\Common\\Job\\JobInterface;\nuse Flowpack\\JobQueue\\Common\\Job\\JobManager;\nuse Neos\\Flow\\Annotations as Flow;\n\nclass SomeClass {\n\n    /**\n     * @Flow\\Inject\n     * @var JobManager\n     */\n    protected $jobManager;\n\n    /**\n     * @return void\n     */\n    public function queueJob()\n    {\n        $job = new SendEmailJob('some@email.com');\n        $this-\u003ejobManager-\u003equeue('queue-name', $job);\n    }\n}\n```\n\n## Command Line Interface\n\nUse the `flowpack.jobqueue.common:queue:*` and `flowpack.jobqueue.common:job:*` commands to interact with the job queues:\n\n| Command         | Description                                                                |\n| --------------- |----------------------------------------------------------------------------|\n| queue:list      | List configured queues                                                     |\n| queue:describe  | Shows details for a given queue (settings, ..)                             |\n| queue:setup     | Initialize a queue (i.e. create required db tables, check connection, ...) |\n| queue:flush     | Remove all messages from a queue (requires --force flag)                   |\n| queue:submit    | Submit a message to a queue (mainly for testing)                           |\n| job:work        | Work on a queue and execute jobs                                           |\n| job:list        | List queued jobs                                                           |\n\n## Signal \u0026 Slots\n\nWhen working with JobQueues proper monitoring is crucial as failures might not be visible immediately.\nThe `JobManager` emits signals for all relevant events, namely:\n\n* messageSubmitted\n* messageTimeout\n* messageReserved\n* messageFinished\n* messageReleased\n* messageFailed\n\nThose can be used to implement some more sophisticated logging for example:\n\n```php\n\u003c?php\nnamespace Your\\Package;\n\nuse Flowpack\\JobQueue\\Common\\Job\\JobManager;\nuse Flowpack\\JobQueue\\Common\\Queue\\Message;\nuse Flowpack\\JobQueue\\Common\\Queue\\QueueInterface;\nuse Neos\\Flow\\Core\\Bootstrap;\nuse Neos\\Flow\\Log\\SystemLoggerInterface;\nuse Neos\\Flow\\Package\\Package as BasePackage;\n\nclass Package extends BasePackage\n{\n\n    /**\n     * @param Bootstrap $bootstrap\n     * @return void\n     */\n    public function boot(Bootstrap $bootstrap)\n    {\n        $dispatcher = $bootstrap-\u003egetSignalSlotDispatcher();\n\n        $dispatcher-\u003econnect(\n            JobManager::class, 'messageFailed',\n            function(QueueInterface $queue, Message $message, \\Exception $jobExecutionException = null) use ($bootstrap) {\n                $additionalData = [\n                    'queue' =\u003e $queue-\u003egetName(),\n                    'message' =\u003e $message-\u003egetIdentifier()\n                ];\n                if ($jobExecutionException !== null) {\n                    $additionalData['exception'] = $jobExecutionException-\u003egetMessage();\n                }\n                $bootstrap-\u003egetObjectManager()-\u003eget(SystemLoggerInterface::class)-\u003elog('Job failed', LOG_ERR, $additionalData);\n            }\n        );\n    }\n}\n```\n\nThis would log every failed message to the system log.\n\n## License\n\nThis package is licensed under the MIT license\n\n## Contributions\n\nPull-Requests are more than welcome. Make sure to read the [Code Of Conduct](CodeOfConduct.rst).\n\n---\n\n[1] The `FakeQueue` actually executes Jobs *synchronously* unless the `async` flag is set (requires Flow 3.3+)\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fflowpack%2Fjobqueue-common","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fflowpack%2Fjobqueue-common","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fflowpack%2Fjobqueue-common/lists"}