{"id":15287233,"url":"https://github.com/mcfedr/queue-manager-bundle","last_synced_at":"2025-04-13T00:21:43.235Z","repository":{"id":15249025,"uuid":"17978024","full_name":"mcfedr/queue-manager-bundle","owner":"mcfedr","description":"A bundle for running background jobs in Symfony","archived":false,"fork":false,"pushed_at":"2024-09-25T14:46:51.000Z","size":704,"stargazers_count":9,"open_issues_count":5,"forks_count":11,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-24T07:43:12.027Z","etag":null,"topics":["beanstalkd","php","resque","scheduled-jobs","sqs","symfony"],"latest_commit_sha":null,"homepage":"","language":"PHP","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mcfedr.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2014-03-21T11:36:17.000Z","updated_at":"2024-09-25T14:46:35.000Z","dependencies_parsed_at":"2024-10-14T18:20:53.656Z","dependency_job_id":"67e6c1ce-8e19-4855-9a45-daa9470fae5c","html_url":"https://github.com/mcfedr/queue-manager-bundle","commit_stats":{"total_commits":162,"total_committers":9,"mean_commits":18.0,"dds":0.4629629629629629,"last_synced_commit":"562f1a39cc390270c4e9ba1af58439821897c9db"},"previous_names":[],"tags_count":79,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mcfedr%2Fqueue-manager-bundle","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mcfedr%2Fqueue-manager-bundle/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mcfedr%2Fqueue-manager-bundle/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mcfedr%2Fqueue-manager-bundle/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mcfedr","download_url":"https://codeload.github.com/mcfedr/queue-manager-bundle/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248648466,"owners_count":21139305,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["beanstalkd","php","resque","scheduled-jobs","sqs","symfony"],"created_at":"2024-09-30T15:27:02.790Z","updated_at":"2025-04-13T00:21:43.204Z","avatar_url":"https://github.com/mcfedr.png","language":"PHP","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Queue Manager Bundle\n\nA bundle for running background jobs in [Symfony](symfony.com).\n\n[![Latest Stable Version](https://poser.pugx.org/mcfedr/queue-manager-bundle/v/stable.png)](https://packagist.org/packages/mcfedr/queue-manager-bundle)\n[![License](https://poser.pugx.org/mcfedr/queue-manager-bundle/license.png)](https://packagist.org/packages/mcfedr/queue-manager-bundle)\n[![Build Status](https://travis-ci.org/mcfedr/queue-manager-bundle.svg?branch=master)](https://travis-ci.org/mcfedr/queue-manager-bundle)\n\nThis bundle provides a consistent queue interface, with plugable 'drivers' that can schedule jobs using a number of\ndifferent queue types:\n\n- [Beanstalkd](https://beanstalkd.github.io/)\n- [Amazon SQS](https://aws.amazon.com/sqs/)\n- [Google Cloud Pub/Sub](https://cloud.google.com/pubsub)\n\nThere are also a number of 'helper' plugins:\n\n- [Doctrine Delay Queue](https://www.doctrine-project.org/)\n  \n  This plugins can schedule jobs far in advance and move them into a real time queue when they should be run. Use in\n  combination with SQS or Beanstalkd which don't support scheduling jobs.\n   \n- Perioidic Jobs\n\n  Automatically schedule a jobs to run every hour/day/week or other period. Randomizes the actual time to keep an even\n  server load.\n\n## Overview\n\nA job is a Symfony service that implements the `Worker` interface. This has a single method `execute(array $arguments)`.\nThe name of the job is the service name.\n\nYou add jobs to the queue by calling `$container-\u003eget(QueueManagerRegistry::class)-\u003eput($name, $arguments)`.\n\nCheck the documentation of the driver you are using as to how to run the daemon process(es).\n\n## Install\n\n### Composer\n\n```bash\ncomposer require mcfedr/queue-manager-bundle\n```\n\n### AppKernel\n\nInclude the bundle in your AppKernel\n\n```php\n    public function registerBundles()\n    {\n        $bundles = [\n            ...\n            new Mcfedr\\QueueManagerBundle\\McfedrQueueManagerBundle(),\n```\n\n## Config\n\nYou must configure one (or more) drivers to use. Generally you will have just one and call it 'default'.\n\n### Beanstalk\n\n#### Usage\n\nThe beanstalk runner is a Symfony command. You can runner multiple instances if you need to\nhandle higher numbers of jobs.\n\n```bash\n./bin/console mcfedr:queue:{name}-runner\n```\n\nWhere `{name}` is what you used in the config. Add `-v` or more to get detailed logs.\n\n#### Config\n\n```yaml\nmcfedr_queue_manager:\n    managers:\n        default:\n            driver: beanstalkd\n            options:\n                host: 127.0.0.1\n                port: 11300\n                default_queue: mcfedr_queue\n```\n\n#### Supported options to `QueueManager::put`\n\n* `queue` - The name of the queue to put the job in.\n* `priority` - The job priority.\n* `ttr` - Time to run, the time given for a job to finish before it is repeated.\n* `time` - A `\\DateTime` object of when to schedule this job.\n* `delay` - Number of seconds from now to schedule this job.\n\n### AWS SQS\n\n#### Usage\n\nThe sqs runner is a Symfony command. You can runner multiple instances if you need to\nhandle higher numbers of jobs.\n\n```bash\n./bin/console mcfedr:queue:{name}-runner\n```\n\nWhere `{name}` is what you used in the config. Add `-v` or more to get detailed logs.\n\n#### Config\n\n```yaml\nmcfedr_queue_manager:\n    managers:\n        default:\n            driver: sqs\n            options:\n                default_url: https://sqs.eu-west-1.amazonaws.com/...\n                region: eu-west-1\n                credentials:\n                    key: 'my-access-key-id'\n                    secret: 'my-secret-access-key'\n                queues:\n                    name: https://sqs.eu-west-1.amazonaws.com/...\n                    name2: https://sqs.eu-west-1.amazonaws.com/...\n```\n\n* `default_url` - Default SQS queue url.\n* `region` - The region where your queue is. Required if not passing `sqs_client`.\n* `credentials` *optional* - [Specify your key and secret](http://docs.aws.amazon.com/aws-sdk-php/v3/guide/guide/credentials.html#using-hard-coded-credentials)\n  This is optional because the SDK can pick up your credentials from a [variety of places](http://docs.aws.amazon.com/aws-sdk-php/v3/guide/guide/credentials.html).\n* `sqs_client` - Name of `SQSClient` service to use.\n* `queues` *optional* - Allows you to setup a mapping of short names for queues, this makes it easier to use multiple queues and keep the config in one place.\n\n#### Supported options to `QueueManager::put`\n\n* `url` - A `string` with the url of a queue.\n* `queue` - A `string` with the name of a queue in the config.\n* `time` - A `\\DateTime` object of when to schedule this job. **Note:** SQS can delay jobs up to 15 minutes. \n* `delay` - Number of seconds from now to schedule this job. **Note:** SQS can delay jobs up to 15 minutes.\n* `ttr` - Number of seconds during which Amazon SQS prevents other consumers from receiving and processing the message (SQS Visibility Timeout).\n\n### GCP Pub/Sub\n\n#### Usage\n\nThe pub/sub runner is a Symfony command. You can runner multiple instances if you need to\nhandle higher numbers of jobs.\n\n```bash\n./bin/console mcfedr:queue:{name}-runner\n```\n\nWhere `{name}` is what you used in the config. Add `-v` or more to get detailed logs.\n\n#### Config\n\n```yaml\nmcfedr_queue_manager:\n    managers:\n        default:\n            driver: pub_sub\n            options:\n                default_subscription: 'test_sub'\n                default_topic: 'projects/project/topics/test-topic'\n                pub_sub_queues:\n                    name1:\n                        topic: 'projects/project/topics/test-topic'\n                        subscription: 'test_sub'\n```\n\n* `default_subscription` - Default Pub/Sub subscription to listen to.\n* `default_topic` - Default Pub/Sub topic to push to.\n* `key_file_path` *optional* - Specify your key file. This is optional because\n  the SDK can pick up your credentials from a [variety of places](https://googleapis.github.io/google-cloud-php/#/docs/google-cloud/v0.107.1/guides/authentication).\n* `pub_sub_client` - Name of `PubSubClient` service to use.\n* `pub_sub_queues` *optional* - Allows you to setup a mapping of short names for queues,\n   this makes it easier to use multiple queues and keep the config in one place.\n   Each queue should have a `topic` and `subscription`.\n\n#### Supported options to `QueueManager::put`\n\n* `topic` - A `string` with the url of a queue.\n* `queue` - A `string` with the name of a queue in the config.\n\n### Periodic\n\nThis driver doesn't run jobs, it requires another driver to actually process jobs.\n\n#### Usage\n\nThere is no runner daemon for this driver as it just plugs into other drivers. Use it by\n`put`ting jobs into this driver with the `period` option.\n\n#### Config\n\n```yaml\nmcfedr_queue_manager:\n    managers:\n        periodic:\n            driver: periodic\n            options:\n                default_manager: delay\n                default_manager_options: []\n```\n\nThis will create a `QueueManager` service named `\"mcfedr_queue_manager.periodic\"`.\n\n* `default_manager` - Default job processor, must support delayed jobs, for example [Doctrine Delay](https://packagist.org/packages/mcfedr/doctrine-delay-queue-driver-bundle).\n* `default_manager_options` - Default options to pass to job processor `put`.\n\n#### Supported options to `QueueManager::put`\n\n* `period` - The average number of seconds between job runs.\n* `manager` - Use a different job processor for this job.\n* `manager_options` - Options to pass to the processors `put` method.\n\n### Doctrine Delay\n\nThis driver doesn't run jobs, it requires another driver to actually process jobs.\n\nIt currently **only** works with MySQL as a native query is required to find jobs in a concurrency safe way.\n\n#### Usage\n\nYou should run the daemon for delay in addition to any other daemons you are using.\nThis runner simply moves jobs from Doctrine into your other job queues. Because its \nnot doing much work generally a single instance can cope with a high number of jobs.\n\n```bash\n./bin/console mcfedr:queue:{name}-runner\n```\n\nWhere `{name}` is what you used in the config. Add `-v` or more to get detailed logs.\n\n#### Config\n\n```yaml\nmcfedr_queue_manager:\n    managers:\n        delay:\n            driver: doctrine_delay\n            options:\n                entity_manager: default\n                default_manager: default\n                default_manager_options: []\n```\n\nThis will create a `QueueManager` service named `\"mcfedr_queue_manager.delay\"`.\n\n* `entity_manager` - Doctrine entity manager to use.\n* `default_manager` - Default job processor.\n* `default_manager_options` - Default options to pass to job processor `put`.\n\n#### Supported options to `QueueManager::put`\n\n* `time` - A `\\DateTime` object of when to schedule this job.\n* `delay` - Number of seconds from now to schedule this job.\n* `force_delay` - A boolean that forces the job to be delayed by the specified number of seconds.\n* `manager` - Use a different job processor for this job.\n* `manager_options` - Options to pass to the processors `put` method.\n\n#### Note\n\nIf `delay` or `time` option is less then 30 seconds the job will be scheduled for immediate execution unless the `force_delay` option is set to `true`\n\n#### Tables\n\nAfter you have installed you will need to do a schema update so that the table of delayed tasks is created.\n\n### Additional options\n\nThese are the defaults for a number of other options.\n\n```yaml\nmcfedr_queue_manager:\n    retry_limit: 3\n    sleep_seconds: 5\n    report_memory: false\n    doctrine_reset: true\n```\n\n| Option | Means |\n|--------|-------|\n| `retry_limit` | The number of times a job will be retried when it fails, unless it throws `UnrecoverableJobExceptionInterface` |\n| `sleep_seconds` | When a queue doesnt have any jobs it will wait this long before checking again |\n| `report_memory` | Enable a listener that reports current memory usage between each job, useful for debugging leaks |\n| `doctrine_reset` | This listener will reset doctrine connect between jobs. Be careful with your memory usage if disabled. | \n\n### Doctrine\n\nTo avoid memory leaks entity manager is being reset after job execution.\n\nResetting a non-lazy manager service is deprecated since Symfony 3.2 and will throw an exception in version 4.0.\nSo if you use Symfony 3.2 or greater you need to install symfony/proxy-manager-bridge to support [Lazy Services](https://symfony.com/doc/current/service_container/lazy_services.html).\n\n```bash\ncomposer require proxy-manager-bridge\n```\n\n## Usage\n\nYou can access the `QueueManagerRegistry` for simple access to your queue.\nJust inject `QueueManagerRegistry::class` and call `put` to add new jobs to the queue.\n\nAlso, each manager will be a service you can access with the name `\"mcfedr_queue_manager.$name\"`.\nIt implements the `QueueManager` interface, where you can call just 2 simple methods.\n\n    /**\n     * Put a new job on a queue\n     *\n     * @param string $name The service name of the worker that implements {@link \\Mcfedr\\QueueManagerBundle\\Queue\\Worker}\n     * @param array $arguments Arguments to pass to execute - must be json serializable\n     * @param array $options Options for creating the job - these depend on the driver used\n     */\n    public function put(string $name, array $arguments = [], array $options = []): Job\n    \n    /**\n     * Remove a job, you should call this to cancel a job\n     *\n     * @param $job\n     * @throws WrongJobException\n     * @throws NoSuchJobException\n     */\n    public function delete(Job $job): void;\n\n## Jobs\n\nJobs to run are Symfony services that implement `Mcfedr\\QueueManagerBundle\\Queue\\Worker`\nThere is one method, that is called with the arguments you passed to `QueueManager::put`.\n\n    /**\n     * Called to start the queued task\n     *\n     * @param array $arguments\n     * @throws \\Exception\n     */\n    public function execute(array $arguments): void;\n\nIf your job throws an exception it will be retried (assuming the driver supports retrying),\nunless the exception thrown is an instance of `UnrecoverableJobExceptionInterface`.\n\nWorkers should be tagged with `mcfedr_queue_manager.worker`, if you are using autowiring this will\nhappen automatically.\n\nBy default the job name is the class, but you can also add tags with specific ids, e.g.\n\n```yaml\nWorker:\n  tags:\n  - { name: 'mcfedr_queue_manager.worker', id: 'test_worker' }\n```\n\nNow you can schedule this job with both:\n\n```php\n$queueManager-\u003eput(Worker::class, ...)\n$queueManager-\u003eput('test_worker', ...)\n```\n\n## Events\n\nA number of events are triggered during the running of jobs.\n\n| Name | Event Object |\n|------|--------------|\n| mcfedr_queue_manager.job_start | `StartJobEvent` |\n| mcfedr_queue_manager.job_finished | `FinishedJobEvent` | \n| mcfedr_queue_manager.job_failed | `FailedJobEvent` |\n| mcfedr_queue_manager.job_batch_start | `StartJobBatchEvent` |\n| mcfedr_queue_manager.job_batch_finished | `FinishedJobBatchEvent` |\n\n## Creating your own driver\n\nFirstly a driver needs to implement a `QueueManager`. This should put tasks into queues.\n\nThe options argument can be used to accept any extra parameters specific to your implementation.\nFor example, this might include a `delay` or a `priority` if you support that.\n\nYou also need to create a `Job` class, many drivers can just extend `AbstractJob` but you can add any extra data you need.\n\n### Creating a runner\n\nMany drivers can use the `RunnerCommand` as a base, implementing the `getJob` method.\n\nOther queue servers have their own runners, in which case you need to write the code such that the correct worker is called.\nThe service `mcfedr_queue_manager.job_executor` can help with this.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmcfedr%2Fqueue-manager-bundle","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmcfedr%2Fqueue-manager-bundle","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmcfedr%2Fqueue-manager-bundle/lists"}