{"id":22092092,"url":"https://github.com/krakphp/job","last_synced_at":"2025-07-24T20:31:57.381Z","repository":{"id":57009026,"uuid":"75369587","full_name":"krakphp/job","owner":"krakphp","description":"Clean, simple, powerful Queued Jobs library","archived":false,"fork":false,"pushed_at":"2017-08-08T15:05:36.000Z","size":90,"stargazers_count":6,"open_issues_count":4,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2024-11-24T10:19:09.587Z","etag":null,"topics":["asynchronous","queued-jobs","redis","scheduling","sqs-queue","workers"],"latest_commit_sha":null,"homepage":null,"language":"PHP","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/krakphp.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2016-12-02T06:56:36.000Z","updated_at":"2024-11-06T17:21:09.000Z","dependencies_parsed_at":"2022-08-21T14:31:36.756Z","dependency_job_id":null,"html_url":"https://github.com/krakphp/job","commit_stats":null,"previous_names":[],"tags_count":10,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/krakphp%2Fjob","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/krakphp%2Fjob/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/krakphp%2Fjob/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/krakphp%2Fjob/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/krakphp","download_url":"https://codeload.github.com/krakphp/job/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":227476144,"owners_count":17779417,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["asynchronous","queued-jobs","redis","scheduling","sqs-queue","workers"],"created_at":"2024-12-01T03:08:19.405Z","updated_at":"2024-12-01T03:08:20.397Z","avatar_url":"https://github.com/krakphp.png","language":"PHP","readme":"# Jobs\n\nSimple yet powerful implementation of Queued Jobs.\n\n## Features\n\n- Consume from multiple queues asynchronously\n- Easy to setup\n- Easy to integrate with any project\n- Incredibly extendable\n- Powerful extensions\n\n## Installation\n\nInstall with composer at `krak/job`.\n\n## Usage\n\n### Create the Kernel\n\nThe kernel is the core manager of the Job library. It's simply a Cargo\\\\Container decorator with helper methods.\n\n```php\n\u003c?php\n\n$kernel = new Krak\\Job\\Kernel();\n```\n\nYou can also pass an optional Container instance if you want to use a special container.\n\n#### Configuring the Kernel\n\nYou can configure the kernel by wrapping any of the services defined in the container. In addition to the configuration provided by the Cargo\\\\Container, the kernel provides helper methods to ease customization.\n\n```php\n// configure the scheduling loop\n$kernel-\u003econfig([\n    'queue' =\u003e 'jobs', // name of the queue\n    'sleep' =\u003e 10, // duration in seconds the scheduler will sleep for after every iteration\n    'ttl' =\u003e 50, // max duration of the scheduler before dying\n    'max_jobs' =\u003e 10, // max number of processes to launch at once\n    'max_retry' =\u003e 3, // max number of retries before just giving up on a failed job\n    'batch_size' =\u003e 5, // max number of jobs to process in a given batch\n]);\n\n// configure the queue provider\n$kernel['Predis\\ClientInterface'] = function() {\n    return new Predis\\Client();\n};\n$kernel['krak.job.queue_provider'] = 'redis';\n\n// configure the consumer stack\n$kernel-\u003ewrap('krak.job.pipeline.consumer', function($consumer) {\n    return $consumer-\u003epush(myConsumer());\n});\n// and so on...\n```\n\n### Define a Job\n\nEvery Job must implement the empty interface `Krak\\Job\\Job` and have a `handle` method. The `handle` method will be executed once the Job has been consumed for processing.\n\n```php\n\u003c?php\n\nnamespace Acme\\Jobs;\n\nuse Krak\\Job\\Job;\nuse Acme\\ServiceA;\n\nclass ProcessJob implements Job\n{\n    private $id;\n\n    public function __construct($id) {\n        $this-\u003eid = $id;\n    }\n\n    public function handle(ServiceA $service) {\n        process($this-\u003eid);\n    }\n}\n```\n\nArguments will automatically wired into the handle method using the AutoArgs package. The Job instance will be serialized, so make sure that the properties of the Job are serializable. It'd also be a good idea to keep the amount of data in a job as small as possible.\n\nYou can also implement the `Krak\\Job\\PipeWrappedJob` interface if you want to customize the wrapped job.\n\n```php\n\u003c?php\n\nuse Krak\\Job;\n\nclass ProcessJob implements Job\\Job, Job\\PipeWrappedJob\n{\n\n    public function handle() {}\n\n    public function pipe(Job\\WrappedJob $wrapped) {\n        return $wrapped-\u003ewithName('my_custom_job_name')\n            -\u003ewithQueue('my_custom_queue')\n            -\u003ewithDelay(3600)\n            -\u003ewithAddedPayload([\n                'custom_data' =\u003e 1,\n            ]);\n    }\n}\n```\n\nWhen the job is dispatched and produced, the pipe function will be called and will configure the wrapped job instance.\n\nAnd then as a final convenience, we provide the `Krak\\Job\\AbstractJob` which you can extend which already implements `Krak\\Job\\Job` and `Krak\\Job\\PipeWrappedJob` and provides default implementations.\n\n### Dispatch a Job\n\nDispatching jobs is easy using the `Krak\\Job\\Dispatch`.\n\n```php\n\u003c?php\n\nuse Krak\\Job;\n\n// use the kernel to create a dispatch instance\n$dispatch = $kernel['dispatch']; // or $kernel[Job\\Dispatch::class];\n\n$dispatch-\u003ewrap(new Acme\\Jobs\\ProcessJob(1))\n    -\u003eonQueue('process') // this is optional\n    -\u003ewithName('process')\n    -\u003edelay(3600) // will delay the sending of this job for 1 hour (not all queues support delay)\n    -\u003edispatch();\n```\n\n### Consuming the Jobs\n\nIn order to start consuming jobs, you need to do a few things:\n\n1. Register the Commands with your Symfony Console Application\n\n    ```php\n    \u003c?php\n    // in bin/console\n\n    $app = new Symfony\\Component\\Console\\Application();\n    Krak\\Job\\registerConsole($app, $kernel);\n    ```\n\n    At this point, we've registered all of the job commands and added the JobHelper to the helper set.\n\n2. Start the consumer\n\n    ```bash\n    ./bin/console job:consume -vvv\n    ```\n\n    You can change the verbosity level to suite your needs\n\n### Restarting \u0026 Stopping Jobs\n\nThere are times when you want to restart the running consumer or even restart the system. To enable this, you need to integrate `Psr\\SimpleCache` into the kernel.\n\nEnabling cache:\n\n```php\n\u003c?php\n\n$kernel['Psr\\SimpleCache\\CacheInterface'] = function($c) {\n    // return any CacheInterface\n    return new Symfony\\Component\\Cache\\Simple\\RedisCache($c['Predis\\ClientInterface']);\n};\n```\n\nOnce cache is enabled, then you'll have access to the following commands: `job:stop`, `job:restart`, `job:status`, and `job:reset`.\n\n## Concepts\n\nThe Job library is broken up into several parts: The Kernel, ScheduleLoop, Queues, Dispatch, Console, Pipeline, Jobs\n\n### Kernel\n\nAn Kernel implements the `Krak\\Job\\Kernel` interface and are responsible for configuring and managing everything.\n\n### Scheduler\n\nThe scheduler is responsible for scheduling the tasks to run. The `Krak\\Job\\Scheduler` class actually doesn't contain much logic, it simply starts an infinite loop and passes control the **Schedule Loop**.\n\nThe Schedule Loop is a handler that accepts a set of parameters like a logger and configuration and does stuff. The implementation can be anything. We have two main types of schedule loops, Queue loops and Scheduler loops.\n\nThe Queue loops manage the scheduling of jobs. They dispatch jobs from the queue to a worker and reap any completed jobs. They also manage the failing of jobs.\n\nScheduler Loops are responsible for managing other Schedulers. This allows recursive scheduling and asynchronous processing of different queues. Because you can have one scheduler that's managing to two distinct queue schedulers in their own processes.\n\n### Worker\n\nThe worker is a simple class that takes a WrappedJob and runs the Consumers on it.\n\n### Consumer\n\nA consumer is a handler that takes a `WrappedJob` instance and returns a Result. We use the Krak\\\\Mw library to transform a set of Consumer middleware into a single consumer to allow full customization with the Consumers.\n\n### Producer\n\nA producer is the antithesis of a Consumer. A producer is designed to produce a job and push it into a queue. Like consumers, producers are implemented via set of producer middleware.\n\n### Dispatch\n\nThe Dispatch is a very simple class/interface designed to wrap Job's into `WrappedJob` and then dispatch them to the producer.\n\n### Queue\n\nThe Queuing module handles the actual queueing implementations. There are two main components: Queue Managers and Queues.\n\n**Supported Queues**\n\n- Doctrine\n- Redis\n- Sqs\n- Stub\n- Sync\n\n#### Doctrine\n\nDoctrine requires the `doctrine/dbal` library to be installed.\n\n```php\n$kernel['Doctrine\\DBAL\\Connection'] = function() {\n    return Doctrine\\DBAL\\DriverManager::getConnection([\n        /* connection params */\n    ]);\n};\n$kernel['krak.job.queue_provider'] = 'doctrine';\n$kernel['krak.job.queue.doctrine.table_name'] = 'krak_jobs';\n```\n\nOnce that's setup, you'll need to perform the database migration to initialize the jobs table. The `Krak\\Job\\Queue\\Doctrine\\JobMigration` class is a utility that will facilitate running the migration.\n\nIf you already using Doctrine Migrations in your project, you can use simply use the following methods:\n\n```php\n// in your migration class\npublic function up(Schema $schema) {\n    $migration = new Krak\\Job\\Queue\\Doctrine\\JobMigration('krak_jobs');\n    $migration-\u003eup($schema);\n}\n\npublic function down(Schema $schema) {\n    $migration = new Krak\\Job\\Queue\\Doctrine\\JobMigration('krak_jobs');\n    $migration-\u003edown($schema);\n}\n```\n\nAlso, you can simply run the following php code to migrate your table up or down\n\n```php\n$conn = $kernel['Doctrine\\DBAL\\Connection'];\n$migration = $kernel['Krak\\Job\\Queue\\Doctrine\\JobMigration'];\n\n// up\n$migration-\u003emigrateUp($conn);\n// or down\n// $migration-\u003emigrateDown($conn);\n```\n\n#### Redis\n\nRedis requires the `predis/predis` library to be installed. You then just set the queue manager via:\n\n```php\n$kernel['Predis\\ClientInterface'] = function() {\n    return new Predis\\Client();\n};\n$kernel['krak.job.queue_provider'] = 'redis';\n```\n\n#### Sqs\n\nSqs requires the aws sdk to be installed `aws/aws-sdk-php`. You can set the queue manager via:\n\n```php\n$kernel['Aws\\Sqs\\SqsClient'] = function() {\n    return new Aws\\Sqs\\SqsClient();\n};\n$kernel['krak.job.queue.sqs.queue_url_map'] = ['queue-name' =\u003e '{queue-url}'];\n$kernel['krak.job.queue.sqs.receive_options'] = ['VisibilityTimeout' =\u003e 10];\n$kernel['krak.job.queue_provider'] = 'sqs';\n```\n\nThe `queue_url_map` is a cache that will be used to lookup the sqs queue url from the queue name given. This cache is optional and will be populated at runtime if not set.\n\n##### Message Configuration\n\nYou can configure how you send messages with configuration when you wrap and dispatch the job.\n\n```php\n$dispatch-\u003ewrap(new MyJob())-\u003ewith('sqs', ['AnySendMessageParamer' =\u003e 'Value'])-\u003edispatch();\n```\n\n#### Stub\n\nThe stub queue is essentially a noop queue provider. It doesn't enqueue or consume any jobs given to it.\n\n```php\n$kernel['krak.job.queue_provider'] = 'stub';\n```\n\n#### Sync\n\nThe sync(chronous) queue provider will consume your jobs synchronously in the calling thread instead of dispatching them to an external service to be consumed in a different process asynchronously. This is useful for debugging and development purposes.\n\nThis is also the default queue provider since it will work out of the box and requires no configuration.\n\n```php\n$kernel['krak.job.queue_provider'] = 'sync';\n```\n\n## Configuration Options\n\n### queue\n\n`queue` represents the name of the queue to use with the queue provider.\n\n### queue_provider\n\nThis defines the queue provider at the queue level. Instead of having using only queue provider for every queue, you might have some queues on one queue provider and others on another. A good use case for this would be splitting admin type jobs on the doctrine queue and then splitting the frontend registration jobs on something more robust/faster like sqs.\n\n### sleep\n\nThe longer the process will sleep, the less resources it will take. The queue provider will be pulled once every `sleep` number of seconds. If you plan on only processing a few jobs on the given queue, then you can set this to a higher value. Conversely, if the queue will be processing at a high throughput, you'll need to set this to smaller value like 1 or 2 seconds.\n\n### ttl\n\nThis is how long in seconds the scheduler should run before stopping. This is useful in conjunction with the `respawn` option to have the application state refresh. For example, in development, you might want to set a short ttl with a restart so that you don't have to keep restarting the scheduler to test changes.\n\n### respawn\n\nThe respawn is a boolean that determines if a parent scheduler will respawn a child queue scheduler once it's been killed.\n\n### max_jobs\n\nThis is the maximum number of worker processes that will be running at the same time. If your jobs take a longer time for completion or if the queue will be consuming a very high number of jobs, then setting this value to greater than 1 can greatly speed up the overall processing of the jobs because they will be done in parallel.\n\nKeep in mind that each process consumes memory and has a bit of overhead so this should be tuned with that in mind.\n\n### max_retry\n\nMax number of retries before just giving up on a failed job\n\n### batch_size\n\nThis is max number of jobs to process in a given batch. Every worker process that is created handles a batch of jobs. The higher the batch size helps lower the memory footprint of the system since fewer processes will be created when the batch size is higher.\n\nThis works great for jobs that finish execution relatively quickly (less than 5 seconds); however, if the jobs take much longer to execute, then you're better off increasing the max_jobs and lowering this value to around 1.\n\n## Cookbook\n\n### Async Scheduling\n\nTo perform schedule multiple queues at a time, update the kernel config like this:\n\n```php\n$kernel-\u003econfig([\n    'name' =\u003e 'Master Scheduler',\n    'sleep' =\u003e 10,\n    'schedulers' =\u003e [\n        [\n            'queue' =\u003e 'emails',\n            'max_jobs' =\u003e 20,\n            'respawn' =\u003e true, // will be respawned after exiting\n            'ttl' =\u003e 50,\n        ],\n        [\n            'queue' =\u003e 'orders',\n            'max_retry' =\u003e 3,\n        ]\n    ]\n]);\n```\n\nThis will create a master scheduler that will then manage two schedulers which manage a different queue. This will launch two separate processes that manage each queue, so the processing of each queue will be completely asynchronous.\n\n### Additional Logging\n\nYou can enable logging by defining the `Psr\\Log\\LoggerInterface` service into the kernel container.\n\n```php\n$kernel[Psr\\Log\\LoggerInterface::class] = function() {\n    return MyPsrLogger();\n};\n```\n\nAny scheduler logging will then also go to defined logger.\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkrakphp%2Fjob","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkrakphp%2Fjob","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkrakphp%2Fjob/lists"}