{"id":15713667,"url":"https://github.com/azutoolkit/joobq","last_synced_at":"2025-05-07T23:01:46.002Z","repository":{"id":51112143,"uuid":"273625384","full_name":"azutoolkit/joobq","owner":"azutoolkit","description":"JoobQ is a fast, efficient asynchronous reliable job queue and job scheduler library processing. Jobs are submitted to a job queue, where they reside until they are able to be scheduled to run in a computing environment.","archived":false,"fork":false,"pushed_at":"2024-12-13T13:17:08.000Z","size":6458,"stargazers_count":31,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-03-31T15:00:56.530Z","etag":null,"topics":["background-jobs","crystal-lang","jobs","queue","redis","scheduler","server","tasks"],"latest_commit_sha":null,"homepage":"","language":"Crystal","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"lgpl-2.1","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/azutoolkit.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-06-20T02:37:41.000Z","updated_at":"2024-12-13T13:17:12.000Z","dependencies_parsed_at":"2024-10-24T10:51:50.170Z","dependency_job_id":"9f0a100a-36b4-489a-932e-aa69a3337230","html_url":"https://github.com/azutoolkit/joobq","commit_stats":null,"previous_names":[],"tags_count":12,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/azutoolkit%2Fjoobq","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/azutoolkit%2Fjoobq/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/azutoolkit%2Fjoobq/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/azutoolkit%2Fjoobq/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/azutoolkit","download_url":"https://codeload.github.com/azutoolkit/joobq/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252968052,"owners_count":21833249,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["background-jobs","crystal-lang","jobs","queue","redis","scheduler","server","tasks"],"created_at":"2024-10-03T21:32:50.420Z","updated_at":"2025-05-07T23:01:45.919Z","avatar_url":"https://github.com/azutoolkit.png","language":"Crystal","funding_links":[],"categories":[],"sub_categories":[],"readme":"# JoobQ\n\n![JoobQ Logo](./joobq-logo.png)\n\n[![Crystal CI](https://github.com/azutoolkit/joobq/actions/workflows/crystal.yml/badge.svg?branch=master)](https://github.com/azutoolkit/joobq/actions/workflows/crystal.yml)\n\n## Overview\n\nWelcome to **JoobQ** – is a great solution for fast, efficient, and reliable asynchronous job queue scheduling! Whether you're building a small application or a large-scale system, JoobQ is designed to handle your job processing needs with ease and precision.\n\n### Why JoobQ?\n\nJoobQ is more than just a job queue scheduler; it's a robust framework that ensures your jobs are managed and executed seamlessly. Here's why developers love JoobQ:\n\n- **Blazing Fast Performance**: JoobQ leverages the power of Crystal language and Redis to deliver lightning-fast job processing.\n- **Reliability**: With built-in retry mechanisms, dead letter queues, and job expiration handling, JoobQ ensures that no job is left behind.\n- **Flexibility**: Configure your job queues, set custom retry policies, and schedule jobs with cron-like syntax – all with minimal effort.\n- **Scalability**: JoobQ is designed to scale with your application, handling millions of jobs effortlessly.\n- **Developer Friendly**: With a clean and intuitive API, JoobQ makes it easy for developers to define, enqueue, and manage jobs.\n\n### Key Features\n\n- **Priority Queues**: Assign different priorities to your job queues based on the number of workers.\n- **Error Handling**: Robust error handling with detailed logging and retry mechanisms.\n- **Cron-like Scheduling**: Schedule recurring jobs with cron syntax.\n- **Delayed Jobs**: Delay job execution to a specific time in the future.\n- **Throttle Control**: Manage the rate at which jobs are processed to prevent system overload.\n- **REST API**: Interact with JoobQ through a comprehensive REST API for job management and metrics.\n- **Graceful Termination**: Stop workers gracefully without losing jobs in progress.\n\n### Get Started!\n\nReady to dive in? Follow our [Getting Started](#getting-started) guide to set up JoobQ in your project and start processing jobs like a pro!\n\n### Table of Contents\n\n- [JoobQ](#joobq)\n  - [Overview](#overview)\n    - [Why JoobQ?](#why-joobq)\n    - [Key Features](#key-features)\n    - [Get Started!](#get-started)\n    - [Table of Contents](#table-of-contents)\n  - [Getting Started](#getting-started)\n    - [Prerequisites](#prerequisites)\n    - [Setting Up](#setting-up)\n  - [Installation](#installation)\n  - [Usage](#usage)\n    - [Environment Variables](#environment-variables)\n    - [Configuration](#configuration)\n    - [Configuration Properties](#configuration-properties)\n    - [Example Configuration with All Properties](#example-configuration-with-all-properties)\n    - [Defining Queues](#defining-queues)\n    - [Queue Throttling](#queue-throttling)\n      - [Queue Throttle Limit Property](#queue-throttle-limit-property)\n      - [How They Work Together](#how-they-work-together)\n      - [Summary](#summary)\n    - [Defining Jobs](#defining-jobs)\n      - [Best Practices for Defining Jobs](#best-practices-for-defining-jobs)\n        - [Idempotency](#idempotency)\n        - [Simple Primitive Types for Arguments](#simple-primitive-types-for-arguments)\n        - [Number of Arguments](#number-of-arguments)\n  - [JoobQ HTTP Server](#joobq-http-server)\n    - [Rest API](#rest-api)\n    - [GET /joobq/jobs/registry](#get-joobqjobsregistry)\n    - [POST /joobq/jobs](#post-joobqjobs)\n  - [Performance](#performance)\n    - [Performance Comparison](#performance-comparison)\n      - [Disclaimer](#disclaimer)\n  - [Contributing](#contributing)\n  - [Testing](#testing)\n  - [Deployment](#deployment)\n  - [License](#license)\n  - [Acknowledgments](#acknowledgments)\n\n## Getting Started\n\nThis section will help you get started with JoobQ. Follow the instructions below to set up and run the project.\n\n### Prerequisites\n\n- Crystal language (\u003e= 0.34.0)\n- Redis\n\n### Setting Up\n\n1. Clone the repository:\n\n   ```sh\n   git clone https://github.com/azutoolkit/joobq.git\n   cd joobq\n   ```\n\n2. Install dependencies:\n\n   ```sh\n   shards install\n   ```\n\n3. Start Redis\n\n   ```sh\n   redis-server\n   ```\n\n## Installation\n\nAdd the following to your `shard.yml`:\n\n```yaml\ndependencies:\n  joobq:\n    github: azutoolkit/joobq\n```\n\nThen run:\n\n```crystal\nshards install\n```\n\n## Usage\n\n```crystal\nrequire \"joobq\"\n```\n\n### Environment Variables\n\n```crystal\nREDIS_HOST=localhost\nREDIS_PORT=6379\nREDIS_POOL_SIZE=50\nREDIS_PASS=somepass\nREDIS_TIMEOUT=0.2\n```\n\n### Configuration\n\nJoobQ can be configured using the `JoobQ.configure` method. Here is an example configuration:\n\n```crystal\nJoobQ.configure do\n  queue \"default\", 10, EmailJob\n\n  scheduler do\n    cron(\"*/1 * * * *\") { # Do Something }\n    every 1.hour, EmailJob, email_address: \"john.doe@example.com\"\n  end\nend\n```\n\n### Configuration Properties\n\nThe `JoobQ::Configure` struct provides several properties that can be customized to configure the JoobQ system:\n\n- `default_queue`: The name of the default queue. Defaults to `\"default\"`.\n- `retries`: The number of retries for a job. Defaults to `3`.\n- `expires`: The expiration time for a job. Defaults to `3.days`.\n- `timeout`: The maximum execution time allowed for a job. Defaults to `2.seconds`.\n- `failed_ttl`: The time-to-live for failed jobs. Defaults to `3.milliseconds`.\n- `dead_letter_ttl`: The time-to-live for jobs in the dead letter queue. Defaults to `7.days`.\n- `job_registry`: An instance of `JobSchemaRegistry` for managing job schemas.\n- `store`: The store instance used for job storage and retrieval. Defaults to `RedisStore`.\n\n### Example Configuration with All Properties\n\n```crystal\nJoobQ.configure do |config|\n  config.default_queue = \"default\"\n  config.retries = 5\n  config.expires = 2.days\n  config.timeout = 5.seconds\n  config.failed_ttl = 1.minute\n  config.dead_letter_ttl = 14.days\n\n  queue \"default\", 10, EmailJob\n  queue \"priority\", 5, PriorityJob, throttle: { limit: 100, period: 1.minute }\n\n  scheduler do\n    cron(\"*/1 * * * *\") { # Do Something }\n    every 1.hour, EmailJob, email_address: \"john.doe@example.com\"\n  end\nend\n```\n\n### Defining Queues\n\nQueues are of type Hash(String, Queue(T)) where the name of the key matches the name of the Queue.\n\n**Example:**\n\n```crystal\nJoobQ.configure do\n  queue name: \"single\", workers: 10, job: Job1, throttle: { limit: 20, period: 1.minute }\n  queue \"example\", 10, ExampleJob | FailJob\n\n  # Scheduling Recurring Jobs\n  scheduler do\n    cron(\"*/1 * * * *\") { # Do Something }\n    cron(\"*/5 20-23 * * *\") { # Do Something }\n    every 1.hour, ExampleJob, x: 1\n  end\nend\n```\n\n### Queue Throttling\n\nThe worker throttle mechanism in JoobQ works in conjunction with the Queue Throttle Limit property to manage the rate at\nwhich jobs are processed. Here's how it works:\n\n#### Queue Throttle Limit Property\n\nThe Queue Throttle limit property sets a maximum number of jobs that can be processed within a specific time frame. This\nhelps to control the load on the system and ensures that the job processing rate does not exceed a certain threshold.\n\n#### How They Work Together\n\n- **Job Availability and Throttle Limit:** The worker checks the queue for available jobs. If the number of jobs processed\n  within the specified time frame reaches the throttle limit, the worker will wait until it is allowed to process more\n  jobs. This prevents the system from being overwhelmed by too many jobs at once.\n\n- **Job Expiration:** Before processing a job, the worker checks if the job has expired. If the job has expired, it is\n  marked as expired and not processed. This ensures that only valid jobs are processed, reducing unnecessary work.\n\n- **Controlled Shutdown:** The worker can be stopped gracefully by sending a termination signal. This allows for a\n  controlled shutdown, ensuring that no jobs are abruptly terminated.\n\n**Example:**\n\nHere is an example of how you might configure the Queue Throttle limit property:\n\n```crystal\nJoobQ.configure do\n  queue \"default\", 10, EmailJob, throttle: { limit: 100, period: 60.seconds }\nend\n```\n\nIn this example, the throttle limit is set to 100 jobs per 60 seconds. This means that the worker will process up to 100\njobs every 60 seconds. If the limit is reached, the worker will wait until the next period to continue processing jobs.\n\n#### Summary\n\nThe worker throttle mechanism, combined with the Queue Throttle limit property, ensures that job processing is controlled\nand efficient. By managing job availability, expiration, and processing rate, JoobQ provides a robust system for\nhandling job queues without overwhelming the system.\n\n### Defining Jobs\n\nTo define Jobs, include the JoobQ::Job module, and implement the perform method.\n\n```crystal\nstruct EmailJob\n  include JoobQ::Job\n\n  # Name of the queue to be processed by\n  @queue   = \"default\"\n  # Number Of Retries for this job\n  @retries = 0\n  # Job Expiration\n  @expires = 1.days.total_seconds.to_i\n\n  # Initialize as normal with or without named tuple arguments\n  def initialize(@email_address : String)\n  end\n\n  def perform\n    # Logic to handle job execution\n  end\nend\n```\n\n**Executing Job:**\n\n```crystal\n# Enqueue the job (Async)\nEmailJob.enqueue(email_address: \"john.doe@example.com\")\n\n# Batch enqueue jobs\nEmailJob.batch_enqueue([job1, job2, job3])\n\n# Perform Immediately\nEmailJob.new(email_address: \"john.doe@example.com\").perform\n\n# Delayed\nEmailJob.delay(for: 1.hour, email_address: \"john.doe@example.com\")\nEmailJob.enqueue_at(time: 3.minutes.from_now, email_address: \"john.doe@example.com\")\n\n# Recurring at given interval\nEmailJob.schedule(every: 1.second, email_address: \"john.doe@example.com\")\n```\n\n#### Best Practices for Defining Jobs\n\nWhen defining jobs in JoobQ, it's important to follow certain best practices to ensure reliability and maintainability. Here are some key recommendations:\n\n##### Idempotency\n\nJobs must be idempotent. This means that running the same job multiple times should produce the same result. Idempotency is crucial for ensuring that jobs can be retried safely without causing unintended side effects. To achieve idempotency:\n\n- Avoid modifying external state directly within the job.\n- Use unique identifiers to track job execution and prevent duplicate processing.\n- Ensure that any side effects (e.g., database updates, API calls) are safe to repeat.\n\n##### Simple Primitive Types for Arguments\n\nJob arguments must be simple primitive types such as integers, strings, and booleans. This ensures that the job data can be easily serialized and deserialized, and reduces the risk of errors during job execution. Complex objects or data structures should be avoided as job arguments.\n\n##### Number of Arguments\n\nKeep the number of arguments for jobs to a minimum. Having too many arguments can make the job definition complex and harder to maintain. As a best practice:\n\n- Limit the number of arguments to 3-5.\n- Group related arguments into a single object if necessary.\n- Use default values for optional arguments to simplify job invocation.\n\nBy following these best practices, you can ensure that your jobs are reliable, maintainable, and easy to work with in the JoobQ framework.\n\n## JoobQ HTTP Server\n\nThe `APIServer` class provides a REST API to interact with the JoobQ job queue system. It listens for HTTP requests and handles job enqueuing, job registry retrieval, and queue metrics.\n\nTo start the API server.\n\n```crystal\nAPIServer.start\n```\n\n### Rest API\n\nJoobQ provides a REST API to interact with the job queue. Below are the available endpoints:\n\n### GET /joobq/jobs/registry\n\nThis endpoint returns all available registered jobs and their JSON schemas that can be enqueued via the REST API.\n\n**Request:**\n\n```http\nGET /joobq/jobs/registry HTTP/1.1\nHost: localhost:8080\n```\n\n**Response:**\n\n```json\n{\n  \"EmailJob\": {\n    \"type\": \"object\",\n    \"properties\": {\n      \"email_address\": {\n        \"type\": \"string\"\n      }\n    },\n    \"required\": [\"email_address\"]\n  }\n}\n```\n\n### POST /joobq/jobs\n\nThis endpoint allows users to enqueue jobs.\n\n**Request:**\n\n```http\nPOST /joobq/jobs HTTP/1.1\nHost: localhost:8080\nContent-Type: application/json\nContent-Length: 175\n\n{\n    \"jid\": \"a13324f4-bdd8-4cf5-b566-c0c9c312f68b\",\n    \"queue\": \"queue:test\",\n    \"retries\": 3,\n    \"expires\": {{timestamp_plus_30}},\n    \"status\": \"enqueued\",\n    // Job initialization arguments\n    \"x\": 10\n}\n```\n\n**Response:**\n\n```json\n{\n  \"status\": \"Job enqueued\",\n  \"queue\": \"default\",\n  \"job_id\": \"some-unique-job-id\"\n}\n```\n\n## Performance\n\nJoobQ is designed for high performance and scalability. In our benchmarks, JoobQ has easily achieved processing rates of 35,000 jobs per second. This performance is made possible by leveraging Crystal's concurrency model and efficient job handling mechanisms.\n\n### Performance Comparison\n\nTo provide a clearer picture of JoobQ's performance, we have compared it with other popular job queue processing tools\nin various programming languages, including Sidekiq (Ruby), Celery (Python), Laravel Queue (PHP), and Quartz (Java).\nThe results are summarized in the table below:\n\n| Job Queue Tool | Language | Jobs per Second |\n| -------------- | -------- | --------------- |\n| JoobQ          | Crystal  | 35,000          |\n| Sidekiq        | Ruby     | 23,500          |\n| Celery         | Python   | 15,000          |\n| Laravel Queue  | PHP      | 10,000          |\n| Quartz         | Java     | 25,000          |\n\nJoobQ Hardware benchmarks performed\n\n```info\nHardware Overview:\n  Model Name:             MacBook Pro\n  Model Identifier:       Mac15,10\n  Model Number:           MRX53LL/A\n  Chip:                   Apple M3 Max\n  Total Number of Cores: 14 (10 performance and 4 efficiency)\n  Memory:                 36 GB\n```\n\nAs shown in the table, JoobQ outperforms many of the popular job queue processing tools, making it an excellent\nchoice for high-throughput job processing needs.\n\n#### Disclaimer\n\nJoobQ out of the box provides great performance, achieving processing rates of up to 35,000 jobs per second in our\nbenchmarks. However, it's important to note that with the right environment and settings, any job scheduler can be\nperformant. Factors such as hardware specifications, network conditions, and job complexity can significantly impact\nthe performance of job queue processing tools. Therefore, it's essential to optimize your environment and configurations\nto achieve the best possible performance for your specific use case.\n\n## Contributing\n\n1. Fork it (\u003chttps://github.com/azutoolkit/joobq/fork\u003e)\n2. Create your feature branch (git checkout -b my-new-feature)\n3. Commit your changes (git commit -am 'Add some feature')\n4. Push to the branch (git push origin my-new-feature)\n5. Create a new Pull Request\n\n## Testing\n\nTo run the tests, use the following command:\n\n```bash\ncrystal spec\n```\n\n## Deployment\n\nTo deploy JoobQ, ensure that you have a running Redis instance. You can use the provided docker-compose.yml to set\nup Redis.\n\n```bash\ndocker-compose up -d\n```\n\n## License\n\nThe MIT License (MIT). Please see License File for more information.\n\n## Acknowledgments\n\n[Elias J. Perez](https://github.com/eliasjpr/) - creator and maintainer\n[Crystal Language](https://crystal-lang.org/)\n[Redis](https://redis.io/)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fazutoolkit%2Fjoobq","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fazutoolkit%2Fjoobq","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fazutoolkit%2Fjoobq/lists"}