{"id":27627204,"url":"https://github.com/treasure-data/perfectqueue","last_synced_at":"2025-04-23T13:53:31.317Z","repository":{"id":677655,"uuid":"2255601","full_name":"treasure-data/perfectqueue","owner":"treasure-data","description":"Highly available distributed queue built on RDBMS","archived":false,"fork":false,"pushed_at":"2022-10-04T14:24:07.000Z","size":414,"stargazers_count":124,"open_issues_count":5,"forks_count":17,"subscribers_count":83,"default_branch":"master","last_synced_at":"2025-03-23T11:19:39.364Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/treasure-data.png","metadata":{"files":{"readme":"README.md","changelog":"ChangeLog","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2011-08-23T14:27:29.000Z","updated_at":"2024-06-20T03:11:42.000Z","dependencies_parsed_at":"2022-08-16T10:40:27.179Z","dependency_job_id":null,"html_url":"https://github.com/treasure-data/perfectqueue","commit_stats":null,"previous_names":[],"tags_count":93,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/treasure-data%2Fperfectqueue","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/treasure-data%2Fperfectqueue/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/treasure-data%2Fperfectqueue/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/treasure-data%2Fperfectqueue/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/treasure-data","download_url":"https://codeload.github.com/treasure-data/perfectqueue/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250444199,"owners_count":21431601,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-04-23T13:53:30.715Z","updated_at":"2025-04-23T13:53:31.299Z","avatar_url":"https://github.com/treasure-data.png","language":"Ruby","funding_links":[],"categories":["Ruby"],"sub_categories":[],"readme":"# PerfectQueue\n\n[![Build Status](https://travis-ci.org/treasure-data/perfectqueue.svg?branch=master)](https://travis-ci.org/treasure-data/perfectqueue)\n[![Coverage Status](https://coveralls.io/repos/treasure-data/perfectqueue/badge.svg?branch=master\u0026service=github)](https://coveralls.io/github/treasure-data/perfectqueue?branch=master)\n\nPerfectQueue is a highly available distributed queue built on top of RDBMS.\nPerfectQueue provides similar API to Amazon SQS, while PerfectQueue focuses on reliability and flexible scheduling rather than scalability.\n\nPerfectQueue introduces following concepts:\n\n  * **At-least-once semantics:** Even if a worker node fails during processing a task, another worker takes over the task.\n  * **Multiuser-aware fair scheduling**: PerfectQueue schedules tasks submitted by users who have larger resource assignment.\n  * **Decider:** Decider is a simple mechanism to implement complex workflows on top of queues while keeping loose coupling.\n  * **Graceful and live restarting:** PerfectQueue continues processing of long-running tasks even during restarting.\n  * **Idempotent task submission:** All tasks have unique identifier and PerfectQueue prevents storing same task twice.\n    * Note: client applications should consider how to always generate a same string for a (semantically) same task.\n  * **Idempotent task processing support:** Workers can use a constant unique identifier to process tasks idempotently.\n\nAll you have to consider is implementing idempotent worker programs. PerfectQueue manages the other problems.\n\n## API overview\n\n```\n# open a queue\nPerfectQueue.open(config, \u0026block)  #=\u003e #\u003cQueue\u003e\n\n# submit a task\nQueue#submit(task_id, type, data, options={})\n\n# poll a task\n# (you don't have to use this method directly. see following sections)\nQueue#poll  #=\u003e #\u003cAcquiredTask\u003e\n\n# get data associated with a task\nAcquiredTask#data  #=\u003e #\u003cHash\u003e\n\n# finish a task\nAcquiredTask#finish!\n\n# retry a task\nAcquiredTask#retry!\n\n# create a task reference\nQueue#[](key)  #=\u003e #\u003cTask\u003e\n\n# chack the existance of the task\nTask#exists?\n\n# force finish a task\n# be aware that worker programs can't detect it\nTask#force_finish!\n```\n\n### Error classes\n\n```\nTaskError\n\n##\n# Workers may get these errors:\n#\n\nAlreadyFinishedError \u003c TaskError\n\nPreemptedError \u003c TaskError\n\nProcessStopError \u003c RuntimeError\n\nImmediateProcessStopError \u003c ProcessStopError\n\nGracefulProcessStopError \u003c ProcessStopError\n\n##\n# Client or other situation:\n#\n\nConfigError \u003c RuntimeError\n\nNotFoundError \u003c TaskError\n\nAlreadyExistsError \u003c TaskError\n\nNotSupportedError \u003c TaskError\n```\n\n\n###  Example\n\n```ruby\n# submit tasks\nPerfectQueue.open(config) {|queue|\n  data = {'key'=\u003e\"value\"}\n  queue.submit(\"task-id\", \"type1\", data)\n}\n```\n\n\n## Writing a worker application\n\n### 1. Implement PerfectQueue::Application::Base\n\n```ruby\nclass TestHandler \u003c PerfectQueue::Application::Base\n  # implement run method\n  def run\n    # do something ...\n    puts \"acquired task: #{task.inspect}\"\n\n    # call task.finish!, task.retry! or task.release!\n    task.finish!\n  end\nend\n```\n\n### 2. Implement PerfectQueue::Application::Dispatch\n\n```ruby\nclass Dispatch \u003c PerfectQueue::Application::Dispatch\n  # describe routing\n  route \"type1\" =\u003e TestHandler\n  route /^regexp-.*$/ =\u003e :TestHandler  # String or Regexp =\u003e Class or Symbol\nend\n```\n\n### 3. Run the worker\n\nIn a launcher script or rake file:\n\n```ruby\nsystem('perfectqueue run -I. -rapp/workers/dispatch Dispatch')\n```\n\nor:\n\n```ruby\nrequire 'perfectqueue'\nrequire 'app/workers/dispatch'\n\nPerfectQueue::Worker.run(Dispatch) {\n  # this method is called when the worker process is restarted\n  raw = File.read('config/perfectqueue.yml')\n  yml = YAJL.load(raw)\n  yml[ENV['RAILS_ENV'] || 'development']\n}\n```\n\n### Signal handlers\n\n- **TERM:** graceful shutdown\n- **QUIT:** immediate shutdown\n- **USR1:** graceful restart\n- **HUP:** immediate restart\n- **USR2:** reopen log files\n- **INT:** detach process for live restarting\n\n## Configuration\n\n- **type:** backend type (required; see following sections)\n- **log:** log file path (default: use stderr)\n- **processors:** number of child processes (default: 1)\n- **processor_type:** type of processor ('process' or 'thread') (default: 'process')\n- **poll_interval:** interval to poll tasks in seconds (default: 1.0 sec)\n- **retention_time:** duration to retain finished tasks (default: 300 sec)\n- **task_heartbeat_interval:** interval to send heartbeat requests (default: 2 sec)\n- **alive_time:** duration to continue a heartbeat request (default: 300 sec)\n- **retry_wait:** duration to retry a retried task (default: 300 sec)\n- **child_kill_interval:** interval to send signals to a child process (default: 2.0 sec)\n- **child_graceful_kill_limit:** threshold time to switch SIGTERM to SIGKILL (default: never)\n- **child_heartbeat_interval:** interval to send heartbeat packets to a child process (default: 2 sec)\n- **child_heartbeat_limit:** threshold time to detect freeze of a child process (default: 10.0 sec)\n- **detach_wait:** sleep for this seconds before detaching process for live restarting (default: 10.0 sec)\n\n## Backend types\n\n### rdb\\_compat\n\nadditional configuration:\n\n- **url:** URL to the RDBMS (example: 'mysql://user:password@host:port/database')\n- **table:** name of the table to use\n\n### rdb\n\nNot implemented yet.\n\n## config/perfectqueue.yml Example\n\n```\ndevelopment:\n  type: rdb_compat\n  url: mysql2://root:@localhost:3306/perfectqueue\n  table: queues\n```\n\n## Command line management tool\n\n```\nUsage: perfectqueue [options] \u003ccommand\u003e\n\ncommands:\n    list                             Show list of tasks\n    submit \u003ckey\u003e \u003ctype\u003e \u003cdata\u003e       Submit a new task\n    force_finish \u003ckey\u003e               Force finish a task\n    run \u003cclass\u003e                      Run a worker process\n    init                             Initialize a backend database\n\noptions:\n    -e, --environment ENV            Framework environment (default: development)\n    -c, --config PATH.yml            Path to a configuration file (default: config/perfectqueue.yml)\n\noptions for submit:\n    -u, --user USER                  Set user\n    -t, --time UNIXTIME              Set time to run the task\n\noptions for run:\n    -I, --include PATH               Add $LOAD_PATH directory\n    -r, --require PATH               Require files before starting\n```\n\n\n### initializing a database\n\n    # assume that the config/perfectqueue.yml exists\n    $ perfectqueue init\n\n### submitting a task\n\n    $ perfectqueue submit k1 user_task '{\"uid\":1}' -u user_1\n\n### listing tasks\n\n    $ perfectqueue list\n                               key            type               user             status                   created_at                      timeout   data\n                                k1       user_task             user_1            waiting    2012-05-18 13:05:31 -0700    2012-05-18 14:27:36 -0700   {\"uid\"=\u003e1, \"type\"=\u003e\"user_task\"}\n                                k2       user_task             user_2            waiting    2012-05-18 13:35:33 -0700    2012-05-18 14:35:33 -0700   {\"uid\"=\u003e2, \"type\"=\u003e\"user_task\"}\n                                k3     system_task                               waiting    2012-05-18 14:04:02 -0700    2012-05-22 15:04:02 -0700   {\"task_id\"=\u003e32, \"type\"=\u003e\"system_task\"}\n    3 entries.\n\n### force finish a tasks\n\n    $ perfectqueue force_finish k2\n\n### running a worker\n\n    $ perfectqueue run -I. -Ilib -rconfig/boot.rb -rapps/workers/task_dispatch.rb TaskDispatch\n\n## Test\n\nRunning spec utilize 'mysql2://root:@localhost/perfectqueue_test' as the connection string.\nPlease install MySQL server at localhost then run;\n\n    $ mysql -h localhost -u root -e 'create database perfectqueue_test;'\n\nYou can run spec.\n\n    $ bundle exec rake spec\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftreasure-data%2Fperfectqueue","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftreasure-data%2Fperfectqueue","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftreasure-data%2Fperfectqueue/lists"}