{"id":13792487,"url":"https://github.com/tarantool/queue","last_synced_at":"2025-04-02T03:10:23.305Z","repository":{"id":867842,"uuid":"9913655","full_name":"tarantool/queue","owner":"tarantool","description":"Create task queues, add and take jobs, monitor failed tasks","archived":false,"fork":false,"pushed_at":"2024-10-09T10:16:17.000Z","size":555,"stargazers_count":236,"open_issues_count":31,"forks_count":52,"subscribers_count":40,"default_branch":"master","last_synced_at":"2024-12-22T18:05:12.048Z","etag":null,"topics":["lua","queue","tarantool"],"latest_commit_sha":null,"homepage":"","language":"Lua","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tarantool.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2013-05-07T14:23:26.000Z","updated_at":"2024-12-13T10:26:12.000Z","dependencies_parsed_at":"2024-03-14T14:50:01.220Z","dependency_job_id":"a58c971c-92f2-41b5-845e-fb6fab5e79c8","html_url":"https://github.com/tarantool/queue","commit_stats":{"total_commits":244,"total_committers":45,"mean_commits":"5.4222222222222225","dds":0.7786885245901639,"last_synced_commit":"ea8449b6fa06c373124cee0530e71c7fd7a11c78"},"previous_names":[],"tags_count":22,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tarantool%2Fqueue","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tarantool%2Fqueue/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tarantool%2Fqueue/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tarantool%2Fqueue/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tarantool","download_url":"https://codeload.github.com/tarantool/queue/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246746934,"owners_count":20827061,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["lua","queue","tarantool"],"created_at":"2024-08-03T22:01:12.749Z","updated_at":"2025-04-02T03:10:23.278Z","avatar_url":"https://github.com/tarantool.png","language":"Lua","readme":"\u003ca href=\"http://tarantool.org\"\u003e\n   \u003cimg src=\"https://avatars2.githubusercontent.com/u/2344919?v=2\u0026s=250\"\nalign=\"right\"\u003e\n\u003c/a\u003e\n\n[![fast_testing][testing-actions-badge]][testing-actions-url]\n[![packaging][packaging-actions-badge]][packaging-actions-url]\n[![publish][publish-actions-badge]][publish-actions-url]\n\n# A collection of persistent queue implementations for Tarantool\n\n## Table of contents\n\n* [Queue types](#queue-types)\n  * [fifo \\- a simple queue](#fifo---a-simple-queue)\n  * [fifottl \\- a simple priority queue with support for task time to live](#fifottl---a-simple-priority-queue-with-support-for-task-time-to-live)\n  * [utube \\- a queue with sub\\-queues inside](#utube---a-queue-with-sub-queues-inside)\n  * [utubettl \\- extension of utube to support ttl](#utubettl---extension-of-utube-to-support-ttl)\n* [The underlying spaces](#the-underlying-spaces)\n  * [Fields of the \\_queue space](#fields-of-the-_queue-space)\n  * [Fields of the \\_queue\\_consumers space](#fields-of-the-_queue_consumers-space)\n  * [Fields of the \\_queue\\_taken\\_2 space](#fields-of-the-_queue_taken_2-space)\n  * [Fields of the \\_queue\\_session\\_ids space](#fields-of-the-_queue_session_ids-space)\n  * [Fields of the space associated with each queue](#fields-of-the-space-associated-with-each-queue)\n* [Task state diagram](#task-state-diagram)\n* [Queue state diagram](#queue-state-diagram)\n* [Installing](#installing)\n* [Using the queue module](#using-the-queue-module)\n  * [Initialization](#initialization)\n  * [Get the module version](#get-the-module-version)\n  * [Creating a new queue](#creating-a-new-queue)\n  * [Set queue settings](#set-queue-settings)\n  * [Session identify](#session-identify)\n  * [Putting a task in a queue](#putting-a-task-in-a-queue)\n  * [Taking a task from the queue (\"consuming\")](#taking-a-task-from-the-queue-consuming)\n  * [Acknowledging the completion of a task](#acknowledging-the-completion-of-a-task)\n  * [Releasing a task](#releasing-a-task)\n  * [Peeking at a task](#peeking-at-a-task)\n  * [Burying a task](#burying-a-task)\n  * [Kicking a number of tasks](#kicking-a-number-of-tasks)\n  * [Deleting a task](#deleting-a-task)\n  * [Dropping a queue](#dropping-a-queue)\n  * [Releasing all taken tasks](#releasing-all-taken-tasks)\n  * [Getting statistics](#getting-statistics)\n  * [Queue and replication](#queue-and-replication)\n* [Implementation details](#implementation-details)\n  * [Queue drivers](#queue-drivers)\n  * [Driver API](#driver-api)\n\n# Queue types\n\n## `fifo` - a simple queue\n\nFeatures:\n\n* If there is only one consumer, tasks are scheduled in strict FIFO order.\n* If there are many concurrent consumers, FIFO order is preserved on\naverage, but is less strict: concurrent consumers may complete tasks in\na different order.\n\nThe following options can be specified when creating a `fifo` queue:\n  * `temporary` - boolean - if true, the contents do not persist on disk\n(the queue is in-memory only)\n  * `if_not_exists` - boolean - if true, no error will be returned if the tube\nalready exists\n  * `on_task_change` - function name - a callback to be executed on every\noperation; the expected function syntax is `function(task, stats_data)`, where\n`stats_data` is the operation type, and `task` is normalized task data.\n    NOTE: It's better to use `:on_task_change()` function.\n\n`fifo` queue does not support:\n  * task priorities (`pri`)\n  * task time to live (`ttl`)\n  * task time to execute (`ttr`)\n  * delayed execution (`delay`)\n\nExample:\n\n```lua\n\n-- add a log record on task completion\nlocal function otc_cb(task, stats_data)\n    if stats_data == 'delete' then\n        log.info(\"task %s is done\", task[1])\n    end\nend\n\nqueue.create_tube('tube_name', 'fifo', {temporary = true, on_task_change = otc_cb})\nqueue.tube.tube_name:put('my_task_data_1')\nqueue.tube.tube_name:put('my_task_data_2')\n\n```\n\nIn the example above, the `otc_cb` function will be called 2 times, on each task\ncompletion. Values for the callback arguments will be taken from the queue.\n\n## `fifottl` - a simple priority queue with support for task time to live\n\nThe following options can be specified when creating a `fifottl` queue:\n  * `temporary` - boolean - if true, the contents of the queue do not persist\non disk\n  * `if_not_exists` - boolean - if true, no error will be returned if the tube\nalready exists\n  * `on_task_change` - function name - a callback to be executed on every\noperation\n\nThe following options can be specified when putting a task in a `fifottl` queue:\n  * `pri` - task priority (`0` is the highest priority and is the default)\n  * `ttl` - numeric - time to live for a task put into the queue, in\nseconds. if `ttl` is not specified, it is set to infinity\n    (if a task exists in a queue for longer than ttl seconds, it is removed)\n  * `ttr` - numeric - time allotted to the worker to work on a task, in\nseconds; if `ttr` is not specified, it is set to the same as `ttl`\n    (if a task is being worked on for more than `ttr` seconds, its status\nis changed to 'ready' so another worker may take it)\n  * `delay` - time to wait before starting to execute the task, in seconds\n\nExample:\n\n```lua\n\nqueue.create_tube('tube_name', 'fifottl', {temporary = true})\nqueue.tube.tube_name:put('my_task_data', { ttl = 60.1, delay = 80 })\n\n```\n\nIn the example above, the task has 60.1 seconds to live, but the start\nof execution is delayed for 80 seconds. Thus the task actually will\nexist for up to (60.1 + 80) 140.1 seconds.\n\nA smaller priority value indicates a higher priority, so a task with\npriority 1 will be executed after a task with priority 0, if all other\noptions are equal.\n\n## `limfifottl` - a simple size-limited priority queue with support for task time to live\n\nWorks same as fifottl, but has limitied size and put operation timeout.\n\nThe following options can be specified when creating a `fifottl` queue:\n  * `temporary` - boolean - if true, the contents of the queue do not persist\non disk\n  * `if_not_exists` - boolean - if true, no error will be returned if the tube\nalready exists\n  * `on_task_change` - function name - a callback to be executed on every\noperation\n  * `capacity` - number - limit size of the queue\n\nThe following options can be specified when putting a task in a `fifottl` queue:\n  * `pri` - task priority (`0` is the highest priority and is the default)\n  * `ttl` - numeric - time to live for a task put into the queue, in\nseconds. if `ttl` is not specified, it is set to infinity\n    (if a task exists in a queue for longer than ttl seconds, it is removed)\n  * `ttr` - numeric - time allotted to the worker to work on a task, in\nseconds; if `ttr` is not specified, it is set to the same as `ttl`\n    (if a task is being worked on for more than `ttr` seconds, its status\nis changed to 'ready' so another worker may take it)\n  * `delay` - time to wait before starting to execute the task, in seconds\n  * `timeout` - numeric - seconds to wait until queue has free space; if\n  `timeout` is not specified or time is up, and queue has no space, method return Nil\n\n\n## `utube` - a queue with sub-queues inside\n\nThe main idea of this queue backend is the same as in a `fifo` queue:\nthe tasks are executed in FIFO order.\nHowever, tasks may be grouped into sub-queues.\n\nIt is advised not to use `utube` methods inside transactions with\n`read-confirmed` isolation level. It can lead to errors when trying to make\nparallel tube methods calls with mvcc enabled.\n\nThe following options can be specified when creating a `utube` queue:\n  * `temporary` - boolean - if true, the contents of the queue do not persist\non disk\n  * `if_not_exists` - boolean - if true, no error will be returned if the tube\nalready exists\n  * `on_task_change` - function name - a callback to be executed on every\noperation\n  * `storage_mode` - string - one of\n    * `queue.driver.utube.STORAGE_MODE_DEFAULT` (\"default\") - default\n      implementation of `utube`\n    * `queue.driver.utube.STORAGE_MODE_READY_BUFFER`\n      (\"ready_buffer\") - allows processing `take` requests faster, but\n      by the cost of `put` operations speed. Right now this option is supported\n      only for `memtx` engine.\n      WARNING: this is an experimental storage mode.\n\n    Here is a benchmark comparison of these two modes:\n    * Benchmark for simple `put` and `take` methods. 30k utubes are created\n      with a single task each. Task creation time is calculated. After that\n      30k consumers are calling `take` + `ack`, each in the separate fiber.\n      Time to ack all tasks is calculated. The results are as follows:\n\n      |         | put (30k) | take+ack |\n      |---------|-----------|----------|\n      | default | 180ms     | 1.6s     |\n      | ready   | 270ms     | 1.7s     |\n    * Benchmark for the busy utubes. 10 tubes are created.\n      Each contains 1000 tasks. After that, 10 consumers are created (each works\n      on his tube only, one tube — one consumer). Each consumer will \n      `take`, then `yield` and then `ack` every task from their utube \n      (1000 tasks each).\n      After that, we can also run this benchmark with 10k tasks on each utube,\n      100k tasks and 150k tasks. But all that with 10 utubes and 10 consumers.\n      The results are as follows:\n\n      |         | 1k    | 10k  | 50k  | 150k  |\n      |---------|-------|------|------|-------|\n      | default | 53s   | 1.5h | 100h | 1000h |\n      | ready   | 450ms | 4.7s | 26s  | 72s   |\n\nThe following options can be specified when putting a task in a `utube`\nqueue:\n  * `utube` - the name of the sub-queue.\nSub-queues split the task stream according to the sub-queue name: it is\nnot possible to take two tasks\nout of a sub-queue concurrently, each sub-queue is executed in strict\nFIFO order, one task at a time.\n\n`utube` queue does not support:\n  * task priorities (`pri`)\n  * task time to live (`ttl`)\n  * task time to execute (`ttr`)\n  * delayed execution (`delay`)\n\nExample:\n\nImagine a web crawler, fetching pages from the Internet and finding URLs\nto fetch more pages.\nThe web crawler is based on a queue, and each task in the queue refers\nto a URL which the web crawler must download and process.\nIf the web crawler is split into many worker processes, then the same URL may\nshow up in the queue many times, because a single URL may be referred to by many\nlinking pages.\nAnd the worker processes, working in parallel, can cause a denial-of-service on\nthe site of the  URL. As a result, the web crawler can end up in the web\nserver's user-agent ban list -- not a desirable outcome.\n\nIf the URL's domain name is used as a sub-queue name, this problem can be\nsolved: all the URLs with the same domain name can be fetched and processed\nin strict FIFO order.\n\n## `utubettl` - extension of `utube` to support `ttl`\n\nThis queue type is effectively a combination of `fifottl` and `utube`.\n\nIt is advised not to use `utubettl` methods inside transactions with\n`read-confirmed` isolation level. It can lead to errors when trying to make\nparallel tube methods calls with mvcc enabled.\n\nThe following options can be specified when creating a `utubettl` queue:\n  * `temporary` - boolean - if true, the contents of the queue do not persist\non disk\n  * `if_not_exists` - boolean - if true, no error will be returned if the tube\nalready exists\n  * `on_task_change` - function name - a callback to be executed on every\noperation\n  * `storage_mode` - string - one of\n    * `queue.driver.utubettl.STORAGE_MODE_DEFAULT` (\"default\") - default\n      implementation of `utubettl`\n    * `queue.driver.utubettl.STORAGE_MODE_READY_BUFFER`\n      (\"ready_buffer\") - allows processing `take` requests faster, but\n      by the cost of `put` operations speed. Right now this option is supported\n      only for `memtx` engine.\n      WARNING: this is an experimental storage mode.\n\n    Here is a benchmark comparison of these two modes:\n    * Benchmark for simple `put` and `take` methods. 30k utubes are created\n      with a single task each. Task creation time is calculated. After that\n      30k consumers are calling `take` + `ack`, each in the separate fiber.\n      Time to ack all tasks is calculated. The results are as follows:\n\n      |         | put (30k) | take+ack |\n      |---------|-----------|----------|\n      | default | 200ms     | 1.7s     |\n      | ready   | 320ms     | 1.8s     |\n    * Benchmark for the busy utubes. 10 tubes are created.\n      Each contains 1000 tasks. After that, 10 consumers are created (each works\n      on his tube only, one tube — one consumer). Each consumer will\n      `take`, then `yield` and then `ack` every task from their utube\n      (1000 tasks each).\n      After that, we can also run this benchmark with 10k tasks on each utube,\n      100k tasks and 140k tasks. But all that with 10 utubes and 10 consumers.\n      The results are as follows:\n\n      |         | 1k    | 10k  | 50k  | 140k  |\n      |---------|-------|------|------|-------|\n      | default | 80s   | 1.6h | 100h | 1000h |\n      | ready   | 520ms | 5.4s | 28s  | 83s   |\n\nThe following options can be specified when putting a task in a\n`utubettl` queue:\n  * `pri` - task priority (`0` is the highest priority and is the default)\n  * `utube` - the name of the sub-queue\n  * `ttl` - numeric - time to live for a task put into the queue, in\nseconds. if `ttl` is not specified, it is set to infinity\n    (if a task exists in a queue for longer than ttl seconds, it is removed)\n  * `ttr` - numeric - time allotted to the worker to work on a task, in\nseconds; if `ttr` is not specified, it is set to the same as `ttl`\n    (if a task is being worked on for more than `ttr` seconds, its status\nis changed to 'ready' so another worker may take it)\n  * `delay` - time to wait before starting to execute the task, in seconds\n\n# The underlying spaces\n\nThe queue system consists of fibers, IPC channels, functions, and spaces.\nHere is how queues map to spaces in a Tarantool database.\n\nThe `_queue` space contains tuples for each queue and its properties.\nThis space is created automatically when the queue system is initialized for\nthe first time (for example, by \"require 'queue'\"), and is re-used on\nlater occasions.\n\n## Fields of the `_queue` space\n\n1. `tube` - the name of the queue\n1. `tube_id` - queue ID, numeric\n1. `space` - the name of a space associated with the queue, which\ncontains one tuple for each queue task\n1. `type` - the queue type ('fifo', 'fifottl', 'utube', 'utubettl')\n1. `opts` - additional options supplied when creating the queue, for\nexample 'ttl'\n\nThe `_queue_consumers` temporary space contains tuples for each job\nwhich is working on a queue.\nConsumers may be simply waiting for tasks to be put in the queues.\n\n## Fields of the `_queue_consumers` space\n\n1. `connection_id` - connection ID of the client\n1. `fid` - client fiber ID\n1. `tube_id` - queue ID, referring to the `tube_id` field in the `_queue`\nspace; the client waits for tasks in this queue\n1. `timeout` - the client wait timeout\n1. `time` - the time when the client took a task\n\nThe `_queue_taken_2` (`_queue_taken` is deprecated) space contains\ntuples for each job which is processing a task in the queue.\n\n## Fields of the `_queue_taken_2` space\n\n1. `tube_id` - queue ID, to which the task belongs\n1. `task_id` - task ID (of the task being taken)\n1. `connection_id` - connection ID of the client, referring to the\n`connection_id` field of the `_queue_consumers` space\n1. `session_uuid` - session UUID (string)\n1. `time` - the time when the client began to execute the task\n\nThe `_queue_session_ids` space contains a map: connection id (box\nsession id) to the session UUID. This space is temporary if `in_replicaset`\nis set to false.\n\n## Fields of the `_queue_session_ids` space\n\n1. `connection_id` - connection id (numeric)\n2. `session_uuid` - session UUID (string)\n\n## Fields of the `_queue_shared_sessions` space\n\n1. `uuid` - session UUID (string)\n2. `exp_time` - session expiration time (numeric)\n3. `active` - session state (boolean)\n\nThis space is temporary if `in_replicaset` is set to false.\n\nAlso, there is a space which is associated with each queue,\nwhich is named in the `space` field of the `_queue` space.\nThe associated space contains one tuple for each task.\n\n## Fields of the space associated with each queue\n\n1. task_id - numeric - see below\n2. task_state - 'r' for ready, 't' for taken, etc. - see below\n3. task_data - the contents of the task, usually a long string\nx. (additional fields if the queue type has options for ttl, priority,\nor delay)\n\nThe `task_id` value is assigned to a task when it's inserted into a queue.\nCurrently, `task_id` values are simple integers for `fifo` and `fifottl`\nqueues.\n\nThe `task_state` field takes one of the following values\n(different queue types support different\nsets of `task_state` values, so this is a superset):\n\n* 'r' - the task is **ready** for execution (the first consumer executing\na `take` request will get it)\n* 't' - the task has been **taken** by a consumer\n* '-' - the task has been **executed (done)** (a task is removed from the queue\nafter it has been executed, so this value will rarely be seen)\n* '!' - the task is **buried** (disabled temporarily until further changes)\n* '~' - the task is **delayed** for some time.\n\nFor details on the state transitions, refer to [Task state diagram](#task-state-diagram).\n\n# Task state diagram\n\nThe following diagram shows possible transitions between the [task states](#fields-of-the-space-associated-with-each-queue).\nFor information on the transition triggers, refer to:\n\n* [put()](#putting-a-task-in-a-queue)\n* [release()](#releasing-a-task)\n* [take()](#taking-a-task-from-the-queue-consuming)\n* [kick()](#kicking-a-number-of-tasks)\n* [bury()](#burying-a-task)\n* [ack()](#acknowledging-the-completion-of-a-task)\n* [delete()](#deleting-a-task)\n* description of the `timeout`, `ttl timeout`, and `ttr timeout` options in\nthe sections of the corresponding [queue types](#queue-types).\n\n```mermaid\nflowchart LR\n      INIT((\" \"))--\u003e  |\"put()\"| READY\n      INIT((\" \"))--\u003e |\"put('my_task_data', {delay = delay})\"| DELAYED\n      READY--\u003e |\"take()\"| TAKEN\n      READY--\u003e |\"delete() / ttl timeout\"| DONE\n      READY--\u003e |\"bury()\"| BURIED\n      TAKEN--\u003e |\"release() / ttr timeout\"| READY\n      TAKEN--\u003e |\"release\\n(id, {delay = delay})\"| DELAYED\n      TAKEN--\u003e |\"ack() / delete()\"| DONE\n      TAKEN--\u003e |\"bury()\"| BURIED\n      BURIED--\u003e |\"delete() /\\nttl timeout\"| DONE\n      BURIED--\u003e |\"kick()\"| READY\n      DELAYED--\u003e |\"timeout\"| READY\n      DELAYED--\u003e |\"delete()\"| DONE\n```\n\n# Queue state diagram\n\nQueue can be used in a master-replica scheme:\n\nThere are five states for queue:\n* INIT\n* STARTUP\n* RUNNING\n* ENDING\n* WAITING\n\nWhen the tarantool is launched for the first time,\nthe state of the queue is always `INIT` until `box.info.ro` is false.\n\nStates switching scheme:\n```mermaid\nflowchart LR\n      I((\"init\"))--\u003eS[startup]\n      S[startup]--\u003eR[running]\n      W[waiting]--\u003e |\"(ro -\u003erw)\"| S[startup]\n      R[running]--\u003e |\"(rw -\u003ero)\"| E[ending]\n      E[ending]--\u003eW[waiting]\n```\n\nCurrent queue state can be shown by using `queue.state()` method.\n\nIn the `STARTUP` state, the queue is waiting for possible data synchronization\nwith other cluster members by the time of the largest upstream lag multiplied\nby two. After that, all taken tasks are released, except for tasks with\nsession uuid matching shared sessions uuids. This makes possible to take\na task, switch roles on the cluster, and release the task within the timeout\nspecified by the `queue.cfg({ttr = N})` parameter. And the last step in the\n`STARTUP` state is starting tube driver using new method called `start()`.\n\nIn the `RUNNING` state, the queue is working as usually. The `ENDING` state calls\n`stop()` method. in the `WAITING` state, the queue listens for a change in the\nread_only flag.\n\nAll states except `INIT` is controlled by new fiber called `queue_state_fiber`.\n\n# Installing\n\nThere are three alternative ways of installation.\n* Get the `tarantool_queue` package from a repository. For example, on\nUbuntu, say: sudo apt-get install tarantool-queue\n* Take the Lua rock from rocks.tarantool.org.\n* Take the source files from https://github.com/tarantool/queue, then\nbuild and install.\n\n# Using the `queue` module\n\n## Initialization\n```lua\nqueue = require 'queue'\n```\n\nThe request \"require 'queue'\" causes automatic creation of\nthe `_queue` space, unless it already exists. The same request\nalso sets the necessary space triggers and other objects\nassociated with queues.  \nIf the instance hasn't been configured yet (`box.cfg()` hasn't been called),\nthe initialization of the queue module will be deferred until the instance will\nbe configured (\"lazy start\"). For a good work of the queue, it's necessary to\nrun the instance in rw mode. If the instance run in ro mode, the initialization\nof the queue will be deferred until the instance will be configured in rw mode.\nAfter the instance has been started in rw mode and the queue has been\ninitialized, it's a bad idea to switch it to ro mode. In the case, an attempt to\ndo something with a persistent (\"temporary\" option set to false) queue will fail\n(a temporary queue will work fine). In addition, in the case of mode has been\nswitched, triggers may fail (`_on_consumer_disconnect` for example), which may\ncause an inconsistent state of the queue. As for the core drivers that use\nbackground fibers (fifottl, limfifottl, utubettl) - they check the instance mode\non each iteration and will wait until the instance will be switched to rw mode.\n\n## Get the module version\n```lua\nqueue._VERSION\n```\n\nReturns the current version of the module.\n\n## Creating a new queue\n\n```lua\nqueue.create_tube(queue name, queue type [, {options} ])\n```\n\nCreates a queue.\n\nThe queue name must be alphanumeric and be up to 32 characters long.\n\nThe queue type must be 'fifo', 'fifottl', 'utube', or 'utubettl'.\n\nThe options, if specified, must be one or more of the options described above\n(`temporary` and/or `ttl` and/or `ttr` and/or `pri`, depending on the queue\ntype).\nThe `ttr` and `ttl` options can be regarded as defaults, which may be overridden\nwhen a task is put in a queue.\n\nEffect: a tuple is added in the `_queue` space, and a new associated space is\ncreated.\n\nExample: `queue.create_tube('list_of_sites', 'fifo', {temporary = true})`\n\n## Set queue settings\n\n```lua\nqueue.cfg({options})\n```\n\nSet queue settings.  \nIf an invalid value or an unknown option is used, an error will be thrown.  \nAvailable `options`:\n* `ttr` - time to release in seconds. The time after which, if there is no active\nconnection in the session, it will be released with all its tasks.\n* `in_replicaset` - enable replication mode. Must be true if the queue is used\nin master and replica mode. With replication mode enabled, the potential loss of\nperformance can be ~20% compared to single mode. Default value is false.\n\n## Session identify\n\n```lua\nqueue.identify(session_uuid)\n```\n\nIn the queue the session has a unique UUID and many connections may share one\nlogical session. Also, the consumer can reconnect to the existing session during\nthe`ttr` time.  \nTo get the UUID of the current session, call the `queue.identify()`\nwithout parameters.  \nTo connect to the existing session, call the `queue.identify(session_uuid)`\nwith the UUID of the session.  \nIn case of attempt to use an invalid format UUID or expired UUID, an error will\nbe thrown.\n\nBe careful, UUID here is a 16-bit string generated by\n[uuid.bin()](https://www.tarantool.io/en/doc/latest/reference/reference_lua/uuid/#lua-function.uuid.bin),\nnot an object of type UUID.\n\nUsage example:  \nSometimes we need an ability to acknowledge a task after reconnect (because\nretrying it is undesirable) or even acknowlegde using another connection.\n\nExample of code for connecting to the old session in case of reconnection:\n```lua\nlocal netbox = require('net.box')\n\nlocal conn = netbox.connect('localhost:1918', { reconnect_after = 5 })\nlocal session_uuid = conn:call('queue.identify')\nconn:on_connect(function()\n    conn:call('queue.identify', {session_uuid})\nend)\n```\n\n## Putting a task in a queue\n\nTo insert a new task into a queue, use:\n\n```lua\nqueue.tube.tube_name:put(task_data [, {options} ])\n```\n\nThe `tube_name` must be the name which was specified by `queue.create_tube`.\n\nThe `task_data` contents are the user-defined description of the task,\nusually a long string.\n\nThe options, if specified, must be one or more of the options described\nabove\n(`ttl` and/or `ttr` and/or `pri` and/or `delay` and/or `utube`, depending on the queue\ntype).\nIf an option is not specified, the default is what was specified during\n`queue.create_tube`, and if that was not specified, then the default is what\nwas described above for the queue type. Note: if the `delay` option is\nspecified, the delay time is added to the ttl time.\n\nEffect: a new tuple is created in the queue's associated space, with\ntask_id = a number which is equal to the largest `task_id` so far, plus 1\ntask_state = 'r' (ready)\ntask_data = whatever the user put in the `task_data` parameter\n\nReturns: the value of the new tuple in the queue's associated space,\nalso called the \"created task\".\n\nExample: queue.tube.list_of_sites:put('Your task is to do something',\n{pri=2})\n\nAfter a task has been put in a queue, one of these things may happen:\nit may be removed from the queue because its ttl (time to live) expires,\nor it may be acted on by a worker (usually with a `take` request).\n\n## Taking a task from the queue (\"consuming\")\n\n```lua\nqueue.tube.tube_name:take([timeout])\n```\n\nTake a queue task.\n\nThe `take` request searches for a task in the queue or sub-queue\n(that is, a tuple in the queue's associated space)\nwhich has `task_state` = 'r' (ready), and `task_id` = a value lower\nthan any other tuple which also has `task_state` = 'r'.\n\nIf there is no such task, and timeout was specified, then\nthe job waits until a task becomes ready or the timeout expires.\n\nEffect: the value of `task_state` changes to 't' (taken).\nThe `take` request tells the system that the task is being worked on.\nIt should be followed by an `ack` request when the work is finished.\nAdditional effect: a tuple is added to the `_queue_taken_2` space.\n\nReturns: the value of the taken tuple, or nil if none was found.\nThe value of the first field in the tuple (`task_id`) is important\nfor further requests. The value of the second field in the tuple\n(`task_data`) is important as it presumably contains user-defined\ninstructions for what to do with the task.\n\nExample: t_value = queue.tube.list_of_sites:take(15)\n\n## Increasing TTR and/or TTL for tasks\n\n```lua\nqueue.tube.tube_name:touch(task_id, increment)\n```\n\nIncrease `ttr` of running task. Useful if you can't predict in advance\ntime needed to work on task.\n\nEffect: the value of `ttr` and `ttl` increased by `increment` seconds. If queue\ndoes not support ttr, error will be thrown. If `increment` is lower than zero,\nerror will be thrown. If `increment` is zero or nil effect is noop. If current\n`ttr` of task is 500 years or greater then operation is noop.\n\nExample: t_value = queue.tube.list_of_sites:touch(15, 60)\n\n## Acknowledging the completion of a task\n\n```lua\nqueue.tube.tube_name:ack(task_id)\n```\n\nThe worker which has used 'take' to take the task should\nuse 'ack' to signal that the task has been completed.\nThe current `task_state` of the tuple should be 't' (taken),\nand the worker issuing the `ack` request must have the same ID\nas the worker which issued the `take` request.\n\nEffect: the value of `task_state` changes to '-' (acknowledged).\nShortly after this, it may be removed from the queue automatically.\n\nIf 'take' occurs but is not soon followed by 'ack'\n-- that is, if ttr (time to run) expires, or if the worker disconnects --\nthe effect is: `task_state` is changed from\n't' (taken) back to 'r' (ready).\nThis effect is the same as what would happen with a `release` request.\n\nExample: queue.tube.list_of_sites:ack(15)\n\n## Releasing a task\n\n```lua\nqueue.tube.tube_name:release(task_id, opts)\n```\n\nPut the task back in the queue.\nA worker which has used 'take' to take a task,\nbut cannot complete it, may make a `release` request\ninstead of an `ack` request. Effectively, 'ack' implies\nsuccessful completion of a taken task, and 'release' implies\nunsuccessful completion of a taken task.\n\nEffect: the value of `task_state` changes to 'r' (ready).\nAfter this, another worker may take it.\nThis is an example of a situation where, due to user intervention,\na task may not be successfully completed in strict FIFO order.\n\nExample: queue.tube.list_of_sites:release(15, {delay=10})\n\nNote: in the above example, the delay option means\n\"the task cannot be executed again for 10 seconds\".\n\n## Peeking at a task\n\n```lua\nqueue.tube.tube_name:peek(task_id)\n```\n\nLook at a task without changing its state.\n\nEffect: this is the same as getting a tuple from the space\nassociated with the queue: box.space.tube_name:select(task_id).\n\nReturns: the tuple of the task.\n\nExample: queue.tube.list_of_sites:peek(15)\n\n## Burying a task\n\n```lua\nqueue.tube.tube_name:bury(task_id)\n```\n\nIf it becomes clear that a task cannot be executed\nin the current circumstances, you can \"bury\" the task\n-- that is, disable it until the circumstances change.\n\nEffect: the value of `task_state` changes to '!' (buried).\nSince '!' is not equal to 'r' (ready), the task can no longer be taken.\nSince '!' is not equal to '-' (complete), the task will not be deleted.\nThe only thing that can affect a buried task is a `kick` request.\n\nReturns: the tuple value.\n\nExample: queue.tube.list_of_sites:bury(15)\n\n## Kicking a number of tasks\n\n```lua\nqueue.tube.tube_name:kick(count)\n```\n\nReverse the effect of a `bury` request on one or more tasks.\n\nEffect: the value of `task_state` changes from '!' (buried)\nto 'r' (ready), for one or more tasks.\n\nReturns: number of tasks actually kicked.\n\nExample: queue.tube.list_of_sites:kick(99)\n(this will change up to 99 buried tasks)\n\n## Deleting a task\n\n```lua\nqueue.tube.tube_name:delete(task_id)\n```\n\nDelete the task identified by `task_id`.\n\nEffect: the current state of `task_state` is not checked.\nThe task is removed from the queue.\n\nExample: queue.tube.list_of_sites:delete(15)\n\n## Dropping a queue\n\n```lua\nqueue.tube.tube_name:drop()\n```\n\nReverse the effect of a `create` request.\n\nEffect: remove the tuple from the `_queue` space,\nand drop the space associated with the queue.\n\n## Releasing all taken tasks\n\n```lua\nqueue.tube.tube_name:release_all()\n```\n\nForcibly returns all taken tasks to a ready state.\n\n## Getting statistics\n\n```lua\nqueue.statistics( [queue name] )\n```\n\nShow the number of tasks in a queue broken down by `task_state`, and the number\nof requests broken down by the type of request. If the queue name is not\nspecified, show these numbers for all queues.\nStatistics are temporary, they are reset whenever the Tarantool server restarts.\n\nExample:\n\n```lua\nqueue.tube.tube_name:on_task_change(callback)\n```\n\nReplace old `on_task_change` callback or set the new one. Previously set\ncallback is returned.\n\nGet statistics for given tube:\n\n```lua\nqueue.statistics('list_of_sites')\n---\n- tasks:\n     taken: 0\n     buried: 0\n     ready: 0\n     done: 2\n     delayed: 0\n     total: 0\n   calls:\n     ack: 1\n     take: 1\n     kick: 1\n     bury: 1\n     put: 2\n     delete: 1\n...\n```\n\n## Queue and replication\n\nUsage example:\n\n```lua\n-- Instance file for the master.\nqueue = require(\"queue\")\n-- Queue is in replicaset.\n-- Clean up session after 5 minutes after disconnect.\nqueue.cfg({ttr = 300, in_replicaset = true})\n\nbox.cfg{\n  listen = 3301,\n  replication = {'replicator:password@127.0.0.1:3301',  -- Master URI.\n                 'replicator:password@127.0.0.1:3302'}, -- Replica URI.\n  read_only = false,\n}\n\nbox.once(\"schema\", function()\n   box.schema.user.create('replicator', {password = 'password'})\n   box.schema.user.grant('replicator', 'replication') -- grant replication role\nend)\n\nrequire('console').start()\nos.exit()\n```\n\n```lua\n-- Instance file for the replica.\nqueue = require(\"queue\")\n-- Queue is in replicaset.\n-- Clean up session after 5 minutes after disconnect.\nqueue.cfg({ttr = 300, in_replicaset = true})\nbox.cfg{\n  listen = 3302,\n  replication = {'replicator:password@127.0.0.1:3301',  -- Master URI.\n                 'replicator:password@127.0.0.1:3302'}, -- Replica URI.\n  read_only = true\n}\n\nrequire('console').start()\nos.exit()\n```\n\nStart master and replica instances and check queue state:\n\nMaster:\n```sh\ntarantool\u003e queue.state()\n---\n- RUNNING\n...\n```\n\nReplica:\n```sh\ntarantool\u003e queue.state()\n---\n- INIT\n...\n```\n\nNow reverse the `read_only` setting of the master and replica and check the\nstatus of the queue again.\n\nMaster:\n```sh\ntarantool\u003e box.cfg({read_only = true})\ntarantool\u003e queue.state()\n---\n- WAITING\n...\n```\n\nReplica:\n```sh\ntarantool\u003e box.cfg({read_only = false})\ntarantool\u003e queue.state()\n---\n- RUNNING\n...\n```\n\n# Implementation details\n\nThe implementation is based on the common functions for all queues:\n\n1. controlling the `consumers` (watching connection state/wakeup)\n1. similarities of the API\n1. spaces to support each tube\n1. etc\n\nEach new queue has a \"driver\" to support it.\n\n## Queue drivers\n\nMandatory requirements\n\n1. The driver works with tuples. The only thing the driver needs\nto know about the tuples is their first two fields: `id` and `state`.\n1. Whenever the driver notices that a task state has changed, it must\nnotify the framework about the change.\n1. The driver must not throw exceptions, unless the driver API is misused.\nI.e. for normal operation, even errors during normal operation, there\nshould be no exceptions.\n\nRegistering a custom driver\n\n`register_driver(driver_name, tube_ctr)` - queue method is used to register\n  a custom driver. The arguments are:\n  * driver_name: unique driver name. Must be different from the core drivers\n  names.\n  * tube_ctr: implementation of tube control methods(\"create_space\" and \"new\").\n\n## Driver API\n\nDriver class must implement the following API:\n\n1. `new` (constructs an instance of a driver), takes:\n    * the space object, in which the driver must store its tasks\n    * a callback to notify the main queue framework on a task state change\n  (`on_task_change`)\n    * options of the queue (a Lua table)\n1. `create_space` - creates the supporting space. The arguments are:\n    * space name\n    * space options\n1. `start` - initialize internal resources if any, e.g. start fibers.\n1. `stop` - clean up internal resources if any, e.g. stop fibers.\n\nTo sum up, when the user creates a new queue, the queue framework\npasses the request to the driver, asking it to create a space to\nsupport this queue, and then creates a driver instance, passing to it\nthe created space object.\n\nThe same call sequence is used when the queue is \"restarted\" after\nTarantool server restart.\n\nThe driver instance returned by the `new` method must provide the following\nAPI:\n\n* `tube:normalize_task(task)` - converts the task tuple to the object\nwhich is passed on to the user (removes the administrative fields)\n* `tube:put(data[, opts])` - puts a task into the queue.\nReturns a normalized task which represents a tuple in the space\n* `tube:take()` - sets the task state to 'in progress' and returns the task.\nIf there are no 'ready' tasks in the queue, returns nil.\n* `tube:delete(task_id)` - deletes a task from the queue.\nReturns the original task with a state changed to 'done'\n* `tube:release(task_id, opts)` - puts a task back to the queue (in the 'ready'\n   state)\n* `tube:bury(task_id)` - buries a task\n* `tube:kick(count)` - digs out `count` tasks\n* `tube:peek(task_id)` - return the task state by ID\n* `tube:touch(task_id, delta)` - increases `ttr` and `ttl` of the task by delta\nseconds. If queue does not support `ttr`, error will be thrown. Returns the task\n* `tube:tasks_by_state(task_state)` - return the iterator to tasks in a certain state\n* `tube:truncate()` - delete all tasks from the tube. Note that `tube:truncate`\nmust be called only by the user who created this tube (has space ownership) OR\nunder a `setuid` function. Read more about `setuid` functions\n[here](http://tarantool.org/doc/book/box/authentication.html?highlight=function#functions-and-the-func-space).\n\n\n[testing-actions-badge]: https://github.com/tarantool/queue/actions/workflows/fast_testing.yml/badge.svg\n[testing-actions-url]: https://github.com/tarantool/queue/actions/workflows/fast_testing.yml\n\n[packaging-actions-badge]: https://github.com/tarantool/queue/actions/workflows/packaging.yml/badge.svg\n[packaging-actions-url]: https://github.com/tarantool/queue/actions/workflows/packaging.yml\n\n[publish-actions-badge]: https://github.com/tarantool/queue/actions/workflows/publish.yml/badge.svg\n[publish-actions-url]: https://github.com/tarantool/queue/actions/workflows/publish.yml\n","funding_links":[],"categories":["Packages"],"sub_categories":["Database"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftarantool%2Fqueue","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftarantool%2Fqueue","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftarantool%2Fqueue/lists"}