{"id":20564415,"url":"https://github.com/tarantool/sharded-queue","last_synced_at":"2025-04-14T15:13:20.501Z","repository":{"id":48506623,"uuid":"230119178","full_name":"tarantool/sharded-queue","owner":"tarantool","description":"Sharded queue","archived":false,"fork":false,"pushed_at":"2024-04-17T09:30:58.000Z","size":167,"stargazers_count":12,"open_issues_count":4,"forks_count":5,"subscribers_count":26,"default_branch":"master","last_synced_at":"2025-04-14T15:13:07.988Z","etag":null,"topics":["lua","queue","tarantool"],"latest_commit_sha":null,"homepage":"","language":"Lua","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tarantool.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2019-12-25T15:14:10.000Z","updated_at":"2023-09-06T13:56:14.000Z","dependencies_parsed_at":"2022-09-04T15:04:12.260Z","dependency_job_id":"df4e557f-abfa-46da-ad5f-5fa8fa36ce56","html_url":"https://github.com/tarantool/sharded-queue","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tarantool%2Fsharded-queue","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tarantool%2Fsharded-queue/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tarantool%2Fsharded-queue/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tarantool%2Fsharded-queue/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tarantool","download_url":"https://codeload.github.com/tarantool/sharded-queue/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248904637,"owners_count":21180835,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["lua","queue","tarantool"],"created_at":"2024-11-16T04:26:27.605Z","updated_at":"2025-04-14T15:13:20.477Z","avatar_url":"https://github.com/tarantool.png","language":"Lua","readme":"# Tarantool Sharded Queue Application\n\nThis module provides roles for the Tarantool 3 and for the Tarantool\nCartridge implementing of a distributed queue compatible with [Tarantool queue](https://github.com/tarantool/queue) (*fifiottl driver*)\n\n```mermaid\nflowchart TB\n    customers((customers)):::blue\n    customers \u003c--\u003e router1\n    customers \u003c--\u003e router2\n\n    subgraph Queue\n    router1\n    router2\n\n        subgraph shard1\n        storage11[(\"Storage\\n master\")]\n        storage12[(\"Storage\\n replica\")]\n        storage13[(\"Storage\\n replica\")]\n        storage11\u003c--\u003estorage12\n        storage11\u003c--\u003estorage13\n        storage12\u003c--\u003estorage13\n        end\n\n        subgraph shard2\n        storage21[(\"Storage\\n master\")]\n        storage22[(\"Storage\\n replica\")]\n        storage23[(\"Storage\\n replica\")]\n        storage21\u003c--\u003estorage22\n        storage21\u003c--\u003estorage23\n        storage22\u003c--\u003estorage23\n        end\n\n        subgraph shard3\n        storage31[(\"Storage\\n master\")]\n        storage32[(\"Storage\\n replica\")]\n        storage33[(\"Storage\\n replica\")]\n        storage31\u003c--\u003estorage32\n        storage31\u003c--\u003estorage33\n        storage32\u003c--\u003estorage33\n        end\n\n        router1 \u003c--\u003e shard1\n        router1 \u003c--\u003e shard2\n        router1 \u003c--\u003e shard3\n        router2 \u003c--\u003e shard1\n        router2 \u003c--\u003e shard2\n        router2 \u003c--\u003e shard3\n\n    end\n```\n\n## Usage in a Tarantool 3 application\n\nYou need to install the Tarantool 3.0.2+ or 3.1+.\n\n1. Add a dependency to your application rockspec.\n2. Enable `roles.sharded-queue-router` role on all sharding `router` instances.\n3. Enable `roles.sharded-queue-storage` role on all sharding `storage`\n   instances.\n4. Configure tubes for the `roles.sharded-queue-storage` role the same for all\n   storage instances.\n5. Do not forget to bootstrap the vshard in your application. See\n   [init.lua](./init.lua) as an example.\n\nYou could see a full example of the configuration in the [config.yaml](./config.yaml).\n\nBe careful, it is impossible to create or drop tubes dynamically by API calls\nwith Tarantool 3. You need to update the role configuration instead.\n\n## Usage in a Tarantool Cartridge application\n\n1. Add a dependency to your application rockspec. You need to make sure that\nthe dependency of the Tarantool Cartridge 2.9.0 is enabled because the\n`sharded-queue` does not have it by default.\n2. Add roles to your application:\n```init.lua\ncartridge.cfg({\n    ...\n    roles = {        \n        'sharded_queue.storage',\n        'sharded_queue.api'\n        ...\n    },\n    ...\n})\n```\n3. Enable the `sharded_queue.storage` role on all storage nodes. Be careful,\n   there should be no replicasets with `cartridge.roles.vshard-storage` role,\n   but without the `sharded_queue.storage` role.\n4. Queue API will be available on all nodes where the `sharded_queue.api` role\n   is enabled.\n\n## Usage as a ready-to-deploy service\n\nPrepare `tt` environment:\n```shell\ntt init\ngit clone https://github.com/tarantool/sharded-queue instances.enabled/sharded-queue\n```\n\nRun:\n```shell\ntt pack --app-list sharded-queue rpm --version 1.0.0\n```\n\nNow you could install resulting package on your target servers.\n\nFor more details refer to [tt](https://github.com/tarantool/tt/)\n\n## Usage from client perspective\n\nThe good old queue api is located on all instances of the router masters that\nwe launched. For a test configuration, this is one router on `localhost:3301`\n\n```shell\ntarantool@user:~/sharded_queue$ tarantool\nTarantool 1.10.3-6-gfbf53b9\ntype 'help' for interactive help\ntarantool\u003e netbox = require('net.box')\n---\n...\ntarantool\u003e queue_conn = netbox.connect('localhost:3301', {user = 'admin',password = 'secret-cluster-cookie'})\n---\n...\ntarantool\u003e queue_conn:call('queue.create_tube', { 'test_tube' })   \n---\n...\ntarantool\u003e queue_conn:call('queue.tube.test_tube:put', { 'task_1' })\n---\n- [3653, 'r', 'task_1']\n...\ntarantool\u003e queue_conn:call('queue.tube.test_tube:take')\n---\n- [3653, 't', 1]\n...\n\n```\n\nYou may also set up tubes using Cartridge cluster-wide config:\n```config.yml\ntubes:\n     tube_1:\n        temporary: true\n        ttl: 60\n     tube_2:\n        driver: my_app.my_driver\n     cfg:\n        metrics: false\n```\n\nBe careful, `cfg` field in a cluster-wide config acts as a configuration\nvalue for `api.cfg()` call. These configuration options are currently\nsupported:\n\n* `metrics` - enable or disable stats collection by metrics.\n  metrics \u003e= 0.11.0 is required. It is enabled by default.\n\n## Running locally with Tarantool 3 (as an example)\n\nPrepare `tt` environment:\n```shell\ntt init\ngit clone https://github.com/tarantool/sharded-queue instances.enabled/sharded-queue\n```\n\nInstall dependencies:\n```shell\ntt build sharded-queue\n```\n\nStart default configuration:\n```shell\ntt start sharded-queue\n```\n\nTo stop, say:\n```shell\ntt stop sharded-queue\n```\n\n## Launching tests\n    \nSay:\n\n```shell\nmake deps\nmake deps-cartridge # For Tarantool \u003c 3.\nmake deps-metrics # For Tarantool \u003c 3.\nmake test\n```\n\n## Metrics\n\nThe module exports several metrics if the module `metrics` \u003e= 0.11 is\ninstalled and the feature is not disabled by the configuration.\n\n### Router (`roles.sharded-queue-router` or `sharded_queue.api` for the Cartridge)\n\n* Metric `tnt_sharded_queue_router_statistics_calls_total` is a counter with\n  the number of requests broken down by [the type of request][queue-statistics].\n  The metric has labels in the following format:\n\n  `{name = \"tube_name\", state = \"request_type\"}`\n\n  A list of possible request types: `done`, `take`, `kick`, `bury`, `put`,\n  `delete`, `touch`, `ack`, `release`. The metric on the `sharded_queue.api`\n  role accumulates values from all buckets.\n\n* Metric `tnt_sharded_queue_router_statistics_tasks` is a gauge with\n  the number of tasks in a queue broken down by [a task state][queue-statistics].\n  The metric has labels in the following format:\n\n  `{name = \"tube_name\", state = \"task_state\"}`\n\n  A list of possible task states: `taken`, `buried`, `ready`, `done`,\n  `delayed`, `total`. The metric on the `sharded_queue.api` role accumulates\n  values from all buckets.\n\n* Metric `tnt_sharded_queue_router_role_stats` is a [summary][metrics-summary]\n  with quantiles of `sharded_queue.api` role API calls. The metric includes a\n  counter of API calls and errors and has labels in the following format:\n\n  `{name = \"tube_name\", method = \"api_call_method\", status = \"ok\" or \"error\"}`\n\n  A list of possible call methods: `put`, `take`, `delete`, `release`, `touch`,\n  `ack`, `bury`, `kick`, `peek`, `drop`.\n\n### Storage (`roles.sharded-queue-storage` or `sharded_queue.storage` for the Cartridge)\n\n* Metric `tnt_sharded_queue_storage_statistics_calls_total` is a counter with\n  the number of requests broken down by [the type of request][queue-statistics].\n  The metric has labels in the following format:\n\n  `{name = \"tube_name\", state = \"request_type\"}`\n\n  A list of possible request types: `done`, `take`, `kick`, `bury`, `put`,\n  `delete`, `touch`, `ack`, `release`. The metric on the `sharded_queue.storage`\n  role shows actual values on the instance.\n\n* Metric `tnt_sharded_queue_storage_statistics_tasks` is a gauge with\n  the number of tasks in a queue broken down by [a task state][queue-statistics].\n  The metric has labels in the following format:\n\n  `{name = \"tube_name\", state = \"task_state\"}`\n\n  A list of possible task states: `taken`, `buried`, `ready`, `done`,\n  `delayed`, `total`. The metric on the `sharded_queue.storage` role shows\n  actual values on the instance.\n\n* Metric `tnt_sharded_queue_storage_role_stats` is a [summary][metrics-summary]\n  with quantiles of `sharded_queue.api` role API calls. The metric includes a\n  counter of API calls and errors and has labels in the following format:\n\n  `{name = \"tube_name\", method = \"api_call_method\", status = \"ok\" or \"error\"}`\n\n  A list of possible call methods: `statistic`, `put`, `take`, `delete`,\n  `release`, `touch`, `ack`, `bury`, `kick`, `peek`, `drop`.\n\n## API extensions (compared to tarantool/queue)\n\n* ``tube:take`` method has additional table argument ``options``. It may be used to provide additional logic in some\n    drivers.\n    \n    Example:\n    ```lua\n      tube:take(3, {cumstom_driver_option='test'})\n    ```\n\n* **Logging**: There are 2 ways to log your api method calls:\n\n    1. with the parameter `log_request` during tube creation;\n\n    2. with the exact same parameter on each operation (each of methods (`take`, `put`, `ack`, `release`, `delete`, `touch`, `bury`, `peek`) has additional table argument ``options``).\n    In this case `log_request` will override the tube's log parameter for only 1 operation.\n\n    Examples:\n\n    ```lua\n    conn:call('queue.create_tube', { mytube, {\n        log_request = true, -- log all operations\n    }})\n\n    conn:call(\"queue.tube.mytube:put\", { data, {\n        log_request = false, -- this PUT will not be logged\n    }})\n\n    conn:call(\"queue.tube.mytube:put\", { anoter_data }) -- and this PUT will be logged\n    ```\n\n    If you use **fifottl** driver (default), you can log driver's method calls with `log_request` (log router's and storage's operations).\n\n* You can not create or drop tubes by API calls with Tarantool 3. You need\n  to update the role configuration instead.\n\n[metrics-summary]: https://www.tarantool.io/en/doc/latest/book/monitoring/api_reference/#summary\n[queue-statistics]: https://github.com/tarantool/queue?tab=readme-ov-file#getting-statistics\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftarantool%2Fsharded-queue","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftarantool%2Fsharded-queue","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftarantool%2Fsharded-queue/lists"}