{"id":47185086,"url":"https://github.com/dkam/tuber","last_synced_at":"2026-03-13T09:02:15.773Z","repository":{"id":343888215,"uuid":"1178548038","full_name":"dkam/tuber","owner":"dkam","description":"Rust implementation of Beanstalkd","archived":false,"fork":false,"pushed_at":"2026-03-12T12:27:42.000Z","size":306,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-03-12T13:02:38.490Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dkam.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-03-11T05:59:55.000Z","updated_at":"2026-03-12T12:27:46.000Z","dependencies_parsed_at":"2026-03-12T13:03:33.484Z","dependency_job_id":null,"html_url":"https://github.com/dkam/tuber","commit_stats":null,"previous_names":["dkam/tuber"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/dkam/tuber","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dkam%2Ftuber","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dkam%2Ftuber/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dkam%2Ftuber/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dkam%2Ftuber/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dkam","download_url":"https://codeload.github.com/dkam/tuber/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dkam%2Ftuber/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30463559,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-13T06:34:02.089Z","status":"ssl_error","status_checked_at":"2026-03-13T06:33:49.182Z","response_time":60,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-03-13T09:02:13.457Z","updated_at":"2026-03-13T09:02:15.748Z","avatar_url":"https://github.com/dkam.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# tuber\n\nAn experimental, simple, fast work queue — a Rust rewrite of [beanstalkd](https://github.com/beanstalkd/beanstalkd). Wire-compatible with existing beanstalkd clients.\n\n## Quick Start\n\n```bash\n# Start the server\ntuber server\n\n# Put a job\ntuber put \"hello world\"\n\n# Put jobs from stdin (one per line)\necho -e \"job1\\njob2\\njob3\" | tuber put\n\n# List tubes\ntuber tubes\n\n# Check stats\ntuber stats\n```\n\n## The case for Tuber / Beanstalkd\n\nRedis-backed queues are popular and performant, but Redis isn't a natural fit for job queues. You're bolting priorities, delays, reservations, and timeouts onto a general-purpose data structure server — complexity that grows with every edge case.\n\nSQLite-backed queues are simple and fast, but limited to a single host. PostgreSQL and MySQL-backed queues can scale beyond one host, but a job queue should be separate from your application database for capacity planning — which means another instance to manage with connection pooling, tuning, vacuuming, backups, and restores.\n\nTuber and Beanstalkd are purpose-built for this. A single binary, easy to deploy in Docker, with optional write-ahead log for durability. No capacity planning, no tuning, no surprises. Workers wait efficiently at any scale.\n\n## Features\n\nAll the great hits from beanstalkd - backwards compatible, plus:\n\n### Weighted Reserve\n\nBy default, `reserve` picks the highest-priority job across all watched tubes (FIFO). Switch to weighted mode and each tube is chosen randomly in proportion to its weight:\n\n```text\nwatch email\nwatch notifications 2\nwatch another-tube 6\nreserve-mode weighted\nreserve\n```\n\nTubes default to weight 1. Here, `another-tube` is selected 3x as often as `notifications` and 6x as often as `email`.\n\n### Unique Jobs (Idempotency)\n\nPrevent duplicate jobs with an `idp:` key on `put`. If a live job with the same key already exists in the tube, the original job ID is returned along with the existing job's state:\n\n```text\nput 100 0 30 5 idp:my-key\n\u003cbody\u003e\n→ INSERTED 1\n\nput 100 0 30 5 idp:my-key\n\u003cbody\u003e\n→ INSERTED 1 READY       (dedup hit — job is ready)\n```\n\nThe response state tells you exactly what happened to the original job:\n\n| Response | Meaning |\n|---|---|\n| `INSERTED \u003cid\u003e` | Fresh insert, new job created |\n| `INSERTED \u003cid\u003e READY` | Dedup hit — original job is waiting to be reserved |\n| `INSERTED \u003cid\u003e RESERVED` | Dedup hit — original job is being processed |\n| `INSERTED \u003cid\u003e DELAYED` | Dedup hit — original job is delayed |\n| `INSERTED \u003cid\u003e BURIED` | Dedup hit — original job is buried |\n| `INSERTED \u003cid\u003e DELETED` | Dedup hit during TTL cooldown (see below) |\n\nThe state suffix only appears on dedup hits — a `put` without `idp:` always returns plain `INSERTED \u003cid\u003e`, keeping the response fully backwards-compatible with standard beanstalkd clients.\n\nThe key is scoped to the tube and cleared when the job is deleted, so the same key can be reused afterwards.\n\n#### Cooldown TTL\n\nBy default, the idempotency key is removed as soon as the job is deleted. Add a TTL with `idp:key:N` to keep deduplicating for N seconds after deletion:\n\n```text\nput 0 0 30 5 idp:report:300\n\u003cbody\u003e\n→ INSERTED 1\n\n(reserve → delete job 1)\n\nput 0 0 30 5 idp:report:300\n\u003cbody\u003e\n→ INSERTED 1 DELETED     (still deduped — within 300s cooldown)\n```\n\nAfter the cooldown expires, the key is freed and a new job will be created. `idp:key` (no TTL) keeps the original behaviour — key removed immediately on delete.\n\n### Job Groups (Fan-out / Fan-in)\n\nGroup related jobs together with `grp:` and chain dependent work with `aft:`. After-jobs are held until every job in the group they depend on has been deleted:\n\n```text\nput 0 0 30 11 grp:import\nimport-row-1\nput 0 0 30 11 grp:import\nimport-row-2\nput 0 0 60 14 aft:import\nsend-summary\n```\n\nThe `send-summary` job stays held until both `import` group jobs are deleted. Buried jobs block group completion — kick them to let the group finish. If an `aft:` job isn't running and you're not sure why, use `stats-group \u003cname\u003e` to check whether the group still has pending or buried members.\n\nChain stages together by combining `aft:` and `grp:` on the same job — the after-job becomes a member of the next group:\n\n```text\nput 0 0 30 5 grp:extract\nrow-1\nput 0 0 30 5 grp:extract\nrow-2\nput 0 0 30 5 aft:extract grp:transform\ntransform\nput 0 0 30 5 aft:transform\nload\n```\n\nHere `transform` waits for the extract group to finish, then becomes part of the `transform` group. `load` waits for `transform` to complete — giving you a simple DAG pipeline.\n\nUse `stats-group \u003cname\u003e` to inspect group state — useful for debugging why `aft:` jobs aren't running:\n\n```text\nstats-group import\n→ OK \u003cbytes\u003e\n---\nname: \"import\"\npending: 2\nburied: 1\ncomplete: false\nwaiting-jobs: 1\n```\n\nA buried job blocks group completion (`complete: false`). Kick it to let the group finish.\n\nGroup names are global — jobs in the same group can span multiple tubes. Note that the server does not detect cycles: if two groups depend on each other, the waiting jobs will be held indefinitely. Cycle avoidance is the client's responsibility.\n\n\n\n### Concurrency Keys\n\nLimit parallel processing of related jobs. When a job with a `con:` key is reserved, other ready jobs sharing the same key are hidden from `reserve` until the reservation ends (via delete, release, bury, TTR timeout, or disconnect):\n\n```text\nput 0 0 30 7 con:user-42\npayload1\nput 0 0 30 7 con:user-42\npayload2\n```\n\nOnly one `con:user-42` job can be reserved at a time, ensuring serial processing per key.\n\nSet a higher limit with `con:key:N` to allow N concurrent reservations:\n\n```text\nput 0 0 30 7 con:api:3\npayload1\nput 0 0 30 7 con:api:3\npayload2\n```\n\nUp to 3 `con:api` jobs can be reserved simultaneously. `con:key` (no `:N`) defaults to a limit of 1.\n\nBurying or releasing-with-delay a job frees its concurrency slot immediately — the slot is only held while the job is reserved. Delayed jobs don't occupy a slot until they become ready and are reserved. Use `stats-job \u003cid\u003e` to check a job's current state if reserves are unexpectedly blocked.\n\n### Prometheus Metrics\n\nExpose a `/metrics` endpoint for Prometheus scraping:\n\n```bash\ntuber server -l 0.0.0.0 -p 11300 -V --metrics-port 9100\n```\n\n## Server\n\n```bash\ntuber server [OPTIONS]\n```\n\n| Option | Default | Description |\n|---|---|---|\n| `-l`, `--listen` | `0.0.0.0` | Listen address |\n| `-p`, `--port` | `11300` | Listen port |\n| `-b`, `--binlog-dir` | — | WAL directory (enables persistence) |\n| `-z`, `--max-job-size` | `65535` | Max job size in bytes |\n| `-V` | warn | Verbosity (`-V` info, `-VV` debug) |\n| `--metrics-port` | — | Prometheus metrics endpoint port |\n\n```bash\n# Listen on a custom port with persistence\ntuber server -p 11301 -b /var/lib/tuber\n\n# Verbose mode with metrics\ntuber server -VV --metrics-port 9100\n```\n\n### Durability \u0026 fsync\n\nWhen persistence is enabled (`-b`), tuber appends job mutations to a write-ahead log (WAL). The WAL is fsynced every 100ms as part of the server's internal tick — not on every write. This means:\n\n- **At most 100ms of data can be lost on a crash.** Jobs written in the last tick interval may not have been fsynced to disk yet.\n- **fsync overhead is constant regardless of throughput.** Whether you're doing 10 jobs/sec or 100,000 jobs/sec, tuber calls fsync ~10 times per second. On NVMe/SSD storage this adds negligible latency; on spinning disks it costs ~50–150ms/sec of I/O time.\n\nThis is a different trade-off from databases like PostgreSQL or MySQL, which fsync on every transaction commit to guarantee durability of each acknowledged write (the \"D\" in ACID). Tuber's `INSERTED` response means the job is buffered in the WAL but not necessarily fsynced — similar to PostgreSQL's `synchronous_commit = off` mode. For most queue workloads, losing a fraction of a second of jobs on a hard crash is acceptable, and the throughput benefit is significant.\n\nWithout `-b`, all state is in-memory only and lost on restart.\n\n## Put\n\n```bash\ntuber put [OPTIONS] [BODY]\n```\n\n| Option | Default | Description |\n|---|---|---|\n| `-t`, `--tube` | `default` | Tube name |\n| `-p`, `--pri` | `0` | Priority (0 is most urgent) |\n| `-d`, `--delay` | `0` | Delay in seconds before job becomes ready |\n| `--ttr` | `60` | Time-to-run in seconds |\n| `-i`, `--idp` | — | Idempotency key — `key` or `key:ttl` (TTL seconds keeps deduping after delete) |\n| `-g`, `--grp` | — | Group name (for job grouping) |\n| `--aft` | — | After-group dependency (wait for this group to complete) |\n| `-c`, `--con` | — | Concurrency key — `key` or `key:N` (N = max concurrent reservations, default 1) |\n| `-a`, `--addr` | `localhost:11300` | Server address |\n\n```bash\n# Put a job on a specific tube with priority\ntuber put -t emails --pri 100 \"send welcome email\"\n\n# Pipe jobs from a file\ncat jobs.txt | tuber put -t batch\n\n# Put a job with a concurrency key\ntuber put -c deploy \"deploy-service-a\"\n\n# Put grouped jobs with a dependent follow-up\ntuber put -g import \"import-row-1\"\ntuber put -g import \"import-row-2\"\ntuber put --aft import \"send-summary\"\n```\n\n## Work\n\nReserve and execute jobs as shell commands.\n\n```bash\ntuber work [OPTIONS]\n```\n\n| Option | Default | Description |\n|---|---|---|\n| `-t`, `--tube` | `default` | Tube to watch |\n| `-j`, `--parallel` | `1` | Number of parallel workers |\n| `-a`, `--addr` | `localhost:11300` | Server address |\n\n```bash\n# Process jobs from the \"emails\" tube with 4 workers\ntuber work -t emails -j 4\n```\n\n## Tubes\n\nList all tubes with a summary of job counts.\n\n```bash\ntuber tubes [OPTIONS]\n```\n\n| Option | Default | Description |\n|---|---|---|\n| `-a`, `--addr` | `localhost:11300` | Server address |\n\n```bash\n$ tuber tubes\ndefault: ready=4 reserved=0 delayed=0 buried=0\nmy-tube: ready=16 reserved=0 delayed=0 buried=0\n```\n\n## Stats\n\nShow global server statistics or per-tube statistics.\n\n```bash\ntuber stats [OPTIONS]\n```\n\n| Option | Default | Description |\n|---|---|---|\n| `-t`, `--tube` | — | Tube name (omit for global stats) |\n| `-a`, `--addr` | `localhost:11300` | Server address |\n\n```bash\n# Global stats\ntuber stats\n\n# Per-tube stats\ntuber stats -t emails\n```\n\n## Shell Jobs\n\nTuber's `work` command reserves jobs and executes each job body as a shell command. Combined with `put`, this gives you a simple distributed task runner from the command line.\n\n```bash\n# Start a server and two workers\ntuber server \u0026\ntuber work -j 2 \u0026\n\n# Queue some work\ntuber put \"echo 'hello world'\"\ntuber put \"curl -s https://example.com/api/webhook -d '{\\\"event\\\": \\\"done\\\"}'\"\n```\n\n### Preventing duplicate work with idempotency keys\n\nUse `-i` to ensure a job is only queued once. If a live job with the same key already exists, tuber returns the original job ID instead of creating a duplicate.\n\n```bash\n# Transcode a video — safe to retry without double-processing\ntuber put -i \"transcode-video-42\" \"ffmpeg -i /data/video-42.raw -c:v libx264 /data/video-42.mp4\"\n\n# Generate a nightly report — re-running the script won't queue it twice\ntuber put -i \"report-2026-03-12\" \"./generate-report.sh 2026-03-12\"\n\n# Keep deduplicating for 5 minutes after the job completes\ntuber put -i \"report-2026-03-12:300\" \"./generate-report.sh 2026-03-12\"\n```\n\n### Serialising work with concurrency keys\n\nUse `-c` to ensure only one job with a given key is processed at a time. Other jobs sharing the key wait in the queue until the current one finishes.\n\n```bash\n# Deploy to each host — one deploy per host at a time, but different hosts in parallel\ntuber put -c \"deploy-web1\" \"./deploy.sh web1\"\ntuber put -c \"deploy-web1\" \"./deploy.sh web1\"  # waits for the first to finish\ntuber put -c \"deploy-web2\" \"./deploy.sh web2\"  # runs in parallel with web1\n\n# Crawl pages from a site without hammering it — serialise per-domain\ntuber put -c \"example.com\" \"curl -o /data/page1.html https://example.com/page1\"\ntuber put -c \"example.com\" \"curl -o /data/page2.html https://example.com/page2\"\ntuber put -c \"other.com\"   \"curl -o /data/index.html https://other.com/\"\n\n# Allow up to 3 concurrent API calls per service\ntuber put -c \"api-svc:3\" \"curl -X POST https://api.example.com/job1\"\ntuber put -c \"api-svc:3\" \"curl -X POST https://api.example.com/job2\"\ntuber put -c \"api-svc:3\" \"curl -X POST https://api.example.com/job3\"\n```\n\n### Chaining pipelines with groups\n\nUse `-g` to group related jobs and `--aft` to hold a follow-up until the group completes.\n\n```bash\n# Import rows in parallel, then send a summary when all are done\ntuber put -g import \"./import-row.sh 1\"\ntuber put -g import \"./import-row.sh 2\"\ntuber put -g import \"./import-row.sh 3\"\ntuber put --aft import \"./send-import-summary.sh\"\n```\n\n## Protocol Reference\n\nTuber speaks the [beanstalkd protocol](https://github.com/beanstalkd/beanstalkd/blob/master/doc/protocol.txt), so any beanstalkd client library works out of the box. Commands marked with **⚡** are tuber extensions beyond standard beanstalkd.\n\nAll commands are `\\r\\n`-terminated. `\u003cid\u003e` is a 64-bit job ID, `\u003cpri\u003e` is a 32-bit priority (0 = most urgent), `\u003cdelay\u003e` and `\u003cttr\u003e` are seconds, `\u003cbytes\u003e` is body length.\n\n### Producer commands\n\n| Command | Description |\n|---|---|\n| `put \u003cpri\u003e \u003cdelay\u003e \u003cttr\u003e \u003cbytes\u003e [tags]\\r\\n\u003cbody\u003e\\r\\n` | Submit a job. Returns `INSERTED \u003cid\u003e` or `BURIED \u003cid\u003e`. |\n| `use \u003ctube\u003e\\r\\n` | Set the tube for subsequent `put` commands. Returns `USING \u003ctube\u003e`. |\n\n**⚡ Put extension tags** — append space-separated tags after `\u003cbytes\u003e`:\n\n| Tag | Effect |\n|---|---|\n| `idp:\u003ckey\u003e` or `idp:\u003ckey\u003e:\u003cttl\u003e` | Idempotency — deduplicates jobs by key within the tube. Optional TTL (seconds) keeps deduplicating after deletion. See [Unique Jobs](#unique-jobs-idempotency). |\n| `grp:\u003cname\u003e` | Assigns the job to a group for fan-out/fan-in. See [Job Groups](#job-groups-fan-out--fan-in). |\n| `aft:\u003cname\u003e` | Holds the job until all jobs in the named group are deleted. See [Job Groups](#job-groups-fan-out--fan-in). |\n| `con:\u003ckey\u003e` or `con:\u003ckey\u003e:\u003climit\u003e` | Concurrency key — limits how many jobs per key can be reserved at once (default 1). See [Concurrency Keys](#concurrency-keys). |\n\n### Worker commands\n\n| Command | Description |\n|---|---|\n| `reserve\\r\\n` | Block until a job is available. Returns `RESERVED \u003cid\u003e \u003cbytes\u003e\\r\\n\u003cbody\u003e`. |\n| `reserve-with-timeout \u003cseconds\u003e\\r\\n` | Like `reserve` but times out. Returns `RESERVED …` or `TIMED_OUT`. |\n| `reserve-job \u003cid\u003e\\r\\n` | Reserve a specific job by ID. Returns `RESERVED …` or `NOT_FOUND`. |\n| **⚡** `reserve-mode \u003cmode\u003e\\r\\n` | Set reserve strategy: `default` (priority-first) or `weighted` (random by tube weight). See [Weighted Reserve](#weighted-reserve). |\n| `delete \u003cid\u003e\\r\\n` | Delete a job. Returns `DELETED` or `NOT_FOUND`. |\n| `release \u003cid\u003e \u003cpri\u003e \u003cdelay\u003e\\r\\n` | Release a reserved job back to ready (or delayed). Returns `RELEASED`. |\n| `bury \u003cid\u003e \u003cpri\u003e\\r\\n` | Bury a reserved job. Returns `BURIED`. |\n| `touch \u003cid\u003e\\r\\n` | Reset the TTR timer on a reserved job. Returns `TOUCHED`. |\n| `watch \u003ctube\u003e [weight]\\r\\n` | Add a tube to the watch list. Optional **⚡** weight for weighted mode. Returns `WATCHING \u003ccount\u003e`. |\n| `ignore \u003ctube\u003e\\r\\n` | Remove a tube from the watch list. Returns `WATCHING \u003ccount\u003e` or `NOT_IGNORED`. |\n\n### Peek / inspect commands\n\n| Command | Description |\n|---|---|\n| `peek \u003cid\u003e\\r\\n` | Peek at a job by ID. Returns `FOUND \u003cid\u003e \u003cbytes\u003e\\r\\n\u003cbody\u003e` or `NOT_FOUND`. |\n| `peek-ready\\r\\n` | Peek at the next ready job in the used tube. |\n| `peek-delayed\\r\\n` | Peek at the next delayed job in the used tube. |\n| `peek-buried\\r\\n` | Peek at the next buried job in the used tube. |\n\n### Admin commands\n\n| Command | Description |\n|---|---|\n| `kick \u003cbound\u003e\\r\\n` | Kick up to `\u003cbound\u003e` buried/delayed jobs in the used tube. Returns `KICKED \u003ccount\u003e`. |\n| `kick-job \u003cid\u003e\\r\\n` | Kick a specific buried or delayed job. Returns `KICKED` or `NOT_FOUND`. |\n| `pause-tube \u003ctube\u003e \u003cdelay\u003e\\r\\n` | Pause a tube for `\u003cdelay\u003e` seconds. Returns `PAUSED`. |\n| **⚡** `flush-tube \u003ctube\u003e\\r\\n` | Delete all jobs from a tube. Returns `FLUSHED \u003ccount\u003e`. |\n| `stats\\r\\n` | Server-wide statistics in YAML. |\n| `stats-job \u003cid\u003e\\r\\n` | Statistics for a single job in YAML. |\n| `stats-tube \u003ctube\u003e\\r\\n` | Statistics for a tube in YAML. |\n| **⚡** `stats-group \u003cname\u003e\\r\\n` | Statistics for a job group in YAML (pending, buried, complete, waiting-jobs). |\n| `list-tubes\\r\\n` | List all existing tubes in YAML. |\n| `list-tube-used\\r\\n` | Show the currently used tube. Returns `USING \u003ctube\u003e`. |\n| `list-tubes-watched\\r\\n` | List watched tubes in YAML. |\n| `quit\\r\\n` | Close the connection. |\n\n## Building\n\n```bash\ncargo build --release\n```\n\nThe binary will be at `target/release/tuber`.\n\n## License\n\nMIT — see [LICENSE](LICENSE).\n\nOriginally created by Keith Rarick and contributors. The original beanstalkd is licensed under the [MIT License](https://github.com/beanstalkd/beanstalkd/blob/master/LICENSE).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdkam%2Ftuber","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdkam%2Ftuber","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdkam%2Ftuber/lists"}