{"id":20269903,"url":"https://github.com/teliosdev/tq","last_synced_at":"2025-08-22T07:34:13.547Z","repository":{"id":211659216,"uuid":"723983050","full_name":"teliosdev/tq","owner":"teliosdev","description":"TQ is a Rust library to manage a task queue.","archived":false,"fork":false,"pushed_at":"2025-04-08T02:07:00.000Z","size":106,"stargazers_count":0,"open_issues_count":3,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-07-10T18:19:29.226Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/teliosdev.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2023-11-27T06:45:05.000Z","updated_at":"2025-04-03T16:53:59.000Z","dependencies_parsed_at":"2023-12-18T07:24:53.069Z","dependency_job_id":"4dab2adc-e0f7-4556-b774-7d8ecad56dde","html_url":"https://github.com/teliosdev/tq","commit_stats":null,"previous_names":["teliosdev/tq"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/teliosdev/tq","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/teliosdev%2Ftq","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/teliosdev%2Ftq/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/teliosdev%2Ftq/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/teliosdev%2Ftq/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/teliosdev","download_url":"https://codeload.github.com/teliosdev/tq/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/teliosdev%2Ftq/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":271605659,"owners_count":24788968,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-22T02:00:08.480Z","response_time":65,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-14T12:27:57.718Z","updated_at":"2025-08-22T07:34:13.478Z","avatar_url":"https://github.com/teliosdev.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# TQ\n\nTQ is a Rust library to manage a task queue.\n\nIt is currently built for low-throughput queues, where the number of tasks\nprocessed per second is low, but the number of tasks in the queue is high.\n\nRight now, the only provider for TQ is Redis, but more providers can be added\nin the future.\n\n## Usage\n\nThe library is split into two parts: the producer, and the consumer.  To use\nthe former, the `producer` feature must be enabled; and the latter, `consumer`.\nEach provide the opportunity to specify the name of the queue, with some\nproducer specific options available for both.  This readme will assume you're\nusing the `redis` provider (enabled by the `redis` feature).\n\n### Producer\n\nThe producer is used to add tasks to the queue.  It can be used as follows:\n\n```rust\nuse redis::Client;\nuse tq::producer::redis::RedisProducer;\n\n// A redis client only contains the information to connect to Redis; it does\n// not yet connect to redis.\nlet client = Client::open(\"redis://127.0.0.1/0\")?;\n// The maximum length of the stream.  If the stream is longer than this, the\n// `push` operation may silently truncate the stream in some cases - in others\n// it will error.\nlet stream_max_len = 1000;\n// Create the producer.\nlet mut producer = RedisProducer::new(client, stream_max_len);\n\n// The task must only be serializable/deserializable by serde (and\n// Send + Sync + 'static).\n#[derive(Serialize, Deserialize)]\nstruct Task {\n    id: u64,\n    /* ... */\n}\n\n// Push the task to the queue.  This connects to redis and creates the stream\n// if it does not exist.  The connection only lasts for the length of the\n// push.\nproducer.push(\"my_queue\", \u0026Task { id: 0, /* ... */ }).await?;\n// Alternatively, you can bypass the lag check and push the task to the queue\n// immediately.  This will still silently truncate the stream if it is longer\n// than `stream_max_len`.\nproducer.push_unchecked(\"my_queue\", \u0026Task { id: 1, /* ... */ }).await?;\n```\n\nThere's not much in the way of configuration here, apart from the\n`stream_max_len`.  The only interesting thing we can check is the lag, via\n`producer.current_lag()`, which returns the estimated maximum lag of all of\nthe groups, in messages.  This is an estimate, and may be inaccurate.\n\n### Consumer\n\nThe consumer is used to consume tasks from the queue.  This portion is far\nmore complex than the producer, and has many more options.  It can be used as\nfollows:\n\n```rust\nuse redis::Client;\nuse tq::consumer::redis::RedisConsumer;\n\nlet client = Client::open(\"redis://127.0.0.1/0\")?;\nlet mut consumer = RedisConsumer::new(\n    client,\n    // The name of the queue.  This should match what was passed in to the\n    // producer.\n    \"my_queue\".to_string(),\n    // The name of the consuemr group to use.\n    \"my_group\".to_string(),\n    // The idle time of a message.  If a message is not acknowledged within\n    // this time, it will be requeued (notwithstanding heartbeats).\n    Duration::from_secs(60 * 60)\n);\n\n// This creates the queue and the consumer group within the queue.  Note that\n// the group doesn't begin tracking position until it is created, so lag will\n// always be 0 until this is called. (It will then immediately jump to the\n// length of the queue.)\nconsumer.create().await?;\n\n// The task must only be serializable/deserializable by serde (and\n// Send + Sync + 'static).\n#[derive(Serialize, Deserialize)]\nstruct Task {\n    id: u64,\n    /* ... */\n}\n\n// To process messages, we must create a tower service.  This is a bit\n// complicated, but it's not too bad - `tower::service_fn` makes this much\n// easier.\nlet service = tower::service_fn(|()| async move {\n    Ok::\u003c_, core::convert::Infallible\u003e(tower::service_fn(|message: tq::Message\u003cTask\u003e| async move {\n        // Do something with the message.\n        println!(\"Got message: {:?}\", message);\n        // `Ok` means the message should be ack'd, `Err` means it should be\n        // nack'd.\n        Ok::\u003c(), ()\u003e(())\n    }))\n});\n\nlet consumer = Consumer::build::\u003cTask, _\u003e(consumer)\n    .with_service(service)\n    // The keep alive interval between heartbeats.  A good default is half of\n    // the idle timeout above.\n    .with_keep_alive_interval(Duration::from_secs(60 * 30))\n    // The amount of time to wait for a message from redis before looping\n    // again.\n    .with_poll_interval(Duration::from_secs(60))\n    .build();\n\n// \"main\" is the name of the consumer within the consuemr group to use.\n// This will not be the actual name - it will instead be `\"main-1\"`, `\"main-2\"`,\n// one for each of the consumer this spawns.  The second argument is the number\n// of consumers to spawn, equivalent to the parallelism of the consumer.\n//\n// This does not start driving the future, it only creates it.  To drive it,\n// you will need to call `wait` on spawn.\nlet spawn = consumer.run(\"main\", 1).await?;\n// Begin graceful shutdown on ctrl+c.\nlet spawn = spawn.with_graceful_shutdown(async move {\n    tokio::signal::ctrl_c().await.ok();\n});\nlet spawn = spawn.wait().await?;\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fteliosdev%2Ftq","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fteliosdev%2Ftq","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fteliosdev%2Ftq/lists"}