{"id":20327612,"url":"https://github.com/tokahuke/yaque","last_synced_at":"2025-04-07T05:12:11.410Z","repository":{"id":44610590,"uuid":"269489006","full_name":"tokahuke/yaque","owner":"tokahuke","description":"Yaque is yet another disk-backed persistent queue for Rust.","archived":false,"fork":false,"pushed_at":"2024-04-08T15:05:35.000Z","size":192,"stargazers_count":83,"open_issues_count":13,"forks_count":11,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-03-30T22:11:22.382Z","etag":null,"topics":["async","asynchronous","asyncio","disk","filesystem","persistence","persistent-storage","queue","resilience","rust","rust-lang"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tokahuke.png","metadata":{"files":{"readme":"readme.md","changelog":"changelog.md","contributing":null,"funding":null,"license":"license","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-06-04T23:50:08.000Z","updated_at":"2025-03-24T17:31:31.000Z","dependencies_parsed_at":"2022-08-31T02:00:47.802Z","dependency_job_id":"1222de90-3ff5-4048-a0b3-cb96b3b1a2c2","html_url":"https://github.com/tokahuke/yaque","commit_stats":{"total_commits":99,"total_committers":7,"mean_commits":"14.142857142857142","dds":0.4949494949494949,"last_synced_commit":"88bbf1833ba466bfe48c298104aec114112858eb"},"previous_names":[],"tags_count":15,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tokahuke%2Fyaque","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tokahuke%2Fyaque/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tokahuke%2Fyaque/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tokahuke%2Fyaque/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tokahuke","download_url":"https://codeload.github.com/tokahuke/yaque/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247595335,"owners_count":20963943,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["async","asynchronous","asyncio","disk","filesystem","persistence","persistent-storage","queue","resilience","rust","rust-lang"],"created_at":"2024-11-14T19:48:18.345Z","updated_at":"2025-04-07T05:12:11.391Z","avatar_url":"https://github.com/tokahuke.png","language":"Rust","readme":"[![crates.io](https://img.shields.io/crates/d/yaque)](https://crates.io/crates/yaque)\n[![docs.rs](https://img.shields.io/docsrs/yaque)](https://docs.rs/yaque/latest/yaque/)\n\n\n# Yaque: Yet Another QUEue\n\nYaque is yet another disk-backed persistent queue (and mutex) for Rust. It\nimplements an SPSC channel using your OS' filesystem. Its main advantages\nover a simple `VecDeque\u003cT\u003e` are that\n* You are not constrained by your RAM size, just by your disk size. This\nmeans you can store gigabytes of data without getting OOM killed.\n* Your data is safe even if you program panics. All the queue state is\nwritten to the disk when the queue is dropped.\n* Your data can *persist*, that is, can exist through multiple executions\nof your program. Think of it as a very rudimentary kind of database.\n* You can pass data between two processes.\n\nYaque is _asynchronous_ and built directly on top of `mio` and `notify`.\nIt is therefore completely agnostic to the runtime you are using for you\napplication. It will work smoothly with `tokio`, with `async-std` or any\nother executor of your choice.\n\n## Sample usage\n\nTo create a new queue, just use the `channel` function, passing a\ndirectory path on which to mount the queue. If the directory does not exist\non creation, it (and possibly all its parent directories) will be created.\n```rust\nuse yaque::channel;\n\nfutures::executor::block_on(async {\n    let (mut sender, mut receiver) = channel(\"data/my-queue\").unwrap();\n})\n```\nYou can also use `Sender::open` and `Receiver::open` to open only one\nhalf of the channel, if you need to.\n\nThe usage is similar to the MPSC channel in the standard library, except\nthat the receiving method, `Receiver::recv` is asynchronous. Writing to\nthe queue with the sender is basically lock-free and atomic.\n```rust\nuse yaque::{channel, queue::try_clear};\n\nfutures::executor::block_on(async {\n    // Open using the `channel` function or directly with the constructors.\n    let (mut sender, mut receiver) = channel(\"data/my-queue\").unwrap();\n    \n    // Send stuff with the sender...\n    sender.send(b\"some data\").await.unwrap();\n\n    // ... and receive it in the other side.\n    let data = receiver.recv().await.unwrap();\n\n    assert_eq!(\u0026*data, b\"some data\");\n\n    // Call this to make the changes to the queue permanent.\n    // Not calling it will revert the state of the queue.\n    data.commit();\n});\n\n// After everything is said and done, you may delete the queue.\n// Use `clear` for awaiting for the queue to be released.\ntry_clear(\"data/my-queue\").unwrap();\n```\nThe returned value `data` is a kind of guard that implements `Deref` and\n`DerefMut` on the underlying type.\n\n## `queue::RecvGuard` and transactional behavior\n\nOne important thing to notice is that reads from the queue are\n_transactional_. The `Receiver::recv` returns a `queue::RecvGuard` that acts as\na _dead man switch_. If dropped, it will revert the dequeue operation,\nunless `queue::RecvGuard::commit` is explicitly called. This ensures that\nthe operation reverts on panics and early returns from errors (such as when\nusing the `?` notation). However, it is necessary to perform one more\nfilesystem operation while rolling back. During drop, this is done on a\n\"best effort\" basis: if an error occurs, it is logged and ignored. This is done\nbecause errors cannot propagate outside a drop and panics in drops risk the\nprogram being aborted. If you _have_ any cleanup behavior for an error from\nrolling back, you may call `queue::RecvGuard::rollback` which _will_ return the\nunderlying error.\n\n## Batches\n\nYou can use the `yaque` queue to send and receive batches of data ,\ntoo. The guarantees are the same as with single reads and writes, except\nthat you may save on OS overhead when you send items, since only one disk\noperation is made. See `Sender::send_batch`, `Receiver::recv_batch` and\n`Receiver::recv_until` for more information on receiver batches.\n\n## Tired of `.await`ing? Timeouts are supported\n\nIf you need your application to not stall when nothing is being put on the\nqueue, you can use `Receiver::recv_timeout` and\n`Receiver::recv_batch_timeout` to receive data, awaiting up to a\ncompletion of a provided future, such as a delay or a channel. Here is an\nexample:\n```rust\nuse yaque::channel;\nuse std::time::Duration;\nuse futures_timer::Delay;\n\nfutures::executor::block_on(async {\n    let (mut sender, mut receiver) = channel(\"data/my-queue-2\").unwrap();\n    \n    // receive some data up to a second\n    let data = receiver\n        .recv_timeout(Delay::new(Duration::from_secs(1)))\n        .await\n        .unwrap();\n\n    // Nothing was sent, so no data...\n    assert!(data.is_none());\n    drop(data);\n    \n    // ... but if you do send something...\n    sender.send(b\"some data\").await.unwrap();\n \n    // ... now you receive something:\n    let data = receiver\n        .recv_timeout(Delay::new(Duration::from_secs(1)))\n        .await\n        .unwrap();\n\n    assert_eq!(\u0026*data.unwrap(), b\"some data\");  \n});\n```\n\n## `Ctrl+C` and other unexpected events\n\nFirst of all, \"Don't panic©\"! Writing to the queue is an atomic operation.\nTherefore, unless there is something really wrong with your OS, you should be\nfine in terms of data corruption most of the time.\n\nIn case of a panic (the program's, not the programmer's), the queue is\nguaranteed to save all the most up-to-date metadata for the receiver. For\nthe reader it is even simpler: there is nothing to be saved in the first\nplace. The only exception to this guarantee is if the saving operation fails\ndue to an IO error. Remember that the program is not allowed to panic during\na panic. Therefore in this case, `yaque` will not attempt to recover from an\nerror.\n\nThe same thing cannot be said from OS signals. Signals from the OS are *not*\nhandled automatically by this library. It is understood that the application\nprogrammer knows best how to handle them. If you chose to close queue on\n`Ctrl+C` or other signals, you are in luck! Saving both sides of the queue\nis [async-signal-safe](https://man7.org/linux/man-pages/man7/signal-safety.7.html)\nso you may set up a bare signal hook directly using, for example,\n`signal_hook`(https://docs.rs/signal-hook/), if you are the sort of person\nthat enjoys `unsafe` code. If not, there are a ton of completely safe\nalternatives out there. Choose the one that suits you the best.\n\nUnfortunately, there are also times when you get `aborted` or `killed`. These\nsignals cannot be handled by any library whatsoever. When this happens, not\neverything is lost yet. We provied a whole module, `recovery`,\nto aid you in automatic queue recovery. Please check the module for the\nspecific function names. From an architectural perspective, we offer two\ndifferent approaches to queue recovery, which may be suitable to different\nuse cases:\n\n1. Recover with replay (the standard): we can reconstruct a _lower bound_\nof the actual state of the queue during the crash, which consists of the\n_maximum_ of the following two positions:\n    * the bottom of the smallest segment still present in the directory.\n    * the position indicated in the metadata file.\n\nSince this is a lower bound, some elements may be replayed. If your\nprocessing is _idempotent_, this will not be an issue and you lose no data\nwhatsoever.\n\n2. Recover with loss: we can also reconstruct an _upper bound_ for the\nactual state of the queue: the bottom of the second smallest segment in\nthe queue. In this case, the smallest segment is simply erased and the\nreceiver caries on as if nothing has happened. If replays are intollerable,\nbut some data loss is, this might be the right alternative for you. You can\nlimit data loss by constraining the segment size, configuring this option on\n`SenderBuilder`.\n\nIf you really want to err on the safer side, you may use `Receiver::save`\nto periodically back the receiver state up. Just choose you favorite timer\nimplementation and set a simple periodical task up every hundreds of milliseconds.\nHowever, be warned that this is only a _mitigation_ of consistency problems, not\na solution.\n\n## Known issues and next steps\n\n* ~~This is a brand new project. Although I have tested it and it will\ncertainly not implode your computer, don't trust your life on it yet.~~ This code\nis running in production for non-critical applications.\n* Wastes too much kernel time when the queue is small enough and the sender\nsends many frequent small messages non-atomically. You can mitigate that by\nwriting in batches to the queue.\n* There are probably unknown bugs hidden in some corner case. If you find\none, please [fill an issue on GitHub](https://github.com/tokahuke/yaque/issues/new).\nPull requests and contributions are also greatly appreciated.\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftokahuke%2Fyaque","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftokahuke%2Fyaque","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftokahuke%2Fyaque/lists"}