{"id":16340732,"url":"https://github.com/casey/tokio-by-hand","last_synced_at":"2025-03-23T00:32:25.180Z","repository":{"id":57670055,"uuid":"106893677","full_name":"casey/tokio-by-hand","owner":"casey","description":"My attempt to understand Rust futures","archived":false,"fork":false,"pushed_at":"2017-10-14T04:32:40.000Z","size":32,"stargazers_count":13,"open_issues_count":0,"forks_count":1,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-03-18T15:48:57.545Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/casey.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-10-14T03:06:44.000Z","updated_at":"2022-12-22T20:18:28.000Z","dependencies_parsed_at":"2022-09-26T20:40:44.080Z","dependency_job_id":null,"html_url":"https://github.com/casey/tokio-by-hand","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/casey%2Ftokio-by-hand","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/casey%2Ftokio-by-hand/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/casey%2Ftokio-by-hand/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/casey%2Ftokio-by-hand/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/casey","download_url":"https://codeload.github.com/casey/tokio-by-hand/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245040235,"owners_count":20551297,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-10T23:57:41.273Z","updated_at":"2025-03-23T00:32:24.833Z","avatar_url":"https://github.com/casey.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"tokio by hand\n=============\n\n[![docs.rs](https://docs.rs/tokio-by-hand/badge.svg)](https://docs.rs/tokio-by-hand)\n\nI've been trying to wrap my head around futures for a while, and it's been a pretty rough ride.\n\nThe project that I started off working on was to consume a cryptocurrency exchange market data websocket feed and write it to a database.\n\nI started using https://crates.io/crates/websocket and decided to make it async. Although I could have gotten away with using threads, locks, and channels, I had never used tokio and futures in any depth before, so I used it as an excuse to learn.\n\nMy experience was basically that the provided combinators were easy enough to use, but whenever I needed to go beyond them, things got difficult very quickly. The learning curve was very uneven, easy things were manageable, but all of a sudden when things couldn't be done with the basic combinators, I was pretty stuck.\n\nThere were a couple reasons that sometimes the combinators weren't enough:\n\n1. Program organization. I had a huge mass of combinator calls that did everything I wanted, but it was all in a single function and was difficult to read and debug. I was reading from the websocket, writing back control messages to the websocket, deserializing text from the websocket into my local structures, and serializing higher-level control messages to text and writing them back into the websocket.\n  \n    My thought was that I could abstract deserialization, serialization, and low level management of the websocket into another Sink + Stream over the higher level, deserialized types, but this meant implementing a future myself, which is much more difficult than successfully using combinators.\n\n2. Crazy types. I found that combinators generally produced incomprehensible types. If I implemented my own concrete futures over concrete types, the program would be much much simpler. Boxing sort of worked, but sometimes type inference failed in baffling ways, and I needed a lot of annotations to coax things in the correct types\n\nTrying to implement, as my first future, a very complicated mess of deserialization, serialization, and websocket connection management was, as you can imagine, very difficult.\n\nSo, I stepped back and started implementing a bunch of simpler futures, so that I could understand how things worked.\n\nI implemented the following futures:\n- `standard::sleeper::Sleeper`, a future that produces `()` after a timeout by sleeping on a spawned thread.\n- `standard::instant::{Producer,Consumer}` futures that produce and consume a `u8`. A future that consumes a `u8` is a bit unusual, but I used it as the basis for implementing sinks.\n- `standard::delayed::{Producer,Consumer}` same as above, but operate after a timeout\n- `standard::instant_series::{Producer,Consumer}` a Stream and a Sink that produce an infinite sequence of random `u8`s\n- `standard::instant_series::{Producer,Consumer}` same as above, but with a one second delay after each item\n- `standard::buffered::Consumer` same as the above consumer, but contains a fixed size buffer in case items are received more rapidly than they can be processed.\n\nWhen implementing them, my main difficulties in understanding the futures model were as follows:\n\n1. I found the lack of an explicit task argument to be very confusing. It wasn't obvious to me what a task was, how one was created, what it represented, and how a task could be notified.\n\n    I thought about commenting on https://github.com/alexcrichton/futures-rs/issues/129 , but I felt like it would mostly be a \"me too\" comment, so I held off.\n\n2. Understanding when a future had \"done enough\" to ensure that it would make progress in the future. More than a few times I would implement a future and expect it work, but hadn't actually arranged, directly or indirectly, for the task to be notified, so everything would hang.\n\nAfter figuring out where my difficulties lay, I created a modified futures API that would make it hard to avoid these same mistakes.\n\nYou can see the API in the `extended` module. I re-implemented everything in the `standard` module with the `extended` API, and found it much easier to understand why everything was happening.\n\nIt adds an explicit `TaskHandle` to `poll`, `start_send`, and `poll_complete`. This was to help with #1. To help with #2, I extended the `Async` and `AsyncSink` types to also, when not ready, contain an `AgreementToNotify` value.\n\nAn `AgreementToNotify` value is is obtained by calling `TaskHandle::i_will_notify()`, which returns `(Task, AgreementToNotify)`. (Internally, the `Task` is obtained from `task::current()`.) The name of the method is admittadly a bit on-the-nose, but the idea is to mark in your future implementation the point where you're actually getting the current task so that you can notify it in the future.\n\nIn practice, I only did this once in the codebase, in the implementation of `extended::sleeper::Sleeper`. However, this was still very useful, since it made clear whether a future was the last in a chain of futures, and was thus responsible for notifying the task, or whether it was merely delegating to another future.\n\nIf delegating to another future, it would have to get the `AgreementToNotify` from a `NotReady` from another future, and it was thus impossible to return `NotReady` without directly or indirectly arranging for the task to be notified.\n\nThe extended API made things a _lot_ clearer to me. I understand that it's an extremely verbose API, but I think that it aids greatly in understanding the futures model, makes the contracts of the the traits extremely clear, and moves many errors to compile time. At least, I found that to be the case for me.\n\nI was able to implement `extended::adapter::Adapter` pretty easily, which is sort of a simple version of the future I need to write for the original program, which I'm pretty confident that I'll be able to do now.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcasey%2Ftokio-by-hand","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcasey%2Ftokio-by-hand","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcasey%2Ftokio-by-hand/lists"}