{"id":19958732,"url":"https://github.com/conradludgate/jenner","last_synced_at":"2025-09-03T01:45:15.271Z","repository":{"id":45591786,"uuid":"434526421","full_name":"conradludgate/jenner","owner":"conradludgate","description":"Experiments with generators","archived":false,"fork":false,"pushed_at":"2023-03-22T21:39:34.000Z","size":99,"stargazers_count":8,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-08-30T04:31:08.904Z","etag":null,"topics":["async","generator","iterator","rust","streams","yield"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/conradludgate.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2021-12-03T08:48:07.000Z","updated_at":"2024-01-29T03:30:54.000Z","dependencies_parsed_at":"2025-05-03T21:41:02.026Z","dependency_job_id":null,"html_url":"https://github.com/conradludgate/jenner","commit_stats":null,"previous_names":[],"tags_count":12,"template":false,"template_full_name":null,"purl":"pkg:github/conradludgate/jenner","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/conradludgate%2Fjenner","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/conradludgate%2Fjenner/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/conradludgate%2Fjenner/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/conradludgate%2Fjenner/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/conradludgate","download_url":"https://codeload.github.com/conradludgate/jenner/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/conradludgate%2Fjenner/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":273377160,"owners_count":25094528,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-02T02:00:09.530Z","response_time":77,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["async","generator","iterator","rust","streams","yield"],"created_at":"2024-11-13T01:44:29.054Z","updated_at":"2025-09-03T01:45:15.238Z","avatar_url":"https://github.com/conradludgate.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# jenner\n\nA proc-macro to make use of nightly generator syntax in order to create and manipulate\nstreams using a much easier syntax, much akin to how async/await futures work today.\n\n## Example\n\n```rust\n#![feature(generators, generator_trait, never_type, into_future, async_iterator)] // required nightly feature\nuse jenner::generator;\nuse std::{future::Future, async_iter::AsyncIterator};\n\n/// Creating brand new streams\n#[generator]\n#[yields(u32)]\nasync fn countdown() {\n    yield 5;\n    for i in (0..5).rev() {\n        // futures can be awaited in these streams\n        tokio::time::sleep(std::time::Duration::from_secs(1)).await;\n        // yielding values corresponds to the stream item\n        yield i;\n    }\n}\n\n/// Consuming streams to create new streams (akin to input.map())\n#[generator]\n#[yields(u32)]\nasync fn double(input: impl AsyncIterator\u003cItem = u32\u003e) {\n    // custom async for syntax handles the polling of the stream automatically for you\n    for i in input {\n        yield i * 2;\n    }.await;\n}\n\n/// Futures are also supported\n#[generator]\nasync fn collect\u003cT: std::fmt::Debug\u003e(input: impl AsyncIterator\u003cItem = T\u003e) -\u003e Vec\u003cT\u003e {\n    let mut v = vec![];\n    for i in input {\n        println!(\"{:?}\", i);\n        v.push(i)\n    }.await;\n    /// Return value of the stream is the output of the future\n    v\n}\n```\n\n## Breakdown\n\nThe `generator` attribute macro works in a very simple way, making a few simple but crucial transformations.\n\n### Generator\n\nFirstly, the function signature is re-written to\n\n```rust\nfn countdown() -\u003e impl ::jenner::AsyncGenerator\u003cu32, ()\u003e;\nfn double(input: impl AsyncIterator\u003cItem = u32\u003e) -\u003e impl ::jenner::AsyncGenerator\u003cu32, ()\u003e;\nfn collect(input: impl AsyncIterator\u003cItem = u32\u003e) -\u003e impl ::jenner::AsyncGenerator\u003c!, Vec\u003cu32\u003e\u003e; // never yields\n```\n\nThen, the function body is wrapped in this expression\n\n```rust\nunsafe {\n    ::jenner::GeneratorImpl::new_async(|mut __cx: ::jenner::UnsafeContextRef|{\n        $body\n    })\n}\n```\n\nThe `new_async` function is fairly simple.\nIt accepts a `Generator\u003cYield = Poll\u003cY\u003e, Return = R\u003e` and returns an `AsyncGenerator\u003cY, R\u003e` type,\nwhich implements both `AsyncIterator\u003cItem = Y\u003e` and `Future\u003cOutput = R\u003e`.\n\n### Yields\n\nAny `yield` keywords in the body are modified from\n\n```rust\nyield $expr\n```\n\ninto\n\n```rust\nyield Poll::Ready($expr)\n```\n\nThis allows the generator to tell the stream that a new value is now ready.\n\n### Awaits\n\nCurrently, with the state of generators in nightly, you cannot mix `yield`s and `await`s.\nTo get around this, the following rule is applied\n\nAny `.await` keywords in the body are modified from\n\n```rust\n$expr.await\n```\n\ninto\n\n```rust\n{\n    let fut = $expr;\n    let mut fut = IntoFuture::into_future(fut);\n    loop {\n        let pinned = unsafe { Pin::new_unchecked(\u0026mut fut) };\n        let polled = Future::poll(pinned, \u0026mut *__cx);\n        match polled {\n            Poll::Ready(r) =\u003e break r,\n            Poll::Pending =\u003e {\n                __cx = yield Poll::Pending,\n            }\n        }\n    }\n}\n```\n\nThis change is quite big in comparison to the `yield`.\n\nWe create a loop to allow us to repeatedly poll the future.\nIf the future is still pending, then we just yield that back up to the stream.\nThis tells the stream that it's currently waiting for some asynchronous task to complete.\n\nIf the future's output is now ready, we `break` the value from the loop. This uses the fact\nthat loops are an expression. This allows us to assign the value from the future into our stream's scope.\n\nThis is pretty close to how `await` works in regular rust's `async` blocks.\n\n### For Await\n\nIterating over streams is currently a very poor experience.\nInstead, we provide a simple syntax to iterate any generator asynchronously.\n\n```rust\nlet output = for i in $stream {\n    $body\n}.await;\n```\n\nOne thing of note, the for loop now returns a value. This is not like standard for loops, but is similar to the `loop` keyword.\nThe idea here is that generators both have their iterator part, as well as a final output. We may want to capture that.\n\nHowever, we cannot rely on the loop completing every time, the user could have their own conditional break statement. We deal\nwith this by returning a `jenner::ForResult\u003cBreak, Complete\u003e` enum type, not too different from `Result`.\n\nYou can use `result.finished()` to turn a `ForResult\u003cBreak, Complete\u003e` into `Result\u003cComplete, Break\u003e`. There's also a helper function\n`fn complete(self) -\u003e Complete` if there are no `break`s inside of the loop.\n\nWhen processed, the code turns into\n\n```rust\n{\n    let gen = #stream; // evaluate the stream\n    let mut gen = {\n        // weak form of specialisation.\n        use ::jenner::{__private::IntoAsyncGenerator, AsyncGenerator};\n        gen.into_async_generator()\n    };\n    let res: ::jenner::ForResult\u003c_, _\u003e = loop {\n        let next = loop {\n            let pinned = unsafe { Pin::new_unchecked(\u0026mut fut) };\n            let polled = AsyncIterator::poll_resume(pinned, \u0026mut* __cx);\n            match polled {\n                Poll::Ready(r) =\u003e break r,\n                Poll::Pending =\u003e {\n                    _cx = yield Poll::Pending;\n                }\n            }\n        };\n        match next {\n            GeneratorState::Yielded(i) =\u003e #body,\n            GeneratorState::Complete(c) =\u003e break ForResult::Complete(c),\n        }\n    };\n    res\n}\n```\n\nThis is pretty similar to the `await` case, but repeated.\n\n### Futures\n\nWhile these stream generators are automatically valid futures,\nand edge case occurs when you never actually call `yield` since the\n`Yield` type cannot be inferred from the context.\n\nWe solve this by counting the number of `yield` statements we see in the body.\nIf no `yield` tokens are found, we hard encode the `Yield` type in the function to `()`.\nThis is similar to how omitting a return from a function results in `()` being the returned value.\n\n### Error Handling\n\nSince these generators are also functions that can return value,\nwe can use the try `?` syntax to return early from functions.\n\n```rust\n#[generator]\n#[yields(u32)]\nfn make_requests() -\u003e Result\u003c(), \u0026'static str\u003e {\n    for i in 0..5 {\n        let resp = async move {\n            // imagine this makes a http request that could fail\n            let req = if i == 4 { Err(\"4 is a random number\") } else { Ok(i) };\n            req\n        }.await;\n\n        // Using the `?` syntax to return early with the error\n        // but continue with any good values. (can be used anywhere and not exclusively with yields)\n        yield resp?;\n    }\n\n    // we don't care about the return value, but rust needs one anyway\n    Ok(())\n}\n```\n\nThis requires no extra special code, except for ensuring that the return type is well defined.\nIn this case, that's performed by ensuring the return value is both `AsyncIterator + Future`, specifying the\noutput of the future to be a result.\n\nThis is also not exclusive to `Result`, any it supports anything that the regular `try` syntax supports.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fconradludgate%2Fjenner","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fconradludgate%2Fjenner","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fconradludgate%2Fjenner/lists"}