{"id":13648544,"url":"https://github.com/oxidecomputer/serde_tokenstream","last_synced_at":"2025-07-22T23:38:36.870Z","repository":{"id":38328317,"uuid":"263850941","full_name":"oxidecomputer/serde_tokenstream","owner":"oxidecomputer","description":"serde::Deserializer for proc_macro/proc_macro2::TokenStream","archived":false,"fork":false,"pushed_at":"2024-11-11T21:52:04.000Z","size":205,"stargazers_count":73,"open_issues_count":9,"forks_count":3,"subscribers_count":23,"default_branch":"main","last_synced_at":"2025-07-14T00:09:46.395Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/oxidecomputer.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-05-14T07:53:39.000Z","updated_at":"2025-05-05T08:19:52.000Z","dependencies_parsed_at":"2023-10-11T20:47:13.344Z","dependency_job_id":"f03ff842-51fd-4ec6-b3a1-1afbdbff6838","html_url":"https://github.com/oxidecomputer/serde_tokenstream","commit_stats":{"total_commits":111,"total_committers":4,"mean_commits":27.75,"dds":"0.33333333333333337","last_synced_commit":"b687538444e79adae2abffefb1708549c280d33a"},"previous_names":[],"tags_count":8,"template":false,"template_full_name":null,"purl":"pkg:github/oxidecomputer/serde_tokenstream","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oxidecomputer%2Fserde_tokenstream","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oxidecomputer%2Fserde_tokenstream/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oxidecomputer%2Fserde_tokenstream/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oxidecomputer%2Fserde_tokenstream/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/oxidecomputer","download_url":"https://codeload.github.com/oxidecomputer/serde_tokenstream/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oxidecomputer%2Fserde_tokenstream/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":266591232,"owners_count":23953082,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-22T02:00:09.085Z","response_time":66,"last_error":null,"robots_txt_status":null,"robots_txt_updated_at":null,"robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-02T01:04:20.275Z","updated_at":"2025-07-22T23:38:36.846Z","avatar_url":"https://github.com/oxidecomputer.png","language":"Rust","funding_links":[],"categories":["Rust"],"sub_categories":[],"readme":"# `serde_tokenstream`\n\nThis Rust crate is intended for use with macros that need bespoke configuration.\nIt's implemented as a `serde::Deserializer` that operates on a\n`proc_macro2::TokenSteam` (easily converted from the standard\n`proc_macro::TokenStream`).\n\n## Usage\n\nSay we're building an attribute proc macro that you want consumers to use like\nthis:\n\n```rust\n#[MyMacro {\n    name = \"SNPP\",\n    owner = \"Canary M Burns\",\n    details = {\n        kind = Fission,\n        year_of_opening = 1968,\n    }\n}]\nfn some_func() {\n    ...\n}\n```\n\nIt's also useful for function-like macros:\n\n```rust\nmy_macro!(\n    name = \"SNPP\",\n    owner = \"Hans\",\n    layoffs_in_alphabetical_order = [\n        \"Simpson, Homer\"\n    ]\n);\n```\n\nThe function that implements the proc macro must have two parameters (both of\ntype `proc_macro::TokenStream`): attributes (the tokens with the braces that\nfollow the name of the macro), and the item (the function, type, etc. to\nwhich the macro is applied):\n\n```rust\n#[proc_macro_attribute]\npub fn MyMacro(\n    attr: proc_macro::TokenStream,\n    item: proc_macro::TokenStream,\n) -\u003e proc_macro::TokenStream {\n    ...\n}\n```\n\nWe'll first define the `struct` type that represents the configuration and\n`derive` a `serde::Deserialize`:\n\n```rust\n#[derive(Deserialize)]\nstruct Config {\n    name: String,\n    owner: String,\n    details: ConfigDetails,\n}\n\n#[derive(Deserialize)]\nstruct ConfigDetails {\n    kind: ConfigDetailsType,\n    year_of_opening: usize,\n}\n\n#[derive(Deserialize)]\nenum ConfigDetailsType {\n    Coal,\n    Fission,\n    Hydroelectric,\n}\n```\n\nNow we can parse `attr` into the `Config` struct with\n`serde_tokenstream::from_tokenstream`:\n\n```rust\nuse proc_macro2::TokenStream;\nuse serde_tokenstream::from_tokenstream;\n\n#[allow(non_snake_case)]\n#[proc_macro_attribute]\npub fn MyMacro(\n    attr: proc_macro::TokenStream,\n    item: proc_macro::TokenStream,\n) -\u003e proc_macro::TokenStream {\n    let config = match from_tokenstream::\u003cConfig\u003e(\u0026TokenStream::from(attr)) {\n        Ok(c) =\u003e c,\n        Err(err) =\u003e return err.to_compile_error().into(),\n    };\n\n    ...\n}\n```\n\nSee the `serde` documentation for the full range of controls that can be\napplied to types and their members.\n\n## Error Handling\n\nErrors indicate the problematic portion of consuming code to assist the macro\nconsumer:\n\n```rust\n#[MyMacro{\n    name = \"Rocinante\",\n    owner = \"Rocicorp\",\n    details = {\n        kind = Fusion,\n        year_of_opening = 2347\n    }\n}]\nfn deploy() {\n    ...\n}\n```\n\n```\nerror: unknown variant `Fusion`, expected one of `Coal`, `Fission`, `Hydroelectric`\n --\u003e tests/test_err1.rs:7:16\n  |\n7 |         kind = Fusion,\n  |                ^^^^^^\n```\n\n## Nested attributes\n\nFor parsing attributes nested inside an outer macro, use\n`from_tokenstream_spanned`. This function provides better span attribution for\nerrors at the top level.\n\nThe most common use is with `syn::MetaList`. For example, if your macro is a\nderive macro:\n\n```rust\n#[derive(MyRobot)]\n#[robot {\n    name = \"Mawhrin-Skel\",\n    kind = Drone,\n    planet = \"Eä\",\n}]\nfn monitor() {\n    ...\n}\n```\n\nThen `robot` can be interpreted as a `syn::MetaList` instance. With that:\n\n```rust\nuse serde_tokenstream::from_tokenstream;\n\n#[derive(Deserialize)]\nstruct Robot {\n    ...\n}\n\n#[proc_macro_derive(MyRobot, attributes(robot))]\npub fn my_robot(input: proc_macro::TokenStream) -\u003e proc_macro::TokenStream {\n    let list = /* obtain the `syn::MetaList` from the input */;\n\n    let config = match from_tokenstream_spanned::\u003cRobot\u003e(\n        list.delimiter.span(),\n        \u0026list.tokens\n    ) {\n        Ok(c) =\u003e c,\n        Err(err) =\u003e return err.to_compile_error().into(),\n    };\n}\n```\n\n## TokenStream and syn::\\* values\n\nIn some cases, it's useful to pass TokenStream values as parameters to a macro.\nIn this case we can use the `TokenStreamWrapper` which is a wrapper around\n`TokenStream` that implements `Deserialize` or `ParseWrapper` which is a\nwrapper around `syn::Parse` that implements `Deserialize`. The latter is useful\nfor passing in, for example, a `syn::Path`, or other specific entities from the\n`syn` crate.\n\n## OrderedMap\n\nYou may want to use the map syntax with keys that cannot be used by types such\nas `HashMap` or `BTreeMap` because they don't implement `Hash` or `Ord`. In\nthose cases, you can use an `OrderedMap` and extract the pairs as an iterator\nof tuples.\n\nLet's say we we want our \"keys\" to be `serde_json::Value`s and our value to\nbe... whatever... `String`s! You can't use `serde_json::Value` as the key in a\n`HashMap` or `BTreeMap`, but we can for an `OrderedMap`:\n\n```rust\nlet config = from_tokenstream::\u003cOrderedMap\u003cserde_json::Value, String\u003e\u003e(tokens)?;\n```\n\nThe macro can then be invoked like this:\n\n```rust\nmy_macro!(\n    {\n        \"type\" = \"string\",\n        \"format\" = \"uuid\",\n    } = \"uuid::Uuid\",\n    {\n        \"type\" = \"string\",\n        \"format\" = \"ip\",\n    } = \"std::net::IpAddr\",\n);\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foxidecomputer%2Fserde_tokenstream","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Foxidecomputer%2Fserde_tokenstream","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foxidecomputer%2Fserde_tokenstream/lists"}