{"id":50725202,"url":"https://github.com/fasterthanlime/bilbo","last_synced_at":"2026-06-10T03:30:48.204Z","repository":{"id":358910731,"uuid":"1242199824","full_name":"fasterthanlime/bilbo","owner":"fasterthanlime","description":"Deserialize Rust types from JSON, no derive macro needed","archived":false,"fork":false,"pushed_at":"2026-05-20T06:31:22.000Z","size":1020,"stargazers_count":7,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-09T03:30:26.999Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/fasterthanlime.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE-APACHE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-05-18T08:01:20.000Z","updated_at":"2026-06-05T08:06:11.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/fasterthanlime/bilbo","commit_stats":null,"previous_names":["fasterthanlime/bilbo"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/fasterthanlime/bilbo","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fasterthanlime%2Fbilbo","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fasterthanlime%2Fbilbo/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fasterthanlime%2Fbilbo/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fasterthanlime%2Fbilbo/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/fasterthanlime","download_url":"https://codeload.github.com/fasterthanlime/bilbo/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fasterthanlime%2Fbilbo/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34136112,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-10T02:00:07.152Z","response_time":89,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-06-10T03:30:47.637Z","updated_at":"2026-06-10T03:30:48.198Z","avatar_url":"https://github.com/fasterthanlime.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# bilbo\n\nA Cargo workspace for recovering a value's type **at runtime by reading\nthe program's own DWARF debug info**, guided by the stack frame.\n\nWho hangs out with elves and dwarves all day? That's right — Bilbo the\nhobbit. So:\n\n- **`bilbo`** — the ELF/DWARF/unwinding support layer: capture a frame,\n  unwind it, find which local a pointer aliases, and classify its type\n  into a DWARF-free `plan::Ty`. Format-agnostic.\n- **`bilbo-json`** — a JSON deserializer built on `bilbo` that\n  **JIT-compiles a specialized parser with cranelift** — and beats\n  `serde_json` on the nativejson-benchmark trio. (Room for a\n  `bilbo-postcard` etc. later — they'd share `bilbo`'s cold path.)\n\nIt started as a stupid idea:\n\n```rust\nlet mut e: MaybeUninit\u003cEndpoint\u003e = MaybeUninit::uninit();\nbilbo_json::from_json(r#\"{ \"host\": \"rustweek.org\", \"port\": 443 }\"#,\n                      \u0026mut e as *mut _ as *mut u8);\nlet e = unsafe { e.assume_init() };\n```\n\n`from_json` is handed a `*mut u8` and a string. It has *no generic\nparameter*, no `Deserialize` impl, nothing. It figures out that the\npointer aliases an `Endpoint { host: String, port: u16, … }` — and how\nthat type is laid out in memory — by unwinding its own stack and parsing\nits own debug info. Then it builds the value by poking bytes.\n\nThis is not a serious library. It is, however, faster than serde_json.\n\n## Numbers\n\n`\u0026str` → owned Rust value, release, vs **default** `serde_json` (divan\nmedians on Apple M-series; also runs on x86_64 Linux, not yet timed there):\n\n| input | serde_json | bilbo-json | |\n|---|---|---|---|\n| `citm_catalog.json` (1.7 MB) | ~1.06 ms | **~845 µs** | ~1.25× |\n| `canada.json` (2.3 MB) | ~2.23 ms | **~1.34 ms** | ~1.7× |\n| `twitter.json` (632 KB) | ~397 µs | **~322 µs** | ~1.2× |\n\nOutput is byte-for-byte identical to `serde_json` (citm, twitter —\nincluding twitter's recursive `retweeted_status`). On `canada` we're\nactually *more* correct: serde_json's default float\nparser is best-effort (~1 ULP off on some coordinates); ours\n(`fast-float2`) is correctly-rounded, so the gate is \"structure exact +\ncoords within serde's own error.\"\n\n## How it works\n\nTwo phases. The cursed part happens once; the hot path is boring and\nfast. The cold phase is all `bilbo`; the hot phase is `bilbo-json`.\n\n**Cold — `bilbo` (once per type, cached):**\n\n1. `platform` — capture registers and unwind exactly one frame with\n   [`framehop`](https://crates.io/crates/framehop) (real CFI: macOS\n   compact-unwind / `.eh_frame`; Linux `.eh_frame` / `.eh_frame_hdr`).\n   This gives the caller's de-ASLR'd PC, SP and FP — correct even under\n   `-O3`, where a hand-rolled frame-pointer walk breaks.\n2. `dwarf.rs` — load our own DWARF once into a process-global `Store`\n   (macOS: the `.dSYM`; Linux: the `.debug_*` embedded in\n   `current_exe()`). Map that PC to its `DW_TAG_subprogram`, then\n   evaluate every local's DWARF location expression against the caller's\n   frame to find *which local the pointer aliases* (it's the\n   `MaybeUninit\u003cT\u003e` one). Recover its type DIE.\n3. `classify` turns the type DIE into a self-contained `plan::Ty`:\n   field offsets, primitive sizes, the real `ptr`/`cap`/`len` offsets\n   inside `String`/`Vec` (Rust does not promise their order — we learn\n   it from DWARF), niche *and* tagged `Option`, `()`, tuples, `Box\u003cT\u003e`,\n   `BTreeMap`, and recursive types (a cycle in the DIE graph becomes a\n   back-edge in `Ty`, tied off with `Arc\u003cOnceLock\u003cTy\u003e\u003e`).\n4. `resolve.rs` — a two-level cache: call-site PC → type → `Resolved`\n   (the `Ty` plus a generic per-type `ext` slot). The deserializer is a\n   property of the *type*, not the call site, so two sites filling the\n   same type share one classify — and one compile, because the consumer\n   stashes its compiled artifact in `Resolved::ext`, keyed by that same\n   per-type cache.\n\n**Hot — `bilbo-json` (every call, no DWARF, no file I/O):**\n\n- `jit.rs` — cranelift compiles a function specialized to the `Ty`:\n  field-name bytes and offsets baked in as constants, a `memchr`/\n  hybrid-SIMD scanner, the whole parse+bind in one pass, no\n  intermediate `Json` tree. Object keys dispatch through a linear\n  word-compare chain for narrow structs, but a wide struct (twitter's\n  `User` is 41 fields) instead dispatches in O(1) on the key *length*\n  via a `br_table`, then a tiny per-length chain — the difference\n  between losing and winning on twitter. Recursive types compile one\n  function per cycle and call into it. Or `interp.rs`, a plain\n  interpreter over the same `Ty`, kept as a baseline. The compiled\n  parser lands in `bilbo::Resolved::ext` (one compile per type).\n- `jitdump.rs` — emits `/tmp/jit-\u003cpid\u003e.dump` (perf jitdump) so\n  profilers (e.g. [stax](https://github.com/bearcove/stax)) can name\n  and disassemble the JIT'd code instead of showing `\u003cunresolved\u003e`.\n\nThe one honest caveat: a `BTreeMap` has no DWARF-discoverable layout we\ncan poke our way into (B-tree nodes, unstable). So for maps we call the\n*real* `std::collections::BTreeMap` through thin `#[inline(never)]`\ntrampolines (`bilbo-json`'s `tramp.rs`), monomorphized once per value\ntype, whose addresses `bilbo` resolves — from DWARF, like everything\nelse.\n\n## Running it\n\n```sh\ncargo run -p bilbo-json                 # demo: Endpoint + tagged-Option + Box + recursive, narrated\ncargo bench -p bilbo-json --bench de    # small struct vs serde_json / facet-json\ncargo bench -p bilbo-json --bench citm  # citm_catalog.json\ncargo bench -p bilbo-json --bench canada\ncargo bench -p bilbo-json --bench twitter  # full fidelity: enums, Box, recursion\n```\n\n`BILBO_JSON_PROFILE=1 cargo run -p bilbo-json --release` prints the\nJIT'd parser's address and hammers it forever, for `stax` to sample.\n\nEach bench gates against `serde_json` before timing: byte-for-byte for\n`citm`/`twitter`/`de`, and structure-exact + correctly-rounded floats\nfor `canada` (where serde's default parser is the less accurate one).\n\n## Supported types\n\nStructs, tuples (positional arrays), `Vec\u003cT\u003e`, `String`, `\u0026str`,\n`bool`, `char`, `u8..u64`/`i8..i64`, `f32`/`f64`, `()`, `Option\u003cT\u003e`\n(both niche — `Option\u003cString\u003e`, `Option\u003cBox\u003c_\u003e\u003e` — and tagged —\n`Option\u003cu64\u003e`, `Option\u003cbool\u003e`), an absent struct key meaning `None`\n(serde's rule), `Box\u003cT\u003e`, `BTreeMap\u003cString, V\u003e`, and **recursive\ntypes** (`Status::retweeted_status: Option\u003cBox\u003cStatus\u003e\u003e` ties the knot\nvia `Arc\u003cOnceLock\u003cTy\u003e\u003e`; the JIT emits one function per cycle). This is\nexactly what the full-fidelity `twitter.json` benchmark exercises.\n\n## Caveats\n\n- **Two targets only: macOS/aarch64 and Linux/x86_64.** macOS uses\n  dyld, a Mach-O `.dSYM`, NEON, and framehop's aarch64 unwinder; Linux\n  uses `dl_iterate_phdr`, the ELF's embedded `.debug_*`, and framehop's\n  x86_64 unwinder. Each backend has its own arch inline asm. Nothing\n  else is supported.\n- Needs debug info, configured in `.cargo/config.toml` / `[profile.*]`:\n  macOS keeps a packed `.dSYM`; Linux forces `split-debuginfo=off` so\n  the full `.debug_*` stays embedded in the ELF. Both force frame\n  pointers.\n- Wildly `unsafe` by construction. `from_json(s, ptr)` writes a fully\n  reconstructed value through a raw pointer based on what it *believes*\n  the type is. This is a toy / proof of cursedness, not a crate to\n  depend on.\n- Optimized builds reuse stack slots; we disambiguate by preferring the\n  `MaybeUninit\u003cT\u003e` local at the matched address (which is exactly the\n  API contract).\n\n## Module map\n\n`crates/bilbo` — the ELF/DWARF support layer:\n\n| file | role |\n|---|---|\n| `platform/mod.rs` | platform-agnostic surface; cfg-selects one backend |\n| `platform/darwin.rs` | aarch64 + Mach-O + dyld + `.dSYM` |\n| `platform/linux.rs` | x86_64 + ELF + `dl_iterate_phdr` + embedded `.debug_*` |\n| `frame.rs` | thin re-export of the active backend's regs/unwind |\n| `dwarf.rs` | DWARF store, PC→subprogram, local-by-pointer, classify |\n| `plan.rs` | `Ty` — the cached, DWARF-free layout artifact |\n| `resolve.rs` | two-level cache: callsite → type → `Resolved` (`Ty` + generic `ext`) |\n\n`crates/bilbo-json` — the JSON consumer:\n\n| file | role |\n|---|---|\n| `jit.rs` | cranelift backend (specialized parser) + runtime shims |\n| `interp.rs` | plain interpreter backend (baseline) |\n| `json.rs` | tiny lenient JSON parser (interp/baseline only) |\n| `jitdump.rs` | perf jitdump emitter for profilers |\n| `tramp.rs` | real-`BTreeMap` trampolines, resolved via DWARF |\n\n## License\n\nLicensed under either of\n\n- Apache License, Version 2.0 ([LICENSE-APACHE](LICENSE-APACHE) or\n  \u003chttp://www.apache.org/licenses/LICENSE-2.0\u003e)\n- MIT license ([LICENSE-MIT](LICENSE-MIT) or\n  \u003chttp://opensource.org/licenses/MIT\u003e)\n\nat your option. It's a stupid idea; do whatever you want with it.\n\nUnless you explicitly state otherwise, any contribution intentionally\nsubmitted for inclusion in the work by you, as defined in the Apache-2.0\nlicense, shall be dual licensed as above, without any additional terms or\nconditions.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffasterthanlime%2Fbilbo","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffasterthanlime%2Fbilbo","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffasterthanlime%2Fbilbo/lists"}