{"id":16851904,"url":"https://github.com/cad97/sorbus","last_synced_at":"2025-04-11T07:02:43.696Z","repository":{"id":66119482,"uuid":"243120783","full_name":"CAD97/sorbus","owner":"CAD97","description":"An experimental reimplementation of rowan, focused on size efficiency","archived":false,"fork":false,"pushed_at":"2020-09-04T05:42:14.000Z","size":1005,"stargazers_count":24,"open_issues_count":2,"forks_count":1,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-02-03T17:54:24.850Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://cad97.github.io/sorbus/sorbus/index.html","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/CAD97.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE/APACHE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2020-02-25T22:57:47.000Z","updated_at":"2022-08-17T17:56:10.000Z","dependencies_parsed_at":null,"dependency_job_id":"0994771a-8ae5-4b33-b15d-5a8bb8bdcf89","html_url":"https://github.com/CAD97/sorbus","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CAD97%2Fsorbus","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CAD97%2Fsorbus/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CAD97%2Fsorbus/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CAD97%2Fsorbus/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/CAD97","download_url":"https://codeload.github.com/CAD97/sorbus/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":239570569,"owners_count":19661113,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-13T13:34:03.794Z","updated_at":"2025-02-18T23:31:20.897Z","avatar_url":"https://github.com/CAD97.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# sorbus\n\nA generic \"green\" syntax tree implementation.\nExtracted from [Rust-analyzer]'s [rowan], inspired by Swift's [libSyntax] and .NET's [Roslyn].\n\nRowan currently still uses its own green tree implementation, but is expected to eventually re-export this one.\n\nSorbus is the [genus and subgenus of the Rowan tree](https://en.wikipedia.org/wiki/Sorbus).\n\n  [rust-analyzer]: \u003chttps://github.com/rust-analyzer/rust-analyzer/\u003e\n  [rowan]: \u003chttps://github.com/rust-analyzer/rowan\u003e\n  [libSyntax]: \u003chttps://github.com/apple/swift/tree/swift-5.2.4-RELEASE/lib/Syntax\u003e\n  [Roslyn]: \u003chttps://github.com/dotnet/roslyn\u003e\n  [ericlippert-red-green-trees]: \u003chttps://ericlippert.com/2012/06/08/red-green-trees/\u003e\n  [oilshell-lst]: \u003chttps://www.oilshell.org/blog/2017/02/11.html\u003e\n\n## Red/Green Trees\n\nAs [made popular by Roslyn][ericlippert-red-green-trees].\n\nThe \"green\" tree is an immutable, persistent, [_lossless_][oilshell-lst] syntax tree.\nIt is also uniformly typed; any language semantics are layered on top of the uniformly typed tree.\n\nImportantly for the IDE use case, any source file, even incorrect ones, can be represented.\nAdditionally, due to persistence, incremental changes can typically reuse much of the tree.\n\nConceptually, the green tree is isomorphic to the below definition:\n\n```rust\n#[derive(Copy, Eq)]\nstruct Kind(pub u16);\n\n#[derive(Clone, Eq)]\nstruct Node {\n    kind: Kind,\n    text_len: usize,\n    children: Vec\u003cEither\u003cArc\u003cNode\u003e, Arc\u003cToken\u003e\u003e\u003e,\n}\n\n#[derive(Clone, Eq)]\nstruct Token {\n    kind: SyntaxKind,\n    text: String,\n}\n```\n\nThe \"red\" tree is a transient view of the green tree that remembers parent pointers and absolute textual position within the tree.\nIt is also at this level that language semantics are typically layered on top of the uniformly typed green tree. \nRowan provides the red tree built on top of sorbus's green tree.\n\n## Implementation Tricks\n\n  [DSTs]: \u003chttps://doc.rust-lang.org/reference/dynamically-sized-types.html\u003e\n  [miri]: \u003chttps://github.com/rust-lang/miri\u003e\n  [serde]: \u003clib.rs/serde\u003e\n  [`ArcSwap`]: \u003chttps://docs.rs/arc-swap/0.4/arc_swap/type.ArcSwap.html\u003e\n  [`ArcBorrow`]: \u003chttps://docs.rs/rc-borrow/1/rc_borrow/struct.ArcBorrow.html\u003e\n  [`Thin`]: \u003chttps://docs.rs/erasable/1/erasable/struct.Thin.html\u003e\n  [green::Builder]: \u003chttps://cad97.github.io/sorbus/sorbus/green/struct.Builder.html\u003e\n  [green::TreeBuilder]: \u003chttps://cad97.github.io/sorbus/sorbus/green/struct.TreeBuilder.html\u003e\n  [TreeBuilder::checkpoint]: \u003chttps://cad97.github.io/sorbus/sorbus/green/struct.TreeBuilder.html#method.checkpoint\u003e\n  [Builder::gc]: \u003chttps://cad97.github.io/sorbus/sorbus/green/struct.Builder.html#method.gc\u003e\n\nThe green tree is immutable and persistent, so nodes can be deduplicated.\nThis is achieved in sorbus by proxying all creation of green tree elements through\n[a builder cache][green::Builder], at which point they are deduplicated.\n\nTo reduce the number of allocations required for a green tree and increase the locality,\nthe green nodes and tokens are [DSTs] laid out linearly in memory, roughly as the following:\n\n```text\nNode\n+----------+------+----------+----------------+--------+--------+-----+--------+\n| refcount | kind | text len | children count | child1 | child2 | ... | childN |\n+----------+------+----------+----------------+--------+--------+-----+--------+\n\nToken\n+----------+------+----------+---------+\n| refcount | kind | text len | text... |\n+----------+------+----------+---------+\n```\n\n(though not necessarily in that order). As a result, green tree elements are only usable behind\nan indirection: `Arc` for owned nodes, and `\u0026` for borrowed nodes. Additionally, we use the\n[`ArcBorrow`] type as a \"+0\" reference counted pointer to reduce reference counting overhead.\n\nBecause `Node` and `Token` are DSTs, the indirections to them are fat pointers, taking two words.\nTo mitigate this, the length of the trailing arrays are kept inline. This enables use of [`Thin`]\nto create thin pointers to the green element types. This (and more generally type erasure) are\nused liberally within the library, but can also be used externally if the pointer size is an issue.\n\nNote that `Thin` can be used for (most) any pointer type, including `Thin\u003cArc\u003c_\u003e\u003e` for owned,\n`Thin\u003c\u0026_\u003e` for borrowed, and `Thin\u003cArcBorrow\u003c'_, _\u003e\u003e` for \"+0\" pointers.\n\nPointers that may be to a node or a token are packed into a single word using alignment tagging.\nHowever, the child stored in each node's children array is not just the tagged pointer — instead,\nit also includes the cumulative offset of that child from the parent node. This array is then\npacked tightly by alternating the alignment of each child — one is `(usize, u32)` and the next\nis `(u32, usize)` — such that everything stays nicely aligned and without padding.\n\nThis extra somewhat redundant storage of node offsets allows asymptotically faster tree traversal —\neach node's children can be binary searched, allowing top-down finding of the node at some offset in\njust ***O*(*d* log *w*)** time rather than ***O*(*d* *w*)** time (for tree depth *d* and node width\n*w*). Syntax trees don't have specific bounds on tree depth, but in practice they roughly resemble a\n\"well balanced\" tree with **log *n*** tree depth, so this (for well balanced trees) is a reduction\nfrom ***O*(log *n* × log *n*)** to ***O*(log *n* × log log *n*)**. This is about as much better as\n***O*(log *n*)** is than ***O*(*n*)**.\n\nCaching is done in linear time on the number of direct children by caching based on identity.\nThis is possible due to pervasive caching. Additionaly, the only allocation involved in creating\na tree element is if the node is a newly seen node (both to create the node itself, and to store\nit in the cache (the latter of which is amortized)); cache hits are allocation-free.\n\n## Cool Things\n\nWhile the main [builder cache][green::Builder] is bottom-up, we also provide a convenience top-down\n[tree builder][green::TreeBuilder]. It handles the stack of elements required to build the tree, and\nprovides an API [specifically for pratt parsing][TreeBuilder::checkpoint].\n\nThe green tree supports full serialization and deserialization via [serde], even in\nnon-self-describing formats. (No deduplication is done in the serialized form, however.\nDeduplication is done again at deserialization time.)\n\nComplicated tree structures often suffer from recursive destructors. Sorbus implements destruction\nwith an explicit loop and stack, and thus doesn't risk overflowing a small stack with a large tree.\nAs a result, all of sorbus's tests can (and are 🎉) run under Rust's const evaluator, [miri],\nwhich sanitizes for most forms of UB but has a restricted stack size compared to runtime.\n\n## Future Work\n\nWhile the builder does provide [simplistic garbage collection][Builder::gc], it's _very_ simple:\nit just removes any elements that aren't referenced outside the cache itself. Nodes higher in the\ntree are very unlikely to have identical duplicates, so we could skip caching them. We should\nexplore more interesting [cache strategies] alongside the current \"cache everything\" appraoch.\n\n  [cache strategies]: \u003chttps://medium.com/@bparli/64dc973d5857\u003e\n\n\u003e **Author Note:**  \n\u003e Although, perfect deduplication does make the \"duplicated code\" inspection trivial!\n\u003e Of the extra-information cache strategies, I expect Greedy Dual-Size to outperform a\n\u003e Least Frequently Used strategy (even with aging), as the node's textual length is a very\n\u003e decent predictor of duplication chance. But also because of that, I think a strategy of just\n\u003e not caching nodes with a textual length above some threshold (1024 bytes?) to do just as well.\n\u003e Plus, not having to store all the addtional state for the dynamic cache eviction. We just have\n\u003e to be careful that this doesn't hurt incrementality, as higher nodes _do_ get duplicated when\n\u003e a single file is reparsed with only minor edits.\n\nFurther incrementallity: while sorbus is a persistent, cached, and deduplicated data structure,\nand thus inherently shares state when trees are reconstructed, it could better support one specific\nform of incrementallity: partial or multi-pass parsing. Specifically, it'd be nice to be able to\nparse a source file but only the \"root level,\" leaving items unparsed until a query requires it.\n\n\u003e **Author Note:**  \n\u003e As of current, I'm yet unsure whether this needs specific support at the green (sorbus) level, or\n\u003e just the red (rowan) level. Either way, this is adding back in a form of mutability into the\n\u003e immutable tree, and the red tree pointers have very _interesting_ ownership semantics (that I should\n\u003e write up a design document for once I've given the red tree the same rework and polish that I've given\n\u003e the green tree in sorbus), so it needs to be done carefully. I suspect the best design will end up\n\u003e being to store the root as [`ArcSwap`], using the already existing \"replacement\" API, and swapping\n\u003e the new node back in. This doesn't require any support at the green level. Using `ArcSwap`\n\u003e directly in the green tree is likely impossible because erasure uses unsynronized access.\n\n## License\n\nLicensed under either of\n\n * Apache License, Version 2.0\n   ([LICENSE/APACHE](LICENSE/APACHE) or http://www.apache.org/licenses/LICENSE-2.0)\n * MIT license\n   ([LICENSE/MIT](LICENSE/MIT) or http://opensource.org/licenses/MIT)\n\nat your option.\n\n## Contribution\n\nUnless you explicitly state otherwise, any contribution intentionally submitted\nfor inclusion in the work by you, as defined in the Apache-2.0 license, shall be\ndual licensed as above, without any additional terms or conditions.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcad97%2Fsorbus","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcad97%2Fsorbus","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcad97%2Fsorbus/lists"}