{"id":26010470,"url":"https://github.com/jdonszelmann/incremental-query","last_synced_at":"2025-03-05T22:50:43.250Z","repository":{"id":252669180,"uuid":"841094987","full_name":"jdonszelmann/incremental-query","owner":"jdonszelmann","description":"implementation of an incremental compilation algorithm similar to rustc's","archived":false,"fork":false,"pushed_at":"2024-08-29T12:37:07.000Z","size":77,"stargazers_count":7,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-03T17:05:37.242Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jdonszelmann.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE-APACHE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-08-11T16:17:20.000Z","updated_at":"2024-08-29T12:37:10.000Z","dependencies_parsed_at":"2024-08-14T23:07:58.850Z","dependency_job_id":null,"html_url":"https://github.com/jdonszelmann/incremental-query","commit_stats":null,"previous_names":["jdonszelmann/incremental-query"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jdonszelmann%2Fincremental-query","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jdonszelmann%2Fincremental-query/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jdonszelmann%2Fincremental-query/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jdonszelmann%2Fincremental-query/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jdonszelmann","download_url":"https://codeload.github.com/jdonszelmann/incremental-query/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":242117657,"owners_count":20074436,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-03-05T22:50:42.721Z","updated_at":"2025-03-05T22:50:43.240Z","avatar_url":"https://github.com/jdonszelmann.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n[\u003cimg alt=\"github\" src=\"https://img.shields.io/badge/github-jdonszelmann/incremental-query?style=for-the-badge\u0026labelColor=555555\u0026logo=github\" height=\"20\"\u003e](https://github.com/jdonszelmann/incremental-query)\n[\u003cimg alt=\"crates.io\" src=\"https://img.shields.io/crates/v/incremental-query.svg?style=for-the-badge\u0026color=fc8d62\u0026logo=rust\" height=\"20\"\u003e](https://crates.io/crates/incremental-query)\n[\u003cimg alt=\"docs.rs\" src=\"https://img.shields.io/badge/docs.rs-incremental-query?style=for-the-badge\u0026labelColor=555555\u0026logo=docs.rs\" height=\"20\"\u003e](https://docs.rs/incremental-query)\n[\u003cimg alt=\"build status\" src=\"https://img.shields.io/github/actions/workflow/status/jdonszelmann/incremental-query/ci.yml?branch=master\u0026style=for-the-badge\" height=\"20\"\u003e](https://github.com/jdonszelmann/incremental-query/actions)\n\n# Incremental Query\n\nThis library wasn't ever supposed to be a library.\nIt started with me wanting to learn how [rustc's query system](https://rustc-dev-guide.rust-lang.org/query.html) worked, \nand I decided to implement a system very similar to it myself.\nTurns out, I'm quite happy with it and started documenting it and now it's this library.\n\nNote: this crate uses a nightly feature: `#![feature(macro_metavar_expr)]`, so it only works on nightly for now.\n\n# What is this?\n\nCore to this library is the concept of a query. Just like in rustc.\nA query is just like a normal function, and you can call it as if it is.\nThere are just some restrictions to what the parameters and return type of the function can be.\nHowever, when you call them, a lot more happens than with a normal function.\n\nThe first time, the code in a query runs as normal, but the inputs and outputs are cached and hashed.\nFurthermore, any other query you call inside this query (and transitively so) is kept track of.\nOn subsequent invocations of the same query with the same inputs, the cached result is returned.\n\nThis explanation might make it seem like it's just performing memoization. \nHowever, memoized functions must be [pure](https://en.wikipedia.org/wiki/Pure_function)! \nOtherwise, we cannot decide whether the output is the same by just looking at the inputs.\n\nInstead, using queries, functions don't have to be pure.\nPure functions are still very nice though! We don't have to rerun them as often.\nHowever, if an impure query changes its result, this is noticed by the system, and\nany (pure) query depending on it will be automatically rerun when its result is needed.\nAlso, when an impure query is rerun, and a dependent chain of queries updates, but\nat any point we notice that a result didn't change, the chain of reruns is stopped to \nsave as much computation time as possible.\n\nYou might see how this is useful for incremental compilation. One query reads a file, another parses an ast,\nand yet another does a part of typechecking. Rustc has hundreds of queries. If a file hasn't changed, the read file\nquery notices. Then we don't have to run the parse query again, awesome! Typechecking might depend on multiple files,\nbut if none of the files it does depend on change, we don't have to rerun it either.\n\nAt this moment, this library doesn't yet support serializing the query cache to a file. \nSome day it might. If it did, the cache would be useful for even longer.\n\nThe following section is about this library specifically. Even if you're just interested in [how it works](#how-does-it-work),\nit might stil be useful to read this so you understand the notation used in the examples.\n\n# How can I use it?\n\nQueries are defined *almost* like normal functions.\nThe only restriction is that they must be tagged with a `#[query]` attribute.\n\n```rust\nuse incremental_query::{Context, query, rerun};\n# use rand::{thread_rng, Rng};\n\n/// This is a query that retuns a random number.\n#[query]\n#[rerun(always)]\nfn random\u003c'cx\u003e(_cx: \u0026Context\u003c'cx\u003e) -\u003e u64 {\n    thread_rng().gen_range(0..u64::MAX)\n}\n```\n\n\u003e Note: There are some restrictions on what the signature of a query can look like:\n\u003e The function *must* have one lifetime argument.\n\u003e Furthermore, the first parameter *must* be a reference to a [`Context`]\n\u003e Any input *must* be a reference to some `T` which implements [`QueryParameter`].\n\u003e\n\u003e Also, even though inputs to queries are references, when you invoke a query you must\n\u003e provide corresponding *owned values*, not references. The owned values are stored in the cache,\n\u003e while the function gets references to those values *that are now in the cache*.\n\nTo invoke a query, you can now simply call it:\n\n```\nuse incremental_query::{Context, Storage};\n# use incremental_query::{query, rerun};\n# use rand::{thread_rng, Rng};\n# \n# \n# /// This is a query that retuns a random number.\n# #[query]\n# #[rerun(always)]\n# fn random\u003c'cx\u003e(_cx: \u0026Context\u003c'cx\u003e) -\u003e u64 {\n#     thread_rng().gen_range(0..u64::MAX)\n# }\n# \n#\n// instantiate a storage object that holds the cached data\nlet storage = Storage::new();\n// create a context with it\nlet cx = Context::new(\u0026storage);\n\n// call the query\nprintln!(\"{}\", random(\u0026cx));\n```\n\nWhen you call a query, you'll get back a reference to the declared return type.\nThat's again because the actual return value is now stored in the cache. \nYou just get a reference into that cache.\n\n## Function purity and generations\n\nYou may have noticed the `#[rerun(always)]` attribute on the query defined above.\n\nWithout that attribute, the query is expected to be pure. \nThat means that, without an attribute, \na query should not read from a file, or generate random numbers, or call another function that might do that.\nHowever, it may stil call other queries which *are* marked as impure.\n\nA query can be marked as impure in two different ways. Either through `#[rerun(always)]`, \nwhich will never assume its result stayed the same (in fact, the library won't even cache the output).\nThese can be quite bad for performance, so there's an alternative: `#[rerun(generation)]`.\n\nThe context has a generation counter. It starts at `0`, and you can change it with [`Context::next_generation`](Context::next_generation).\nEach cached query also stores in which generation is was cached. Generational queries will be cached as normal,\nbut will be rerun when the generation number changes.\n\nThus, you can make all impure queries generational, and when you, through some means know that what they \ndepend on may have changed (like files on disk have updated), you can increment the generation number and\nthe results will be reflected.\n\nFor example, here:\n```rust\n# use incremental_query::{Context, Storage, query, rerun};\n#[query]\n#[rerun(generation)]\nfn boolean_query\u003c'cx\u003e(_cx: \u0026Context\u003c'cx\u003e) -\u003e bool {\n    // first return true, then in the 2nd generation false\n#   true\n}\n\n#[query]\nfn one\u003c'cx\u003e(_cx: \u0026Context\u003c'cx\u003e) -\u003e u64 {\n    1\n}\n#[query]\nfn two\u003c'cx\u003e(_cx: \u0026Context\u003c'cx\u003e) -\u003e u64 {\n    2\n}\n\n#[query]\nfn conditional\u003c'cx\u003e(cx: \u0026Context\u003c'cx\u003e, ) -\u003e u64 {\n    if *boolean_query(cx) {\n        *one(cx)\n    } else {\n        *two(cx)\n    }\n}\n\nlet storage = Storage::new();\nlet cx = Context::new(\u0026storage);\n\nconditional(\u0026cx);\nconditional(\u0026cx);\nconditional(\u0026cx);\ncx.next_generation();\nconditional(\u0026cx);\nconditional(\u0026cx);\nconditional(\u0026cx);\n```\n\n`boolean_query` is marked as generational. Let's say that it returned `true` in the first generation\nbut `false` in the second. We call `conditional` three times. The first three will all see the result \nto be `true` and will output `1` since the generation is still at `0`, so there's no reason to assume the output may have changed.\nAfter the generation is changed, `conditional` will return `2` on each of the three following invocations.\nin total, `boolean_query` is run 2 times here, `one` and `two` are each run once, and `conditional` is run twice. \nAll other results are cached.\n\n# How does it work?\n\nYou might notice that a lot of the following explanation lines up with the explanation of salsa,\nand rust's query system explained in the [rustc dev guide](https://rustc-dev-guide.rust-lang.org/queries/incremental-compilation.html).\nThat's not a coincidence, as this library started out as me wanting to implement that algorithm.\nI'll try my best to explain the algorithm here again, and maybe simplify some of it a little.\n\n\u003e If you've read the rustc dev guide, you may have wondered the following: it uses the word query for two\n\u003e different concepts interchangably. It took me a while to realize that.\n\u003e First of all, a query is a definition of a query, a function tagged with `#[query]`\n\u003e However, in the guide it also refers to the instance of a query. When it says that a query cannot depend on itself,\n\u003e it doesn't mean that some `type_of(a)` query cannot execute `type_of(b)`. It just means that you can't make a cycle where\n\u003e `type_of(a)` depends recursively on `type_of(a)` again. Similarly, when the guide says a query is cached,\n\u003e it means that a query and a given input to it is cached.\n\nWhen a query is invoked, it actually immediately delegates to the [`Context`]. \nThe [`Context`] *might* then later decide to actually run the function. \nThe [`Context`] consists of a cache, but also of dependency graph, which is a\n[directed acyclic graph or DAG](https://en.wikipedia.org/wiki/Directed_acyclic_graph), \nand a datastructure to detect cycles while running queries.\n\nBefore a query can be run, it first performs the following checks:\n\n1. Hash the input (together with the query type, to make hashes for different queries unique).\n    * If this hash was never seen before, add a new node to the dependency graph and immediately run the query.\n      cache its output, and return.\n2. If the hash already exist, find the corresponding node in the dependency graph\n    * IF the node doesn't have a cached result, run the query and cache the result now. \n3. If it has no cached result, we're still not done because we need to make sure if the cache hasn't been invalidated.\n   There are three kinds of queries: always rerunning, generational and pure.\n    * The cache result of a pure query can, even though its pure, still be invalidated. \n      That can happen when a dependency of the pure query isn't pure. \n      We solve this using what rustc calls the `try_mark_green` algorithm. More on that later\n    * The cache result of generational queries can be invalidated just like a pure query, \n      when a dependency changes. However, its result also gets outdated when the global generation is changed.\n    * The cache result of an always rerunning query is never valid. \n      In fact we never even store its output.\n\n## The try mark green algorithm, or recursively evaluating cache validity\n\nFor every query instance (that is, a query type, its input, hashed input, possibly cached output, and output hash)\ne also store one extra piece of information: is it red or is it green.\n\nThis field can be changed by all kinds of things:\n1. When a query is actually executed, and its output isn't the same as on the previous execution,\n   (or there is no previous output) the query turns red.\n2. When a query is actually executed, and its output is the same as on the previous execution,\n   the query turns green.\n3. When a query is evaluated for cache invalidation, and all its dependencies are green,\n   its marked green.\n\nA cache invalidation subsequently goes as follows:\n\n1. If a query is `rerun(always)` its cache is instantly invalid.\n2. If a query is `rerun(generational)` its cache is instantly invalid when the generation changed.\n3. If a query is already marked green at the start of the validity check, it does not (transitively) \n   depend on any `rerun(always)` query and the generation number hasn't changed, its cache is instantly valid.\n4. For all the dependencies of the query, run this check recursively. \n   This might mean running one of the dependent queries, which could change their red/green status through one of the 3 ways outlined above.\n   * If after recursively running this algorithm for all dependencies, any of the dependencies is red we have to rerun\n     this query, which might change its red/green status.\n   * If after recursively running all dependent queries, all of the dependencies are marked green,\n     we do not have to rerun it, its cache is valid. Actually, we also immediately mark this query as green itself.\n\nNote that here, for some query `Q`, if we have to rerun a dependency because it *could* have changed, \nbut it actually hasn't, then that dependency will be marked green and we won't have to rerun `Q` itself.\n\n## Keeping track of dependencies\n\nEach query instance has a list of dependencies. Whenever a query `Q` is executed,\nthe context actually updates a field called `curr` that says what the current \"parent\"\nquery is. when `Q` runs a query of its own, it will pass through the context again,\nand this dependency list is updated.\n\nNote that when a query is rerun, because some dependency changed, we have to wipe its \ndependency list. That's because the dependency list might change! \nTo adapt an example of the [rustc dev guide](https://rustc-dev-guide.rust-lang.org/queries/incremental-compilation.html#the-query-dag):\n\n```rust\n# use incremental_query::{Context, Storage, query, rerun};\n#[query]\n#[rerun(generation)]\nfn boolean_query\u003c'cx\u003e(_cx: \u0026Context\u003c'cx\u003e) -\u003e bool {\n    // first return true, then in the 2nd generation false\n#   true\n}\n\n#[query]\nfn one\u003c'cx\u003e(_cx: \u0026Context\u003c'cx\u003e) -\u003e u64 {\n    1\n}\n#[query]\nfn two\u003c'cx\u003e(_cx: \u0026Context\u003c'cx\u003e) -\u003e u64 {\n    2\n}\n\n#[query]\nfn conditional\u003c'cx\u003e(cx: \u0026Context\u003c'cx\u003e, ) -\u003e u64 {\n    if *boolean_query(cx) {\n        *one(cx)\n    } else {\n        *two(cx)\n    }\n}\n\nlet storage = Storage::new();\nlet cx = Context::new(\u0026storage);\n\nconditional(\u0026cx);\ncx.next_generation();\nconditional(\u0026cx);\n```\n\nHere, on the first execution of `conditional`, the dependency list will include `one`.\nHowever, after the first generation, the dependency list no longer includes `one`, and will\ninstead contain `two`.\n\n## License\n\nLicensed under either of \u003ca href=\"LICENSE-APACHE\"\u003eApache License, Version 2.0\u003c/a\u003e or \u003ca href=\"LICENSE-MIT\"\u003eMIT license\u003c/a\u003e at your option.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjdonszelmann%2Fincremental-query","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjdonszelmann%2Fincremental-query","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjdonszelmann%2Fincremental-query/lists"}