{"id":13667925,"url":"https://github.com/nrc/stupid-stats","last_synced_at":"2025-03-24T02:32:55.388Z","repository":{"id":27606195,"uuid":"31089790","full_name":"nrc/stupid-stats","owner":"nrc","description":"Tutorial and demo of rust compiler replacement tooling","archived":false,"fork":false,"pushed_at":"2021-03-23T00:53:03.000Z","size":35,"stargazers_count":56,"open_issues_count":0,"forks_count":10,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-19T02:44:17.665Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/nrc.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-02-20T23:12:41.000Z","updated_at":"2024-11-12T05:12:47.000Z","dependencies_parsed_at":"2022-09-02T17:30:21.047Z","dependency_job_id":null,"html_url":"https://github.com/nrc/stupid-stats","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nrc%2Fstupid-stats","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nrc%2Fstupid-stats/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nrc%2Fstupid-stats/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nrc%2Fstupid-stats/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/nrc","download_url":"https://codeload.github.com/nrc/stupid-stats/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245198782,"owners_count":20576444,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-02T07:00:54.605Z","updated_at":"2025-03-24T02:32:55.141Z","avatar_url":"https://github.com/nrc.png","language":"Rust","funding_links":[],"categories":["Rust"],"sub_categories":[],"readme":"# A tutorial on creating a drop-in replacement for rustc.\n\nMany tools benefit from being a drop-in replacement for a compiler. By this, I\nmean that any user of the tool can use `mytool` in all the ways they would\nnormally use `rustc` - whether manually compiling a single file or as part of a\ncomplex make project or Cargo build, etc. That could be a lot of work;\nrustc, like most compilers, takes a large number of command line arguments which\ncan affect compilation in complex and interacting ways. Emulating all of this\nbehaviour in your tool is annoying at best, especically if you are making many\nof the same calls into librustc that the compiler is.\n\nThe kind of things I have in mind are tools like rustdoc or a future rustfmt.\nThese want to operate as closely as possible to real compilation, but have\ntotally different outputs (documentation and formatted source code,\nrespectively). Another use case is a customised compiler. Say you want to add a\ncustom code generation phase after macro expansion, then creating a new tool\nshould be easier than forking the compiler (and keeping it up to date as the\ncompiler evolves).\n\nI have gradually been trying to improve the API of librustc to make creating a\ndrop-in tool easier to produce (many others have also helped improve these\ninterfaces over the same time frame). It is now pretty simple to make a tool\nwhich is as close to rustc as you want it to be. In this tutorial I'll show\nhow.\n\nNote/warning, everything I talk about in this tutorial is internal API for\nrustc. It is all extremely unstable and likely to change often and in\nunpredictable ways. Maintaining a tool which uses these APIs will be non-\ntrivial, although hopefully easier than maintaining one that does similar things\nwithout using them.\n\nThis tutorial starts with a very high level view of the rustc compilation\nprocess and of some of the code that drives compilation. Then I'll describe how\nthat process can be customised. In the final section of the tutorial, I'll go\nthrough an example - stupid-stats - which shows how to build a drop-in tool.\n\n\n## Overview of the compilation process\n\nCompilation using rustc happens in several phases. We start with parsing, this\nincludes lexing. The output of this phase is an AST (abstract syntax tree).\nThere is a single AST for each crate (indeed, the entire compilation process\noperates over a single crate). Parsing abstracts away details about individual\nfiles which will all have been read in to the AST in this phase. At this stage\nthe AST includes all macro uses, attributes will still be present, and nothing\nwill have been eliminated due to `cfg`s.\n\nThe next phase is configuration and macro expansion. This can be thought of as a\nfunction over the AST. The unexpanded AST goes in and an expanded AST comes out.\nMacros and syntax extensions are expanded, and `cfg` attributes will cause some\ncode to disappear. The resulting AST won't have any macros or macro uses left\nin.\n\nThe code for these first two phases is in [libsyntax](https://github.com/rust-lang/rust/tree/master/src/libsyntax).\n\nAfter this phase, the compiler allocates ids to each node in the AST\n(technically not every node, but most of them). If we are writing out\ndependencies, that happens now.\n\nThe next big phase is analysis. This is the most complex phase and\nuses the bulk of the code in rustc. This includes name resolution, type\nchecking, borrow checking, type and lifetime inference, trait selection, method\nselection, linting, and so forth. Most error detection is done in this phase\n(although parse errors are found during parsing). The 'output' of this phase is\na bunch of side tables containing semantic information about the source program.\nThe analysis code is in [librustc](https://github.com/rust-lang/rust/tree/master/src/librustc)\nand a bunch of other crates with the 'librustc_' prefix.\n\nNext is translation, this translates the AST (and all those side tables) into\nLLVM IR (intermediate representation). We do this by calling into the LLVM\nlibraries, rather than actually writing IR directly to a file. The code for this is in\n[librustc_trans](https://github.com/rust-lang/rust/tree/master/src/librustc_trans).\n\nThe next phase is running the LLVM backend. This runs LLVM's optimisation passes\non the generated IR and then generates machine code. The result is object files.\nThis phase is all done by LLVM, it is not really part of the rust compiler. The\ninterface between LLVM and rustc is in [librustc_llvm](https://github.com/rust-lang/rust/tree/master/src/librustc_llvm).\n\nFinally, we link the object files into an executable. Again we outsource this to\nother programs and it's not really part of the rust compiler. The interface is\nin [librustc_back](https://github.com/rust-lang/rust/tree/master/src/librustc_back)\n(which also contains some things used primarily during translation).\n\nAll these phases are coordinated by the driver. To see the exact sequence, look\nat the `compile_input` function in [librustc_driver/driver.rs](https://github.com/rust-lang/rust/tree/master/src/librustc_driver/driver.rs).\nThe driver (which is found in [librust_driver](https://github.com/rust-lang/rust/tree/master/src/librustc_driver))\nhandles all the highest level coordination of compilation - handling command\nline arguments, maintaining compilation state (primarily in the `Session`), and\ncalling the appropriate code to run each phase of compilation. It also handles\nhigh level coordination of pretty printing and testing. To create a drop-in\ncompiler replacement or a compiler replacement, we leave most of compilation\nalone and customise the driver using its APIs.\n\n\n## The driver customisation APIs\n\nThere are two primary ways to customise compilation - high level control of the\ndriver using `CompilerCalls` and controlling each phase of compilation using a\n`CompileController`. The former lets you customise handling of command line\narguments etc., the latter lets you stop compilation early or execute code\nbetween phases.\n\n\n### `CompilerCalls`\n\n`CompilerCalls` is a trait that you implement in your tool. It contains a fairly\nad-hoc set of methods to hook in to the process of processing command line\narguments and driving the compiler. For details, see the comments in\n[librustc_driver/lib.rs](https://github.com/rust-lang/rust/tree/master/src/librustc_driver/lib.rs).\nI'll summarise the methods here.\n\n`early_callback` and `late_callback` let you call arbitrary code at different\npoints - early is after command line arguments have been parsed, but before\nanything is done with them; late is pretty much the last thing before\ncompilation starts, i.e., after all processing of command line arguments, etc. is\ndone. Currently, you get to choose whether compilation stops or continues at\neach point, but you don't get to change anything the driver has done. You can\nrecord some info for later, or perform other actions of your own.\n\n`some_input` and `no_input` give you an opportunity to modify the primary input\nto the compiler (usually the input is a file containing the top module for a\ncrate, but it could also be a string). You could record the input or perform\nother actions of your own.\n\nIgnore `parse_pretty`, it is unfortunate and hopefully will get improved. There\nis a default implementation, so you can pretend it doesn't exist.\n\n`build_controller` returns a `CompileController` object for more fine-grained\ncontrol of compilation, it is described next.\n\nWe might add more options in the future.\n\n\n### `CompilerController`\n\n`CompilerController` is a struct consisting of `PhaseController`s and flags.\nCurrently, there is only flag, `make_glob_map` which signals whether to produce\na map of glob imports (used by save-analysis and potentially other tools). There\nare probably flags in the session that should be moved here.\n\nThere is a `PhaseController` for each of the phases described in the above\nsummary of compilation (and we could add more in the future for finer-grained\ncontrol). They are all `after_` a phase because they are checked at the end of a\nphase (again, that might change), e.g., `CompilerController::after_parse`\ncontrols what happens immediately after parsing (and before macro expansion).\n\nEach `PhaseController` contains a flag called `stop` which indicates whether\ncompilation should stop or continue, and a callback to be executed at the point\nindicated by the phase. The callback is called whether or not compilation\ncontinues.\n\nInformation about the state of compilation is passed to these callbacks in a\n`CompileState` object. This contains all the information the compiler has. Note\nthat this state information is immutable - your callback can only execute code\nusing the compiler state, it can't modify the state. (If there is demand, we\ncould change that). The state available to a callback depends on where during\ncompilation the callback is called. For example, after parsing there is an AST\nbut no semantic analysis (because the AST has not been analysed yet). After\ntranslation, there is translation info, but no AST or analysis info (since these\nhave been consumed/forgotten).\n\n\n## An example - stupid-stats\n\nOur example tool is very simple, it simply collects some simple and not very\nuseful statistics about a program; it is called stupid-stats. You can find\nthe (more heavily commented) complete source for the example on [Github](https://github.com/nick29581/stupid-stats/blob/master/src).\nTo build, just do `cargo build`. To run on a file `foo.rs`, do `cargo run\nfoo.rs` (assuming you have a Rust program called `foo.rs`. You can also pass any\ncommand line arguments that you would normally pass to rustc). When you run it\nyou'll see output similar to\n\n```\nIn crate: foo,\n\nFound 12 uses of `println!`;\nThe most common number of arguments is 1 (67% of all functions);\n25% of functions have four or more arguments.\n```\n\nTo make things easier, when we talk about functions, we're excluding methods and\nclosures.\n\nYou can also use the executable as a drop-in replacement for rustc, because\nafter all, that is the whole point of this exercise. So, however you use rustc\nin your makefile setup, you can use `target/stupid` (or whatever executable you\nend up with) instead. That might mean setting an environment variable or it\nmight mean renaming your executable to `rustc` and setting your PATH. Similarly,\nif you're using Cargo, you'll need to rename the executable to rustc and set the\nPATH. Alternatively, you should be able to use\n[multirust](https://github.com/brson/multirust) to get around all the PATH stuff\n(although I haven't actually tried that).\n\n(Note that this example prints to stdout. I'm not entirely sure what Cargo does\nwith stdout from rustc under different circumstances. If you don't see any\noutput, try inserting a `panic!` after the `println!`s to error out, then Cargo\nshould dump stupid-stats' stdout to Cargo's stdout).\n\nLet's start with the `main` function for our tool, it is pretty simple:\n\n```\nfn main() {\n    let args: Vec\u003c_\u003e = std::env::args().collect();\n    rustc_driver::run_compiler(\u0026args, \u0026mut StupidCalls::new());\n    std::env::set_exit_status(0);\n}\n```\n\nThe first line grabs any command line arguments. The second line calls the\ncompiler driver with those arguments. The final line sets the exit code for the\nprogram.\n\nThe only interesting thing is the `StupidCalls` object we pass to the driver.\nThis is our implementation of the `CompilerCalls` trait and is what will make\nthis tool different from rustc.\n\n`StupidCalls` is a mostly empty struct:\n\n```\nstruct StupidCalls {\n    default_calls: RustcDefaultCalls,\n}\n```\n\nThis tool is so simple that it doesn't need to store any data here, but usually\nyou would. We embed a `RustcDefaultCalls` object to delegate to in our impl when\nwe want exactly the same behaviour as the Rust compiler. Mostly you don't want\nto do that (or at least don't need to) in a tool. However, Cargo calls rustc\nwith the `--print file-names`, so we delegate in `late_callback` and `no_input`\nto keep Cargo happy.\n\nMost of the rest of the impl of `CompilerCalls` is trivial:\n\n```\nimpl\u003c'a\u003e CompilerCalls\u003c'a\u003e for StupidCalls {\n    fn early_callback(\u0026mut self,\n                        _: \u0026getopts::Matches,\n                        _: \u0026config::Options,\n                        _: \u0026diagnostics::registry::Registry,\n                        _: ErrorOutputType)\n                      -\u003e Compilation {\n        Compilation::Continue\n    }\n\n    fn late_callback(\u0026mut self,\n                     t: \u0026TransCrate,\n                     m: \u0026getopts::Matches,\n                     s: \u0026Session,\n                     c: \u0026CrateStore,\n                     i: \u0026Input,\n                     odir: \u0026Option\u003cPathBuf\u003e,\n                     ofile: \u0026Option\u003cPathBuf\u003e)\n                     -\u003e Compilation {\n        self.default_calls.late_callback(t, m, s, c, i, odir, ofile);\n        Compilation::Continue\n    }\n\n    fn some_input(\u0026mut self,\n                  input: Input,\n                  input_path: Option\u003cPath\u003e)\n                  -\u003e (Input, Option\u003cPath\u003e) {\n        (input, input_path)\n    }\n\n    fn no_input(\u0026mut self,\n                m: \u0026getopts::Matches,\n                o: \u0026config::Options,\n                odir: \u0026Option\u003cPath\u003e,\n                ofile: \u0026Option\u003cPath\u003e,\n                r: \u0026diagnostics::registry::Registry)\n                -\u003e Option\u003c(Input, Option\u003cPath\u003e)\u003e {\n        self.default_calls.no_input(m, o, odir, ofile, r);\n\n        // This is not optimal error handling.\n        panic!(\"No input supplied to stupid-stats\");\n    }\n\n    fn build_controller(\u0026mut self, _: \u0026Session) -\u003e driver::CompileController\u003c'a\u003e {\n        ...\n    }\n}\n```\n\nWe don't do anything for either of the callbacks, nor do we change the input if\nthe user supplies it. If they don't, we just `panic!`, this is the simplest way\nto handle the error, but not very user-friendly, a real tool would give a\nconstructive message or perform a default action.\n\nIn `build_controller` we construct our `CompileController`. We only want to\nparse, and we want to inspect macros before expansion, so we make compilation\nstop after the first phase (parsing). The callback after that phase is where the\ntool does it's actual work by walking the AST. We do that by creating an AST\nvisitor and making it walk the AST from the top (the crate root). Once we've\nwalked the crate, we print the stats we've collected:\n\n```\nfn build_controller(\u0026mut self, _: \u0026Session) -\u003e driver::CompileController\u003c'a\u003e {\n    // We mostly want to do what rustc does, which is what basic() will return.\n    let mut control = driver::CompileController::basic();\n    // But we only need the AST, so we can stop compilation after parsing.\n    control.after_parse.stop = Compilation::Stop;\n\n    // And when we stop after parsing we'll call this closure.\n    // Note that this will give us an AST before macro expansions, which is\n    // not usually what you want.\n    control.after_parse.callback = box |state| {\n        // Which extracts information about the compiled crate...\n        let krate = state.krate.unwrap();\n\n        // ...and walks the AST, collecting stats.\n        let mut visitor = StupidVisitor::new();\n        visit::walk_crate(\u0026mut visitor, krate);\n\n        // And finally prints out the stupid stats that we collected.\n        let cratename = match attr::find_crate_name(\u0026krate.attrs[]) {\n            Some(name) =\u003e name.to_string(),\n            None =\u003e String::from_str(\"unknown_crate\"),\n        };\n        println!(\"In crate: {},\\n\", cratename);\n        println!(\"Found {} uses of `println!`;\", visitor.println_count);\n\n        let (common, common_percent, four_percent) = visitor.compute_arg_stats();\n        println!(\"The most common number of arguments is {} ({:.0}% of all functions);\",\n                 common, common_percent);\n        println!(\"{:.0}% of functions have four or more arguments.\", four_percent);\n    };\n\n    control\n}\n```\n\nThat is all it takes to create your own drop-in compiler replacement or custom\ncompiler! For the sake of completeness I'll go over the rest of the stupid-stats\ntool.\n\n```\nstruct StupidVisitor {\n    println_count: usize,\n    arg_counts: Vec\u003cusize\u003e,\n}\n```\n\nThe `StupidVisitor` struct just keeps track of the number of `println!`s it has\nseen and the count for each number of arguments. It implements\n`syntax::visit::Visitor` to walk the AST. Mostly we just use the default\nmethods, these walk the AST taking no action. We override `visit_item` and\n`visit_mac` to implement custom behaviour when we walk into items (items include\nfunctions, modules, traits, structs, and so forth, we're only interested in\nfunctions) and macros:\n\n```\nimpl\u003c'v\u003e visit::Visitor\u003c'v\u003e for StupidVisitor {\n    fn visit_item(\u0026mut self, i: \u0026'v ast::Item) {\n        match i.node {\n            ast::Item_::ItemFn(ref decl, _, _, _, _) =\u003e {\n                // Record the number of args.\n                self.increment_args(decl.inputs.len());\n            }\n            _ =\u003e {}\n        }\n\n        // Keep walking.\n        visit::walk_item(self, i)\n    }\n\n    fn visit_mac(\u0026mut self, mac: \u0026'v ast::Mac) {\n        // Find its name and check if it is \"println\".\n        let ast::Mac_::MacInvocTT(ref path, _, _) = mac.node;\n        if path_to_string(path) == \"println\" {\n            self.println_count += 1;\n        }\n\n        // Keep walking.\n        visit::walk_mac(self, mac)\n    }\n}\n```\n\nThe `increment_args` method increments the correct count in\n`StupidVisitor::arg_counts`. After we're done walking, `compute_arg_stats` does\nsome pretty basic maths to come up with the stats we want about arguments.\n\n\n## What next?\n\nThese APIs are pretty new and have a long way to go until they're really good.\nIf there are improvements you'd like to see or things you'd like to be able to\ndo, let me know in a comment or [GitHub issue](https://github.com/rust-lang/rust/issues).\nIn particular, it's not clear to me exactly what extra flexibility is required.\nIf you have an existing tool that would be suited to this setup, please try it\nout and let me know if you have problems.\n\nIt'd be great to see Rustdoc converted to using these APIs, if that is possible\n(although long term, I'd prefer to see Rustdoc run on the output from save-\nanalysis, rather than doing its own analysis). Other parts of the compiler\n(e.g., pretty printing, testing) could be refactored to use these APIs\ninternally (I already changed save-analysis to use `CompilerController`). I've\nbeen experimenting with a prototype rustfmt which also uses these APIs.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnrc%2Fstupid-stats","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnrc%2Fstupid-stats","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnrc%2Fstupid-stats/lists"}