{"id":16005985,"url":"https://github.com/dsanders11/chromium-include-cleanup","last_synced_at":"2025-03-17T20:30:46.718Z","repository":{"id":74467083,"uuid":"453598852","full_name":"dsanders11/chromium-include-cleanup","owner":"dsanders11","description":"Scripts to help guide cleanup of #include lines in the Chromium codebase","archived":false,"fork":false,"pushed_at":"2025-02-28T09:45:15.000Z","size":4798,"stargazers_count":3,"open_issues_count":4,"forks_count":2,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-03-16T14:11:33.791Z","etag":null,"topics":["chromium","clangd","cpp","scripts"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dsanders11.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-01-30T05:47:52.000Z","updated_at":"2025-02-28T09:45:19.000Z","dependencies_parsed_at":"2025-02-07T03:31:34.710Z","dependency_job_id":null,"html_url":"https://github.com/dsanders11/chromium-include-cleanup","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dsanders11%2Fchromium-include-cleanup","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dsanders11%2Fchromium-include-cleanup/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dsanders11%2Fchromium-include-cleanup/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dsanders11%2Fchromium-include-cleanup/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dsanders11","download_url":"https://codeload.github.com/dsanders11/chromium-include-cleanup/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244105515,"owners_count":20398870,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["chromium","clangd","cpp","scripts"],"created_at":"2024-10-08T11:22:42.823Z","updated_at":"2025-03-17T20:30:46.703Z","avatar_url":"https://github.com/dsanders11.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# chromium-include-cleanup\n\nScripts to help guide cleanup of #include lines in a codebase, using `clangd`\n\n## Scripts\n\n* `apply_include_changes.py` - Apply include changes to files in the source\n  tree\n* `filter_include_changes.py` - Filter include changes output\n* `list_includers.py` - List includers of a file\n* `list_transitive_includes.py` - List transitive (and direct) includes of a file\n* `post_process_compilation_db.py` - Post-process the clang compilation\n  database for analysis\n* `set_edge_weights.py` - Set edge weights in include changes output\n* `suggest_include_changes.py` - Suggests includes to add and remove\n\n## Prerequisites\n\nTo use these scripts, you'll need:\n\n* A [release of `clangd`][clangd-releases] which has \"IncludeCleaner\" with\n  support for missing includes (17.0.0+)\n* The full output of `//tools/clang/scripts/analyze_includes.py`, see\n  [discussion on the mailing list][include-analysis] for how to generate it\n* A compilation database for `clangd` to use, which can be generated with\n  `gn gen . --export-compile-commands` in the Chromium output directory\n  * The generated `compile_commands.json` should be post-processed with\n    the `post_process_compilation_db.py` script for best results\n\n### Install Dependencies\n\n```shell\n$ pip install -r ~/chromium-include-cleanup/requirements.txt\n```\n\n### Patching `clangd`\n\nTo get suggestions for includes to add, and other tweaks, `clangd` needs to be\npatched with the patches in `clangd_patches` and built from source.\n\n## `clangd` Configuration\n\nYou need to enable `MissingIncludes` and `UnusedIncludes` diagnostics in a\n`clangd` [config file][clangd-config]:\n\n```yaml\nDiagnostics:\n  MissingIncludes: Strict\n  UnusedIncludes: Strict\n```\n\n## Finding Unused Includes\n\nThese instructions assume you've already built and processed the build\nlog with `//tools/clang/scripts/analyze_includes.py`, if you haven't, see the link above under\n\"Prerequisites\". It assumes the output is at `~/include-analysis.js`, so\nadjust to taste.\n\nThis also assumes you have `clangd` on your `$PATH`.\n\n```shell\n$ cd ~/chromium/src/out/Default\n$ gn gen . --export-compile-commands\n$ python3 ~/chromium-include-cleanup/post_process_compilation_db.py compile_commands.json \u003e compile_commands-fixed.json\n$ mv compile_commands-fixed.json compile_commands.json\n$ cd ../../\n$ python3 ~/chromium-include-cleanup/suggest_include_changes.py --compile-commands-dir=out/Default ~/include-analysis.js \u003e ~/unused-edges.csv\n$ python3 ~/chromium-include-cleanup/set_edge_weights.py ~/unused-edges.csv ~/include-analysis.js --config ~/chromium-include-cleanup/configs/chromium.json \u003e ~/weighted-unused-edges.csv\n```\n\nAnother useful option is `--filename-filter=^base/`, which lets you filter the\nfiles which will be analyzed, which can speed things up considerably if it is\nlimited to a subset of the codebase.\n\nEdge weights are set in a separate script to allow quick iteration, since\n`suggest_include_changes.py` takes many hours to run. The default metric\nfor edge weights pulls the \"Added Size\" metric from the include analysis\noutput. This means new weights can be easily be applied to the output of\n`suggest_include_changes.py` by downloading the latest hosted include\nanalysis output at \u003chttps://commondatastorage.googleapis.com/chromium-browser-clang/include-analysis.js\u003e,\nbut mileage may vary since you're combining output from your local build\nand the hosted build.\n\n## Performance\n\nFor a full codebase run of the `suggest_include_changes.py` script on Ubuntu,\nit takes 7 hours on a 4 core, 8 thread machine. `clangd` is highly parallel\nthough, and the script is configured to use all available logical CPUs, so it\nwill scale well on beefier machines.\n\n## Current Limitations\n\nCurrently the `suggest_include_changes.py` script has problems with suggesting\nincludes to remove when the filename in the `#include` line does not match the\nfilename in the include analysis output, which could happen for includes\ninside third-party code which is including relative to itself, not the source\nroot.\n\nWhen suggesting includes to add, `clangd` will sometimes suggest headers which\nare internal to the standard library, like `\u003c__hash_table\u003e`, rather than the\npublic header. Unfortunately these cases can't be disambiguated by this script,\nsince there's not enough information to work off of.\n\n## Accuracy of Output\n\nThese scripts rely on `clangd` and specifically the \"IncludeCleaner\" feature\nto determine which includes are unused, and which headers need to be added.\nWith the Chromium codebase, there are many places where `clangd` will return\nfalse positives, suggesting that an include is not used when it actually is.\nAs such, the output is more of a guide than something which can be used as-is\nin an automated situation.\n\nKnown situations in Chromium where `clangd` will produce false positives:\n\n* When an include is only used for a `friend class` declaration\n* When the code using an include is inside an `#ifdef` not used on the system\n  which built the codebase\n* Macros in general are often a struggle point\n* Umbrella headers\n* Certain forward declarations seem to be flagged incorrectly as the canonical\n  location for a symbol, such as \"base/callback_forward.h\"\n* Forward declarations in the file being analyzed\n  * `clangd` won't consider an include unused even if forward declarations\n    exist which make it unnecessary\n  * `clangd` will still suggest an include even if a forward declaration makes it\n    unnecessary\n\n[clangd-releases]: https://github.com/clangd/clangd/releases\n[include-analysis]: https://groups.google.com/a/chromium.org/g/chromium-dev/c/0ZME4DuE06k\n[clangd-config]: https://clangd.llvm.org/config#files\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdsanders11%2Fchromium-include-cleanup","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdsanders11%2Fchromium-include-cleanup","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdsanders11%2Fchromium-include-cleanup/lists"}