{"id":13823136,"url":"https://github.com/WolframResearch/codeparser","last_synced_at":"2025-07-08T16:31:05.507Z","repository":{"id":38883564,"uuid":"254169948","full_name":"WolframResearch/codeparser","owner":"WolframResearch","description":"Parse Wolfram Language source code as abstract syntax trees (ASTs) or concrete syntax trees (CSTs)","archived":false,"fork":false,"pushed_at":"2024-07-14T00:13:51.000Z","size":7183,"stargazers_count":120,"open_issues_count":3,"forks_count":14,"subscribers_count":19,"default_branch":"master","last_synced_at":"2024-08-04T09:01:05.104Z","etag":null,"topics":["abstract-syntax-tree","ast","asts","codeparser","codetools","concrete-syntax-trees","cst","csts","mathematica","parsing","pratt","pratt-parser","syntax-trees","wolfram-language","wolfram-mathematica"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/WolframResearch.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-04-08T18:34:11.000Z","updated_at":"2024-08-04T09:01:08.369Z","dependencies_parsed_at":"2024-01-18T04:52:54.498Z","dependency_job_id":"09c5b959-fca7-42b7-92b7-84dbd820d5bd","html_url":"https://github.com/WolframResearch/codeparser","commit_stats":null,"previous_names":[],"tags_count":10,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/WolframResearch%2Fcodeparser","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/WolframResearch%2Fcodeparser/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/WolframResearch%2Fcodeparser/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/WolframResearch%2Fcodeparser/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/WolframResearch","download_url":"https://codeload.github.com/WolframResearch/codeparser/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":225449301,"owners_count":17476069,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["abstract-syntax-tree","ast","asts","codeparser","codetools","concrete-syntax-trees","cst","csts","mathematica","parsing","pratt","pratt-parser","syntax-trees","wolfram-language","wolfram-mathematica"],"created_at":"2024-08-04T09:00:22.288Z","updated_at":"2024-11-20T00:30:52.815Z","avatar_url":"https://github.com/WolframResearch.png","language":"Rust","funding_links":[],"categories":["Code Parsing"],"sub_categories":[],"readme":"# CodeParser\n\nCodeParser is a package for parsing Wolfram Language source code as abstract syntax trees (ASTs) or concrete syntax trees (CSTs).\nCodeParser is useful for inspecting code, formatting code, and instrumenting code (for e.g., coverage reporting or profiling), and much more!\n\nCodeParser has many key features:\n* Understands practically entire Wolfram Language syntax.\n* Fast native library implementation.\n* Tested with combination of suite of hand-written tests and fuzz testing.\n* Gracious error handling and recovery\n\n\n```\nNeeds[\"CodeParser`\"]\n\nCodeParse[\"1+1\"]\n```\n```\nOut[2]= ContainerNode[String, {CallNode[LeafNode[Symbol, \"Plus\", \u003c||\u003e], {LeafNode[Integer, \"1\", \u003c|Source -\u003e {{1, 1}, {1, 2}}|\u003e], LeafNode[Integer, \"1\", \u003c|Source -\u003e {{1, 3}, {1, 4}}|\u003e]}, \u003c|Source -\u003e {{1, 1}, {1, 4}}|\u003e]}, \u003c||\u003e]\n```\n\n[\"CodeParser and CodeInspector\" on community.wolfram.com](https://community.wolfram.com/groups/-/m/t/1931315)\n\n[Parsing the Wolfram Language from WTC 2019: Watch Video (youtube)](https://www.youtube.com/watch?v=rOa5IntICFA)\n\n[Parsing the Wolfram Language from WTC 2019: Watch Video (wolfram.com)](https://www.wolfram.com/broadcast/video.php?v=2908)\n\n[Parsing the Wolfram Language from WTC 2019: Download Presentation](https://files.wolframcdn.com/pub/www.wolfram.com/technology-conference/2019/Thursday/2019BrentonBostickParsingTheWL.nb)\n\n\n## Setup\n\nCodeParser is included in Mathematica 12.2 and above.\n\nFor older versions, install CodeParser paclet from the public paclet server:\n```\nPacletInstall[\"CodeParser\"]\n```\n\n[Build and install the CodeParser paclet locally](HowToBuild.md)\n\n\n## Using CodeParser\n\nAfter CodeParser is installed, it can be used.\n\n```\nNeeds[\"CodeParser`\"]\n\nCodeParse[\"1+1\"]\n```\n```\nOut[2]= ContainerNode[String, {CallNode[LeafNode[Symbol, \"Plus\", \u003c||\u003e], {LeafNode[Integer, \"1\", \u003c|Source -\u003e {{1, 1}, {1, 2}}|\u003e], LeafNode[Integer, \"1\", \u003c|Source -\u003e {{1, 3}, {1, 4}}|\u003e]}, \u003c|Source -\u003e {{1, 1}, {1, 4}}|\u003e]}, \u003c||\u003e]\n```\n\nThe input to `CodeParse` may be a string, a `File`, or a list of bytes.\n\n\n### Command-line tool (Optional)\n\nAn optional `codeparser` command-line tool is also built and can be used.\n\n```\ncmake -DBUILD_EXE=ON ..\ncmake --build . --target codeparser-exe\n\n$cpp/src/exe/codeparser\n\u003e\u003e\u003e 1+1\nInfixNode[Plus, {LeafNode[Integer, \"1\", \u003c|Source-\u003e{{1, 2}, {1, 2}}|\u003e], LeafNode[Integer, 1, \u003c|Source-\u003e{{1, 3}, {1, 4}}|\u003e]}, \u003c|Source-\u003e{{1, 1}, {1, 4}}|\u003e]\n\n\u003e\u003e\u003e\n```\n\n\n## Troubleshooting\n\nMake sure that the CodeParser can be found on your system:\n```\nNeeds[\"CodeParser`\"]\n```\n\nand try a basic example:\n```\nCodeParse[\"1+1\"]\n```\n\nYou may get `LibraryFunction` messages:\n```\nIn[1]:= Needs[\"CodeParser`\"]\n\nIn[2]:= CodeParse[\"1+1\"]\n\nDuring evaluation of In[2]:= LibraryFunction::version: The version number 7 of the library is not consistent with the current or any previous WolframLibraryVersion.\n\nDuring evaluation of In[2]:= LibraryFunction::initerr: A nonzero error code 7 was returned during the initialization of the library /Users/user/Library/Mathematica/Paclets/Repository/CodeParser-1.6/LibraryResources/MacOSX-x86-64/CodeParser.dylib.\n\nDuring evaluation of In[2]:= LibraryFunction::libload: The function ConcreteParseBytes_Listable_LibraryLink was not loaded from the file /Users/user/Library/Mathematica/Paclets/Repository/CodeParser-1.6/LibraryResources/MacOSX-x86-64/CodeParser.dylib.\n\nOut[2]= $Failed\n```\n\nThis means that CodeParser was built with a newer version of Wolfram System than your system supports.\n\nTo fix this, build CodeParser from source with the version of Wolfram System that you will use.\n\n## Benchmarks\n\n\u003e Some of the benchmarks test large data files. Those files are tracked in this\n\u003e repository to ensure that benchmarks are always run against identical input.\n\u003e [Git LFS](https://git-lfs.github.com/) is used to ensure that a basic checkout\n\u003e of this repository remains small, which is important in CI/CD builds.\n\nTo run the benchmarks, first ensure that the large benchmark files have been\nchecked out locally using:\n\n```shell\n$ git lfs pull --exclude=\"\" --include=\"*\"\n```\n\nThis will override the default settings in [`.lfsconfig`](./.lfsconfig).\n\nThen, to begin running the benchmarks, execute:\n\n```shell\n$ cargo bench\n```\n\n## File Overview\n\n* [Tests/files/large/](./Tests/files/large/) contains files\n  managed by [`Git LFS`](https://git-lfs.github.com/). The files in this\n  directory are used by the benchmarks. These files should never be modified, to\n  ensure that benchmark comparisions between different revisions of this\n  repository can be meaningfully compared.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FWolframResearch%2Fcodeparser","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FWolframResearch%2Fcodeparser","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FWolframResearch%2Fcodeparser/lists"}