{"id":15144342,"url":"https://github.com/exafunction/codeium-parse","last_synced_at":"2025-09-12T04:41:54.198Z","repository":{"id":133099540,"uuid":"611466291","full_name":"Exafunction/codeium-parse","owner":"Exafunction","description":"A command line tool for parsing code syntax","archived":false,"fork":false,"pushed_at":"2024-10-10T21:20:35.000Z","size":73,"stargazers_count":132,"open_issues_count":2,"forks_count":7,"subscribers_count":9,"default_branch":"main","last_synced_at":"2025-07-23T02:44:38.773Z","etag":null,"topics":["command-line","command-line-interface","command-line-tool","syntax-tree","tree-sitter"],"latest_commit_sha":null,"homepage":"","language":"Scheme","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Exafunction.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-03-08T22:08:57.000Z","updated_at":"2025-06-25T01:57:43.000Z","dependencies_parsed_at":null,"dependency_job_id":"cc84bca5-b558-44ef-9c71-2709b0e3864d","html_url":"https://github.com/Exafunction/codeium-parse","commit_stats":{"total_commits":27,"total_committers":6,"mean_commits":4.5,"dds":0.2222222222222222,"last_synced_commit":"5302215fe2ed239b425d3e2a66058f563ab23972"},"previous_names":[],"tags_count":16,"template":false,"template_full_name":null,"purl":"pkg:github/Exafunction/codeium-parse","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Exafunction%2Fcodeium-parse","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Exafunction%2Fcodeium-parse/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Exafunction%2Fcodeium-parse/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Exafunction%2Fcodeium-parse/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Exafunction","download_url":"https://codeload.github.com/Exafunction/codeium-parse/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Exafunction%2Fcodeium-parse/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":274754500,"owners_count":25343035,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-12T02:00:09.324Z","response_time":60,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["command-line","command-line-interface","command-line-tool","syntax-tree","tree-sitter"],"created_at":"2024-09-26T10:40:42.323Z","updated_at":"2025-09-12T04:41:54.157Z","avatar_url":"https://github.com/Exafunction.png","language":"Scheme","readme":"\u003cp align=\"center\"\u003e\n  \u003cimg width=\"300\" alt=\"Codeium\" src=\"codeium.svg\"/\u003e\n\u003c/p\u003e\n\n---\n\n[![Discord](https://img.shields.io/discord/1027685395649015980?label=community\u0026color=5865F2\u0026logo=discord\u0026logoColor=FFFFFF)](https://discord.gg/3XFf78nAx5)\n[![Twitter Follow](https://img.shields.io/badge/style--blue?style=social\u0026logo=twitter\u0026label=Follow%20%40codeiumdev)](https://twitter.com/intent/follow?screen_name=codeiumdev)\n![License](https://img.shields.io/github/license/Exafunction/codeium-parse)\n[![Docs](https://img.shields.io/badge/Codeium%20Docs-09B6A2)](https://docs.codeium.com)\n[![Canny Board](https://img.shields.io/badge/Feature%20Requests-6b69ff)](https://codeium.canny.io/feature-requests/)\n[![built with Codeium](https://codeium.com/badges/main)](https://codeium.com?repo_name=exafunction%2Fcodeium-parse)\n\n[![Visual Studio](https://img.shields.io/visual-studio-marketplace/i/Codeium.codeium?label=Visual%20Studio\u0026logo=visualstudio)](https://marketplace.visualstudio.com/items?itemName=Codeium.codeium)\n[![JetBrains](https://img.shields.io/jetbrains/plugin/d/20540?label=JetBrains)](https://plugins.jetbrains.com/plugin/20540-codeium/)\n[![Open VSX](https://img.shields.io/open-vsx/dt/Codeium/codeium?label=Open%20VSX)](https://open-vsx.org/extension/Codeium/codeium)\n[![Google Chrome](https://img.shields.io/chrome-web-store/users/hobjkcpmjhlegmobgonaagepfckjkceh?label=Google%20Chrome\u0026logo=googlechrome\u0026logoColor=FFFFFF)](https://chrome.google.com/webstore/detail/codeium/hobjkcpmjhlegmobgonaagepfckjkceh)\n\n# codeium-parse\n\n## _A command line tool for parsing code syntax_\n\nThis repository contains a binary built with [tree-sitter](https://github.com/tree-sitter/tree-sitter) that lets you:\n* Inspect the concrete syntax tree of a source file\n* Use pre-written tree-sitter query files to locate important symbols in source code\n* Format output in JSON to use the results in your own applications\n\nIn particular, this repo provides a binary prepackaged with:\n* A recent version of the tree-sitter library\n* A large number of tree-sitter grammars\n* An implementation of many common query predicates\n\nContributions are welcome and we encourage using this tool for any applications that involve code syntax analysis. For example, these queries are used by [Codeium Search](https://www.codeium.com/about_codeium_search) to index code locally for repo-wide semantic search. If you use Codeium Search, adding queries for your language here will enable it to work better on your own code!\n\n## Example\n\n(Requires [fd](https://github.com/sharkdp/fd) and [jq](https://github.com/stedolan/jq).)\n\n```shell\n# Print all names and arguments from function definitions.\nfd -e js \\\n  | xargs -i ./parse -quiet -use_tags_query -json -json_include_path -file '{}' \\\n  | jq -r '.\n    | select(.captures.\"definition.function\" != null)\n    | .file + \":\" + .captures.name[0].text + .captures.\"codeium.parameters\"[0].text'\n# Output:\n# examples/example.js:add(a, b)\n```\n\n## Getting started\n\n```console\n$ ./download_parse.sh\n$ ./parse -file examples/example.js -named_only\nprogram [0, 0] - [4, 0] \"// Adds two numbers.\\n…\"\n  comment [0, 0] - [0, 20] \"// Adds two numbers.\"\n  function_declaration [1, 0] - [3, 1] \"function add(a, b) {\\n…\"\n    name: identifier [1, 9] - [1, 12] \"add\"\n    parameters: formal_parameters [1, 12] - [1, 18] \"(a, b)\"\n      identifier [1, 13] - [1, 14] \"a\"\n      identifier [1, 16] - [1, 17] \"b\"\n    body: statement_block [1, 19] - [3, 1] \"{\\n…\"\n      return_statement [2, 4] - [2, 17] \"return a + b;\"\n        binary_expression [2, 11] - [2, 16] \"a + b\"\n          left: identifier [2, 11] - [2, 12] \"a\"\n          right: identifier [2, 15] - [2, 16] \"b\"\n$ ./parse -file examples/example.js -use_tags_query -json | jq \".captures.doc[0].text\"\n\"// Adds two numbers.\"\n```\n\n## Support status\n\n### Queries\n\nQueries try to follow the [conventions established by tree-sitter.](https://tree-sitter.github.io/tree-sitter/code-navigation-systems)\n\nMost captures also include documentation as `@doc`. `@definition.function` and `@definition.method` also capture `@codeium.parameters`.\n\n| Top-level capture         | Python | TypeScript | JavaScript | Go  | Java | C++   | PHP | Ruby | C#  | Perl  | Kotlin | Dart  | Bash | C   |\n| ------------------------- | ------ | ---------- | ---------- | --- | ---- | ----- | --- | ---- | --- | ----- | ------ | ----- | ---- | --- |\n| `@definition.class`       | ✓      | ✓          | ✓          | ✓   | ✓    | ✓     | ✓   | ✓    | ✓   | ✓     | ✓      | ✓     | ✗    | ✗   |\n| `@definition.function`    | ✓      | ✓[^3]      | ✓          | ✓   | N/A  | ✓     | ✓   | N/A  | N/A | ✓     | ✓      | ✓     | ✓    | ✓   |\n| `@definition.method`      | ✓[^1]  | ✓[^3]      | ✓          | ✓   | ✓    | ✓[^1] | ✓   | ✓    | ✓   | ✓[^1] | ✓      | ✓[^1] | ✓    | ✓   |\n| `@definition.constructor` | ✓      | ✓          | ✓          | N/A | ✗    | ✗     | ✗   | ✗    | ✓   | ✗     | ✓      | ✗     | N/A  | N/A |\n| `@definition.interface`   | N/A    | ✓          | N/A        | ✓   | ✓    | N/A   | ✓   | ✗    | ✓   | N/A   | ✗      | ✗     | N/A  | N/A |\n| `@definition.namespace`   | N/A    | ✓          | N/A        | N/A | N/A  | ✓     | ✓   | N/A  | ✓   | ✗     | ✗      | N/A   | N/A  | N/A |\n| `@definition.module`      | N/A    | ✓          | N/A        | N/A | N/A  | ✗     | N/A | ✓    | N/A | N/A   | N/A    | ✗     | N/A  | N/A |\n| `@definition.type`        | N/A    | ✓          | N/A        | ✓   | N/A  | ✗     | ✗   | N/A  | N/A | N/A   | N/A    | ✗     | N/A  | N/A |\n| `@definition.constant`    | ✗      | ✗          | ✗          | ✗   | ✗    | ✗     | ✗   | ✗    | ✗   | ✗     | ✗      | ✗     | N/A  | ✗   |\n| `@definition.enum`        | ✗      | ✗          | ✗          | ✗   | ✗    | ✗     | ✗   | N/A  | ✓   | N/A   | ✗      | ✗     | N/A  | ✗   |\n| `@definition.import`      | ✓      | ✓          | ✓          | ✗   | ✗    | ✗     | N/A | ✓    | ✗   | ✓     | ✓      | ✗     | N/A  | ✓   |\n| `@definition.include`     | N/A    | N/A        | N/A        | N/A | N/A  | ✗     | ✗   | N/A  | N/A | N/A   | N/A    | N/A   | N/A  | N/A |\n| `@definition.package`     | N/A    | N/A        | N/A        | ✓   | ✓    | N/A   | N/A | N/A  | N/A | N/A   | N/A    | N/A   | N/A  | N/A |\n| `@reference.call`         | ✓      | ✓          | ✓          | ✓   | ✗    | ✗     | ✗   | ✓    | ✗   | ✗     | ✗      | ✗     | ✗    | ✗   |\n| `@reference.class`        | ✓[^2]  | ✓          | ✓          | ✓   | ✗    | ✗     | ✗   | ✗    | ✗   | ✗     | ✗      | ✗     | N/A  | N/A |\n\n| Language | Supported injections   |\n| -------- | ---------------------- |\n| Vue      | JavaScript, TypeScript |\n| HTML     | JavaScript             |\n\n[^1]: Currently functions and methods are not distinguished.\n[^2]: Function calls and class instantiation are indistinguishable in Python.\n[^3]: Function and method signatures are captured individually in TypeScript. Therefore, the `@doc` capture may not exist on all nodes.\n\nWant to write a query for a new language? `tags.scm` and other queries in each language's tree-sitter repository, [like tree-sitter-javascript](https://github.com/tree-sitter/tree-sitter-javascript/blob/5720b249490b3c17245ba772f6be4a43edb4e3b7/queries/tags.scm), are a good place to start.\n\n### Query predicates\n\n```console\n$ ./parse -supported_predicates\n#eq?/#not-eq?\n    (#eq? \u003c@capture|\"literal\"\u003e \u003c@capture|\"literal\"\u003e)\n    Checks if two values are equal.\n\n#has-parent?/#not-has-parent?\n    (#has-parent? @capture node_type...)\n    Checks if @capture has a parent node of any of the given types.\n\n#has-type?/#not-has-type?\n    (#has-type? @capture node_type...)\n    Checks if @capture has a node of any of the given types.\n\n#lineage-from-name!\n    (#lineage-from-name! \"literal\")\n    If the name captures scopes, split by \"literal\" and retain the last element\n    as the name. The other elements are appended to the lineage.\n\n#match?/#not-match?\n    (#match? @capture \"regex\")\n    Checks if the text for @capture matches the given regular expression.\n\n#select-adjacent!\n    (#select-adjacent! @capture @anchor)\n    Selects @capture nodes contiguous with @anchor (all starting and ending on\n    adjacent lines).\n\n#set!\n    (#set! key \u003c@capture|\"literal\"\u003e)\n    Store metadata as a side effect of a match.\n\n#strip!\n    (#strip! @capture \"regex\")\n    Removes all matching text from all @capture nodes.\n```\n\nNeed a predicate which hasn't been implemented? [File an issue!](https://github.com/Exafunction/codeium-parse/issues/new) We try to use [predicates from nvim-treesitter.](https://github.com/nvim-treesitter/nvim-treesitter/blob/980f0816cc28c20e45715687a0a21b5b39af59eb/lua/nvim-treesitter/query_predicates.lua)\n\n### Grammars\n\n```console\n$ ./parse -supported_languages\nada\nc\ncpp\ncsharp\ncss\ndart\ngo\nhcl\nhtml\njava\njavascript\njson\njulia\nkotlin\nlatex\nmarkdown\nocaml\nocaml_interface\nperl\nphp\nprotobuf\npython\nruby\nrust\nshell\nsvelte\nswift\ntoml\ntree_sitter_query\ntsx\ntypescript\nvue\nyaml\n```\n\nLooking for support for another language? [File an issue](https://github.com/Exafunction/codeium-parse/issues/new) with a link to the repo that contains the grammar.\n\n## Contributing\n\nPull requests are welcome. For non-issue discussions about `codeium-parse`, [join\nour Discord.](https://discord.gg/3XFf78nAx5)\n\n### Adding and testing queries\n\n* You can create new source files with patterns you want to target in `test_files/`.\n* Look at the syntax tree using `./parse -file test_files/\u003cyour file\u003e` to get a sense of how to capture the pattern.\n* Learn the query syntax from [tree-sitter documentation.](https://tree-sitter.github.io/tree-sitter/using-parsers#pattern-matching-with-queries)\n* Run `./goldens.sh` to see what your query captures.\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fexafunction%2Fcodeium-parse","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fexafunction%2Fcodeium-parse","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fexafunction%2Fcodeium-parse/lists"}