{"id":15134900,"url":"https://github.com/jetbrains/grammar-kit","last_synced_at":"2025-05-15T03:04:43.269Z","repository":{"id":41398038,"uuid":"2154376","full_name":"JetBrains/Grammar-Kit","owner":"JetBrains","description":"Grammar files support \u0026 parser/PSI generation for IntelliJ IDEA","archived":false,"fork":false,"pushed_at":"2025-01-29T15:36:13.000Z","size":70218,"stargazers_count":735,"open_issues_count":57,"forks_count":132,"subscribers_count":43,"default_branch":"master","last_synced_at":"2025-04-14T01:57:38.220Z","etag":null,"topics":["grammar","intellij","lexer","parser"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/JetBrains.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2011-08-04T12:28:11.000Z","updated_at":"2025-04-05T15:43:27.000Z","dependencies_parsed_at":"2024-01-14T19:12:22.902Z","dependency_job_id":"98adfe6a-71ee-451d-9e63-438f29674874","html_url":"https://github.com/JetBrains/Grammar-Kit","commit_stats":{"total_commits":1193,"total_committers":19,"mean_commits":62.78947368421053,"dds":0.09136630343671415,"last_synced_commit":"9b897c18db366400381a7eb56f7a0ef9b6bcd2f0"},"previous_names":[],"tags_count":42,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JetBrains%2FGrammar-Kit","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JetBrains%2FGrammar-Kit/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JetBrains%2FGrammar-Kit/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JetBrains%2FGrammar-Kit/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/JetBrains","download_url":"https://codeload.github.com/JetBrains/Grammar-Kit/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248809032,"owners_count":21164895,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["grammar","intellij","lexer","parser"],"created_at":"2024-09-26T05:40:31.443Z","updated_at":"2025-04-14T01:57:51.548Z","avatar_url":"https://github.com/JetBrains.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"\nGrammar-Kit\n===========\n[![official project](http://jb.gg/badges/official.svg)](https://confluence.jetbrains.com/display/ALL/JetBrains+on+GitHub)\n[![Build Status](https://teamcity.jetbrains.com/app/rest/builds/buildType:(id:IntellijIdeaPlugins_GrammarKit_Build)/statusIcon.svg?guest=1)](https://teamcity.jetbrains.com/viewType.html?buildTypeId=IntellijIdeaPlugins_GrammarKit_Build\u0026guest=1)\n[![GitHub license](https://img.shields.io/badge/license-Apache%20License%202.0-blue.svg?style=flat)](http://www.apache.org/licenses/LICENSE-2.0)\n[![X Follow](https://img.shields.io/badge/follow-%40JBPlatform-1DA1F2?logo=x)][jb:x]\n[![Slack](https://img.shields.io/badge/Slack-%23intellij--platform-blue?style=flat-square\u0026logo=Slack)][jb:slack]\n\nAn [IntelliJ IDEA plugin](http://plugins.jetbrains.com/plugin/6606) for language plugin developers.\n\nAdds BNF Grammars and JFlex file editing support, and a parser/PSI code generator.\n\nQuick links: [Latest dev build](https://teamcity.jetbrains.com/guestAuth/app/rest/builds/buildType:IntellijIdeaPlugins_GrammarKit_Build,status:SUCCESS/artifacts/content/GrammarKit*.zip),\n[Changelog](CHANGELOG.md), [Tutorial](TUTORIAL.md), [How-to](HOWTO.md)\n\n\u003e [!IMPORTANT]\n\u003e\n\u003e Since 2022.3, Grammar-Kit plugin requires Java 17.\n\nOpen-source plugins built with Grammar-Kit:\n\n* [Clojure-Kit](https://github.com/gregsh/Clojure-Kit), \n  [intellij-rust](https://github.com/intellij-rust/intellij-rust),\n  [intellij-erlang](https://github.com/ignatov/intellij-erlang),\n  [intellij-elm](https://github.com/intellij-elm/intellij-elm)\n* [intellij-elixir](https://github.com/KronicDeth/intellij-elixir),\n  [Perl5-IDEA](https://github.com/Camelcade/Perl5-IDEA),\n  [Dart](https://github.com/JetBrains/intellij-plugins/tree/master/Dart), \n  [intellij-haxe](https://github.com/HaxeFoundation/intellij-haxe),\n  [Cypher](https://github.com/neueda/jetbrains-plugin-graph-database-support)\n\nSee also [Custom Language Support Tutorial](https://plugins.jetbrains.com/docs/intellij/custom-language-support-tutorial.html).\n\nGeneral usage instructions\n--------------------------\n1. Create grammar \\*.bnf file, see [Grammar.bnf](grammars/Grammar.bnf) in the plugin code.\n2. Tune the grammar using _Live Preview_ + Structure view (Ctrl-Alt-P / Cmd-Alt-P)\n3. Generate parser/ElementTypes/PSI classes (Ctrl-Shift-G / Cmd-Shift-G)\n4. Generate lexer \\*.flex file and then run JFlex generator (both via context menu) \n5. Implement ParserDefinition and add the corresponding registrations to the plugin.xml\n6. Mix-in resolve and other non-trivial functionality to PSI\n\nUsing with Gradle\n-----------------\n\nInvoking the parser generator from an IDE as described above is the preferred way.\u003cbr/\u003e\nOtherwise use [gradle-grammar-kit-plugin](https://github.com/JetBrains/gradle-grammar-kit-plugin) if the following limitations are not critical:\n\n* Method mixins are not supported (two-pass generation is not implemented)\n* Generic signatures and annotations may not be correct\n\n\nPlugin features\n===============\n\n![Editor support](images/editor.png)\n\n* Refactoring: extract rule (Ctrl-Alt-M/Cmd-Alt-M)\n* Refactoring: introduce token (Ctrl-Alt-C/Cmd-Alt-C)\n* Editing: flip _choice_ branches intention (via Alt-Enter)\n* Editing: Unwrap/remove expression (Ctrl-Shift-Del/Cmd-Shift-Del)\n* Navigation: quick grammar and flex file structure popup (Ctrl-F12/Cmd-F12)\n* Navigation: go to related file (parser and PSI) (Ctrl-Alt-Home/Cmd-Alt-Home)\n* Navigation: navigate to matched expressions (Ctrl-B/Cmd-B inside attribute pattern)\n* Highlighting: customizable colors (via Settings/Colors and Fonts)\n* Highlighting: pinned expression markers (tooltip shows pin value in charge)\n* Highlighting: a number of inspections, the list is available in Settings/Inspections\n* Documentation: rule documentation popup shows FIRST/FOLLOWS/PSI content (Ctrl-Q/Cmd-J)\n* Documentation: attribute documentation popup (Ctrl-Q/Cmd-J)\n* [Live preview](TUTORIAL.md): open language live preview editor (Ctrl-Alt-P/Cmd-Alt-P)\n* [Live preview](TUTORIAL.md): start/stop grammar evaluator highlighting (Ctrl-Alt-F7/Cmd-Alt-F7 in preview editor)\n* Generator: generate parser/PSI code (Ctrl-Shift-G/Cmd-Shift-G)\n* Generator: generate custom _parserUtilClass_ class\n* Generator: generate \\*.flex - JFlex lexer definition\n* Generator: run JFlex generator on a \\*.flex file\n* Diagram: PSI tree diagram (UML plugin required)\n\n\nSyntax overview\n===============\nSee [Parsing Expression Grammar (PEG)](http://en.wikipedia.org/wiki/Parsing_expression_grammar) for basic syntax.\nUse ::= for ← symbol. You can also use [ .. ] for optional sequences and { | | } for choices as these variants are popular in real-world grammars.\nGrammar-Kit source code is the main example of Grammar-Kit application.\nThe grammar for BNF parser and PSI generation can be found [here](grammars/Grammar.bnf).\n\nHere's how it may look like:\n\n````\n// Basic PEG BNF syntax\n\nroot_rule ::= rule_A rule_B rule_C rule_D                // sequence expression\nrule_A ::= token | 'or_text' | \"another_one\"             // choice expression\nrule_B ::= [ optional_token ] and_another_one?           // optional expression\nrule_C ::= \u0026required !forbidden                          // predicate expression\nrule_D ::= { can_use_braces + (and_parens) * }           // grouping and repetition\n\n// Grammar-Kit BNF syntax\n\n{ generate=[psi=\"no\"] }                                  // top-level global attributes\nprivate left rule_with_modifier ::= '+'                  // rule modifiers\nleft rule_with_attributes ::= '?' {elementType=rule_D}   // rule attributes\n\nprivate meta list ::= \u003c\u003cp\u003e\u003e (',' \u003c\u003cp\u003e\u003e) *                // meta rule with parameters\nprivate list_usage ::= \u003c\u003clist rule_D\u003e\u003e                   // meta rule application\n````\n\nThe basic syntax is extended with global attributes and rule attributes.\nAttributes are specified by the list of *name=value* pairs enclosed in braces.\nRule attributes are placed right after the rule definition.\nGlobal attributes are placed on top or separated from a rule definition with a semicolon.\n\nGenerator generates a static method for each BNF expression as follows:\n```\nstatic boolean rule_name(..)               // rule top level expression\nstatic boolean rule_name_0(..)             // rule sub-expression\n...                                        // ...\nstatic boolean rule_name_N1_N2_..._NX      // rule sub-sub-...-sub-expression\n```\nNaming a rule like *rule_name_N1_N2_..._NX* shall be avoided.\n\nOne can specify an attribute for several rules at once in a global attributes block:\n\n````\n{\n  extends(\".*_expr\")=expr        // applies to all .*_expr rules\n  pin(\".*_list(?:_\\d+)*\")=1      // applies to all .*_list rules and their sub-expressions\n}\n````\n\n### Rule modifiers:\n\n1. *private* (PSI tree): skip node creation and let its child nodes be included in its parent. \n2. *left* (PSI tree):  take an AST node on the left (previous sibling) and enclose it by becoming its parent. \n3. *inner* (PSI tree):  take an AST node on the left (previous sibling) and inject itself into it by becoming its child.\n4. *upper* (PSI tree):  take the parent node and replace it by adopting all its children.\n\n5. *meta* (parser):  a parametrized rule; its parse function can take other parse functions as parameters.\n6. *external* (parser):  a rule with a hand-written parse function; no parsing code is generated. \n   \n7. *fake* (PSI classes):  a rule for shaping the generated PSI classes; only PSI classes are generated.\n\nModifiers can be combined, *inner* should only be used together with *left*,\n*private left* is equivalent to *private left inner*, \n*fake* should not be combined with *private*.\n\nBy default, rules are *public*, i.e. *non-private*, *non-fake*, etc.\n\n### Meta rules \u0026 external expressions:\nThe external expression *\u003c\u003c ... \u003e\u003e* is simply an inline variant of an external rule. It can also be used to specify a meta rule along with arguments.\n\nFor example:\n\n````\nmeta comma_separated_list ::= \u003c\u003cparam\u003e\u003e ( ',' \u003c\u003cparam\u003e\u003e ) *\noption_list ::= \u003c\u003ccomma_separated_list (OPTION1 | OPTION2 | OPTION3)\u003e\u003e\n````\n\nExternal rule expression syntax is the same as a body of external expression:\n\n````\n external manually_parsed_rule ::= methodName param1 param2 ...\n````\n\nExternal expressions and external rules interpret double- and single-quoted strings differently.\nGenerally, anything that appears in an external expression after rule or method name is treated\nas a parameter and passed \"as is\" except single-quoted strings which are unquoted first.\nThis helps to pass qualified enum constants, java expressions, etc.\n\nRule references in parameter list are implemented as [GeneratedParserUtilBase.Parser](src/org/intellij/grammar/parser/GeneratedParserUtilBase.java) instances.\n\n### Tokens:\nExplicit tokens are declared via _tokens_ global attribute, e.g. in *token_name=token_value* form. \nA token name is for the IElementType token constant, a token value is usually its string representation in single or double-quotes.\n\nTokens in grammar can be referenced by name or by value in single or double-quotes.\nIt is recommended to use values where possible for better readability.\nNames can be used to resolve conflicts when there is an unquoted token value that also matches some rule.\n\nImplicit tokens are tokens not specified via _tokens_ attribute.\nUnquoted implicit tokens (aka keyword tokens) have names equals to their values.\nQuoted implicit tokens (aka text-matched tokens) are slow because they are matched by text and not by an IElementType constant returned by a lexer.\nText-matched tokens can span more than one real token returned by the lexer.\n\nRules, tokens, and text-matched tokens have different colors.\n\n### Attributes for error recovery and reporting:\n* _pin_ (value: a number or pattern) tunes the parser to handle incomplete matches. \nA sequence matches if its prefix up to a pinned item matches.\nOn successfully reaching the pinned item the parser tries to match the rest items whether they match or not.\nPin value indicates the desired item by either a number *{pin=2}* or pattern  *{pin=\"rule_B\"}*.\nBy default, the pin is applied to the top sequence expression. Sub-expressions can be included using a target pattern:\n*{pin(\".\\*\")=1}* applies to all sub-sequences.\n\n* _recoverWhile_ (value: predicate rule) matches any number of tokens after the rule\nmatching completes with any result. This attribute helps the parser recover when an unmatched token\nsequence is encountered. See [HOWTO section](HOWTO.md#22-using-recoverwhile-attribute) for more.\n\n* _name_ (value: string) specifies a name for a rule to be used in error reports. For example, *name(\"_.*expr\")=expression* changes\nexpression error messages to \"\u0026lt;expression\u0026gt; required\" instead of a long list of tokens.\n\n### Generated parser structure:\nThe generator can split parser code into several classes for better support of large grammars.\n\nIn simple cases, a parser will consist just of several generated classes.\n\nThe actual error recovery and reporting code as well as the parser-based completion provider support code and the basic token matching code reside\nin a _parserUtilClass_ class. It may be altered by specifying some other class that extends or mimics the original [GeneratedParserUtilBase](src/org/intellij/grammar/parser/GeneratedParserUtilBase.java).\nThere's no need to keep a copy of GeneratedParserUtilBase in a project, it is included in *IntelliJ Platform* since version 12.1.\n\nThe manual parsing code, i.e. _external_ rules must be implemented the same way as generated, by a static method in the _parserUtilClass_ class or any other class that will\nbe imported via _parserImports_ attribute like this:\n````\n{\n  parserImports=[\"static org.sample.ManualParsing.*\"]\n}\n````\n\n### Lexer and PSI:\nIElementType constants generated by the parser generator have to be recognized and returned by the lexer.\nThe JFlex-based lexer can be generated from a grammar that defines all the required tokens ( *Generate JFlex Lexer* menu).\n\n*Run JFlex Generator* menu in a \\*.flex file calls JFlex to generate lexer java code.\nKeywords are picked right from usages while tokens like *string*, *identifier* and *comment* can be defined like this (from [TUTORIAL](TUTORIAL.md)):\n\n````\n{\n  tokens=[\n    ...\n    comment='regexp://.*'\n    number='regexp:\\d+(\\.\\d*)?'\n    id='regexp:\\p{Alpha}\\w*'\n    string=\"regexp:('([^'\\\\]|\\\\.)*'|\\\"([^\\\"\\\\]|\\\\.)*\\\")\"\n    ...\n  ]\n  ...\n}\n````\n\nWhile *Live Preview* mode supports full Java RegExp syntax and JFlex supports only a subset (see [JFlex documentation](http://jflex.de/manual.html#SECTION00053000000000000000))\nGrammar-Kit tries to perform some obvious conversions.\n\nThe lexer can be provided separately or one can use the generated \\*.flex file as a base.\n\nParser generator generates token types constants and PSI by default.\nThis can be switched off via *generateTokens* and *generatePSI* global boolean attributes respectively.\n \n*elementType* rule attribute allows mixing the generated code and some existing hand-made PSI.   \n\n[jb:slack]: https://plugins.jetbrains.com/slack\n[jb:x]: https://x.com/JBPlatform","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjetbrains%2Fgrammar-kit","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjetbrains%2Fgrammar-kit","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjetbrains%2Fgrammar-kit/lists"}