{"id":16567875,"url":"https://github.com/sylvainhalle/bullwinkle","last_synced_at":"2025-03-21T11:33:24.675Z","repository":{"id":15411328,"uuid":"18143415","full_name":"sylvainhalle/Bullwinkle","owner":"sylvainhalle","description":"An on-the-fly parser for BNF grammars","archived":false,"fork":false,"pushed_at":"2022-02-17T10:36:22.000Z","size":2238,"stargazers_count":51,"open_issues_count":8,"forks_count":16,"subscribers_count":8,"default_branch":"master","last_synced_at":"2024-10-12T21:07:42.442Z","etag":null,"topics":["bnf","character-string","grammar","parsing"],"latest_commit_sha":null,"homepage":null,"language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sylvainhalle.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE-2.0.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2014-03-26T15:37:51.000Z","updated_at":"2024-07-05T17:39:16.000Z","dependencies_parsed_at":"2022-09-05T21:30:25.708Z","dependency_job_id":null,"html_url":"https://github.com/sylvainhalle/Bullwinkle","commit_stats":null,"previous_names":[],"tags_count":22,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sylvainhalle%2FBullwinkle","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sylvainhalle%2FBullwinkle/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sylvainhalle%2FBullwinkle/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sylvainhalle%2FBullwinkle/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sylvainhalle","download_url":"https://codeload.github.com/sylvainhalle/Bullwinkle/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":221815024,"owners_count":16885097,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bnf","character-string","grammar","parsing"],"created_at":"2024-10-11T21:07:42.804Z","updated_at":"2024-10-28T10:05:53.917Z","avatar_url":"https://github.com/sylvainhalle.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"Bullwinkle: a runtime parser for BNF grammars\n=============================================\n\n[![Travis](https://img.shields.io/travis/sylvainhalle/Bullwinkle.svg?style=flat-square)](https://app.travis-ci.com/github/sylvainhalle/Bullwinkle)\n[![Coverity Scan Build Status](https://img.shields.io/coverity/scan/15155.svg?style=flat-square)](https://scan.coverity.com/projects/sylvainhalle-bullwinkle)\n[![Coverage](https://sonarcloud.io/api/project_badges/measure?project=sylvainhalle_bullwinkle\u0026metric=coverage)](https://sonarcloud.io/dashboard?id=sylvainhalle_bullwinkle)\n\u003cimg src=\"http://leduotang.ca/Bullwinkle.svg\" height=\"20\" alt=\"Downloads\"/\u003e\n\nBullwinkle is a parser for LL(k) languages that operates through recursive descent\nwith backtracking.\n\n[Parser generators](http://en.wikipedia.org/wiki/Parser_generator) such as\nANTLR, Yacc or Bison take a grammar as input and produce code for a parser\nspecific to that grammar, which must then be compiled to be used. On the\ncontrary, Bullwinkle reads the definition of the grammar (expressed in\n[Backus-Naur Form](http://en.wikipedia.org/wiki/Backus-Naur_form) (BNF)) at\n*runtime* and can parse strings on the spot.\n\nOther unique features of Bullwinkle include:\n\n- Instances of the Bullwinkle parser can be safely serialized with\n  [Azrael](https://github.com/sylvainhalle/Azrael).\n- [Partial parsing](#partial), a special mode where input strings can\n  contain non-terminal symbols from the grammar. A string can hence be\n  partially verified for syntactical correctness.\n- [Object builders](#builder), a class of objects that makes it easy to\n  traverse a parse tree and build an output object recursively.\n\nTable of Contents                                                    {#toc}\n-----------------\n\n- [An example](#example)\n- [Compiling and installing Bullwinkle](#install)\n- [Defining a grammar](#grammar)\n- [Using the parse tree](#tree)\n- [Partial parsing](#partial)\n- [Using object builders](#builder)\n- [Command-line usage](#cli)\n- [About the author](#about)\n\nAn example                                                       {#example}\n----------\n\nConsider for example the following simple grammar, taken from the file\n`Examples/Simple-Math.bnf` in the Bullwinkle archive:\n\n    \u003cexp\u003e := \u003cadd\u003e | \u003csub\u003e | \u003cmul\u003e | \u003cdiv\u003e | - \u003cexp\u003e | \u003cnum\u003e;\n    \u003cadd\u003e := \u003cnum\u003e + \u003cnum\u003e | ( \u003cexp\u003e + \u003cexp\u003e );\n    \u003csub\u003e := \u003cnum\u003e - \u003cnum\u003e | ( \u003cexp\u003e - \u003cexp\u003e );\n    \u003cmul\u003e := \u003cnum\u003e × \u003cnum\u003e | ( \u003cexp\u003e × \u003cexp\u003e );\n    \u003cdiv\u003e := \u003cnum\u003e ÷ \u003cnum\u003e | ( \u003cexp\u003e ÷ \u003cexp\u003e );\n    \u003cnum\u003e := ^[0-9]+;\n\nHere is a simple Java program that reads characters strings and tries to parse\nthem against this grammar (a complete working program can be found in the file\n`SimpleExample.java`:\n    \n    try\n    {\n      BnfParser parser = new BnfParser(\"Examples/Simple-Math.bnf\");\n      ParseNode node1 = parser.parse(\"3+4\");\n      ParseNode node2 = parser.parse(\"(10 + (3 - 4))\");\n    }\n    catch (IOException | InvalidGrammarExpression | ParseException)\n    {\n      System.err.println(\"Some error occurred\");\n    }\n\nThe first instruction loads the grammar definition and instantiates an\nobject `parser` for that grammar. Calls to method `parse()` give this parser\na character string, and return an object of class `ParseNode` which points\nto the head of the corresponding parse tree (or null if the input string\ndoes not follow the grammar). These instructions are enclosed in a try/catch\nblock to catch potential exceptions thrown during this process. The whole\nprocess is done dynamically at runtime, without requiring any compiling.\n\nHere is the parse tree returned for the second expression in the previous\nexample:\n\n![Parse tree](Simple-Math.png?raw=true)\n\nCompiling and Installing Bullwinkle                              {#install}\n-----------------------------------\n\nFirst make sure you have the following installed:\n\n- The Java Development Kit (JDK) to compile. Bullwinkle was developed and\n  tested on version 6 of the JDK, but it is probably safe to use any\n  later version.\n- [Ant](http://ant.apache.org) to automate the compilation and build process\n\nDownload the sources for Bullwinkle from\n[GitHub](http://github.com/sylvainhalle/Bullwinkle) or clone the repository\nusing Git:\n\n    git clone git@github.com:sylvainhalle/Bullwinkle.git\n\n### Compiling\n\nCompile the sources by simply typing:\n\n    ant\n\nThis will produce a file called `bullwinkle.jar` in the folder. This\nfile is runnable and stand-alone, or can be used as a library, so it can be\nmoved around to the location of your choice.\n\nIn addition, the script generates in the `doc` folder the Javadoc\ndocumentation for using Bullwinkle. This documentation is also embedded in\nthe JAR file. To show documentation in Eclipse, right-click on the jar,\nclick \"Properties\", then fill the Javadoc location (which is the JAR\nitself).\n\n### Testing\n\nBullwinkle can test itself by running:\n\n    ant test\n\nUnit tests are run with [jUnit](http://junit.org); a detailed report of\nthese tests in HTML format is availble in the folder `tests/junit`, which\nis automatically created. Code coverage is also computed with\n[JaCoCo](http://www.eclemma.org/jacoco/); a detailed report is available\nin the folder `tests/coverage`.\n\n### Coverity Scan\n\nBullwinkle uses [Coverity Scan](https://scan.coverity.com) for static analysis\nof its source code and defect detection. Instructions for using Coverity Scan\nlocally are detailed [here](https://scan.coverity.com/download?tab=java). In\na nutshell, if Coverity Scan is installed, type the following:\n\n    cov-build --dir cov-int ant compile\n\n(Make sure to clean up the directory first by launching `ant clean`.)\n\nDefining a grammar                                               {#grammar}\n------------------\n\nFor Bullwinkle to work, the grammar must be\n[LL(k)](http://en.wikipedia.org/wiki/LL_parser). Roughly, this means that\nit must not contain a production rules of the form\n`\u003cS\u003e := \u003cS\u003e something`. Trying to parse such a rule by recursive descent\ncauses an infinite recursion (which will throw a `ParseException` when the\nmaximum recursion depth is reached).\n\nDefining a grammar can be done in two ways.\n\n### Parsing a string\n\nThe first way is by parsing a character string (taken from a file or created\ndirectly) that contains the grammar declaration. This format uses a fairly\nintuitive syntax, as the example above has shown.\n   \n- Non-terminal symbols are enclosed in `\u003c` and `\u003e` and their names must not\n  contain spaces.\n- Rules are defined with `:=` and cases are separated by the pipe character.\n- A rule can span multiple lines (any whitespace character after the first one\n  is ignored, as in e.g. HTML) and must end by a semicolon.\n- Terminal symbols are defined by typing them directly in a rule, or through\n  regular expressions and begin with the `^` (hat) character. The example above\n  shows both cases: the `+` symbol is typed directly into the rules, while the\n  terminal symbol `\u003cnum\u003e` is defined with a regex. **Look out:**\n  - If a space needs to be used in the regular expression, it must be\n    declared by using the regex sequence `\\s`, and *not* by putting a space.\n  - Beware not to put an extra space before the ending semicolon, or that\n    space will count as part of the regex\n  - Caveat emptor: a few corner cases are not covered at the moment, such as\n    a regex that would contain a semicolon.\n- The left-hand side symbol of the first rule found is assumed to be the start\n  symbol. This can be overridden by calling method `setStartSymbol()` on an\n  instance of the parser.\n- Whitespace acts as a token separator, so there is no need to declare terminal\n  tokens separately. This means that the rule `\u003cnum\u003e + \u003cnum\u003e` matches any string\n  with a number, the symbol +, and another number, separated by any number of\n  spaces, including none. This also means that writing `1+2` defines a *single*\n  token that matches only the string \"1+2\". When declaring rules, tokens *must*\n  be separated by a space. Writing `(\u003cexp\u003e)` is illegal and will throw an\n  exception; one must write `( \u003cexp\u003e )` (note the spaces). However, since\n  whitespace is ignored when parsing, this rule would still match the string\n  \"(1+1)\".\n\nSome symbols or sequences of symbols, such as `:=`, `|`, `\u003c`, `\u003e` and `;`,\nhave a special meaning and cannot be used directly inside terminal symbols\n(note that this limitation applies only when parsing a grammar from a text\nfile). However, these symbols can be included by *escaping* them, i.e.\nreplacing them with their UTF-8 hex code.\n\n- `|` can be replaced by `\\u007c`\n- `\u003c` can be replaced by `\\u003c`\n- `\u003c` can be replaced by `\\u003e`\n- `;` can be replaced by `\\u003b`\n- `:=` can be replaced by `\\u003a\\u003d`\n\nThe characters should appear as is (i.e. unescaped) in the string to parse.\n\n### Building the rules manually\n\nA second way of defining a grammar consists of assembling rules by creating\ninstances of objects programmatically. Roughly:\n\n- A `BnfRule` contains a left-hand side that must be a `NonTerminalToken`, and\n  a right-hand side containing multiple cases that are added through method\n  `addAlternative()`.\n- Each case is itself a `TokenString`, formed of multiple `TerminalToken`s and\n  `NonTerminalToken`s which can be `add`ed. Terminal tokens include\n  `NumberTerminalToken`, `StringTerminalToken` and `RegexTerminalToken`.\n- `BnfRule`s are `add`ed to an instance of the `BnfParser`.\n\nUsing the parse tree                                                {#tree}\n--------------------\n\nOnce a grammar has been loaded into an instance of `BnfParser`, the `parse()`\nmethod is used to parse a given string and produce a parse tree (or null if the\nstring does not parse). This parse tree can then be explored in two ways:\n\n1. In a manner similar to the DOM, by calling the `getChildren()` method of an\n   instance of a `ParseNode` to get the list of its children (and so on,\n   recursively);\n2. Through the [Visitor design\n   pattern](http://en.wikipedia.org/wiki/Visitor_pattern). In that case, one\n   creates a class that implements the `ParseNodeVisitor` interface, and passes\n   this visitor to the `ParseNode`'s `acceptPostfix()` or `acceptPrefix()`\n   method, depending on the desired mode of traversal. The sample code shows an\n   example of a visitor (class `GraphvizVisitor`), which produces a DOT file\n   from the contents of the parse tree.\n\nIf your goal is to create some object out of the parse tree, consider using\nthe [object builder](#builder) class to simplify your work.\n\nPartial parsing                                                  {#partial}\n---------------\n\nPartial parsing is a special mode where the input string is allowed\nto contain non-terminal symbols. For example, consider the following grammar:\n\n    \u003cS\u003e := \u003cA\u003e \u003cB\u003e c;\n    \u003cA\u003e := foo;\n    \u003cB\u003e := bar | \u003cZ\u003e d;\n    \u003cZ\u003e := 0 | 1;\n\nIn partial parsing mode, the string `foo \u003cB\u003e c` is accepted by the\ngrammar. In this case, one of the leaf nodes of the resulting parse tree\nis not a terminal symbol, but rather the non-terminal symbol \u0026lt;B\u0026gt;.\n\nOne particular use of partial parsing is the step-by-step verification of\npartially formed strings. In the previous example, one might create\nan input string by first writing\n\n    \u003cA\u003e \u003cB\u003e c\n\nThis string can be checked to be valid by parsing it with partial parsing\nenabled. Then non-terminal \u0026lt;A\u0026gt; can be expanded, yielding:\n\n    foo \u003cB\u003e c\n\nAgain, one can check that this string is still syntactically valid. Non-terminal\n\u0026lt;B\u0026gt; can be expanded to form `foo \u003cZ\u003e d c`, and then `foo 0 d c`.\n\nTo enable partial parsing, use the method `setPartialParsing()` of class\n`BnfParser`.\n\nUsing object builders                                            {#builder}\n---------------------\n\nMany times, the goal of parsing an expression is to create some \"object\"\nout of the resulting parse tree. The `ParseTreeObjectBuilder` class in\nBullwinkle simplifies the task of creating such objects.\n\nSuppose for example that you created\nobjects to represent simple arithmetical expressions: there is one class\nfor `Add`, another for `Sub`(traction), another for plain `Num`bers, etc.\n(See the `Examples` folder in the sources, where such classes are indeed\nshown in `ArithExp.java`.) You can create and nest such objects\nprogrammatically, for example to represent 10+(6-4):\n\n    ArithExp a = new Add(new Num(10), new Sub(new Num(6), new Num(4));\n\nSuppose you created a simple grammar to represent such expressions in\n\"forward\" Polish notation, such as this:\n\n    \u003cexp\u003e := \u003cadd\u003e | \u003csub\u003e | \u003cnum\u003e;\n    \u003cadd\u003e := + \u003cexp\u003e \u003cexp\u003e;\n    \u003csub\u003e := - \u003cexp\u003e \u003cexp\u003e;\n    \u003cnum\u003e := ^[0-9]+;\n\nUsing such a grammar, the previous expression would be written as\n`+ 10 - 6 4`. You would like to be able to instantiate `ArithExp` objects\nfrom expressions following this syntax.\n\nThe `ParseTreeObjectBuilder` makes such a task simple. It performs a\n*postfix* traversal of a parse tree and maintains a stack of arbitrary\nobjects. When visiting a parse node that corresponds to a non-terminal\ntoken, such as \u0026lt;foo\u0026gt;, it looks for a method that handles this symbol.\nThis is done by adding an annotation `@Builds` to the method, as follows:\n\n    @Builds(rule=\"\u003cfoo\u003e\")\n    public void myMethod(Stack\u003cObject\u003e stack) { ...\n\nThe object builder calls this method, and passes it the current contents\nof the object stack. It is up to this method to pop and push objects\nfrom that stack, in order to recursively create the desired object at the\nend. For example, in the grammar above, the code to handle token \u0026lt;add\u0026gt;\nwould look like:\n\n    @Builds(rule=\"\u003cadd\u003e\")\n    public void handleAdd(Stack\u003cObject\u003e stack) {\n      ArithExp e2 = (ArithExp) stack.pop();\n      ArithExp e1 = (ArithExp) stack.pop();\n      stack.pop(); // To remove the \"+\" symbol\n      stack.push(new Add(e1, e2));\n    }\n\nSince the builder traverses the tree in a postfix fashion, when a parse\nnode for \u0026lt;add\u0026gt; is visited, the object stack should already contain\nthe `ArithExp` objects created from its two operands. As a rule, each method\nshould pop from the stack as many objects as there are tokens in the corresponding case in the grammar. For example, the rule for \u0026lt;add\u0026gt;\nhas three tokens, and so the method handling \u0026lt;add\u0026gt; pops three objects.\n\nThe full example for this parser can be found in `BuildExampleStack` in the\n`Examples` project.\n\nAs one can see, it is possible to create object builders that read\nexpressions in just a few lines of code. This can be even further simplified\nusing the `pop` and `clean` parameters. Instead of popping objects manually,\nand pushing a new object back onto the stack, one can use the `pop` parameter\nto ask for the object builder to already pop the appropriate number of\nobjects from the stack. The method for \u0026lt;add\u0026gt; would then become:\n\n    @Builds(rule=\"\u003cadd\u003e\", pop=true)\n    public ArithExp handleAdd(Object ... parts) {\n      return new Add((ArithExp) parts[1], (ArithExp) parts[2]);\n    }\n\nNotice how this time, the method's arguments is an array of objects; in that\ncase, the array has three elements, corresponding to the three tokens of the\n\u0026lt;add\u0026gt; rule. The first is the \"+\" symbol, and the other two are the\n`ArithExp` objects created from the two sub-expressions. Similarly, instead of\npushing an object to the stack, the method simply returns it; the object builder\ntakes care of pushing it. By not accessing the contents of the stack directly,\nit is harder to make mistakes.\n\nAs a further refinement, the `clean` option can remove from the arguments all\nthe objects that match terminal symbols in the corresponding rule. Consider a\ngrammar for infix arithmetical expressions, where parentheses are optional\naround single numbers. This grammar would look like:\n\n    \u003cexp\u003e := \u003cadd\u003e ...\n    \u003cadd\u003e := \u003cnum\u003e + \u003cnum\u003e | ( \u003cexp\u003e ) + \u003cnum\u003e | \u003cnum\u003e + ( \u003cexp\u003e ) ...\n\nThis time, the rules for each operator must take into account whether any of\ntheir operands is a number or a compound expression. The code handling\n\u0026lt;add\u0026gt; would be more complex, as one would have to carefully pop an\nelement, check if it is a parenthesis, and if so, take care of popping the\nmatching parenthesis later on, etc. However, one can see that each case of\nthe rule has exactly two non-terminal tokens, and that both are `ArithExp`.\nUsing the `clean` option in conjunction with `pop`, the code for handling\n\u0026lt;add\u0026gt; becomes identical as before:\n\n    @Builds(rule=\"\u003cadd\u003e\", pop=true, clean=true)\n    public ArithExp handleAdd(Object ... parts) {\n      return new Add((ArithExp) parts[0], (ArithExp) parts[1]);\n    }\n\nThe array indices become 0 and 1, since only the two `ArithExp` objects remain\nas the arguments. Again, a full example can be found in the `Examples` folder,\ninside `BuildExamplePop.java`.\n\nCommand-line usage                                                   {#cli}\n------------------\n\nThe project comes with `bullwinkle.jar`, a file that can be used\neither as a library inside a Java program (as described above), or as a\nstand-alone command-line application. In that case, the application reads\nthe grammar definition from a file, a string to parse either from the\nstandard input or from another file, and writes to the standard output the\nresulting parse tree. This tree can then be read by another application.\n\nCommand-line usage is as follows:\n\n    java -jar bullwinkle.jar [options] grammar [file]\n\nwhere `grammar` is the path to a file describing the grammar to use, and\n`file` is an optional filename containing the string to be parsed. If no\nfile is given, the string will be read from the standard input.\n\nOptions are:\n\n`-f x`, `--format x`\n:  Output with format x. Supported values are `xml`, `txt` and `dot`. See\n   below for a description of these formats.\n \n`-v x`\n:  Set verbosity to level x (0 = no messages are printed).\n\nThree output formats are supported directly.\n\n### XML\n\nIn the XML format, non-terminal symbols are converted into tags, and\nterminal tokens are surrounded by the `\u003ctoken\u003e` element. In the above\nexample, the expression `3 + 4` becomes the following XML structure:\n\n    \u003cparsetree\u003e\n      \u003cexp\u003e\n        \u003cadd\u003e\n          \u003cnum\u003e\n            \u003ctoken\u003e\n              3\n            \u003c/token\u003e\n          \u003c/num\u003e\n          \u003ctoken\u003e\n            +\n          \u003c/token\u003e\n          \u003cnum\u003e\n            \u003ctoken\u003e\n              4\n            \u003c/token\u003e\n          \u003c/num\u003e\n        \u003c/add\u003e\n      \u003c/exp\u003e\n    \u003c/parsetree\u003e\n\n### Indented text\n\nIndented text merely outputs terminal and non-terminal tokens, indenting\nany subtree by one space, as follows:\n\n    exp\n     add\n      num\n       token\n        3\n      token\n       +\n      token\n      num\n       token\n        4\n\n### DOT\n\nThe DOT format produces a text file suitable for use with the\n[Graphviz](http://www.graphviz.org) package. The picture shown earlier was\nproduced in this way.\n\nProjects that use Bullwinkle                                       {#usage}\n----------------------------\n\n- [BeepBeep 3](https://liflab.github.io/beepbeep-3), an event stream query\n  engine\n- [Cornipickle](https://github.com/liflab/cornipickle), a web testing tool\n- [SealTest](https://liflab.github.io/sealtest), a test sequence generator\n- [SugarSMV](https://github.com/liflab/sugarsmv), a syntactical extension\n  of the NuSMV model checker\n\n\nAbout the author                                                   {#about}\n----------------\n\nBullwinkle was written by [Sylvain Hallé](http://leduotang.ca/sylvain),\nAssociate Professor at [Université du Québec à Chicoutimi](http://www.uqac.ca),\nCanada. It arose from the need to experiment with\nvarious grammars without requiring compilation, as with classical parser\ngenerators.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsylvainhalle%2Fbullwinkle","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsylvainhalle%2Fbullwinkle","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsylvainhalle%2Fbullwinkle/lists"}