{"id":43332950,"url":"https://github.com/nihei9/vartan","last_synced_at":"2026-02-02T00:09:53.986Z","repository":{"id":53924850,"uuid":"371996333","full_name":"nihei9/vartan","owner":"nihei9","description":"An LALR parser generator for golang","archived":false,"fork":false,"pushed_at":"2023-06-14T12:28:02.000Z","size":413,"stargazers_count":15,"open_issues_count":2,"forks_count":3,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-12-27T11:55:22.336Z","etag":null,"topics":["ast","go","golang","lalr","lalr-parser","lalr-parser-generator","parser","parser-generator","syntax-analysis","syntax-analyzer","syntax-tree"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/nihei9.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-05-29T14:39:25.000Z","updated_at":"2025-03-05T21:10:44.000Z","dependencies_parsed_at":"2024-06-20T09:18:57.390Z","dependency_job_id":"be2f5408-df8d-4a78-9759-3118d35facb4","html_url":"https://github.com/nihei9/vartan","commit_stats":null,"previous_names":[],"tags_count":9,"template":false,"template_full_name":null,"purl":"pkg:github/nihei9/vartan","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nihei9%2Fvartan","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nihei9%2Fvartan/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nihei9%2Fvartan/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nihei9%2Fvartan/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/nihei9","download_url":"https://codeload.github.com/nihei9/vartan/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nihei9%2Fvartan/sbom","scorecard":{"id":686862,"data":{"date":"2025-08-11","repo":{"name":"github.com/nihei9/vartan","commit":"16a433b0681a19985c0134d5f9719f0fcdfe634a"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":3.8,"checks":[{"name":"Token-Permissions","score":0,"reason":"detected GitHub workflow tokens with excessive permissions","details":["Warn: no topLevel permission defined: .github/workflows/test.yml:1","Info: no jobLevel write permissions found"],"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"Code-Review","score":0,"reason":"Found 0/30 approved changesets -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"Dangerous-Workflow","score":10,"reason":"no dangerous workflow patterns detected","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Maintained","score":0,"reason":"0 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Pinned-Dependencies","score":0,"reason":"dependency not pinned by hash detected -- score normalized to 0","details":["Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/test.yml:15: update your workflow using https://app.stepsecurity.io/secureworkflow/nihei9/vartan/test.yml/main?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/test.yml:16: update your workflow using https://app.stepsecurity.io/secureworkflow/nihei9/vartan/test.yml/main?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/test.yml:25: update your workflow using https://app.stepsecurity.io/secureworkflow/nihei9/vartan/test.yml/main?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/test.yml:26: update your workflow using https://app.stepsecurity.io/secureworkflow/nihei9/vartan/test.yml/main?enable=pin","Warn: third-party GitHubAction not pinned by hash: .github/workflows/test.yml:29: update your workflow using https://app.stepsecurity.io/secureworkflow/nihei9/vartan/test.yml/main?enable=pin","Info:   0 out of   4 GitHub-owned GitHubAction dependencies pinned","Info:   0 out of   1 third-party GitHubAction dependencies pinned"],"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"SAST","score":0,"reason":"no SAST tool detected","details":["Warn: no pull requests merged into dev branch"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Branch-Protection","score":-1,"reason":"internal error: error during branchesHandler.setup: internal error: githubv4.Query: Resource not accessible by integration","details":null,"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE:0","Info: FSF or OSI recognized license: MIT License: LICENSE:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Vulnerabilities","score":10,"reason":"0 existing vulnerabilities detected","details":null,"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}}]},"last_synced_at":"2025-08-22T01:10:03.532Z","repository_id":53924850,"created_at":"2025-08-22T01:10:03.532Z","updated_at":"2025-08-22T01:10:03.532Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28996569,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-01T23:10:54.274Z","status":"ssl_error","status_checked_at":"2026-02-01T23:10:47.298Z","response_time":56,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ast","go","golang","lalr","lalr-parser","lalr-parser-generator","parser","parser-generator","syntax-analysis","syntax-analyzer","syntax-tree"],"created_at":"2026-02-02T00:09:53.406Z","updated_at":"2026-02-02T00:09:53.979Z","avatar_url":"https://github.com/nihei9.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# vartan\n\nvartan is an LALR(1) parser generator for golang. vartan also provides a command to perform syntax analysis to allow easy debugging of your grammar.\n\n[![Test](https://github.com/nihei9/vartan/actions/workflows/test.yml/badge.svg?branch=main)](https://github.com/nihei9/vartan/actions/workflows/test.yml)\n\n## Installation\n\nCompiler:\n\n```sh\n$ go install github.com/nihei9/vartan/cmd/vartan@latest\n```\n\nCode Generator:\n\n```sh\n$ go install github.com/nihei9/vartan/cmd/vartan-go@latest\n```\n\n## Usage\n\n### 1. Define your grammar\n\nvartan uses BNF-like DSL to define your grammar. As an example, let's write a grammar that represents a simple expression.\n\n```\n#name expr;\n\n#prec (\n\t#left mul div\n\t#left add sub\n);\n\nexpr\n\t: expr add expr\n\t| expr sub expr\n\t| expr mul expr\n\t| expr div expr\n\t| func_call\n\t| int\n\t| id\n\t| l_paren expr r_paren #ast expr\n\t;\nfunc_call\n\t: id l_paren args r_paren #ast id args\n\t| id l_paren r_paren      #ast id\n\t;\nargs\n\t: args comma expr #ast args... expr\n\t| expr\n\t;\n\nws #skip\n\t: \"[\\u{0009}\\u{0020}]+\";\nint\n\t: \"0|[1-9][0-9]*\";\nid\n\t: \"[A-Za-z_][0-9A-Za-z_]*\";\nadd\n\t: '+';\nsub\n\t: '-';\nmul\n\t: '*';\ndiv\n\t: '/';\nl_paren\n\t: '(';\nr_paren\n\t: ')';\ncomma\n\t: ',';\n```\n\nSave the above grammar to a file in UTF-8. In this explanation, the file name is `expr.vartan`.\n\n⚠️ The input file must be encoded in UTF-8.\n\n### 2. Compile the grammar\n\nNext, generate a parsing table using `vartan compile` command.\n\n```sh\n$ vartan compile expr.vartan -o expr.json\n```\n\n### 3. Debug\n\n#### 3.1. Parse\n\nIf you want to make sure that the grammar behaves as expected, you can use `vartan parse` command to try parse without implementing a driver.\n\n⚠️ An encoding that `vartan parse` command and the driver can handle is only UTF-8.\n\n```sh\n$ echo -n 'foo(10, bar(a)) + 99 * x' | vartan parse expr.json\nexpr\n├─ expr\n│  └─ func_call\n│     ├─ id \"foo\"\n│     └─ args\n│        ├─ expr\n│        │  └─ int \"10\"\n│        └─ expr\n│           └─ func_call\n│              ├─ id \"bar\"\n│              └─ args\n│                 └─ expr\n│                    └─ id \"a\"\n├─ add \"+\"\n└─ expr\n   ├─ expr\n   │  └─ int \"99\"\n   ├─ mul \"*\"\n   └─ expr\n      └─ id \"x\"\n```\n\nWhen `vartan parse` command successfully parses the input data, it prints a CST or an AST (if any).\n\n#### 3.2. Resolve conflicts\n\n`vartan compile` command also generates a report named `*-report.json`. This file describes each state in the parsing table in detail. If your grammar contains conflicts, see `Conflicts` and `States` sections of this file. Using `vartan show` command, you can see the report in a readable format.\n\n```sh\n$ vartan show expr-report.json\n```\n\n### 4. Test\n\n`vartan test` command allows you to test whether your grammar recognizes an input text as a syntax tree with an expected structure. To do so, you need to define a test case as follows.\n\n```\nThis is an example.\n---\na / b * 100\n---\n(expr\n\t(expr\n\t\t(expr (id 'a'))\n\t\t(div '/')\n\t\t(expr (id 'b')))\n\t(mul '*')\n\t(expr (int '100')))\n```\n\nThe test case consists of a description, an input text, and a syntax tree you expect. Each part is separated by the delimiter `---`. The syntax tree is represented by the syntax like an [S-expression](https://en.wikipedia.org/wiki/S-expression).\n\nA text of a token is represented by a string enclosed in `'` or `\"`. Within `\"...\"`, characters can alose be represented by Unicode code points (for instance, `\\u{000A}` is LF).\n\nSave the above test case to `test.txt` file and run the following command.\n\n```sh\n$ vartan test expr.vartan test.txt\nPassed test.txt\n```\n\nWhen you specify a directory as the 2nd argument of `vartan test` command, it will run all test cases in the directory.\n\n### 5. Generate a parser\n\nUsing `vartan-go` command, you can generate a source code of a parser to recognize your grammar.\n\n```sh\n$ vartan-go expr.json\n```\n\nThen you will get the following files.\n\n* `expr_parser.go`\n* `expr_lexer.go`\n* `expr_semantic_action.go`\n\nYou need to implement a driver to use the parser. An example is below.\n\n```go\npackage main\n\nimport (\n\t\"fmt\"\n\t\"io\"\n\t\"os\"\n)\n\nfunc main() {\n\ttoks, err := NewTokenStream(os.Stdin)\n\tif err != nil {\n\t\tfmt.Println(err)\n\t\tos.Exit(1)\n\t}\n\tgram := NewGrammar()\n\ttb := NewDefaultSyntaxTreeBuilder()\n\tp, err := NewParser(toks, gram, SemanticAction(NewASTActionSet(gram, tb)))\n\tif err != nil {\n\t\tfmt.Println(err)\n\t\tos.Exit(1)\n\t}\n\terr = p.Parse()\n\tif err != nil {\n\t\tfmt.Println(err)\n\t\tos.Exit(1)\n\t}\n\tsynErrs := p.SyntaxErrors()\n\tif len(synErrs) \u003e 0 {\n\t\tfor _, synErr := range synErrs {\n\t\t\tprintSyntaxError(os.Stderr, synErr, gram)\n\t\t}\n\t\tos.Exit(1)\n\t}\n\tfmt.Println(\"accepted\")\n\tPrintTree(os.Stdout, tb.Tree())\n}\n\nfunc printSyntaxError(w io.Writer, synErr *SyntaxError, gram Grammar) {\n\tvar msg string\n\ttok := synErr.Token\n\tswitch {\n\tcase tok.EOF():\n\t\tmsg = \"\u003ceof\u003e\"\n\tcase tok.Invalid():\n\t\tmsg = fmt.Sprintf(\"'%v' (\u003cinvalid\u003e)\", string(tok.Lexeme()))\n\tdefault:\n\t\tif term := gram.Terminal(tok.TerminalID()); term != \"\" {\n\t\t\tmsg = fmt.Sprintf(\"'%v' (%v)\", string(tok.Lexeme()), term)\n\t\t} else {\n\t\t\tmsg = fmt.Sprintf(\"'%v'\", string(tok.Lexeme()))\n\t\t}\n\t}\n\tfmt.Fprintf(w, \"%v:%v: %v: %v\", synErr.Row+1, synErr.Col+1, synErr.Message, msg)\n\n\tif len(synErr.ExpectedTerminals) \u003e 0 {\n\t\tfmt.Fprintf(w, \": expected: %v\", synErr.ExpectedTerminals[0])\n\t\tfor _, t := range synErr.ExpectedTerminals[1:] {\n\t\t\tfmt.Fprintf(w, \", %v\", t)\n\t\t}\n\t}\n\n\tfmt.Fprintf(w, \"\\n\")\n}\n```\n\nPlease save the above source code to `main.go` and create a directory structure like the following.\n\n```\n/project_root\n├── expr_parser.go\n├── expr_lexer.go\n├── expr_semantic_action.go\n└── main.go (the driver you implemented)\n```\n\nNow, you can perform the syntax analysis.\n\n```sh\n$ echo -n 'foo+99' | go run .\naccepted\nexpr\n├─ expr\n│  └─ id \"foo\"\n├─ add \"+\"\n└─ expr\n   └─ int \"99\"\n```\n\n```sh\n$ echo -n 'foo+99?' | go run .\n1:7: unexpected token: '?' (\u003cinvalid\u003e): expected: \u003ceof\u003e, add, sub, mul, div\nexit status 1\n```\n\n## Vartan syntax\n\n### Grammar name\n\nA grammar name `#name \u003cIdentifier\u003e` is an identifier that represents a grammar name. For now, this identifier is used as a file name generated like _\u003cgrammar-name\u003e\\_parser.go_.\n\n### Production rules\n\nA production rule consists of a non-terminal symbol and sequences of symbols the non-terminal symbol derives. The first production rule will be the start production rule.\n\nProduction rule:\n\n```\n\u003cnon-terminal-symbol\u003e\n\t: \u003calternative-1\u003e\n\t| \u003calternative-2\u003e\n\t| ...\n\t| \u003calternative-N\u003e\n\t;\n```\n\nor\n\n```\n\u003cterminal-symbol\u003e\n\t: \u003cpattern-or-string-literal\u003e\n\t;\n```\n\nAlternative:\n\n```\n\u003celement-1\u003e \u003celement-2\u003e ... \u003celement-N\u003e\n```\n\nAn element an alternative contains is a terminal symbol or a non-terminal symbol.\n\nIf a production rule satisfies all of the following conditions, it is considered to define a terminal symbol.\n\n* A rule has only one alternative.\n* The alternative has only one pattern or string literal.\n\nFragment:\n\n```\nfragment \u003cterminal-symbol\u003e\n\t: \u003cpattern-or-string\u003e\n\t;\n```\n\nUsing the `fragment` keyword, you can also define a fragment that represents a part of a pattern. You can use a fragment by embedding it into a pattern like `\"\\f{some_fragment}\"`.\n\n### Types\n\n#### Identifier\n\nAn identifier is a string that satisfies all of the following rules:\n\n* Contains only the lower-case letters (`a`-`z`), the digits (`0`-`9`), and the underscore (`_`).\n* The first letter is only the lower-case letters.\n* The last letter is only the lower-case letters or the digits.\n\nexamples:\n\n`expression`, `if_statement`, `parameter1`\n\n#### Pattern\n\nA pattern is a string enclosed with `\"` and represents a regular expression. A pattern that appears in production rules is used in lexical analysis. For more information on the syntax of regular expressions, please see [Regular Expression](#regular-expression).\n\nexamples:\n\n`\"if\"`, `\"\\+\"`, `\"[A-Za-z_][0-9A-Za-z_]*\"`, `[\\u{0009}\\u{0020}]+`\n\n#### String literal\n\nA string literal is a string enclosed with `'`. A string literal is interpreted literally, not as a regular expression.\n\nexamples:\n\n`'if'`, `'+'`, `'=='`\n\n### Directives for non-terminal symbols\n\n#### `#ast {\u003csymbol-or-label: Identifier\u003e}`\n\nA `#ast` directive allows you to define a structure of an AST (Abstract Syntax Tree).\n\nexample 1:\n\nConsider a grammar that accepts comma-separated list of integers. You can avoid including brackets and commas in an AST by specifying only the necessary symbols int the `#ast` directive parameters. Also, you can flatten an AST using `...` operator. `...` operator expands child nodes of a specified symbol.\n\n```\n#name example;\n\nlist\n\t: l_bracket elems r_bracket #ast elems...\n\t;\nelems\n\t: elems comma int #ast elems... int\n\t| int\n\t;\n\nws #skip\n\t: \"[\\u{0009}\\u{0020}]+\";\nl_bracket\n\t: '[';\nr_bracket\n\t: ']';\ncomma\n\t: ',';\nint\n\t: \"0|[1-9][0-9]*\";\n```\n\nThe above grammar generates an AST as follows:\n\n```\n$ echo -n '[1, 2, 3]' | vartan parse example.json\nlist\n├─ int \"1\"\n├─ int \"2\"\n└─ int \"3\"\n```\n\nexample 2:\n\nConsider a grammar that accepts ternary-if expression (`\u003ccondition\u003e ? \u003cvalue if true\u003e : \u003cvalue if false\u003e`). As this grammar contains two `int` symbols, you need to add labels to each symbol to distinguish them. A label consists of `@` + an identifier.\n\n```\n#name example;\n\nif_expr\n\t: id q int@true colon int@false #ast id true false\n\t;\n\nws #skip\n\t: \"[\\u{0009}\\u{0020}]+\";\nq\n\t: '?';\ncolon\n\t: ':';\nid\n\t: \"[a-z_][0-9a-z_]*\";\nint\n\t: \"0|[1-9][0-9]*\";\n```\n\nThe above grammar generates an AST as follows:\n\n```\n$ echo -n 'x? 0 : 99' | vartan parse example.json\nif_expr\n├─ id \"x\"\n├─ int \"0\"\n└─ int \"99\"\n```\n\nLabels are intended to identify elements in directives. An AST doesn't contain labels.\n\n#### `#prec \u003csymbol: Identifier\u003e`\n\nA `#prec` directive gives alternatives the same precedence as `symbol`.\n\nSee [Operator precedence and associativity](#operator-precedence-and-associativity) section for more details on the `#prec` directive.\n\n#### `#recover`\n\nA parser transitions to an error state when an unexpected token appears. By default, the parser recovers from the error state when it shifts three tokens after going to the error state.\n\nWhen the parser reduces a non-terminal symbol having a `#recover` directive, the parser recovers from the error state.\n\nSee [Error recovery](#error-recovery) section for more details on the `#recover` directive.\n\n### Directives for terminal symbols\n\n#### `#mode {\u003cmode-name: Identifier\u003e}`, `#push \u003cmode-name: Identifier\u003e`, and `#pop`\n\nA terminal symbol with a `#mode` directive is recognized only on a specified mode (`mode-name`). The mode is lexer state, which a `#push` directive and a `#pop` directive can switch.\n\nWhen the parser shifts a terminal symbol having the `#push` directive, the current mode of the lexer will change to the specified mode (`mode-name`). Using the `#pop` directive, you can make the lexer revert to the previous mode.\n\nexample:\n\n```\n#name example;\n\ntag_pairs\n\t: tag_pairs tag_pair\n\t| tag_pair\n\t;\ntag_pair\n\t: open_tag tag_pairs close_tag\n\t| open_tag text close_tag\n\t;\nopen_tag\n\t: tag_open name tag_close\n\t;\nclose_tag\n\t: tag_open closing_mark name tag_close\n\t;\n\nws #skip\n\t: \"[\\u{0009}\\u{000A}\\u{000D}\\u{0020}]+\";\ntext\n\t: \"([^\u003c]|\\\\\u003c)+\";\ntag_open #push tag\n\t: '\u003c';\ntag_close #mode tag #pop\n\t: '\u003e';\nclosing_mark #mode tag\n\t: '/';\nname #mode tag\n\t: \"[a-z_-][0-9a-z_-]*\";\n```\n\nThe above grammar accepts XML-like texts such as the following:\n\n```\n\u003cassistant-director\u003e\n\t\u003cname\u003eWalter Skinner\u003c/name\u003e\n\t\u003cborn\u003eJune 3, 1952\u003c/born\u003e\n\u003c/assistant-director\u003e\n```\n\n#### `#skip`\n\nThe parser doesn't shift a terminal symbol having a `#skip` directive. In other words, these terminal symbols are recognized in lexical analysis but not used in syntax analysis. The `#skip` directive helps define delimiters like white spaces.\n\nexample:\n\n```\n#name example;\n\ns\n\t: foo bar\n\t;\n\nws #skip\n\t: \"[\\u{0009}\\u{0020}]+\";\nfoo\n\t: 'foo';\nbar\n\t: 'bar';\n```\n\nThe above grammar accepts the following input:\n\n```\nfoo    bar\n```\n\n```\nfoobar\n```\n\n### Operator precedence and associativity\n\n`#left` and `#right` directives allow you to define precedence and associativiry of symbols. `#left`/`#right` each assign the left/right associativity to symbols.\n\nIf you want to change precedence, `#assign` directive helps you. `#assign` directive changes only precedence, not associativity.\n\nWhen the right-most terminal symbol of an alternative has precedence or associativity defined explicitly, the alternative inherits its precedence and associativity.\n\n`#prec` directive assigns the same precedence as a specified symbol to an alternative and disables associativity.\n\nYou can define an ordered symbol with the form `$\u003cID\u003e`. The ordered symbol is an identifier having only precedence, and you can use it in `#prec` directive applied to an alternative. The ordered symbol helps you to resolve shift/reduce conflicts without terminal symbol definitions.\n\nThe grammar for simple four arithmetic operations and assignment expression can be defined as follows:\n\n```\n#name example;\n\n#prec (\n\t#assign $uminus\n\t#left mul div\n\t#left add sub\n\t#right assign\n);\n\nexpr\n\t: expr add expr\n\t| expr sub expr\n\t| expr mul expr\n\t| expr div expr\n\t| id assign expr\n\t| int\n\t| sub int #prec $uminus // This `sub` means a unary minus symbol.\n\t| id\n\t;\n\nws #skip\n\t: \"[\\u{0009}\\u{0020}]+\";\nint\n\t: \"0|[1-9][0-9]*\";\nid\n\t: \"[a-z_][0-9a-z_]*\";\nadd\n\t: '+';\nsub\n\t: '-';\nmul\n\t: '*';\ndiv\n\t: '/';\nassign\n\t: '=';\n```\n\n`#left` and `#right` can appear multiple times, and the first symbols applied to will have the highest precedence. That is, `mul` and `div` have the highest precedence, and `assign` has the lowest precedence.\n\n⚠️ In many Yacc-like tools, the last symbols defined have the highest precedence. Not that in vartan, it is the opposite.\n\nWhen you compile the above grammar, some conflicts occur. However, vartan can resolve the conflicts following `#left`, `#right`, and `#prec`.\n\n```\n$ echo -n 'foo = bar = x * -1 / -2' | vartan parse example.json\nexpr\n├─ id \"foo\"\n├─ assign \"=\"\n└─ expr\n   ├─ id \"bar\"\n   ├─ assign \"=\"\n   └─ expr\n      ├─ expr\n      │  ├─ expr\n      │  │  └─ id \"x\"\n      │  ├─ mul \"*\"\n      │  └─ expr\n      │     ├─ sub \"-\"\n      │     └─ int \"1\"\n      ├─ div \"/\"\n      └─ expr\n         ├─ sub \"-\"\n         └─ int \"2\"\n```\n\nIncidentally, using no directives, you can define the above example as the following grammar:\n\n```\n#name example;\n\nexpr\n\t: id assign expr\n\t| arithmetic\n\t;\n\narithmetic\n\t: arithmetic add term\n\t| arithmetic sub term\n\t| term\n\t;\nterm\n\t: term mul factor\n\t| term div factor\n\t| factor\n\t;\nfactor\n\t: int\n\t| sub int\n\t| id\n\t;\n\nws #skip\n\t: \"[\\u{0009}\\u{0020}]+\";\nint\n\t: \"0|[1-9][0-9]*\";\nid\n\t: \"[a-z_][0-9a-z_]*\";\nadd\n\t: '+';\nsub\n\t: '-';\nmul\n\t: '*';\ndiv\n\t: '/';\nassign\n\t: '=';\n```\n\nThis grammar expresses precedence and associativity by nesting production rules instead of directives. Also, no conflicts occur in compiling this grammar.\n\nHowever, the more production rules you define, the more time syntax analysis needs. A structure of a mechanically generated parse tree is also more complex, but you can improve the structure of the parse tree using `#ast` directive.\n\n```\n$ echo -n 'foo = bar = x * -1 / -2' | vartan parse example.json\nexpr\n├─ id \"foo\"\n├─ assign \"=\"\n└─ expr\n   ├─ id \"bar\"\n   ├─ assign \"=\"\n   └─ expr\n      └─ arithmetic\n         └─ term\n            ├─ term\n            │  ├─ term\n            │  │  └─ factor\n            │  │     └─ id \"x\"\n            │  ├─ mul \"*\"\n            │  └─ factor\n            │     ├─ sub \"-\"\n            │     └─ int \"1\"\n            ├─ div \"/\"\n            └─ factor\n               ├─ sub \"-\"\n               └─ int \"2\"\n```\n\n### Error recovery\n\nBy default, a parser will stop syntax analysis on a syntax error. If you want to continue semantic actions after syntax errors occur, you can use an `error` symbol and a `#recover` directive.\n\nWhen a syntax error occurs, the parser pops states from a state stack. If a state containing the `error` symbol appears, a parser stops popping the states. It then shifts the `error` symbol and resumes syntax analysis.\n\nConsider grammar of simple equivalent expressions.\n\n```\n#name example;\n\neq_exprs\n\t: eq_exprs eq_expr #ast eq_exprs... eq_expr\n\t| eq_expr\n\t;\neq_expr\n\t: name eq int semi_colon #ast name int\n\t| error semi_colon       #recover\n\t;\n\nws #skip\n\t: \"[\\u{0009}\\u{0020}]\";\neq\n\t: '=';\nsemi_colon\n\t: ';';\nname\n\t: \"[a-z_][0-9a-z_]*\";\nint\n\t: \"0|[1-9][0-9]*\";\n```\n\nThe alternative `error semi_colon` traps a syntax error and discards symbols until the parser shifts `';'`. When the parser shifts `';'` and reduces `eq_expr`, the parser will recover from an error state immediately by the `#recover` directive.\n\nIn the following example, you can see the parser print syntax error messages and an AST. This result means the parser had continued syntax analysis and semantic actions even if syntax errors occurred.\n\n```\n$ vartan compile example.vartan -o example.json\n$ echo -n 'x; x =; x = 1;' | vartan parse example.json\neq_exprs\n├─ eq_expr\n│  ├─ error\n│  └─ semi_colon \";\"\n├─ eq_expr\n│  ├─ error\n│  └─ semi_colon \";\"\n└─ eq_expr\n   ├─ name \"x\"\n   └─ int \"1\"\n1:2: unexpected token: ';' (semi_colon): expected: eq\n1:7: unexpected token: ';' (semi_colon): expected: int\n```\n\n### Regular Expression\n\n⚠️ vartan doesn't allow you to use some code points. See [Unavailable Code Points](#unavailable-code-points).\n\n#### Composites\n\nConcatenation and alternation allow you to combine multiple characters or multiple patterns into one pattern.\n\n| Pattern    | Matches        |\n|------------|----------------|\n| `abc`      | `abc`          |\n| `abc\\|def` | `abc` or `def` |\n\n#### Single Characters\n\nIn addition to using ordinary characters, there are other ways to represent a single character:\n\n* dot expression\n* bracket expressions\n* code point expressions\n* character property expressions\n* escape sequences\n\n##### Dot Expression\n\nThe dot expression matches any one chracter.\n\n| Pattern | Matches           |\n|---------|-------------------|\n| `.`     | any one character |\n\n##### Bracket Expressions\n\nThe bracket expressions are represented by enclosing characters in `[ ]` or `[^ ]`. `[^ ]` is negation of `[ ]`. For instance, `[ab]` matches one of `a` or `b`, and `[^ab]` matches any one character except `a` and `b`.\n\n| Pattern  | Matches                                          |\n|----------|--------------------------------------------------|\n| `[abc]`  | `a`, `b`, or `c`                                 |\n| `[^abc]` | any one character except `a`, `b`, and `c`       |\n| `[a-z]`  | one in the range of `a` to `z`                   |\n| `[a-]`   | `a` or `-`                                       |\n| `[-z]`   | `-` or `z`                                       |\n| `[-]`    | `-`                                              |\n| `[^a-z]` | any one character except the range of `a` to `z` |\n| `[a^]`   | `a` or `^`                                       |\n\n##### Code Point Expressions\n\nThe code point expressions match a character that has a specified code point. The code points consists of a four or six digits hex string.\n\n| Pattern      | Matches                     |\n|--------------|-----------------------------|\n| `\\u{000A}`   | U+000A (LF)                 |\n| `\\u{3042}`   | U+3042 (hiragana `あ`)      |\n| `\\u{01F63A}` | U+1F63A (grinning cat `😺`) |\n\n##### Character Property Expressions\n\nThe character property expressions match a character that has a specified character property of the Unicode. Currently, vartan supports `General_Category`, `Script`, `Alphabetic`, `Lowercase`, `Uppercase`, and `White_Space`. When you omitted the equal symbol and a right-side value, vartan interprets a symbol in `\\p{...}` as the `General_Category` value.\n\n| Pattern                       | Matches                                                |\n|-------------------------------|--------------------------------------------------------|\n| `\\p{General_Category=Letter}` | any one character whose `General_Category` is `Letter` |\n| `\\p{gc=Letter}`               | the same as `\\p{General_Category=Letter}`              |\n| `\\p{Letter}`                  | the same as `\\p{General_Category=Letter}`              |\n| `\\p{l}`                       | the same as `\\p{General_Category=Letter}`              |\n| `\\p{Script=Latin}`            | any one character whose `Script` is `Latin`            |\n| `\\p{Alphabetic=yes}`          | any one character whose `Alphabetic` is `yes`          |\n| `\\p{Lowercase=yes}`           | any one character whose `Lowercase` is `yes`           |\n| `\\p{Uppercase=yes}`           | any one character whose `Uppercase` is `yes`           |\n| `\\p{White_Space=yes}`         | any one character whose `White_Space` is `yes`         |\n\n##### Escape Sequences\n\nAs you escape the special character with `\\`, you can write a rule that matches the special character itself.\nThe following escape sequences are available outside of bracket expressions.\n\n| Pattern | Matches |\n|---------|---------|\n| `\\.`    | `.`     |\n| `\\?`    | `?`     |\n| `\\*`    | `*`     |\n| `\\+`    | `+`     |\n| `\\(`    | `(`     |\n| `\\)`    | `)`     |\n| `\\[`    | `[`     |\n| `\\\\|`   | `\\|`    |\n| `\\\\`    | `\\`     |\n\nThe following escape sequences are available inside bracket expressions.\n\n| Pattern | Matches |\n|---------|---------|\n| `\\^`    | `^`     |\n| `\\-`    | `-`     |\n| `\\]`    | `]`     |\n\n#### Repetitions\n\nThe repetitions match a string that repeats the previous single character or group.\n\n| Pattern | Matches          |\n|---------|------------------|\n| `a*`    | zero or more `a` |\n| `a+`    | one or more `a`  |\n| `a?`    | zero or one `a`  |\n\n#### Grouping\n\n`(` and `)` groups any patterns.\n\n| Pattern     | Matches                                         |\n|-------------|-------------------------------------------------|\n| `a(bc)*d`   | `ad`, `abcd`, `abcbcd`, and so on               |\n| `(ab\\|cd)+` | `ab`, `cd`, `abcd`, `cdab`, `abcdab`, and so on |\n\n#### Unavailable Code Points\n\nLexical specifications and source files to be analyzed cannot contain the following code points.\n\nWhen you write a pattern that implicitly contains the unavailable code points, vartan will automatically generate a pattern that doesn't contain the unavailable code points and replaces the original pattern. However, when you explicitly use the unavailable code points (like `\\u{U+D800}` or `\\p{General_Category=Cs}`), vartan will occur an error.\n\n* surrogate code points: U+D800..U+DFFF\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnihei9%2Fvartan","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnihei9%2Fvartan","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnihei9%2Fvartan/lists"}