{"id":16987240,"url":"https://github.com/yorickpeterse/ruby-ll","last_synced_at":"2025-03-22T15:30:50.897Z","repository":{"id":24432197,"uuid":"27833763","full_name":"yorickpeterse/ruby-ll","owner":"yorickpeterse","description":"An LL(1) parser generator for Ruby.","archived":false,"fork":false,"pushed_at":"2023-09-17T02:34:59.000Z","size":265,"stargazers_count":43,"open_issues_count":0,"forks_count":8,"subscribers_count":5,"default_branch":"main","last_synced_at":"2025-03-18T13:15:16.055Z","etag":null,"topics":["ll","parser","parser-generator","ruby"],"latest_commit_sha":null,"homepage":"","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mpl-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/yorickpeterse.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2014-12-10T18:34:00.000Z","updated_at":"2024-11-11T06:45:00.000Z","dependencies_parsed_at":"2024-10-28T13:23:30.823Z","dependency_job_id":"acc38ee5-be48-46eb-8110-6f2bce3d0589","html_url":"https://github.com/yorickpeterse/ruby-ll","commit_stats":null,"previous_names":[],"tags_count":9,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yorickpeterse%2Fruby-ll","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yorickpeterse%2Fruby-ll/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yorickpeterse%2Fruby-ll/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yorickpeterse%2Fruby-ll/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/yorickpeterse","download_url":"https://codeload.github.com/yorickpeterse/ruby-ll/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244978447,"owners_count":20541857,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ll","parser","parser-generator","ruby"],"created_at":"2024-10-14T02:48:43.048Z","updated_at":"2025-03-22T15:30:50.539Z","avatar_url":"https://github.com/yorickpeterse.png","language":"Ruby","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ruby-ll\n\nruby-ll is a high performance LL(1) table based parser generator for Ruby. The\nparser driver is written in C/Java to ensure good runtime performance, the\ncompiler is written entirely in Ruby.\n\nruby-ll was written to serve as a fast and easy to use alternative to\n[Racc][racc] for the various parsers used in [Oga][oga]. However, ruby-ll isn't\nlimited to just Oga, you can use it to write a parser for any language that can\nbe represented using an LL(1) grammar.\n\nruby-ll is self-hosting, this allows one to use ruby-ll to modify its own\nparser. Self-hosting was achieved by bootstrapping the parser using a Racc\nparser that outputs the same AST as the ruby-ll parser. The Racc parser remains\nin the repository for historical purposes and in case it's ever needed again, it\ncan be found in [bootstrap/parser.y](lib/ll/bootstrap/parser.y).\n\nFor more information on LL parsing, see\n\u003chttps://en.wikipedia.org/wiki/LL_parser\u003e.\n\n## Features\n\n* Support for detecting first/first and first/follow conflicts.\n* clang-like error/warning messages to ease debugging parsers.\n* High performance and a low memory footprint.\n* Support for the `*`, `+` and `?` operators.\n\n## Requirements\n\n| Ruby     | Required\n|:---------|:--------------\n| MRI      | \u003e= 2.6.0\n| JRuby    | \u003e= 9.0\n| Rubinius | Not supported\n| Maglev   | Not supported\n| Topaz    | Not supported\n| mruby    | Not supported\n\nFor MRI you'll need a C90 compatible compiler such as clang or gcc. For JRuby\nyou don't need any compilers to be installed as the .jar is packaged with the\nGem itself.\n\nWhen hacking on ruby-ll you'll also need to have the following installed:\n\n* Ragel 6 for building the grammar lexer\n* javac for building the JRuby extension\n\n## Installation\n\nruby-ll can be installed from RubyGems:\n\n    gem install ruby-ll\n\n## Usage\n\nThe CLI takes a grammar input file (see below for the exact syntax) with the\nextension `.rll` and turns it into a corresponding Ruby file. For example:\n\n    ruby-ll lib/my-gem/parser.rll\n\nThis would result in the parser being written to `lib/my-gem/parser.rb`. If you\nwant to customize the output path you can do so using the `-o` / `--output`\noptions:\n\n    ruby-ll lib/my-gem/parser.rll -o lib/my-gem/my-parser.rb\n\nBy default ruby-ll adds various `require` calls to ensure you can load the\nparser _without_ having to load all of ruby-ll (e.g. the compiler code). If you\nwant to disable this behaviour you can use the `--no-requires` option when\nprocessing a grammar:\n\n    ruby-ll lib/my-gem/parser.rll --no-requires\n\nOnce generated you can use the parser class like any other parser. To start\nparsing simply call the `parse` method:\n\n    parser = MyGem::Parser.new\n\n    parser.parse\n\nThe return value of this method is whatever the root rule (= the first rule\ndefined) returned.\n\n## Parser Input\n\nFor a parser to work it must receive its input from a separate lexer. To pass\ninput to the parser you must define the method `each_token` in an `%inner`\nblock. This method should yield an Array containing two values:\n\n1. The token type as a Symbol (e.g. `:T_STRING`)\n2. The token value, this can be any type of value\n\nThe last Array yielded by this method should be `[-1, -1]` to signal the end of\nthe input. For example:\n\n    def each_token\n      yield [:T_STRING, 'foo']\n      yield [:T_STRING, 'bar']\n      yield [-1, -1]\n    end\n\n## Error Handling\n\nParser errors are handled by `LL::Driver#parser_error`. By default this method\nraises an instance of `LL::ParserError` with a message depending on the current\nparser context and input. If you want to customize this behaviour simply\noverwrite the method (e.g. in an `%inner` block).\n\n## Grammar Syntax\n\nThe syntax of a ruby-ll grammar file is fairly simple and consists out of\ndirectives, rules, comments and code blocks.\n\nDirectives can be seen as configuration options, for example to set the name of\nthe parser class. Rules are, well, the parsing rules. Code blocks can be used to\nassociate Ruby code with either a branch of a rule or a certain section of the\nparser (the header or its inner body).\n\nDirectives and rules must be terminated using a semicolon, this is not needed\nfor `%inner` / `%header` blocks.\n\nFor a full example, see ruby-ll's own parser located at\n[lib/ll/parser.rll](lib/ll/parser.rll).\n\n### Comments\n\nComments start with a hash (`#`) sign and continue until the end of the line,\njust like Ruby. Example:\n\n    # Some say comments are a code smell.\n\n### %name\n\nThe `%name` directive is used to set the full name/namespace of the parser\nclass. The name consists out of a single identifier or multiple identifiers\nseparated by `::` (just like Ruby). Some examples:\n\n    %name A;\n    %name A::B;\n    %name A::B::C;\n\nThe last identifier is used as the actual class name. This class will be nested\ninside a module for every other segment leading up to the last one. For example,\nthis:\n\n    %name A;\n\nGets turned into this:\n\n    class A \u003c LL::Driver\n\n    end\n\nWhile this:\n\n    %name A::B::C;\n\nGets turned into this:\n\n    module A\n    module B\n    class C \u003c LL::Driver\n\n    end\n    end\n    end\n\nBy nesting the parser class in modules any constants in the scope can be\nreferred to without requiring the use of a full namespace. For example, the\nconstant `A::B::X` can just be referred to as `X` in the above example.\n\nMultiple calls to this directive will result in previous values being\noverwritten.\n\n### %terminals\n\nThe `%terminals` directive is used to list one or more terminals of the grammar.\nEach terminal is an identifier separated by a space. For example:\n\n    %terminals A B C;\n\nThis would define 3 terminals: `A`, `B` and `C`. While there's no specific\nrequirement as to how you name your terminals it's common practise to capitalize\nthem and prefix them with `T_`, like so:\n\n    %terminals T_A T_B T_C;\n\nMultiple calls to this directive will result in the terminals being appended to\nthe existing list.\n\n### %inner\n\nThe `%inner` directive can be used to specify a code block that should be placed\ninside the parser's body, just after the section containing all parsing tables.\nThis directive should be used for adding custom methods and such to the parser.\nFor example:\n\n    %inner\n    {\n      def initialize(input)\n        @input = input\n      end\n    }\n\nThis would result in the following:\n\n    class A \u003c LL::Driver\n      def initialize(input)\n        @input = input\n      end\n    end\n\nCurly braces can either be placed on the same line as the `%inner` directive or\non a new line, it's up to you.\n\nUnlike regular directives this directive should not be terminated using a\nsemicolon.\n\n### %header\n\nThe `%header` directive is similar to the `%inner` directive in that it can be\nused to add a code block to the parser. The code of this directive is placed\njust before the `class` definition of the parser. This directive can be used to\nadd documentation to the parser class. For example:\n\n    %header\n    {\n      # Hello world\n    }\n\nThis would result in the following:\n\n    # Hello world\n    class A \u003c LL::Driver\n    end\n\n### Rules\n\nRules consist out of a name followed by an equals sign (`=`) followed by 1 or\nmore branches. Each branch is separated using a pipe (`|`). A branch can consist\nout of 1 or many steps, or an epsilon. Branches can be followed by a code block\nstarting with `{` and ending with `}`. A rule must be terminated using a\nsemicolon.\n\nAn epsilon is represented as a single underscore (`_`) and is used to denote a\nwildcard/nothingness.\n\nA simple example:\n\n    %terminals A;\n\n    numbers = A | B;\n\nHere the rule `numbers` is defined and has two branches. If we wanted a rule\nthat would match terminal `A` or nothing we'd use the following:\n\n    %terminals A;\n\n    numbers = A | _;\n\nCode blocks can also be added:\n\n    numbers\n      = A { 'A' }\n      | B { 'B' }\n      ;\n\nWhen the terminal `A` would be processed the returned value would be `'A'`, for\nterminal `B` the returned value would be `'B'`.\n\nCode blocks have access to an array called `val` which contains the values of\nevery step of a branch. For example:\n\n    numbers = A B { val };\n\nHere `val` would return `[A, B]`. Since `val` is just an Array you can also\nreturn specific elements from it:\n\n    numbers = A B { val[0] };\n\nValues returned by code blocks are passed to whatever other rule called it. This\nallows code blocks to be used for building ASTs and the likes.\n\nIf no explicit code block is defined then ruby-ll will generate one for you. If\na branch consists out of only a single step (e.g. `A = B;`) then only the first\nvalue is returned, otherwise all values are returned.\n\nThis means that in the following example the output will be whatever value `C`\ncontains:\n\n    A = B { p val[0] };\n    B = C;\n\nHowever, here the output would be `[C, D]` as the `B` rule's branch contains\nmultiple steps:\n\n    A = B { p val[0] };\n    B = C D;\n\nTo summarize (`# =\u003e` denotes the return value):\n\n    A = B;   # =\u003e B\n    A = B C; # =\u003e [B, C]\n\nYou can override this behaviour simply by defining your own code block.\n\nruby-ll parsers recurse into rules before unwinding, this means that the\ninner-most rule is processed first.\n\nBranches of a rule can also refer to other rules:\n\n    numbers    = A other_rule;\n    other_rule = B;\n\nThe value for `other_rule` in the `numbers` rule would be whatever the\n`other_rule` below it returns.\n\nThe grammar compiler adds errors whenever it encounters a rule with the same\nname as a terminal, as such the following is invalid:\n\n    %terminals A B;\n\n    A = B;\n\nIt's also an error to re-define an existing rule.\n\n### Operators\n\nGrammars can use two operators to define a sequence of terminals/non-terminals:\nthe star (`*`) and plus (`+`) operators. There's also the `?` (question)\noperator which can be used to indicate something as being optional.\n\nThe star operator indicates that something should occur 0 or more times. Here\nthe \"B\" identifier could occur 0 times, once, twice or many more times:\n\n    A = B*;\n\nThe plus operator indicates that something should occur at least once followed\nby any number of more occurrences. For example, this grammar states that \"B\"\nshould occur at least once but can also occur, say, 10 times:\n\n    A = B+;\n\nThe question operator can be used as an alternative to the following pattern:\n\n    # \"A\" or \"A C\"\n    A = B A_follow;\n\n    A_follow = C | _;\n\nUsing this operator you can simply write the following:\n\n    A = B C?;\n\nOperators can be applied either to a single terminal/rule or a series of\nterminals/rules grouped together using parenthesis. For example, both are\nperfectly valid:\n\n    A = B+;\n    A = (B C)+;\n\nWhen calling an operator on a single terminal/rule the corresponding entry in\nthe `val` array is simply set to the terminal/rule value. For example:\n\n    A = B+ { p val[0] };\n\nFor input `B B B` this would output `[B, B, B]`.\n\nHowever, when grouping multiple terminals/rules using parenthesis every\noccurrence is wrapped in an Array. For example:\n\n    A = (B C)+ { p val[0] };\n\nFor input `B C B C` this would output `[[B, C], [B, C]]`. To work around this\nyou can simply move the group of identifiers to its own rule and only return\nwhatever you need:\n\n    A  = A1+ { p val[0] };\n    A1 = B C { val[0] }; # only return \"B\"\n\nFor input `B C B C` this would output `[B, B]`.\n\n## Conflicts\n\nLL(1) grammars can have two kinds of conflicts in a rule:\n\n* first/first\n* first/follow\n\n### first/first\n\nA first/first conflict means that multiple branches of a rule start with the\nsame terminal, resulting in the parser being unable to choose what branch to\nuse. For example:\n\n    %terminals A B;\n\n    rule = A | A B;\n\nThis would result in the following output:\n\n    example.rll:5:1:error: first/first conflict, multiple branches start with the same terminals\n    rule = A | A B;\n    ^\n    example.rll:5:8:error: branch starts with: A\n    rule = A | A B;\n           ^\n    example.rll:5:12:error: branch starts with: A\n    rule = A | A B;\n               ^\n\nTo solve a first/first conflict you'll have to factor out the common left\nfactor. For example:\n\n    %name Example;\n\n    %terminals A B;\n\n    rule        = A rule_follow;\n    rule_follow = B | _;\n\nHere the `rule` rule starts with terminal `A` and can optionally be followed by\n`B`, without introducing any first/first conflicts.\n\n### first/follow\n\nA first/follow conflict occurs when a branch in a rule starts with an epsilon\nand is followed by one or more terminals and/or rules. An example of a\nfirst/follow conflict:\n\n    %name Example;\n\n    %terminals A B;\n\n    rule       = other_rule B;\n    other_rule = A | _;\n\nThis produces the following errors:\n\n    example.rll:5:14:error: first/follow conflict, branch can start with epsilon and is followed by (non) terminals\n    rule       = other_rule B;\n                 ^\n    example.rll:6:18:error: epsilon originates from here\n    other_rule = A | _;\n                     ^\n\nThere's no specific procedure to solving such a conflict other than simply\nremoving the starting epsilon.\n\n## Performance\n\nOne of the goals of ruby-ll is to be faster than existing parser generators,\nRacc in particular. How much faster ruby-ll will be depends on the use case. For\nexample, for the benchmark\n[benchmark/ll/simple\\_json\\_bench.rb](benchmark/l/simple_json_bench.rb) the\nperformance gains of ruby-ll over Racc are as following:\n\n| Ruby            | Speed |\n|:----------------|:------|\n| MRI 2.2         | 1.75x |\n| Rubinius 2.5.2  | 3.85x |\n| JRuby 1.7.18    | 6.44x |\n| JRuby 9000 pre1 | 7.50x |\n\nThis benchmark was run on a Thinkpad T520 laptop so it's probably best to run\nthe benchmark yourself to see how it behaves on your platform.\n\nDepending on the complexity of your parser you might end up with different\ndifferent numbers. The above metrics are simply an indication of the maximum\nperformance gain of ruby-ll compared to Racc.\n\n## Thread Safety\n\nParsers generated by ruby-ll share an internal, mutable state on a per instance\nbasis. As a result of this a single instance of your parser _can not_ be used by\nmultiple threads in parallel. If it wasn't for MRI's C API (specifically due to\nhow `rb_block_call` works) this wouldn't have been an issue.\n\nTo mitigate the above simply create a new instance of your parser every time you\nneed it and have the GC clean it up once you're done. This _will_ introduce a\nslight allocation overhead but it beats having to deal with race conditions.\n\n## License\n\nAll source code in this repository is subject to the terms of the Mozilla Public\nLicense, version 2.0 unless stated otherwise. A copy of this license can be\nfound the file \"LICENSE\" or at \u003chttps://www.mozilla.org/MPL/2.0/\u003e.\n\nThe following files are licensed under a different license:\n\n* ext/c/khash.h: MIT license (see source code)\n* ext/c/kvec.h: MIT license (see source code)\n\n[racc]: https://github.com/tenderlove/racc\n[oga]: https://github.com/yorickpeterse/oga\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyorickpeterse%2Fruby-ll","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fyorickpeterse%2Fruby-ll","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyorickpeterse%2Fruby-ll/lists"}