{"id":15293344,"url":"https://github.com/teliosdev/yoga","last_synced_at":"2025-03-24T13:44:31.434Z","repository":{"id":17921248,"uuid":"20889319","full_name":"teliosdev/yoga","owner":"teliosdev","description":"A library for ruby parsing assistance.","archived":false,"fork":false,"pushed_at":"2017-03-27T09:50:16.000Z","size":133,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-02-24T05:03:21.272Z","etag":null,"topics":["ll-parser","parsing","ruby","scanner","yoga"],"latest_commit_sha":null,"homepage":"http://www.rubydoc.info/github/medcat/yoga/master","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/teliosdev.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2014-06-16T15:08:08.000Z","updated_at":"2017-03-10T06:43:29.000Z","dependencies_parsed_at":"2022-08-04T21:30:15.087Z","dependency_job_id":null,"html_url":"https://github.com/teliosdev/yoga","commit_stats":null,"previous_names":["medcat/yoga"],"tags_count":8,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/teliosdev%2Fyoga","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/teliosdev%2Fyoga/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/teliosdev%2Fyoga/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/teliosdev%2Fyoga/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/teliosdev","download_url":"https://codeload.github.com/teliosdev/yoga/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245284409,"owners_count":20590306,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ll-parser","parsing","ruby","scanner","yoga"],"created_at":"2024-09-30T16:46:23.018Z","updated_at":"2025-03-24T13:44:31.412Z","avatar_url":"https://github.com/teliosdev.png","language":"Ruby","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Yoga\n[![Build Status][build-status]][build-status-link] [![Coverage Status][coverage-status]][coverage-status-link]\n\nA helper for your Ruby parsers.  This adds helper methods to make parsing\n(and scanning!) easier and more structured.  If you're looking for an LALR\nparser generator, that isn't this.  This is designed to help you construct\nRecursive Descent parsers - which are solely LL(k).  If you want an LALR parser\ngenerator, see [_Antelope_](https://github.com/medcat/antelope) or\n[Bison](https://www.gnu.org/software/bison/).\n\nYoga requires [Mixture](https://github.com/medcat/mixture) for parser node\nattributes.  However, the use of the parser nodes included with Yoga are\ncompletely optional.\n\n## Installation\n\nAdd this line to your application's Gemfile:\n\n```ruby\ngem \"yoga\"\n```\n\nAnd then execute:\n\n    $ bundle\n\n## Usage\n\nTo begin your parser, you will first have to create a scanner.  A scanner\ntakes the source text and generates \"tokens.\"  These tokens are abstract\nrepresentations of the source text of the document.  For example, for the\ntext `class A do`, you could have the tokens `:class`, `:CNAME`, and `:do`.\nThe actual names of the tokens are completely up to you.  These token names\nare later used in the parser to set up expectations - for example, for the\ndefinition of a class, you could expect a `:class`, `:CNAME`, and a `:do`\ntoken.\n\nEssentially, the scanner breaks up the text into usable, bite-sized pieces\nfor the parser to chomp on.  Here's what scanner may look like:\n\n```ruby\nmodule MyLanguage\n  class Scanner\n    # All of the behavior from Yoga for scanners.  This provides the\n    # `match/2` method, the `call/0` method, the `match_line/1` method,\n    # the `location/1` method, and the `emit/2` method.  The major ones that\n    # are used are the `match/2`, the `call/0`, and the `match_line/1`\n    # methods.\n    include Yoga::Scanner\n\n    # This must be implemented.  This is called for the next token.  This\n    # should only return a Token, or true.\n    def scan\n      # Match with a string value escapes the string, then turns it into a\n      # regular expression.\n      match(\"[\") || match(\"]\") ||\n      # Match with a symbol escapes the symbol, and turns it into a regular\n      # expression, suffixing it with `symbol_negative_assertion`.  This is\n      # to prevent issues with identifiers and keywords.\n      match(:class) || match(:func) ||\n      # With a regular expression, it's matched exactly.  However, a token\n      # name is highly recommended.\n      match(/[a-z][a-zA-Z0-9_]*[!?=]?/, :IDENT)\n    end\n  end\nend\n```\n\nAnd that's it!  You now have a fully functioning scanner.  In order to use it,\nall you have to do is this:\n\n```ruby\nsource = \"class alpha [func a []]\"\nMyLanguage::Scanner.new(source).call # =\u003e #\u003cEnumerable ...\u003e\n```\n\nNote that `Scanner#call` returns an enumerable.  `#call` is aliased as `#each`.\nWhat this means is that tokens aren't generated until they're requested by the\nparser - each token is generated from the source incrementally.  If you want\nto retrieve all of the tokens immediately, you have to first convert it into\na string, or perform some other operation on the enumerable (since it isn't\nlazy):\n\n```ruby\nMyLanguage::Scanner.new(source).call.to_a # =\u003e [...]\n```\n\nThe scanner also automatically adds location information to all of the tokens.\nThis is handled automatically by `match/2` and `emit/2` - the only issue being\nthat all regular expressions **must not** include a newline.  Newlines should\nbe matched with `match_line/1`; if lines must be emitted as a token, you can\npass the kind of token to emit to `match_line/1` using the `kind:` keyword.\n\nYou may notice that all of the tokens have `\u003canon\u003e` set as the location's file.\nThis is the default location, which is provided to the initializer:\n\n```ruby\nMyLanguage::Scanner.new(source, \"foo\").call.first.location.to_s # =\u003e \"foo:1.1-6\"\n```\n\nParsers are a little bit more complicated.  Before we can pull up the parser,\nlet's define a grammar and some node classes.\n\n```\n; This is the grammar.\n\u003croot\u003e = *\u003cstatement\u003e\n\u003cstatement\u003e = \u003cexpression\u003e ';'\n\u003cexpression\u003e = \u003cexpression\u003e \u003cop\u003e \u003cexpression\u003e\n\u003cexpression\u003e /= \u003cint\u003e ; here, \u003cint\u003e is defined by the scanner.\n\u003cop\u003e = '+' / '-' / '*' / '/' / '^' / '%' / '='\n```\n\n```ruby\nmodule MyLanguage\n  class Parser\n    class Root \u003c Yoga::Node\n      # An attribute on the node.  This is required for Yoga nodes since the\n      # update syntax requires them.  The type for the attribute is optional.\n      attribute :statements, type: [Yoga::Node]\n    end\n\n    class Expression \u003c Yoga::Node\n    end\n\n    class Operation \u003c Expression\n      attribute :operator, type: ::Symbol\n      attribute :left, type: Expression\n      attribute :right, type: Expression\n    end\n\n    class Literal \u003c Expression\n      attribute :value, type: ::Integer\n    end\n  end\nend\n```\n\nWith those out of the way, let's take a look at the parser itself.\n\n```ruby\nmodule MyLanguage\n  class Parser\n    # This provides all of the parser helpers.  This is the same as adding\n    # `Yoga::Parser::Helpers` as an include statement as well.\n    include Yoga::Parser\n\n    # Like the `scan/0` method on the scanner, this must be implemented.  This\n    # is the entry point for the parser.  However, public usage should use the\n    # `call/0` method.  This should return a node of some sort.\n    def parse_root\n      # This \"collects\" a series of nodes in sequence.  It iterates until it\n      # reaches the `:EOF` token (in this case).  The first parameter to\n      # collect is the \"terminating token,\" and can be any value that\n      # `expect/1` or `peek?/1` accepts.  The second, optional parameter to\n      # collect is the \"joining token,\" and is required between each node.\n      # We're not using the semicolon as a joining token because that is\n      # required for _all_ statements.  The joining token can be used for\n      # things like argument lists.  The parameter can be any value that\n      # `expect/1` or `peek?/1` accepts.\n      children = collect(:EOF) { parse_statement }\n\n      # \"Unions\" the location of all of the statements in the list.\n      location = children.map(\u0026:location).inject(:union)\n      Parser::Root.new(statements: children, location: location)\n    end\n\n    # Parses a statement.  This is the same as the \u003cstatement\u003e rule as above.\n    def parse_statement\n      expression = parse_expression\n      # This says that the next token should be a semicolon.  If the next token\n      # isn't, it throws an error with a detailed error message, denoting\n      # what was expected (in this case, a semicolon), what was given, and\n      # where the error was located in the source file.\n      expect(:\";\")\n\n      expression\n    end\n\n\n    # A switch statement, essentially.  This is defined beforehand to make it\n    # _faster_ (not really; it's just useful).  The first parameter to the\n    # switch function is the name of the switch.  This is used later to\n    # actually perform the switch; it is also used to define a first set with\n    # the allowed tokens for the switch.  The second parameter defines a key\n    # value pair.  The keys are the tokens that are allowed; a symbol or an\n    # array of symbols can be used.  The value is the block or the method that\n    # is executed upon encountering that token.\n    switch(:Operation,\n      \"=\": proc { |left| parse_operation(:\"=\", left) },\n      \"+\": proc { |left| parse_operation(:\"+\", left) },\n      \"-\": proc { |left| parse_operation(:\"-\", left) },\n      \"*\": proc { |left| parse_operation(:\"*\", left) },\n      \"/\": proc { |left| parse_operation(:\"/\", left) },\n      \"^\": proc { |left| parse_operation(:\"^\", left) },\n      \"%\": proc { |left| parse_operation(:\"%\", left) })\n\n    def parse_expression\n      # Parse a literal.  All expressions must contain a literal of some sort;\n      # we're just going to use a numeric literal here.\n      left = parse_expression_literal\n\n      # Whenever the `.switch` function is called, it creates a\n      # \"first set\" that can be used like this.  The first set consists of\n      # a set of tokens that are allowed for the switch statement.  In this\n      # case, it just makes sure that the next token is an operator.  If it\n      # is, it parses it as an operation.\n      if peek?(first(:Operation))\n        # Uses the switch defined below.  If a token is found as a key, its\n        # block is executed; otherwise, it errors, giving a detailed error of\n        # what was expected.\n        switch(:Operation, left)\n      else\n        left\n      end\n    end\n\n    def parse_operation(op, left)\n      token = expect(op)\n      right = parse_expression\n\n      Parser::Operation.new(left: left, op: op, right: right, location:\n        left.location | op.location | right.location)\n    end\n\n    def parse_expression_literal\n      token = expect(:NUMERIC)\n      Parser::Literal.new(value: token.value, location: token.location)\n    end\n  end\nend\n```\n\nThis parser can then be used as such:\n\n```ruby\nsource = \"a = 2;\\nb = a + 2;\\n\"\nscanner = MyLanguage::Scanner.new(source).call\nMyLanguage::Parser.new(scanner).call # =\u003e #\u003cMyLanguage::Parser::Root ...\u003e\n```\n\nThat's about it!  If you have any questions, you can email me at\n\u003cjeremy.rodi@medcat.me\u003e, open an issue, or do what you like.\n\nFor more documentation, see [the Documentation][documentation] - Yoga has a\nrequirement of 100% documentation.\n\n## Contributing\n\nBug reports and pull requests are welcome on GitHub at\n\u003chttps://github.com/medcat/yoga\u003e. This project is intended to be a safe,\nwelcoming space for collaboration, and contributors are expected to adhere to\nthe [Contributor Covenant](http://contributor-covenant.org) code of conduct.\n\n[build-status]: https://travis-ci.org/medcat/yoga.svg?branch=master\n[documentation]: http://www.rubydoc.info/github/medcat/yoga/master\n[coverage-status]: https://coveralls.io/repos/github/medcat/yoga/badge.svg?branch=master\n[build-status-link]: https://travis-ci.org/medcat/yoga\n[coverage-status-link]: https://coveralls.io/github/medcat/yoga?branch=master\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fteliosdev%2Fyoga","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fteliosdev%2Fyoga","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fteliosdev%2Fyoga/lists"}