{"id":17942623,"url":"https://github.com/petercamilleri/lexical_analyzer","last_synced_at":"2026-07-13T16:31:37.883Z","repository":{"id":56881047,"uuid":"145737045","full_name":"PeterCamilleri/lexical_analyzer","owner":"PeterCamilleri","description":"The lexical analyzer component for the Ruby Compiler Toolkit Project (RCTP)","archived":false,"fork":false,"pushed_at":"2021-05-19T15:33:56.000Z","size":64,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-03-11T00:08:46.131Z","etag":null,"topics":["compiler","lexical-analysis","ruby","rubygem"],"latest_commit_sha":null,"homepage":null,"language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/PeterCamilleri.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-08-22T16:47:41.000Z","updated_at":"2023-03-24T05:39:28.000Z","dependencies_parsed_at":"2022-08-20T22:31:22.062Z","dependency_job_id":null,"html_url":"https://github.com/PeterCamilleri/lexical_analyzer","commit_stats":null,"previous_names":[],"tags_count":9,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PeterCamilleri%2Flexical_analyzer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PeterCamilleri%2Flexical_analyzer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PeterCamilleri%2Flexical_analyzer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PeterCamilleri%2Flexical_analyzer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/PeterCamilleri","download_url":"https://codeload.github.com/PeterCamilleri/lexical_analyzer/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247008791,"owners_count":20868424,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["compiler","lexical-analysis","ruby","rubygem"],"created_at":"2024-10-29T03:06:31.755Z","updated_at":"2025-10-08T18:04:38.132Z","avatar_url":"https://github.com/PeterCamilleri.png","language":"Ruby","funding_links":[],"categories":[],"sub_categories":[],"readme":"# LexicalAnalyzer\n\nThe lexical analyzer is a component of the Ruby Compiler Toolkit Project (rctp)\nthat scans an input text against an array of rules and generating the lexical\ntokens that it detects. This process is shown below:\n\n![The Lexical Process](./images/lexical_process.png)\n\nIn general, each time the lexical_analyzer receives the \"get\" message, it tries\nto extract another token from the source text. As such, the lexical analyzer\ngem component is the first stage of the compilation process for a compiler\nbuilt using the rctp. With its array of lexical rules it provides the language\ntokens needed to operate the compiler's parser.\n\nThe lexical analyzer is normally used in conjunction with a parse queue object\nwhich handles queuing of tokens and back tracking of the compile process when\nneeded. In the rcpt this is done by the gem\n[parse_queue](https://github.com/PeterCamilleri/parse_queue).\n\n## Installation\n\nAdd this line to your application's Gemfile:\n\n```ruby\ngem 'lexical_analyzer'\n```\n\nAnd then execute:\n\n    $ bundle\n\nOr install it yourself as:\n\n    $ gem install lexical_analyzer\n\n## Usage\n\nA lexical analyzer object is created with two keyword parameters, the text to\nbe analyzed and an array of rules for performing that task.\n\n```ruby\nlexical_analyser = LexicalAnalyzer.new(text: text, rules: rules)\n\ntoken = lexical_analyser.get\n\n```\n\nIt is sometimes desirable to reuse an existing lexical analyzer. This can be\ndone with the renew method.\n\n```ruby\nlexical_analyser.renew(text: new_text)\n\ntoken = lexical_analyser.get\n\n```\n\nNote: The renew method takes the same arguments as the new method, text and an\narray of rules. If these are omitted, the default is to leave that value\nunchanged. The renew method returns the updated lexical analyzer just like the\nnew method returns the newly created one.\n\n#### Rules\n\nThe rules are an array of LexicalRule objects. Each consists of a symbol, a\nregular expression, and an optional action.\n\n```ruby\n# Rule with default block returns [:equality, \"==\"] on a match.\nLexicalRule.new(:equality, /\\A==/)\n\n# Rule with block equivalent to the default.\nLexicalRule.new(:lparen, /\\A\\(/) {|value| [symbol, value]}\n\n# Rule with an ignore block, ignores matches.\nLexicalRule.new(:spaces, /\\A\\s+/) {|_value| false }\n\n# Rule with an integer block returns [:integer, an_integer] on a match.\nLexicalRule.new(:integer, /\\A\\d+/) {|value| [symbol, value.to_i] }\n\n# Rule with a block that expands of to a sub-rule. Returns the value of the\n# lexical analyzer in the captured variable ka.\nLexicalRule.new(:identifier, /\\A[a-zA-Z_]\\w*(?=\\W|$|\\z)/) {|value| ka.renew(text: value).get}\n```\n\nNotes:\n\n* The regular expression must begin with a \\A clause to ensure correct\noperation of the analyzer.\n* An exception to the above is the use of the expression /.+/ at the end of the\nrule list as a sort of lexical \"else\" catch-all clause.\n* The order of rules is important. For example, if there are two rules\nlooking for \"==\" and \"=\" respectively, if the \"=\" is ahead of the \"==\" rule\nin the array the \"==\" rule will never trigger and the analysis will be\nincorrect.\n* The method LexicalRule#symbol is a read accessor for the symbol property of\nthe lexical rule.\n\n#### Tokens\n\nThe output token is an array with two elements.\n\ntoken[0] - the symbol extracted from the rule that generated this token.\n\ntoken[1] - the text that generated this token or its value.\n\n\n#### Example\n\nThe test file \"lexical_analyzer_test.rb\" has the method\ntest_some_lexical_analyzing that is a really good example of this gem in\naction.\n\n## Contributing\n\n#### Plan A\n\n1. Fork it ( https://github.com/PeterCamilleri/lexical_analyzer/fork )\n2. Create your feature branch (`git checkout -b my-new-feature`)\n3. Commit your changes (`git commit -am 'Add some feature'`)\n4. Push to the branch (`git push origin my-new-feature`)\n5. Create a new Pull Request\n\n#### Plan B\n\nGo to the GitHub repository and raise an\n[issue](https://github.com/PeterCamilleri/lexical_analyzer/issues)\n calling attention to some\naspect that could use some TLC or a suggestion or an idea.\n\n## License\n\nThe gem is available as open source under the terms of the\n[MIT License](./LICENSE.txt).\n\n## Code of Conduct\n\nEveryone interacting in the LexicalAnalyzer project’s codebases, issue\ntrackers, chat rooms and mailing lists is expected to follow the\n[code of conduct](./CODE_OF_CONDUCT.md).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpetercamilleri%2Flexical_analyzer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpetercamilleri%2Flexical_analyzer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpetercamilleri%2Flexical_analyzer/lists"}