{"id":17942638,"url":"https://github.com/petercamilleri/parse_queue","last_synced_at":"2025-04-03T13:27:36.383Z","repository":{"id":56887476,"uuid":"145340152","full_name":"PeterCamilleri/parse_queue","owner":"PeterCamilleri","description":"An object queue for moving compiler tokens with nestable backtrack capability.","archived":false,"fork":false,"pushed_at":"2021-05-19T15:36:00.000Z","size":72,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-02-09T02:30:39.825Z","etag":null,"topics":["backtracking","compiler","queue","ruby","rubygem"],"latest_commit_sha":null,"homepage":null,"language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/PeterCamilleri.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-08-19T21:52:30.000Z","updated_at":"2023-03-24T05:39:27.000Z","dependencies_parsed_at":"2022-08-21T00:20:36.442Z","dependency_job_id":null,"html_url":"https://github.com/PeterCamilleri/parse_queue","commit_stats":null,"previous_names":[],"tags_count":6,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PeterCamilleri%2Fparse_queue","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PeterCamilleri%2Fparse_queue/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PeterCamilleri%2Fparse_queue/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PeterCamilleri%2Fparse_queue/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/PeterCamilleri","download_url":"https://codeload.github.com/PeterCamilleri/parse_queue/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247008776,"owners_count":20868423,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["backtracking","compiler","queue","ruby","rubygem"],"created_at":"2024-10-29T03:06:33.781Z","updated_at":"2025-04-03T13:27:36.354Z","avatar_url":"https://github.com/PeterCamilleri.png","language":"Ruby","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ParseQueue\n\nThe parse queue is a component in the Ruby Compiler Toolkit Project (RCTP). Its\nrole is to facilitate the movement of language tokens from one compiler phase\n(like the lexical analyzer) to the next one (like the parser). More than just a\nsimple queue, it supports backing up or falling back to earlier states allowing\nthe parser to try other paths in the syntax tree when one path runs into a\ndead end.\n\nThe parse queue was created to simplify the design of both the lexical analyzer\nand the parser. Parsers often have a built-in limited \"look-ahead\" of tokens.\nThis can be seen in parser names like LL(0), LL(1), LR(1), LALR(n), etc. The\nparse queue provides for a flexible look-ahead removing this burden from other\ncompiler components.\n\nThe operation of the parse queue is summarized in the following diagram:\n\n![parse queue](./images/queue.png)\n\n## Installation\n\nAdd this line to your application's Gemfile:\n\n```ruby\ngem 'parse_queue'\n```\n\nAnd then execute:\n\n    $ bundle\n\nOr install it yourself as:\n\n    $ gem install parse_queue\n\n## Usage\n\nThe parse queue is designed to be used to bridge phases of the compile process.\nIn particular the lexical analyzer and the parser.\n\n#### Creating a parse queue:\n\nWhen creating a parse queue, an optional block parameter is passed in. This is\ncalled whenever more queue items are required. For example:\n\n```ruby\ndef open_file_tokenized(name)\n  txt = IO.readlines(name, nil)[0]\n  lex = LexicalAnalayzer.new(text: txt, rules: LEXICAL_RULES)\n  ParseQueue.new { lex.get }\nend\n```\nThis example above is a method that reads in the named file, creates an\nanalyzer on it (see the\n[lexical_analyzer](https://rubygems.org/gems/lexical_analyzer)\ngem for more details) and then uses that as the source for the parse queue.\nThe queue is returned for use by the compiler's parser.\n\nNote: The constant LEXICAL_RULES is a set of rules used to define the tokens\nextracted by the lexical analyzer. As such it is not discussed further here.\nSee that gem for more details on how rules are constructed.\n\n#### Getting a queued item:\n\nGetting an item from the queue is done with the get method. For example:\n\n```ruby\n  item = pq.get\n```\nThis method returns the next unread item from the queue. Note that if no items\nare available, the exception **ParseQueueNoFwd** is raised.\n\nNote: The get! method is a get without backtracking. In effect it is a get\nfollowed by a shift (see Shifting below).\n\n```ruby\n  item = pq.get!\n```\n\n#### Backtracking:\n\nSometimes while parsing, it is required to backtrack to an earlier point in the\ntoken stream so that an alternative branch of the syntax tree may be processed.\nThis is often done with the try method.\n\n```ruby\n  pq.try {\n    do_stuff_with(pq.get)\n    # etc\n\n    success\n  }\n```\nNote how the try block returns a value called success. If this value is false\nor nil, the parse queue is rolled back to its condition at the start of the try\nblock. Otherwise, any changes to the parse queue are retained.\n\nManual control of backtracking is possible by setting the position property to\na value saved off at an earlier point of the processing. For example:\n\n```ruby\n  save_point = pq.position\n  do_stuff_with(pq.get)\n  # etc\n\n  pq.position = save_point unless success\n```\n\n#### Shifting\n\nSo far, items have been retained in the queue, even after they are done being\nprocessed. For large files, this may use a large amount of memory. To avoid\nthis, used items need to be shifted out of the parse queue. This can be done as\nfollows:\n\n```ruby\n  pq.try! {\n    do_stuff_with(pq.get)\n    # etc\n\n    success\n  }\n```\nNote how the try! block returns a value called success. If this value is false\nor nil, the parse queue is rolled back to its condition at the start of the\ntry! block. Otherwise, any changes to the parse queue are retained and\nprocessed items are removed.\n\nThis too can be done manually as shown below:\n\n```ruby\n  save_point = pq.position\n  do_stuff_with(pq.get)\n  # etc\n\n  if success\n    pq.shift\n  else\n    pq.position = save_point\n  end\n```\nNote that if an attempt is made to fall back to data that has been shifted out,\na **ParseQueueNoRev** exception is raised.\n\n#### Exceptions\n\nThe parse queue uses the following exception classes:\n\n    Exception              # From Ruby.\n      StandardError        # From Ruby.\n        ParseQueueError    # The abstract root of parse queue exceptions.\n          ParseQueueNoFwd  # Error: Can't go forward.\n          ParseQueueNoRev  # Error: Can't fall back.\n\n## Contributing\n\n#### Plan A\n\n1. Fork it ( https://github.com/PeterCamilleri/parse_queue/fork )\n2. Create your feature branch (`git checkout -b my-new-feature`)\n3. Commit your changes (`git commit -am 'Add some feature'`)\n4. Push to the branch (`git push origin my-new-feature`)\n5. Create a new Pull Request\n\n#### Plan B\n\nGo to the GitHub repository and raise an issue calling attention to some\naspect that could use some TLC or a suggestion or an idea.\n\n## License\n\nThe gem is available as open source under the terms of the\n[MIT License](./LICENSE.txt).\n\n## Code of Conduct\n\nEveryone interacting in the ParseQueue project’s codebases, issue trackers,\nchat rooms and mailing lists is expected to follow the\n[code of conduct](./CODE_OF_CONDUCT.md).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpetercamilleri%2Fparse_queue","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpetercamilleri%2Fparse_queue","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpetercamilleri%2Fparse_queue/lists"}