{"id":23029738,"url":"https://github.com/antononcube/raku-grammar-tokenprocessing","last_synced_at":"2026-02-14T03:03:18.417Z","repository":{"id":63725237,"uuid":"469455031","full_name":"antononcube/Raku-Grammar-TokenProcessing","owner":"antononcube","description":"Raku package for processing (gathering, enhancing, etc.) of token specs in grammar and role classes.","archived":false,"fork":false,"pushed_at":"2024-12-27T18:31:42.000Z","size":423,"stargazers_count":3,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-03-30T00:31:39.818Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Raku","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"artistic-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/antononcube.png","metadata":{"files":{"readme":"README-work.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-03-13T18:12:02.000Z","updated_at":"2024-12-27T18:31:46.000Z","dependencies_parsed_at":"2023-12-21T06:27:23.785Z","dependency_job_id":"8c3b4cc4-43d2-4a54-b25c-6de4a090bd62","html_url":"https://github.com/antononcube/Raku-Grammar-TokenProcessing","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/antononcube%2FRaku-Grammar-TokenProcessing","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/antononcube%2FRaku-Grammar-TokenProcessing/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/antononcube%2FRaku-Grammar-TokenProcessing/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/antononcube%2FRaku-Grammar-TokenProcessing/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/antononcube","download_url":"https://codeload.github.com/antononcube/Raku-Grammar-TokenProcessing/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250452529,"owners_count":21433037,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-15T14:16:58.567Z","updated_at":"2026-02-14T03:03:18.317Z","avatar_url":"https://github.com/antononcube.png","language":"Raku","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Raku Grammar::TokenProcessing\n\n## In brief\n\nRaku package for processing grammar files in order to:\n\n- Extract tokens\n\n- Replace token names\n\n- Add fuzzy matching expressions in token definitions\n\n- Random sentence generation\n\n**Remark:** This package is made mostly to automate tasks for the DSL system of packages, see:\n[Raku-DSL-*](https://github.com/search?q=user%3Aantononcube+Raku-DSL).\nHence, the package is tested \"just\" over files written with a particular style and goal.\n\n------\n\n## Installation\n\nInstallation from [Zef ecosystem](https://raku.land):\n\n```\nzef install Grammar::TokenProcessing\n```\n\nInstallation from GitHub:\n\n```\nzef install https://github.com/antononcube/Raku-Grammar-TokenProcessing.git\n```\n\n-------\n\n## Examples \n\nBelow are shown usages via the Command Line Interface (CLI) of UNIX-like operating systems.\n\n\n### Add fuzzy matching to token specs\n\n```shell\nadd-token-fuzzy-matching --help\n```\n\n### Gather tokens\n\n```shell\nget-tokens --help                                                                                     \n```\n\n### Gather tokens into a hash\n\n```shell\nget-tokens-hash --help                                                                                     \n```\n\n### Replace token names\n\n```shell\nreplace-token-names --help                                                                                     \n```\n\n### Random sentence generation\n\n```shell\nrandom-sentence-generation --help\n```\n\nHere is example of random sentence generation based on the grammar of the package \n[\"DSL::English::QuantileRegressionWorkflows\"](https://raku.land/zef:antononcube/DSL::English::QuantileRegressionWorkflows), [AAp5]:\n\n```shell\nrandom-sentence-generation DSL::English::QuantileRegressionWorkflows::Grammar\n```\n\nHere is another example using the Bulgarian localization of [AAp5] in [AAp7]:\n\n```shell\nrandom-sentence-generation DSL::Bulgarian::QuantileRegressionWorkflows::Grammar  -n=10 --syms='Bulgarian English'\n```\n\nHere we generate sentences with a grammar string (that is a valid Raku definition of a grammar):\n\n```shell\nrandom-sentence-generation -n=5 \"\ngrammar Parser {\n    rule  TOP  { I [ \u003clove\u003e | \u003chate\u003e ] \u003clang\u003e }\n    token love { '♥' ** 1..3 | love }\n    token hate { '🖕' ** 1..2 | hate }\n    token lang { \u003c Raku Perl Rust Go Python Ruby \u003e }\n}\"\n```\n\n### Converting rules to regexes\n\nHere are examples of converting rules to regexes:\n\n```perl6\nuse Grammar::TokenProcessing;\n\nmy %ruleBodies =\n        cookie =\u003e 'generic? chocolate cookie \\w+ \\d+',\n        cookie-limited =\u003e 'crunch bar \\d ** 1..2';\n\nfor %ruleBodies.kv -\u003e $k, $v {\n    say \"rule   : $v\";\n    say \"regex  : {rule-to-regex($v)}\\n\";\n}\n```\n\nMore detailed examples -- with grammar creation for regex verification -- can be found in the test file \n[\"06-rule-to-regex-conversion.rakutest\"](./t/06-rule-to-regex-conversion.rakutest).\n\n--------\n\n## References\n\n### Packages\n\n[AAp1] Anton Antonov,\n[DSL::Shared, Raku package](https://github.com/antononcube/Raku-DSL-Shared),\n(2018-2022),\n[GitHub/antononcube](https://github.com/antononcube).\n\n[AAp2] Anton Antonov,\n[DSL::English::ClassificationWorkflows, Raku package](https://github.com/antononcube/Raku-DSL-General-ClassificationWorkflows),\n(2018-2022),\n[GitHub/antononcube](https://github.com/antononcube).\n\n[AAp3] Anton Antonov,\n[DSL::English::DataQueryWorkflows, Raku package](https://github.com/antononcube/Raku-DSL-English-DataQueryWorkflows),\n(2020-2022),\n[GitHub/antononcube](https://github.com/antononcube).\n\n[AAp4] Anton Antonov,\n[DSL::English::LatentSemanticAnalysisWorkflows, Raku package](https://github.com/antononcube/Raku-DSL-General-LatentSemanticAnalysisWorkflows),\n(2018-2022),\n[GitHub/antononcube](https://github.com/antononcube).\n\n[AAp5] Anton Antonov,\n[DSL::English::QuantileRegressionWorkflows, Raku package](https://github.com/antononcube/Raku-DSL-General-QuantileRegressionWorkflows),\n(2018-2022),\n[GitHub/antononcube](https://github.com/antononcube).\n\n[AAp6] Anton Antonov,\n[DSL::English::RecommenderWorkflows, Raku package](https://github.com/antononcube/Raku-DSL-General-RecommenderWorkflows),\n(2018-2022),\n[GitHub/antononcube](https://github.com/antononcube).\n\n[AAp7] Anton Antonov,\n[DSL::Bulgarian, Raku package](https://github.com/antononcube/Raku-DSL-Bulgarian),\n(2022),\n[GitHub/antononcube](https://github.com/antononcube).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fantononcube%2Fraku-grammar-tokenprocessing","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fantononcube%2Fraku-grammar-tokenprocessing","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fantononcube%2Fraku-grammar-tokenprocessing/lists"}