https://github.com/antononcube/raku-grammar-tokenprocessing
Raku package for processing (gathering, enhancing, etc.) of token specs in grammar and role classes.
https://github.com/antononcube/raku-grammar-tokenprocessing
Last synced: 5 months ago
JSON representation
Raku package for processing (gathering, enhancing, etc.) of token specs in grammar and role classes.
- Host: GitHub
- URL: https://github.com/antononcube/raku-grammar-tokenprocessing
- Owner: antononcube
- License: artistic-2.0
- Created: 2022-03-13T18:12:02.000Z (over 4 years ago)
- Default Branch: main
- Last Pushed: 2024-12-27T18:31:42.000Z (over 1 year ago)
- Last Synced: 2025-03-30T00:31:39.818Z (over 1 year ago)
- Language: Raku
- Size: 413 KB
- Stars: 3
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README-work.md
- License: LICENSE
Awesome Lists containing this project
README
# Raku Grammar::TokenProcessing
## In brief
Raku package for processing grammar files in order to:
- Extract tokens
- Replace token names
- Add fuzzy matching expressions in token definitions
- Random sentence generation
**Remark:** This package is made mostly to automate tasks for the DSL system of packages, see:
[Raku-DSL-*](https://github.com/search?q=user%3Aantononcube+Raku-DSL).
Hence, the package is tested "just" over files written with a particular style and goal.
------
## Installation
Installation from [Zef ecosystem](https://raku.land):
```
zef install Grammar::TokenProcessing
```
Installation from GitHub:
```
zef install https://github.com/antononcube/Raku-Grammar-TokenProcessing.git
```
-------
## Examples
Below are shown usages via the Command Line Interface (CLI) of UNIX-like operating systems.
### Add fuzzy matching to token specs
```shell
add-token-fuzzy-matching --help
```
### Gather tokens
```shell
get-tokens --help
```
### Gather tokens into a hash
```shell
get-tokens-hash --help
```
### Replace token names
```shell
replace-token-names --help
```
### Random sentence generation
```shell
random-sentence-generation --help
```
Here is example of random sentence generation based on the grammar of the package
["DSL::English::QuantileRegressionWorkflows"](https://raku.land/zef:antononcube/DSL::English::QuantileRegressionWorkflows), [AAp5]:
```shell
random-sentence-generation DSL::English::QuantileRegressionWorkflows::Grammar
```
Here is another example using the Bulgarian localization of [AAp5] in [AAp7]:
```shell
random-sentence-generation DSL::Bulgarian::QuantileRegressionWorkflows::Grammar -n=10 --syms='Bulgarian English'
```
Here we generate sentences with a grammar string (that is a valid Raku definition of a grammar):
```shell
random-sentence-generation -n=5 "
grammar Parser {
rule TOP { I [ | ] }
token love { '♥' ** 1..3 | love }
token hate { '🖕' ** 1..2 | hate }
token lang { < Raku Perl Rust Go Python Ruby > }
}"
```
### Converting rules to regexes
Here are examples of converting rules to regexes:
```perl6
use Grammar::TokenProcessing;
my %ruleBodies =
cookie => 'generic? chocolate cookie \w+ \d+',
cookie-limited => 'crunch bar \d ** 1..2';
for %ruleBodies.kv -> $k, $v {
say "rule : $v";
say "regex : {rule-to-regex($v)}\n";
}
```
More detailed examples -- with grammar creation for regex verification -- can be found in the test file
["06-rule-to-regex-conversion.rakutest"](./t/06-rule-to-regex-conversion.rakutest).
--------
## References
### Packages
[AAp1] Anton Antonov,
[DSL::Shared, Raku package](https://github.com/antononcube/Raku-DSL-Shared),
(2018-2022),
[GitHub/antononcube](https://github.com/antononcube).
[AAp2] Anton Antonov,
[DSL::English::ClassificationWorkflows, Raku package](https://github.com/antononcube/Raku-DSL-General-ClassificationWorkflows),
(2018-2022),
[GitHub/antononcube](https://github.com/antononcube).
[AAp3] Anton Antonov,
[DSL::English::DataQueryWorkflows, Raku package](https://github.com/antononcube/Raku-DSL-English-DataQueryWorkflows),
(2020-2022),
[GitHub/antononcube](https://github.com/antononcube).
[AAp4] Anton Antonov,
[DSL::English::LatentSemanticAnalysisWorkflows, Raku package](https://github.com/antononcube/Raku-DSL-General-LatentSemanticAnalysisWorkflows),
(2018-2022),
[GitHub/antononcube](https://github.com/antononcube).
[AAp5] Anton Antonov,
[DSL::English::QuantileRegressionWorkflows, Raku package](https://github.com/antononcube/Raku-DSL-General-QuantileRegressionWorkflows),
(2018-2022),
[GitHub/antononcube](https://github.com/antononcube).
[AAp6] Anton Antonov,
[DSL::English::RecommenderWorkflows, Raku package](https://github.com/antononcube/Raku-DSL-General-RecommenderWorkflows),
(2018-2022),
[GitHub/antononcube](https://github.com/antononcube).
[AAp7] Anton Antonov,
[DSL::Bulgarian, Raku package](https://github.com/antononcube/Raku-DSL-Bulgarian),
(2022),
[GitHub/antononcube](https://github.com/antononcube).