{"id":25067016,"url":"https://github.com/shoyamanishi/nlpregex","last_synced_at":"2025-03-31T14:17:58.928Z","repository":{"id":187800787,"uuid":"234977551","full_name":"ShoYamanishi/nlpregex","owner":"ShoYamanishi","description":"Regular Language representation converter, visualizer, and decoder \u0026 acceptor","archived":false,"fork":false,"pushed_at":"2020-01-26T15:29:17.000Z","size":2012,"stargazers_count":3,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-02-06T20:35:28.743Z","etag":null,"topics":["decoder","dfa","fst","graphviz","regular-expressions","regular-language","visualizer"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ShoYamanishi.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2020-01-19T22:28:42.000Z","updated_at":"2024-06-16T19:45:02.000Z","dependencies_parsed_at":"2023-08-12T07:13:15.125Z","dependency_job_id":null,"html_url":"https://github.com/ShoYamanishi/nlpregex","commit_stats":null,"previous_names":["shoyamanishi/nlpregex"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ShoYamanishi%2Fnlpregex","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ShoYamanishi%2Fnlpregex/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ShoYamanishi%2Fnlpregex/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ShoYamanishi%2Fnlpregex/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ShoYamanishi","download_url":"https://codeload.github.com/ShoYamanishi/nlpregex/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246481004,"owners_count":20784458,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["decoder","dfa","fst","graphviz","regular-expressions","regular-language","visualizer"],"created_at":"2025-02-06T20:29:03.054Z","updated_at":"2025-03-31T14:17:58.904Z","avatar_url":"https://github.com/ShoYamanishi.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# nlpregex \n\nRegular Language Representation Converter, Visualizer, and Decoder \u0026amp; Acceptor.\n\n\n\u003ca href=\"docs/banner.png\"\u003e \u003cimg src=\"docs/banner.png\"\u003e\u003c/a\u003e\n\n# Highlights\n\n* Closed regular language represntation eco-system\n\n* Automatic generation of RE from a flat list\n\n* Common subexpression detection and reduction\n\n* Visualization of AST, NFA, DFA with Grpahviz\n\n* Decoder \u0026amp; acceptor with enhanced regular expression to include output attributes\n\n\n# Description\n\nNlpregex is a set of python tools and components to visualize and convert representation\nof regular language.\nIt is aimed at speech recognition and natural language processing, \nand the alphabets (Σ) are not characters, but tokenized terminals.\nThe conversion of the representations forms a closed ecosystem among the following:\n\n* Regular Expression (RE)\n\n* Abstract Syntax Tree (AST)\n\n* Nondeterministic Finite-state Automaton (NFA) \n\n* Deterministic Finite-state Automaton (DFA)\n\n* Flat list of phrases (provided the language does not contain infinite repetitions)\n\n\nThe visualization of AST, NFA, and DFA are done by GraphViz.\n\n\u003ca href=\"docs/ecosystem.png\"\u003e \u003cimg src=\"docs/ecosystem.png\"\u003e\u003c/a\u003e\n\nThe regular expression is augmented to include the output attributes after a terminal,\nand before and after a group enclosed by '(' and ')' in paires.\nThe attributes are transferred to the output tokens of FA and it is used for the decoder as FST.\n\n# Conversion from FA to AST(s)\n\n\u003ca href=\"docs/conversion.png\"\u003e \u003cimg src=\"docs/conversion.png\"\u003e\u003c/a\u003e\n\nThe technical contribution of this project is the conversion from FA to AST,\nwhich makes it possible to convert a flat list of phrases to a regular expression.\nIt takes the following steps.\n\n## 1. Form a set of equations\n\nThe first step Forms a set of simultaneous equations in [Kleene Algebra](https://en.wikipedia.org/wiki/Kleene_algebra).\nEach state in FA represents a variable, and each input token a coefficient.\nEach equation represents the transitions from a single state to its out-neighbors.\nOne artificial start state and another final state are introduced.\nThe artificial start state has an epsilon transition to the real start state,\nand the real final states have epsilon transitions to the artificial final state.\nThe coefficients are represented by a special kind of ASTs, called [sseForest](regular_language/sse_forrest.py).\n\n## 2. Solve the equations\n\nThe second step solves the equations down to the row that corresponds to the artiicial start state \nby a [solver](regular_language/sse_solver.py).\nThe row reduction is done by substitution similar to Gaussian elimination.\nThe self-recursions are eliminated by [Arden's rule](https://en.wikipedia.org/wiki/Arden%27s_rule).\nThe resultant equation has the following form: \n\n    ```\u003cartificial_start_state\u003e = coeff \u003cartificial_final_state\u003e```\n\nThe remaining coefficient corresponds to the resultant AST.\n\n## 3. Optionally detect and reduce the common subexpressions.\nThe third step optionally tries to detect and reduce common subexpressions.\nIt is not only a [common subtree reduction](regular_language/common_subtree_reducer.py)\nbut it also considers the following two types.\n\n   * Common terms among the union nodes. [(Reducer implementation)](regular_language/common_union_subexpression_reducer.py)\n     Ex. `(abc(a|b) |def|ghi?)` and `(abc(a|b)|jkl|ghi?)` has a common subexpression `(abc(a|b)|ghi?)`.\n\n   * Common substrings among the serial nodes.[(Reducer implementation)](regular_language/common_substring_reducer.py)\n     Ex. `(a b a b c)` and `(b a b c a)` has a longest common substring `(b a b)`.\n\nThose common subexpressions are detected and reduced one-by-one in a greedy manner with the following criteria.\n\n* Tree height\n* Number of terminals under the subtree\n* Number of occurrences of the same subtree\n* Length of the regular expression\n\n# Conversion from a Flat List to RE\n\n0. Initial input file of a flat list of phrases\n\n\u003ca href=\"samples/sample_expanded_04.txt\"\u003e \u003cimg src=\"docs/expanded_list.png\" height=200 \u003e\u003c/a\u003e\n\n\n1. Parse by Lark LR(1) line parser into a trivial AST.\n\n\u003ca href=\"samples/flat_list_parsed_rule.txt\"\u003e \u003cimg src=\"docs/parsed_lines.png\" height=200 \u003e\u003c/a\u003e\n\n\n2. Convert the AST to a trivial lattice NFA \n\n\u003ca href=\"docs/flat_list_nfa.svg\"\u003e \u003cimg src=\"docs/flat_list_nfa.png\" height=200 \u003e\u003c/a\u003e\n\n\n3. Convert the NFA to DFA by OpenFST (determinization and equivalent states reduction)\n\n\u003ca href=\"docs/flat_list_dfa.svg\"\u003e \u003cimg src=\"docs/flat_list_dfa.png\" height=200 \u003e\u003c/a\u003e\n\n\n4. Conversion from FA to a single AST (see above)\n\n\u003ca href=\"samples/sample_auto_generated_single_04.txt\"\u003e \u003cimg src=\"docs/auto_generated_single.png\" height=350 \u003e\u003c/a\u003e\n\n\n5. Reduce the AST to a set of ASTs with nonterminals (see above)\n\n\u003ca href=\"samples/sample_auto_generated_reduced_04.txt\"\u003e \u003cimg src=\"docs/auto_generated_reduced.png\" height=200 \u003e\u003c/a\u003e\n\n\n\n# Regular Expression (Rule) Syntax\n\nA regular expression, or rule, consists of two parts: LHS and RHS.\nLHS designates the nonterminal for the expression.\nRHS is the actual regular expression.\nEx.  `\u003cNT_0\u003e :  a b c ;`\n\n\n## Unary operators\n\n* 0 or more repetition (kleene closure) (`*`) Ex. `a*`, `(a b c)*`\n\n* 1 or more repetition (`+`) (`*`) Ex. `a*`, `(a b c)*`\n\n* Option (`?`) (`*`) Ex. `a?`, `(a b c)?`\n\n* finite repetition (`{min,max}`) Ex. `a{1,3}`, `(a b c){2,5}`\n\n## Binary operators\n\n* Union (selection) (`|`) Ex. `(a|b|c)`\n\n* Concatenation  Ex. `a b c`\n\n## Alphabets\n\n* Terminal : a word token.  Ex. `abc`\n\n* Nonterminal : a token enclosed by angle brackets `\u003c\u003e` Ex. `\u003cNT_0\u003e`\n\n## Others\n\n* Grouping by `(` and `)` Ex. `a b c ( d | f | g | h i ) j k`\n\n* Output attributes enclosed by `[` and `]`\n  Output attributes can be placed immediately after a terminal, or before and after a grouping.\n  Ex. `a b[action_01] c`, ` a b c [prologue_01](d | e | f [action_f] )[epilogue_01] g h i`.\n\n\nPlease see [grammar_nlpregex_rules](regular_language/lark_parser.py) for details.\n\n# Examples\n\n## From a regular expression to pretty-format, flat list, AST, NFA, and DFA\nThis example takes the sample files [sample_input_04.txt](samples/sample_input_04.txt)\nand [sample_input_04_oneline.txt](samples/sample_input_04_oneline.txt)\nand visualize the regular expression designated by `\u003ctop_rule\u003e`.\n\n### Command to pretty-format.\n```shellscript\npython re_formatter.py samples/sample_input_04_oneline.txt pretty.txt\n```\nThe output file is the same as [sample_input_04.txt](samples/sample_input_04.txt).\n\n### Command to generate flat list of phrases.\n```shellscript\npython re_phrase_expander.py samples/sample_input_04_oneline.txt -rule \"\u003ctop_rule\u003e\" expanded.txt\n```\nThe output file is the same as [sample_expanded_04.txt](samples/sample_expanded_04.txt).\nPlease note that the regular expression in sample_input_04.txt and sample_input_04_oneline.txt has \nredundancies and the output file has duplications, which can be removed by `sort | uniq`.\n\n### Command to generate AST, NFA, and DFA.\n```shellscript\n`python re_visualizer.py samples/sample_input_04.txt -rule \"\u003ctop_rule\u003e\" -expand_all_nt -ast ast_fig -nfa nfa_fig -dfa dfa_fig -t svg`\n```\n\nHere are the outputs generated by the command. (Click the figures to see the SVG file.)\n\n\u003ca href=\"docs/ast_fig.svg\"\u003e \u003cimg src=\"docs/ast.png\" width=800 \u003e\u003c/a\u003e\n\n\u003ca href=\"docs/nfa_fig.svg\"\u003e \u003cimg src=\"docs/nfa.png\" width=200 \u003e\u003c/a\u003e\n\n\u003ca href=\"docs/dfa_fig.svg\"\u003e \u003cimg src=\"docs/dfa.png\" width=200 \u003e\u003c/a\u003e\n\n## Auto-generate regex from a flat list\nThis example takes the following [the sample file](samples/sample_expanded_04.txt) that contains a flat list and generate a regular expression \nor a set of regular expressions.\n\n\u003ca href=\"samples/sample_expanded_04.txt\"\u003e \u003cimg src=\"docs/expanded_list.png\" height=400 \u003e\u003c/a\u003e\n\n\n### Command to generate a single regular expression.\n```shellscript\npython flat_list_to_re.py samples/sample_expanded_04.txt auto_gen_single.txt\n```\nThe output file is the same as [sample_auto_generated_single_04.txt](samples/sample_auto_generated_single_04.txt).\n\n\u003ca href=\"samples/sample_auto_generated_single_04.txt\"\u003e \u003cimg src=\"docs/auto_generated_single.png\" height=400 \u003e\u003c/a\u003e\n\n### Command to generate a set of regular expressions by reducing the common subexpressions.\n```shellscript\npython flat_list_to_re.py samples/sample_expanded_04.txt auto_gen_reduced.txt -reduce\n```\nThe output file is the same as [sample_auto_generated_reduced_04.txt](samples/sample_auto_generated_reduced_04.txt).\nThis is the outcome of the most aggressive reduction where any subexpression that has at least two terminals, \nand that occurrs at least twice in the trees are subject to reduction.\n\n\u003ca href=\"samples/sample_auto_generated_reduced_04.txt\"\u003e \u003cimg src=\"docs/auto_generated_reduced.png\" height=400 \u003e\u003c/a\u003e\n\n## Decode as FST\nThe following shows an interactive decoding action for the rules \ndefined in [sample_input_03.txt](samples/sample_input_03.txt).\n\n```shellscript\n$ cat nlpregex\\samples\\sample_input_03.txt\n\u003cnt01\u003e : [out_token1 ]( t01 | t02 | \u003cnt02\u003e )[out_token2] ;\n\u003cnt02\u003e : abc def ( [out_token3] (ghi xyz) [out_token4] ) ? ;\n$ python3 nlpregex\\re_decoder.py -rulefile nlpregex\\samples\\sample_input_03.txt -rule \"\u003cnt01\u003e\" -expand_all_nt\nt01                   # \u003c= Input by the user\nout_token1 out_token2 # \u003c= Output by the decoder\nabc def               # \u003c= Input by the user\nout_token1 out_token2 # \u003c= Output by the decoder\nt03                   # \u003c= Input by the user\n__NOT_ACCEPTED__      # \u003c= Output by the decoder\nabc def ghi xyz       # \u003c= Input by the user\nout_token1 out_token3 out_token4 out_token2 # \u003c= Output by the decoder\n$\n```\n\n# Install\n\n1. Install [GraphViz](https://www.graphviz.org/).\n\n2. Install [OpenFST](http://www.openfst.org/twiki/bin/view/FST/WebHome).\nIf you are using Windows, then these guys [here](https://github.com/kkm000/openfst) have a nice ported version.\n\n3. Add the paths to GraphViz \u0026 OpenFST command tool to PATH.\n\n4. Install Python3. (Probably you already have [Anaconda](https://www.anaconda.com/) if you are working on NLU stuff.)\n\n5. Install GraphViz binding (`pip install graphviz`)\n\n6. Install [Lark parser](`https://github.com/lark-parser/lark`) (`pip install lark-parser`)\n\n7. Add the path to the directory that contains nlpregex to PYTHONPATH\n\n# Command Line Tools\n\n* [re_formatter.py](nlpregex/re_formatter.py)\nThis reformats the rules(expressions) in the given file with nice indentation.\n```\nusage: re_formatter.py [-h]\n                       [-rules [\u003clist of nonterminals\u003e [\u003clist of nonterminals\u003e ...]]]\n                       [-expand [\u003clist of nonterminals\u003e [\u003clist of nonterminals\u003e ...]]]\n                       [-expand_all_nt] [-expand_finite_repeat]\n                       [-width_hint [WIDTH_HINT]] [-indent [INDENT]]\n                       [infile] [outfile]\n\nreformat regular expression with the specified nonterminals replaced with sub-\nexpressions\n\npositional arguments:\n  infile                input file\n  outfile               output file\n\noptional arguments:\n  -h, --help            show this help message and exit\n  -rules [\u003clist of nonterminals\u003e [\u003clist of nonterminals\u003e ...]]\n                        rules to reformat (default ALL)\n  -expand [\u003clist of nonterminals\u003e [\u003clist of nonterminals\u003e ...]]\n                        tries to expand the given nontermals (default NONE)\n  -expand_all_nt        tries to expand all the nonterminals for the specified\n                        rule\n  -expand_finite_repeat\n                        expands finite repeats\n  -width_hint [WIDTH_HINT]\n                        max width at which this tool tries to fold the\n                        selection expression into multiple lines\n  -indent [INDENT]      number of spaces for a single indentation\n```\n\n* [re_phrase_expander.py](nlpregex/re_phrase_expander.py) \nThis expands the specified rule into a list of phrases it accepts.\nThe finite repetisions are not expanded for an obvious reason.\n```\nusage: re_phrase_expander.py [-h] [-rule \u003cnonterminal\u003e]\n                             [-expand [\u003clist of nonterminals\u003e [\u003clist of nonterminals\u003e ...]]]\n                             [-expand_all_nt] [-expand_finite_repeat]\n                             [infile] [outfile]\n\nexpand phrases of regular expression\n\npositional arguments:\n  infile                input file\n  outfile               output file\n\noptional arguments:\n  -h, --help            show this help message and exit\n  -rule \u003cnonterminal\u003e   rule to expand\n  -expand [\u003clist of nonterminals\u003e [\u003clist of nonterminals\u003e ...]]\n                        tries to expand the given nontermals (default NONE)\n  -expand_all_nt        tries to expand all the nonterminals for the specified\n                        rule\n  -expand_finite_repeat\n                        expands finite repeats, otherwise, it will be replaced\n                        with a temporary token\n```\n\n* [re_phrase_visualizer.py](nlpregex/re_visuzlizer.py) \nThis visualizes the specified rule into AST, NFA, and DFA in either SVG or PDF format.\nThese visuals are created by GraphViz's dot command.\n```\nusage: re_visualizer.py [-h] [-rule \u003cnonterminal\u003e]\n                        [-expand [\u003clist of nonterminals\u003e [\u003clist of nonterminals\u003e ...]]]\n                        [-expand_all_nt] [-ast \u003cfile name w/o ext\u003e]\n                        [-nfa \u003cfile name w/o ext\u003e] [-dfa \u003cfile name w/o ext\u003e]\n                        [-t [pdf/svg]] [-s] [-horizontal]\n                        [infile] [outfile]\n\nvisualize regular expression in AST, NFA, and DFA\n\npositional arguments:\n  infile                input file\n  outfile               output file\n\noptional arguments:\n  -h, --help            show this help message and exit\n  -rule \u003cnonterminal\u003e   rule to expand\n  -expand [\u003clist of nonterminals\u003e [\u003clist of nonterminals\u003e ...]]\n                        tries to expand the given nontermals (default NONE)\n  -expand_all_nt        tries to expand all the nonterminals for the specified\n                        rule\n  -ast \u003cfile name w/o ext\u003e\n                        draws abstract syntax tree to the file\n  -nfa \u003cfile name w/o ext\u003e\n                        draws NFA to the file.\n  -dfa \u003cfile name w/o ext\u003e\n                        draws DFA to the file.\n  -t [pdf/svg]          visual output file type. svg or pdf. default: pdf. Use\n                        svg if you have a unicode issue on Windows.\n  -s                    tries to show the visual immediately\n  -horizontal           draw AST horizontally\n```\n\n* [re_decoder.py](nlpregex/re_decoder.py) \nThis decodes the input tokens with the specified rule into output tokens.\nIt works as a finite-state transducer, whose rules are specified by an augmented regular expression.\n```\nusage: re_decoder.py [-h] [-rulefile RULEFILE] [-rule \u003cnonterminal\u003e]\n                     [-expand [\u003clist of nonterminals\u003e [\u003clist of nonterminals\u003e ...]]]\n                     [-expand_all_nt]\n                     [infile] [outfile]\n\ndecode regular language with output attributes\n\npositional arguments:\n  infile                input file\n  outfile               output file\n\noptional arguments:\n  -h, --help            show this help message and exit\n  -rulefile RULEFILE    regular expression rule file\n  -rule \u003cnonterminal\u003e   rule to expand\n  -expand [\u003clist of nonterminals\u003e [\u003clist of nonterminals\u003e ...]]\n                        tries to expand the given nontermals (default NONE)\n  -expand_all_nt        tries to expand all the nonterminals for the specified\n```\n\n\n* [flat_list_to_re.py](nlpregex/flat_list_to_re.py) \nThis generates a rule or a set of rules from the given flat list of phrases.\nIt also detects and reduces the common subexpressions.\n```\nusage: flat_list_to_re.py [-h] [-reduce]\n                          [-min_num_occurrences [MIN_NUM_OCCURRENCES]]\n                          [-min_num_terms [MIN_NUM_TERMS]]\n                          [-width_hint [WIDTH_HINT]] [-indent [INDENT]]\n                          [infile] [outfile]\n\ngenerate regular expression from a flat list\n\n\npositional arguments:\n  infile                input file\n  outfile               output file\n\noptional arguments:\n  -h, --help            show this help message and exit\n  -reduce               tries to reduce the common subexpressions\n  -min_num_occurrences [MIN_NUM_OCCURRENCES]\n                        min number of occurrences of a common subexpression to\n                        be considered for reduction\n  -min_num_terms [MIN_NUM_TERMS]\n                        min number of terminals in a common subexpression to\n                        be considered for reduction\n  -width_hint [WIDTH_HINT]\n                        max width at which this tool tries to fold the\n                        selection expression into multiple lines\n  -indent [INDENT]      number of spaces for a single indentation\n```\n\n\n# Limitation\nIf a regular expression has paired output attributes associated with a group, then the reconstruction\nfrom the corresponding DFA back to an AST is not possible due to the lost associativity during\nthe conversion from NFA to DFA.\nTherefore, if a regular expression has paired output attributes associated to a group, \nthen the reconstruction from the corresponding flat list back to an AST is not possible.\nHowever, the converted DFA with lost associativity still works correctly as a decoder.\n\n# TODO\n\n* Study the possibility of augmenting the expression and the tools to integrate statistical treatment in order for the decoder to work as WFST.\n\n\n# Dependencies\n\n* [OpenFST](http://www.openfst.org) under Apache License\n\n* [GraphViz](https://www.graphviz.org) under Common Public License Version 1.0\n\n* [Lark-Parser](https://github.com/lark-parser/lark) under MIT License\n\n\n# Contact\n\nFor technical and commercial inquiries, please contact: Shoichiro Yamanishi\n\nyamanishi72@gmail.com","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshoyamanishi%2Fnlpregex","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fshoyamanishi%2Fnlpregex","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshoyamanishi%2Fnlpregex/lists"}