{"id":33185844,"url":"https://github.com/c0stya/trre","last_synced_at":"2025-11-27T00:01:06.234Z","repository":{"id":276199842,"uuid":"869963518","full_name":"c0stya/trre","owner":"c0stya","description":"Transductive regular expressions","archived":false,"fork":false,"pushed_at":"2025-05-14T18:45:41.000Z","size":588,"stargazers_count":241,"open_issues_count":2,"forks_count":2,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-05-14T18:51:57.328Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/c0stya.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-10-09T07:54:35.000Z","updated_at":"2025-05-14T18:45:44.000Z","dependencies_parsed_at":"2025-02-14T10:15:14.045Z","dependency_job_id":null,"html_url":"https://github.com/c0stya/trre","commit_stats":null,"previous_names":["c0stya/trre"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/c0stya/trre","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/c0stya%2Ftrre","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/c0stya%2Ftrre/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/c0stya%2Ftrre/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/c0stya%2Ftrre/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/c0stya","download_url":"https://codeload.github.com/c0stya/trre/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/c0stya%2Ftrre/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286079811,"owners_count":27282121,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-11-26T02:00:06.075Z","response_time":193,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-11-16T05:00:20.103Z","updated_at":"2025-11-27T00:01:06.222Z","avatar_url":"https://github.com/c0stya.png","language":"C","funding_links":[],"categories":["\u003ca name=\"text-processing\"\u003e\u003c/a\u003eText processing"],"sub_categories":[],"readme":"# TRRE: transductive regular expressions\n\n#### TLDR: It is an extension of the regular expressions for text editing and a `grep`-like command line tool.\n\n-   [Examples](#examples)\n-   [Language specification](#language-specification)\n-   [Why it works](#why-it-works)\n-   [Precedence](#precedence)\n-   [Modes and greediness](#modes-and-greediness)\n-   [Determinization](#determinization)\n-   [Performance](#performance)\n-   [Installation](#installation)\n-   [TODO](#todo)\n-   [References](#references)\n\n\n## Intro\n\n![automata](docs/automata.png)\n\nRegular expressions is a great tool for searching patterns in text. But I always found it unnatural for text editing. The *group* logic works as a post-processor and can be complicated. Here I propose an extension to the regular expression language for pattern matching and text modification. I call it **transductive regular expressions** or simply **`trre`**.\n\nIt  introduces the `:` symbol to define transformations. The simplest form is `a:b`, which replaces a with b. I call this a `transductive pair` or `transduction`.\n\nTo demonstrate the concept, I have created a command line tool **`trre`**. It feels similar to the `grep -E` command.\n\n## Examples\n\nTo run it first build the program:\n\n```bash\nmake\n```\n\n### Basics\n\nTo change `cat` to `dog` we use the following expression:\n\n```bash\necho 'cat' | ./trre 'cat:dog'\n```\n```\ndog\n```\n\nA more cryptic token-by-token version:\n\n```bash\necho 'cat' | ./trre '(c:d)(a:o)(t:g)'\n```\n```\ndog\n```\n\nIt can be used like `sed` to replace all matches in a string:\n\n```bash\necho 'Mary had a little lamb.' | ./trre 'lamb:cat'\n```\n```\nMary had a little cat.\n```\n\n**Deletion:**\n\nTo delete a string we can use the pattern `string_to_delete:`\n\n```bash\necho 'xor' | ./trre '(x:)or'\n```\n```\nor\n```\n\nThe expression `(x:)` could be interpreted as of translation of `x` to an empty string.\n\nThis expression can be used to delete all the occurances in the default `scan` mode. E.g. expression below removes all occurances of 'a' from a string/\n\n```bash\necho 'Mary had a little lamb.' | ./trre 'a:'\n```\n```\nMry hd  little lmb.\n```\n\nTechnically, here we replace character `a` with empty symbol.\n\nTo remove several characters from this string we can use square brackets construction similar to normal regex:\n\n```bash\necho 'Mary had a little lamb.' | ./trre '[aie]:'\n```\n```\nMry hd  lttl lmb.\n```\n\n**Insertion:**\n\nTo insert a string we can use the pattern `:string_to_insert`\n\n```bash\necho 'or' | ./trre '(:x)or'\n```\n```\nxor\n```\n\nWe could think of the expression `(:x)` as of translation of an empty string into `x`.\n\nWe can insert a word in a context:\n\n```bash\necho 'Mary had a lamb.' | ./trre 'had a (:little )lamb'\n```\n```\nMary had a little lamb.\n```\n\n### Regex over transductions\n\nAs for normal regular expression we could use **alternations** with `|` symbol:\n\n```bash\necho 'cat dog' | ./trre '(c:b)at|(d:h)og'\n```\n```\nbat hog\n```\n\nOr use the **star** over `trre` to repeat the transformation:\n\n```bash\necho 'catcatcat' | ./trre '(cat:dog)*'\n```\n```\ndogdogdog\n```\n\nIn the default `scan` mode, **star** can be omitted:\n\n```bash\necho 'catcatcat' | ./trre 'cat:dog'\n```\n```\ndogdogdog\n```\n\nYou can also use the star in the left part to \"consume\" a pattern infinitely:\n\n```bash\necho 'catcatcat' | ./trre '(cat)*:dog'\n```\n```\ndog\n```\n\n#### Danger zone\n\nAvoid using `*` or `+` in the right part, as it can cause infinite loops:\n\n```bash\necho '' | ./trre ':a*'      # \u003c- do NOT do this\n```\n```\n...\n```\n\nInstead, use finite iterations:\n\n```bash\necho '' | ./trre ':(repeat-10-times){10}'\n\nrepeat-10-timesrepeat-10-timesrepeat-10-timesrepeat-10-timesrepeat-10-timesrepeat-10-timesrepeat-10-timesrepeat-10-timesrepeat-10-timesrepeat-10-times\n```\n\n### Range transformations\n\nTransform ranges of characters:\n\n```bash\necho \"regular expressions\" | ./trre  \"[a:A-z:Z]\"\n```\n```\nREGULAR EXPRESSIONS\n```\n\nAs more complex example, lets create a toy cipher. Below is the Caesar cipher(1) implementation:\n\n```bash\necho 'caesar cipher' | ./trre '[a:b-y:zz:a]'\n```\n```\ndbftbs djqifs\n```\n\nAnd decrypt it back:\n\n```bash\necho 'dbftbs djqifs' | ./trre '[a:zb:a-z:y]'\n```\n```\ncaesar cipher\n```\n\n### Generators\n\n**`trre`** can generate multiple output strings for a single input. By default, it uses the first possible match. You can also generate all possible outputs.\n\n**Binary sequences:**\n```bash\necho '' | ./trre -ma ':(0|1){3}'\n```\n```\n\n000\n001\n010\n011\n100\n101\n110\n111\n```\n\n**Subsets:**\n```bash\necho '' | ./trre -ma ':(0|1){,3}?'\n```\n```\n\n0\n00\n000\n001\n01\n010\n011\n1\n10\n100\n101\n11\n110\n111\n```\n\n## Language specification\n\nInformally, we define a **`trre`** as a pair `pattern-to-match`:`pattern-to-generate`. The `pattern-to-match` can be a string or regexp. The `pattern-to-generate` normally is a string. But it can be a `regex` as well. Moreover, we can do normal regular expression over these pairs.\n\nWe can specify a smiplified grammar of **`trre`** as following:\n\n```\nTRRE    \u003c- TRRE* TRRE|TRRE TRRE.TRRE\nTRRE    \u003c- REGEX REGEX:REGEX\n```\n\nWhere `REGEX` is a usual regular expression, and `.` stands for concatenation.\n\nFor now I consider the operator `:` as non-associative and the expression `TRRE:TRRE` as unsyntactical. There is a natural meaning for this expression as a composition of relations defined by **`trre`**s. But it can make things too complex. Maybe I change this later.\n\n\n## Why it works\n\nUnder the hood, **`trre`** constructs a special automaton called a **Finite State Transducer (FST)**, which is similar to a **Finite State Automaton (FSA)** used in normal regular expressions but handles input-output pairs instead of plain strings.\n\nKey differences:\n\n* **`trre`** defines a binary relation between two regular languages.\n\n* It uses **FST**s instead of **FSA**s for inference.\n\n* It supports on-the-fly determinization for performance (experimental and under construction!).\n\nTo justify the laguage of trunsductive regular expression we need to prove the correspondence between **`trre`** expressions and the corresponding **FST**s. There is my sketch of a the proof: [docs/theory.pdf](docs/theory.pdf).\n\n## Precedence\n\nBelow is the table of precedence (priority) of the `trre` operators from high to low:\n\n| Operator                          | Expression           |\n| --------------------------------- | -------------------- |\n| Escaped characters                | \\\\                   |\n| Bracket expression                | []                   |\n| Grouping                          | ()                   |\n| Iteration                         | * + ? {m,n}          |\n| Concatenation                     |                      |\n| **Transduction**                  | :                    |\n| Alternation                       | \\|                   |\n\n\n## Modes and greediness\n\n**`trre`** supports two modes:\n\n* **Scan Mode (default)**: Applies transformations sequentially.\n\n* **Match Mode**: Checks the entire string against the expression (use `-m` flag).\n\nUse `-a` to generate all possible outputs.\n\nThe `?` modifier makes `*`, `+`, and `{,}` operators non-greedy:\n\n```bash\necho '\u003ccat\u003e\u003cdog\u003e' | ./trre '\u003c(.:)*\u003e'\n```\n```\n\u003c\u003e\n```\n\n```bash\necho '\u003ccat\u003e\u003cdog\u003e' | ./trre '\u003c(.:)*?\u003e'\n```\n```\n\u003c\u003e\u003c\u003e\n```\n\nAnother common example is to change something within tags or brackets. E.g. to change everything within \"\u003c\u003e\" we can use the following expression:\n\n```bash\necho '\u003cdog\u003e \u003cmouse\u003e' | ./trre '\u003c(.*?:cat)\u003e'\n```\n```\n\u003ccat\u003e \u003ccat\u003e\n```\n\n\n## Determinization\n\n\u003cimg src=\"docs/determinization.png\" width=\"80%\"/\u003e\n\nThe important part of the modern regex engines is determinization. This routine converts the non-deterministic automata to the deterministic one. Once converted it has linear time inference on the input string length. It is handy but the convertion is exponential in the worst case.\n\nFor **`trre`** the similar approach is possible. The bad news is that not all the non-deterministic transducers (**NFT**) can be converted to a deterministic (**DFT**). In case of two \"bad\" cycles with same input labels the algorithm is trapped in the infinite loop of a state creation. There is a way to detect such loops but it is expensive (see more in [Allauzen, Mohri, Efficient Algorithms for testing the twins property](https://cs.nyu.edu/~mohri/pub/twins.pdf)).\n\n## Performance\n\nThe default non-deterministic version is a bit slower then `sed`:\n\n```bash\nwget https://www.gutenberg.org/cache/epub/57333/pg57333.txt -O chekhov.txt\n\ntime cat chekhov.txt | ./trre '(vodka):(VODKA)' \u003e /dev/null\n```\n```\nreal\t0m0.046s\nuser\t0m0.043s\nsys\t0m0.007s\n```\n\n```bash\ntime cat chekhov.txt | sed  's/vodka/VODKA/' \u003e /dev/null\n```\n```\nreal\t0m0.024s\nuser\t0m0.020s\nsys\t0m0.010s\n```\n\nFor complex tasks, **`trre_dft`** (deterministic version) can outperform sed:\n\n```bash\ntime cat chekhov.txt | sed -e 's/\\(.*\\)/\\U\\1/' \u003e /dev/null\n```\n```\nreal\t0m0.508s\nuser\t0m0.504s\nsys\t0m0.015s\n```\n\n```bash\ntime cat chekhov.txt | ./trre_dft '[a:A-z:Z]' \u003e /dev/null\n```\n```\nreal\t0m0.131s\nuser\t0m0.127s\nsys\t0m0.009s\n```\n\n## Installation\n\nNo pre-built binaries are available yet. Clone the repository and compile:\n\n```bash\ngit clone git@github.com:c0stya/trre.git trre\ncd trre\nmake \u0026\u0026 sh test.sh\n```\n\nThen move the binary to a directory in your `$PATH`.\n\n## TODO\n\n* Stable *DFT* version\n* Full unicode support\n* Complete the ERE feature set:\n    - negation `^` within `[]`\n    - character classes\n    - '$^' anchoring symbols\n* Efficient range processing\n\n## References\n\n* The approach is strongly inspired by Russ Cox article: [Regular Expression Matching Can Be Simple And Fast](https://swtch.com/~rsc/regexp/regexp1.html)\n* The idea of transducer determinization comes from this paper: [Cyril Allauzen, Mehryar Mohri, \"Finitely Subsequential Transducers\"](https://research.google/pubs/finitely-subsequential-transducers-2/)\n* Parsing approach comes from [Double-E algorithm](https://github.com/erikeidt/erikeidt.github.io/blob/master/The-Double-E-Method.md) by Erik Eidt which is close to the classical [Shunting Yard algorithm](https://en.wikipedia.org/wiki/Shunting_yard_algorithm).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fc0stya%2Ftrre","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fc0stya%2Ftrre","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fc0stya%2Ftrre/lists"}