{"id":13830748,"url":"https://github.com/lhoursquentin/sed-bin","last_synced_at":"2025-07-09T12:31:26.048Z","repository":{"id":48079562,"uuid":"236139096","full_name":"lhoursquentin/sed-bin","owner":"lhoursquentin","description":"sed to C translator written in sed","archived":false,"fork":false,"pushed_at":"2021-08-18T17:46:08.000Z","size":150,"stargazers_count":283,"open_issues_count":1,"forks_count":5,"subscribers_count":10,"default_branch":"master","last_synced_at":"2024-08-04T10:03:52.906Z","etag":null,"topics":["sed"],"latest_commit_sha":null,"homepage":"","language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lhoursquentin.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-01-25T07:24:37.000Z","updated_at":"2024-03-29T18:20:50.000Z","dependencies_parsed_at":"2022-08-12T18:10:30.936Z","dependency_job_id":null,"html_url":"https://github.com/lhoursquentin/sed-bin","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lhoursquentin%2Fsed-bin","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lhoursquentin%2Fsed-bin/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lhoursquentin%2Fsed-bin/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lhoursquentin%2Fsed-bin/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lhoursquentin","download_url":"https://codeload.github.com/lhoursquentin/sed-bin/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":225539517,"owners_count":17485342,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["sed"],"created_at":"2024-08-04T10:01:07.364Z","updated_at":"2024-11-20T11:31:17.255Z","avatar_url":"https://github.com/lhoursquentin.png","language":"C","funding_links":[],"categories":["C"],"sub_categories":[],"readme":"# sed-bin: Compile a sed script\n\nThis project allows to translate **sed** to **C** to be able to compile the\nresult and generate a binary that will have the exact same behavior as the\noriginal sed script, for example `echo foo | sed s/foo/bar/` will be replaced\nby `echo foo | ./sed-bin`.\n\n# Table of contents\n\n* [sed-bin: Compile a sed script](#sed-bin-Compile-a-sed-script)\n* [Quick start](#Quick-start)\n  * [Setup](#Setup)\n  * [How to use](#How-to-use)\n  * [Quick step-by-step](#Quick-step-by-step)\n  * [Full walk-through with a bigger script](#Full-walk-through-with-a-bigger-script)\n* [Sample scripts](#Sample-scripts)\n* [How it works](#How-it-works)\n  * [Some generated code](#Some-generated-code)\n* [Why](#Why)\n* [Translating the translator](#Translating-the-translator)\n* [Using this project as a sed alternative](#Using-this-project-as-a-sed-alternative)\n* [Notes](#Notes)\n\n# Quick start\n\n## Setup\nClone the repo and move inside its directory, you'll need the usual UNIX\ncore and build utils (sed, libc, C compiler, shell, make).\n\n*Note: this project is currently tested with the GNU libc (2.31), GNU sed (4.5)\nand GCC (10.1.1) on Fedora 32. Some additional tests have been done on FreeBSD 12.1*\n\n## How to use\n\n### Quick step-by-step\n\nLet's take a simple example:\n```bash\nsh$ echo foo | sed s/foo/bar/\nbar\n```\nAssuming you want to compile `s/foo/bar/`:\n\n- Use the provided [compile](./compile) shell script, this takes care of the C\n  translation and compilation steps:\n```bash\nsh$ echo s/foo/bar/ | ./compile\n+ cat\n+ ./par.sed\n+ make\ncc    -c -o sed-bin.o sed-bin.c\ncc    -c -o address.o address.c\ncc    -c -o operations.o operations.c\ncc    -c -o read.o read.c\ncc   sed-bin.o address.o operations.o read.o   -o sed-bin\nCompiled sed script available: ./sed-bin\n```\n\n- Once the generated C code is compiled, you can use the resulting `sed-bin`\n  binary in place of `sed s/foo/bar/`:\n```bash\nsh$ echo foo | ./sed-bin\nbar\n```\n\nThat's about it!\n\n### Full walk-through with a bigger script\n\nSay you want to compile the following sed script which is used to generate the\ntable of contents of this project's README (can also be found in the [samples\ndirectory](./samples)):\n\n```sed\n#!/bin/sed -f\n\n# Generate table of contents with links for markdown files\n# Usage: sed -f \u003cthis-script\u003e \u003cmardown file\u003e\n\n# ignore code blocks\n/^```/,/^```/d\n\n# no need to index ourselves\n/^# Table of contents/d\n\n# found heading\n/^#/{\n  # save our line and first work on the actual URI\n  h\n  # strip leading blanks\n  s/^#*[[:blank:]]*//\n  s/[[:blank:]]/-/g\n  # punctuation and anything funky gets lost\n  s/[^-[:alnum:]]//g\n  # swap with hold and work on the displayed title\n  x\n  # get rid of last leading # and potential white spaces\n  s/^\\(#\\)*#[[:blank:]]*/\\1/\n  # the remaining leading # (if any) will be used for indentation\n  s/#/  /g\n  # prepare the first half of the markdown\n  s/\\( *\\)\\(.*\\)/\\1* [\\2](#/\n  # append the link kept and remove the newline\n  G\n  s/\\(.*\\)[[:space:]]\\(.*\\)/\\1\\2)/p\n}\nd\n```\n\nLet's use the provided translator [par.sed](./par.sed) which is a big sed\nscript translating other sed scripts to C code. Redirect the output to a file\nnamed `generated.c`. Another file with some declarations called\n`generated-init.c` will be created by the translator automatically. You'll need\nthose two files to generate a working binary.\n\n```sh\nsh$ sed -f par.sed \u003c samples/generate-table-of-contents.sed \u003e generated.c\n```\n\nIf you take a peek at `generated.c`, you'll note that for simplicity and\nreadability the generated code is mostly functions calls, the actual C code\ndoing the work is not generated but mostly found in\n[operations.c](./operations.c). Now we're ready to compile the generated code:\n\n```sh\nsh$ make\ncc    -c -o sed-bin.o sed-bin.c\ncc    -c -o address.o address.c\ncc    -c -o operations.o operations.c\ncc    -c -o read.o read.c\ncc   sed-bin.o address.o operations.o read.o   -o sed-bin\n```\n\nA binary named `sed-bin` has been generated, it should have the exact same\nbehavior as the sed script:\n\n```sh\nsh$ ./sed-bin \u003c README.md\n* [sed-bin: Compile a sed script](#sed-bin-Compile-a-sed-script)\n* [Quick start](#Quick-start)\n  * [Setup](#Setup)\n  * [How to use](#How-to-use)\n  * [Quick step-by-step](#Quick-step-by-step)\n  * [Full walk-through with a bigger script](#Full-walk-through-with-a-bigger-script)\n* [Sample scripts](#Sample-scripts)\n* [How it works](#How-it-works)\n  * [Some generated code](#Some-generated-code)\n* [Why](#Why)\n* [Translating the translator](#Translating-the-translator)\n* [Using this project as a sed alternative](#Using-this-project-as-a-sed-alternative)\n* [Notes](#Notes)\n```\n\n# Sample scripts\n\nSome example sed scripts are available in the [samples](./samples) directory:\n\n- [samples/binary-add.sed](./samples/binary-add.sed)\n- [samples/generate-table-of-contents.sed](./samples/generate-table-of-contents.sed)\n- [samples/tic-tac-toe.sed](./samples/tic-tac-toe.sed)\n- [par.sed (sed to C translator)](./par.sed)\n\nOther notable sed scripts tested with this project:\n\n- [sokoban.sed](https://github.com/aureliojargas/sokoban.sed), a\n  [sokoban](https://en.wikipedia.org/wiki/Sokoban) game written by Aurelio\n  Jargas\n- [dc.sed](http://sed.sourceforge.net/grabbag/scripts/dc.sed), an arbitrary\n  precision reverse polish notation calculator written by Greg Ubben\n- [chainlint.sed](https://github.com/git/git/blob/master/t/chainlint.sed), a git\n  internal tool to detect broken \"\u0026\u0026\" chains in shell scripts.\n\n# How it works\n\n## Some generated code\nThe translator [par.sed](./par.sed) (which is written in sed itself) converts\nsed commands calls to valid C code:\n\n```bash\nsh$ echo y/o/u/ | sed -f ./par.sed\n```\n\nWill output:\n\n```c\ny(\u0026status, \"o\", \"u\");\n```\n\nThe actual logic to handle `y` (and most other commands) is not generated, we\njust need to translate the sed syntax to valid C code, which here stays fairly\nreadable.\n\nLet's look at a slightly more complex example:\n\n```sed\n/foo/{\n  p;x\n}\n```\n\nTranslates to:\n```c\nstatic Regex reg_1 = {.compiled = false, .str = \"foo\"};\nif (addr_r(\u0026status, \u0026reg_1))\n{\n\np(\u0026status);\nx(\u0026status);\n\n}\n```\n\nAnd an example of how labels are handled:\n\n```sed\nb end\n\n# some comment\ni \\\nDoesn't look like this\\\ncode is reachable\n\n: end\n```\n\nTranslates to:\n```c\ngoto end_label;\n\n\n// some comment\ni(\"Doesn't look like this\\ncode is reachable\");\n\nend_label:;\n```\n\n# Why\n\nNot much practical use to this, here are some thoughts:\n\n- Debugging a sed script is hard, one possible way is to run `sed` in gdb,\n  but this assumes some familiarity with the implementation. Here the generated\n  C code is rather close to the original sed script, which should allow gdb to\n  be easier to use (`make -B CFLAGS=-g` for symbols).\n- Might be useful for obfuscation or maybe to limit the scope of sed? Resulting\n  binaries are usually smaller than a full `sed` binary as well.\n- Better speed? Since the generated code is specific to a script, one might\n  expect it to be much faster than using `sed`, since we can skip parsing,\n  walking the AST etc. I didn't do any serious measurements yet, but so far it\n  seems slightly faster than GNU sed (around 20% faster to translate the\n  translator for instance).\n\n# Translating the translator\n\nThe basic idea of this project is to translate **sed** code to **C** code, to\ncompile it and have a resulting binary with the same behavior as the original\nscript.\n\nNow, since the translator from sed to C is written in sed, we should be able to\ntranslate the translator, compile it, and then be able to use the compiled\nversion to translate other sed scripts.\n\nTranslate the translator (`par.sed`) with itself:\n\n```sh\nsh$ ./par.sed \u003c ./par.sed \u003e generated.c\n```\n\n```sh\nsh$ make\ncc    -c -o sed-bin.o sed-bin.c\ncc    -c -o address.o address.c\ncc    -c -o operations.o operations.c\ncc    -c -o read.o read.c\ncc   sed-bin.o address.o operations.o read.o   -o sed-bin\n```\n\nWe now have a binary that should be able to translate sed code, let's try to\ntranslate the translator with it:\n\n```sh\nsh$ ./sed-bin \u003c ./par.sed | diff -s generated.c -\nFiles generated.c and - are identical\n```\n\nGenerated code is identical, which means that at this point we have a\nstandalone binary that is able to translate other sed scripts to C. We no\nlonger need another sed implementation as a starting point to make the\ntranslation.\n\n# Using this project as a sed alternative\n\nA shell script named [sed](./sed) is available in this repository, providing\nthe same interface as a POSIX sed implementation.\n\n```sh\nsh$ echo foo | ./sed s/foo/bar/\nbar\n```\n\nHere `./sed` automates argument parsing, translation, compilation and execution\nof the resulting binary. On one hand this is much heavier than the usual sed\nimplementation, but on the other hand it provides an easy way to quickly test\nand compare this project with other implementations.\n\nThe default translation is done with the `./par.sed` translator script, which\nwill use the default **sed** binary available on the system. So that means we\nneed to use a full **sed** implementation to provide another **sed**\nimplementation which doesn't make much sense. To get rid of this initial\n**sed** dependency simply translate and compile `par.sed`, save the generated\nbinary and then use the **sed** shell script with `SED_TRANSLATOR` environment\nvariable set to the newly created binary.\n\nFor example:\n\n```sh\nsh$ BIN=compiled-translator ./compile ./par.sed\n+ cat ./par.sed\n+ ./par.sed\n+ make\ncc    -c -o address.o address.c\ncc    -c -o operations.o operations.c\ncc    -c -o read.o read.c\ncc    -c -o sed-bin.o sed-bin.c\ncc address.o operations.o read.o sed-bin.o -o compiled-translator\nCompiled sed script available: compiled-translator\nsh$ echo foo | SED_TRANSLATOR=./compiled-translator ./sed 's/foo/bar/'\nbar\n```\n\n# Notes\n\n- Incomplete features / known issues:\n  - with 2 addresses, the `c` command will be executed every time for each\n  matching line instead of only once when leaving the range\n  - no pattern/hold space overflow checks, currently both limited to 8192 bytes\n  as per the minimum POSIX spec requirement. Going over that limit will most\n  likely cause a segfault.\n\n- The translator does not handle invalid sed scripts, it will just generate\n  invalid C code (and potentially loop endlessly) which will probably fail to\n  compile, make sure you can run your script with an actual `sed` implementation\n  before attempting to translate it.\n\n- Non POSIX support is currently not planned, if you are using GNU sed, you can\n  try to see what is not supported by running your script with the `--posix`\n  option. Also check out the [POSIX specification](https://pubs.opengroup.org/onlinepubs/9699919799/utilities/sed.html).\n\n- Only `-n` (suppress the default output) is accepted as a command line\n  argument of the resulting binary.\n\n- The generated binaries currently only accept data from stdin:\n  `./sed-bin \u003c file` not `./sed-bin file`. If you have multiple files use\n  `cat file1 file2 file3 | ./sed-bin`.\n\n- The C code is very rough around the edges (by that I mean dirty and unsafe,\n  for instance allocating everything on the stack without checking any\n  overflow), I'm still working on it, but contributions (issues/comments/pull\n  requests) are also welcomed :)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flhoursquentin%2Fsed-bin","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flhoursquentin%2Fsed-bin","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flhoursquentin%2Fsed-bin/lists"}