{"id":13625327,"url":"https://github.com/tokenrove/build-your-own-shell","last_synced_at":"2025-04-05T16:08:30.093Z","repository":{"id":47261652,"uuid":"79668095","full_name":"tokenrove/build-your-own-shell","owner":"tokenrove","description":"Guidance for mollusks (WIP)","archived":false,"fork":false,"pushed_at":"2024-02-28T16:53:34.000Z","size":75,"stargazers_count":419,"open_issues_count":7,"forks_count":24,"subscribers_count":9,"default_branch":"master","last_synced_at":"2025-03-29T15:08:37.214Z","etag":null,"topics":["shell","tutorial","unix","workshop"],"latest_commit_sha":null,"homepage":"","language":"Tcl","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tokenrove.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-01-21T19:35:32.000Z","updated_at":"2025-03-29T05:06:47.000Z","dependencies_parsed_at":"2024-02-28T17:56:51.011Z","dependency_job_id":null,"html_url":"https://github.com/tokenrove/build-your-own-shell","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tokenrove%2Fbuild-your-own-shell","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tokenrove%2Fbuild-your-own-shell/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tokenrove%2Fbuild-your-own-shell/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tokenrove%2Fbuild-your-own-shell/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tokenrove","download_url":"https://codeload.github.com/tokenrove/build-your-own-shell/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247361691,"owners_count":20926643,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["shell","tutorial","unix","workshop"],"created_at":"2024-08-01T21:01:54.016Z","updated_at":"2025-04-05T16:08:30.071Z","avatar_url":"https://github.com/tokenrove.png","language":"Tcl","funding_links":[],"categories":["Tcl","Tutorials"],"sub_categories":[],"readme":"\u003e All the broken  \n\u003e Too many broken  \n\u003e Shells  \n\u003e In our shellmounds  \n — [Grayceon, Shellmounds](https://grayceon.bandcamp.com/track/shellmounds)\n\n# Build Your Own Shell\n\nThis is the material for a series of workshops I ran at my workplace\non how to write a Unix shell.\n\nThe focus is slightly more on building an interactive shell than a\nscripting-oriented shell, only because I think this is more\ngratifying, even if it's less useful.\n\nBe warned that some of the suggestions and discussion make opinionated\nchoices without discussing equally-valid alternatives.\n\nThis is a work in progress and there may remain many infelicities.\nPatches Thoughtfully Considered.  Feel free to report issues via\n[Github](https://github.com/tokenrove/build-your-own-shell/issues).\n\n## Why write your own shell?\n\nThe shell is at the heart of Unix.  It's the glue that makes all the\nlittle Unix tools work together so well.  Understanding it sheds light\non many of Unix's important ideas, and writing our own is the best\npath to that understanding.\n\nThis workshop has three goals:\n\n - to give you a better understanding of how Unix processes work;\n   - this will make you better at designing and understanding software\n     that runs on Unix;\n - to clarify some common misunderstandings of POSIX shells;\n   - this will make you more effective at using and scripting\n     ubiquitous shells like bash;\n - to help you build a working implementation of a shell you can be\n   excited about working on.\n   - there are endless personal customizations you can make to your\n     own shell, and can help you think about how you interact with\n     your computer and how it might be different.\n\n(some of this rationale is expanded on in my blog post, [Building\nshells with a grain of salt])\n\n[Building shells with a grain of salt]: https://www.cipht.net/2017/10/17/build-your-own-shell.html\n\n## How to use this repository\n\nI've tried to break this up into progressive stages that cover mostly\northogonal topics.  Each stage contains a description of the\nfacilities that will be discussed, a list of manpages to consult, and\na set of tests.  I've tried to also hint at some functionality that is\nfun but not necessary for the tests to pass.\n\nIn the root of this repository, there is a script called `validate`;\nyou can run all the tests against your shell-in-progress by specifying\nthe path to your shell's executable, like this:\n\n``` shell\n$ ./validate ../mysh/mysh\n```\n\nIt should tell you what stage you need to implement next.  You can\nalso run a stage by itself, or an individual test:\n\n```shell\n$ ./validate ../mysh/mysh stage_2\n$ ./validate ../mysh/mysh stage_3 03\n```\n\nTo run the tests, you will need [`expect`], which is usually in a\npackage called `expect`, and a C compiler.  The way the tests are\nimplemented is less robust than one might hope, but should suffice for\nour pedagogical goals.  They are unfortunately somewhat timing\nsensitive, such that some tests will be flaky.  If you encounter a\nspecifically flaky test, please let me know.\n\nThe tests assume you will be implementing a vanilla Bourne-flavored\nshell with some ksh influences.  Feel free to experiment with\nalternate syntax, but if so, you may need to adjust the tests.  Except\nwhere specifically noted, `bash` (and `ksh`) should pass all the\ntests, so you can \"test the tests\" that way.  (Try `./validate\n/bin/bash`.) Likewise, `cat` should fail all the tests.\n\nOriginally, I targeted plain `/bin/sh`, but I decided the material in\nstage 5 was too important.  Still, `dash` will pass everything but\nstage 5.  There are also some other minor compatibility differences\nwith some existing shells; you may run into them if you try them out.\nAny failure (of a supposedly POSIX shell) that isn't documented in the\ncomments of a test should be reported as a bug.\n\n[`prove`]: http://perldoc.perl.org/prove.html\n[`expect`]: http://wiki.tcl.tk/201\n\n# Stages\n\n## 1: [fork/exec/wait](stage_1.md)\n\nIn which we discuss the basics of Unix processes, write the simplest\npossible shell, and then lay the foundations for the rest of the\nsteps.\n\n## 2: [files and pipes](stage_2.md)\n\nIn which we add pipes and fd redirection to our shell.\n\n## 3: [job control and signals](stage_3.md)\n\nIn which we discuss signals and add support for ever-helpful chords\nlike `^C`, `^\\`, and `^Z`.\n\n## 4: [quoting and expansion](stage_4.md)\n\nIn which we discuss environments, variables, globbing, and other\noft-misunderstood concepts of the shell.\n\n## 5: [interactivity](stage_5.md)\n\nIn which we apply some polish to our shell to make it usable for\ninteractive work.\n\n## \u0026: [where to go next](next-steps.md)\n\nIn which I prompt you to go further.\n\n# Shells written from this workshop\n\nI'll link to some of the shells that were written as a result of this\nworkshop here shortly, including a couple I wrote to serve as examples\nof different approaches.\n\n# Supplementary Material\n\n## Tools\n\n - The [shtepper] is a great resource for understanding shell\n   execution.  You can input an expression and see in excruciating\n   detail how it should be evaluated.\n\n[shtepper](http://shell.cs.pomona.edu/shtepper)\n\n## Documents\n\n - [Advanced Programming in the Unix Environment] by Stevens covers\n   all this stuff and is a must-read.  I call this *APUE* throughout\n   this tutorial.\n - Chet Ramey describes [the Bourne-Again Shell] in [the Architecture\n   of Open Source Applications]; this is probably the best thing to\n   read to understand the structure of a real shell.\n - Michael Kerrisk's [the Linux Programming Interface], though fairly\n   Linux-specific, has some great coverage of many of the topics we'll\n   touch on.  I call this *LPI* throughout this tutorial.\n - [Unix system programming in OCaml] shows the development of a simple shell.\n - [Advanced Unix Programming] by Rochkind; chapter 5 has a simple shell.\n - the [tour of the Almquist shell] is outdated but may help you find\n   where some things are implemented in `dash` and other `ash`\n   descendants.\n\n### Other Tutorials\n\nI wrote this workshop partially because I felt other tutorials don't\ngo far enough, but all of these are worth reading, especially if\nyou're having trouble with a stage they cover:\n\n - Stephen Brennan's [Write a Shell in C] is a more detailed look at\n   what is [stage 1](stage_1.md) here.\n - Jesse Storimer's [A Unix Shell in Ruby] gets as far as pipes;\n - Kamal Marhubi's [Let's Build a Shell] also goes about that far;\n - glibc's [Implementing a Job Control Shell] shows specifically how\n   to implement job control;\n - Nelson Elhage's [Signalling and Job Control] covers some\n   of [stage 3](stage_3.md)'s material.\n\n### References\n\n - the [POSIX standard] explains the expectations for the shell and\n   its utilities in reasonable detail.\n - there are [POSIX conformance test suites] but they don't seem to be\n   available in convenient, non-restricted forms.\n - [yash's posix-shell-tests] are only runnable with yash, but the\n   tests themselves are full of useful ideas.\n\n[A Unix Shell in Ruby]: http://www.jstorimer.com/blogs/workingwithcode/7766107-a-unix-shell-in-ruby\n[Advanced Programming in the Unix Environment]: http://www.apuebook.com/\n[Advanced Unix Programming]: http://basepath.com/aup/\n[Implementing a Job Control Shell]: https://www.gnu.org/software/libc/manual/html_node/Implementing-a-Shell.html\n[Let's Build a Shell]: https://github.com/kamalmarhubi/shell-workshop\n[POSIX standard]: http://pubs.opengroup.org/onlinepubs/9699919799/utilities/contents.html\n[Signalling and Job Control]: https://blog.nelhage.com/2010/01/a-brief-introduction-to-termios-signaling-and-job-control/\n[the Architecture of Open Source Applications]: http://www.aosabook.org/en/index.html\n[the Bourne-Again Shell]: http://www.aosabook.org/en/bash.html\n[the Linux Programming Interface]: http://man7.org/tlpi/index.html\n[tour of the Almquist shell]: http://git.kernel.org/cgit/utils/dash/dash.git/tree/src/TOUR\n[Unix system programming in OCaml]: https://ocaml.github.io/ocamlunix/\n[Write a Shell in C]: https://brennan.io/2015/01/16/write-a-shell-in-c/\n[POSIX conformance test suites]: https://www.opengroup.org/testing/testsuites/vscpcts2003.htm\n[yash's posix-shell-tests]: https://github.com/posix-shell-tests/posix-shell-tests\n\n## Shells to Examine\n\n - [busybox]: C; contains both ash and hush, and test suites.\n - [mksh]: C; non-interactive tests.\n - [rc]: C; fairly minimal.\n - [zsh]: C; extremely maximal.\n - [bash](https://savannah.gnu.org/git/?group=bash): C.\n - [fish]: C++11; has expect-based interactive tests.\n - [Thompson shell]: C; the original Unix shell; very minimal.\n - [scsh]: Scheme and C; intended for scripting.\n - [cash]: OCaml; based on scsh.\n - [eshell]: Emacs Lisp.\n - [oil]: Python and C++; has an extensive test suite.\n - [xonsh](http://xon.sh/): Python.\n - [oh](https://github.com/michaelmacinnis/oh): Go.\n - [yash-rs](https://github.com/magicant/yash-rs): Rust.\n\n[busybox]: https://git.busybox.net/busybox/tree/shell\n[cash]: https://github.com/ShamoX/cash\n[eshell]: https://github.com/emacs-mirror/emacs/tree/master/lisp/eshell\n[fish]: https://github.com/fish-shell/fish-shell\n[mksh]: https://github.com/MirBSD/mksh\n[oil]: https://github.com/oilshell/oil\n[rc]: https://github.com/rakitzis/rc\n[scsh]: https://github.com/scheme/scsh\n[Thompson shell]: https://github.com/dspinellis/unix-history-repo/blob/Research-V5-Snapshot-Development/usr/source/s2/sh.c\n[zsh]: https://github.com/zsh-users/zsh\n\n## Links to Resources by Language\n\nAlthough there is an elegant relationship between C and Unix which\nmakes it attractive to write a shell in the former, to minimize\nfrustration I suggest trying a higher-level language first.  Ideally\nthe language will have good support for:\n\n- making POSIX syscalls\n- string manipulation\n- hash tables\n\nLanguages that provide a lot of their own infrastructure with regards\nsignals or threads may be much more difficult to use.\n\n### C++\n\nhttp://basepath.com/aup/ex/group__Ux.html\n\n### Common Lisp\n\nThe most convenient library would be [iolib], which you can get\nthrough [Quicklisp].  You'll need to install `libfixposix` first.\nThere's also [sb-posix] in `sbcl` for the daring.\n\n[iolib]: https://github.com/sionescu/iolib\n[Quicklisp]: https://www.quicklisp.org/\n[sb-posix]: http://www.sbcl.org/manual/#sb_002dposix\n\n### Haskell\n\n - use [the unix package](https://hackage.haskell.org/package/unix)\n - [Hell] might be a starting point\n\n[Hell]: https://github.com/chrisdone/hell/\n\n### Java / JVM-based languages\n\nYou will probably run into issues related to the JVM, particularly\nwith signals and forking, but as a starting point, you could do worse\nthan loading libc with JNA.\n\nThere's also [jtux](http://basepath.com/aup/jtux/index.htm).\n\n### Lua\n\nThere are a variety of approaches, but [ljsyscall] looks promising.\n[luaposix] might be sufficient.\n\n[ljsyscall]: https://github.com/justincormack/ljsyscall\n[luaposix]: https://github.com/luaposix/luaposix\n\n### OCaml\n\n - [Unix module](https://caml.inria.fr/pub/docs/manual-ocaml/libref/Unix.html)\n - [lambda-term](https://github.com/diml/lambda-term)\n - [parsing with OCamllex and menhir](https://realworldocaml.org/v1/en/html/parsing-with-ocamllex-and-menhir.html)\n\nSee also [Unix system programming in OCaml], [cash].\n\n### perl\n\nSee `perlfunc(3perl)`; all the functions we want are at hand, usually\nwith the same name.\n\n### Python\n\nAlthough Python provides higher-level abstractions like\n[`subprocess`], for the purposes of this workshop you probably want to\nuse the functions in [`os`].\n\nPlease note an important gotcha for stage 2!  Since [Python 3.4], fds\nhave defaulted to non-inheritable, which means you'll need to\nexplicitly `os.set_inheritable(fd, True)` any file descriptor you\nintend to pass down to a child.\n\n[`os`]: https://docs.python.org/3/library/os.html\n[`subprocess`]: https://docs.python.org/3/library/subprocess.html\n[Python 3.4]: https://peps.python.org/pep-0446/\n\n### Racket\n\nThe implementation seems a little too heavy to do this conveniently,\nbut see the Scheme section below for alternatives.\n\n### Ruby\n\n`Process` has most of what you need.  You can use `Shellwords` but you\ndecide if it's cheating or not.\n\n### Rust\n\nAlthough we use few enough calls that you could just create bindings\ndirectly, either\n[to libc with the FFI](https://doc.rust-lang.org/book/ffi.html) or by\n[directly making syscalls](https://crates.io/crates/syscall), for just\ngetting something working, the [nix-rust] library should provide all\nthe necessary facilities.\n\n[nix-rust]: https://github.com/nix-rust/nix\n\n### Scheme\n\nGuile already has all the calls you need; see\n[the POSIX section of the Guile manual].  Another approach would be to\nuse something like [Chibi Scheme] with bindings to libc calls.\n\n[Chibi Scheme]: https://github.com/ashinn/chibi-scheme\n[the POSIX section of the Guile manual]: https://www.gnu.org/software/guile/manual/html_node/POSIX.html#POSIX\n\n### Tcl\n\nAlthough core Tcl doesn't provide what's necessary, `expect` probably\ndoes.  For example, Tcl doesn't have a way to `exec`, but expect\nprovides `overlay` to do this.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftokenrove%2Fbuild-your-own-shell","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftokenrove%2Fbuild-your-own-shell","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftokenrove%2Fbuild-your-own-shell/lists"}