{"id":51019719,"url":"https://github.com/pyrex41/ratatoskr","last_synced_at":"2026-06-21T15:30:30.301Z","repository":{"id":364151231,"uuid":"1265689387","full_name":"pyrex41/ratatoskr","owner":"pyrex41","description":"Ratatoskr: a tree-shaker for Shen programs (descended from Mark Tarver's Yggdrasil) — shakes a program against ShenOSKernel 41.2 and builds standalone artifacts on Common Lisp, LuaJIT, Go, and Rust","archived":false,"fork":false,"pushed_at":"2026-06-12T19:47:00.000Z","size":769,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-12T21:18:13.015Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"HTML","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/pyrex41.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-06-11T02:08:28.000Z","updated_at":"2026-06-12T19:47:04.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/pyrex41/ratatoskr","commit_stats":null,"previous_names":["pyrex41/yggdrasil","pyrex41/ratatoskr"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/pyrex41/ratatoskr","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pyrex41%2Fratatoskr","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pyrex41%2Fratatoskr/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pyrex41%2Fratatoskr/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pyrex41%2Fratatoskr/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/pyrex41","download_url":"https://codeload.github.com/pyrex41/ratatoskr/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pyrex41%2Fratatoskr/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34616509,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-21T02:00:05.568Z","response_time":54,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-06-21T15:30:29.496Z","updated_at":"2026-06-21T15:30:30.290Z","avatar_url":"https://github.com/pyrex41.png","language":"HTML","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Ratatoskr\n\nA tree-shaker for [Shen](https://shenlanguage.org) programs, targeting\nShenOSKernel **41.2**. Descended from Mark Tarver's **Yggdrasil 1.0**\n(3-clause BSD) — in the myth, Ratatoskr is the squirrel that runs the\ntrunk of Yggdrasil, carrying messages between crown and roots; here it\nwalks the kernel call graph and carries a minimal slice of the tree to\neach target runtime.\n\nDr. Tarver's original vision and description, *Using Yggdrasil to Generate\nStand-alone Programs from Shen* (Shen Group, 2023), is preserved here as\n[`yggdrasil.pdf`](yggdrasil.pdf). The Yggdrasil 1.0 distribution this\nrepository started from is archived in [`archive/`](archive/) along with the\n[Wayback Machine capture](https://web.archive.org/web/20240430183437/https://www.shenlanguage.org/Download/Yggdrasil.zip)\nit was retrieved from.\n\nRatatoskr turns a Shen program into a minimal, standalone artifact in a\ntarget language: it computes which of the kernel's 1129 functions the\nprogram can actually reach, emits just that slice as KLambda, and hands the\nresult to a per-target builder that compiles it with the target port's own\nKL compiler.\n\n\u003e **The shaker runs on any of the seven ports.** Stage 1 is pure Shen, but\n\u003e it compiles your program to KLambda with the host's `bootstrap`\n\u003e compiler, so the host must emit fully portable KL. All seven ports —\n\u003e shen-cl, shen-lua, shen-go, shen-rust, ShenScript, shen-julia and\n\u003e shen-swift — are now verified to produce a byte-identical `kernel.kl` +\n\u003e manifest and portable user KL (see the Gotchas section for the per-host\n\u003e launcher invocation and the `*hush*` caveat). shen-cl remains the\n\u003e reference and the fastest host; shen-julia and shen-swift matched it\n\u003e byte-for-byte out of the box (shen-swift via a host-side `pr` override so\n\u003e `*hush*` gates only stdout, never file streams, so shakes run under `-q`).\n\n**See it run:** [`DEMO.md`](DEMO.md) is an executable demo (built with\n[showboat]) that shakes one program and produces a running artifact on all\nfive targets; `showboat verify DEMO.md` re-executes every step.\n\n## CLI (`ratatoskr`)\n\nA single static **Go binary** wraps both stages so you don't hand-write the\nlauncher invocation. It embeds the shaker source + the kernel KLambda slice and\nmaterialises them to a cache dir on first use, so it runs with no checkout.\nInstall it three ways:\n\n```bash\ngo install github.com/pyrex41/ratatoskr@latest        # Go toolchain\n# or download a prebuilt release binary for your OS/arch (GitHub Releases)\nuvx --from git+https://github.com/pyrex41/ratatoskr ratatoskr targets  # uvx (builds Go locally)\n```\n\nThen:\n\n```bash\nratatoskr shake prog.shen out/                 # stage 1: emit the KLambda slice\nratatoskr build prog.shen out/ --target go     # stage 1 + build a Go artifact\nratatoskr run   prog.shen out/ --target js     # build, then run it (prints stdout)\nratatoskr targets                              # list stage-2 targets\n```\n\n(The Python `ratatoskr_cli.py` remains in the repo for reference/dev.)\n\n| subcommand | does |\n|---|---|\n| `shake PROG OUTDIR` | stage 1 — emit `kernel.kl` + `\u003cprog\u003e.kl` + manifest |\n| `build PROG OUTDIR --target T` | stage 1 + the stage-2 builder for target `T` |\n| `run PROG OUTDIR --target T` | build, then execute the artifact |\n| `targets` | list available targets (`lisp`/`lua`/`go`/`rust`/`js`/`julia`/`scheme`/`swift`) |\n\nThe stage-1 **host** defaults to shen-cl (the reference, and shake output is\nhost-independent anyway); override with `--host \"\u003clauncher\u003e\"` (e.g. `--host\n\"node /path/shen.js\" --eval-style sub`, or `--eval-style positional` for\nshen-lua). Set the host via `$RATATOSKR_HOST` or `$BIFROST_SHEN_CL`. Stage-2\nbuilders live in the sibling port repos (`../shen-lua`, `../shen-go`, …),\noverridable per target via `$RATATOSKR_SHEN_*_DIR`; the build/run recipes are\ndata in [`builders.json`](builders.json), which [Bifrost](../bifrost)'s\n`--shake` mode reads too.\n\n**Cross-platform.** The CLI is pure-stdlib Python and runs on Linux, macOS and\nWindows. Launcher resolution matches `shen.exe` (PATHEXT) on Windows, and a\n`.bat`/`.cmd` host or a `.sh` builder (the lisp stage-2 `build.sh`) is\nauto-wrapped (`cmd /c` / `sh` — the latter needs git-bash/WSL/MSYS `sh` on\nPATH). The `portability` CI job exercises this on `windows-latest` too. As ever,\nwhether a given target's *toolchain* (sbcl/luajit/go/cargo/node/julia/chez/swift)\nis available is your environment's call.\n\n## Architecture\n\n**Stage 1 — shake** (this repo; run on any of the seven ports — see the\nhost-portability gotcha for per-host launcher syntax):\n\n```\nshen eval -q -l ratatoskr.shen -e '(ratatoskr.shake [\"prog.shen\"] \"out\")'\n```\n\nwrites to `out/`:\n\n| file | contents |\n|---|---|\n| `kernel.kl` | shaken kernel defuns, load order preserved |\n| `\u003cprog\u003e.kl` | the user program compiled to KLambda |\n| `ratatoskr.manifest.txt` | line-oriented contract (`key=value`) |\n| `ratatoskr.manifest` | same, as s-expressions |\n\nThe manifest also reports the artifact's effectful **capabilities** —\n`reaches=` / `cannot-reach=` over `{eval, read, write, file, clock}` —\nderived from the emitted primitive set. `cannot-reach=eval` is a static,\ncertifiable \"this program can never evaluate code at runtime\". See\n`docs/reachability.md`.\n\n**Stage 2 — build** (one builder per target port, living in that port's\nrepo):\n\n| target | builder | output (eval-stripped fib) |\n|---|---|---|\n| Common Lisp | `builders/lisp/build.sh \u003cdir\u003e \u003cexe\u003e` (this repo; `LISP_IMPL=sbcl\\|clisp\\|ecl`) | saved image (SBCL ~36 MB, CLISP ~7.8 MB) or compiled binary (ECL ~620 KB + libecl) |\n| LuaJIT | `shen-lua/bin/ratatoskr-build.lua \u003cdir\u003e \u003cout.lua\u003e` | self-contained .lua (~640 KB, ~25 ms startup) |\n| Go | `shen-go/cmd/ratatoskr-build \u003cdir\u003e \u003coutdir\u003e` then `go build` | static binary (~4.5 MB, ≤10 ms startup, cross-compiles linux/windows) |\n| Rust | `shen-rust/crates/ratatoskr-build \u003cdir\u003e \u003coutdir\u003e` then `cargo build --release` | static binary (~9 MB, ~40 ms startup) |\n| JavaScript | `node ShenScript/bin/ratatoskr-build.js \u003cdir\u003e \u003cout.js\u003e` (`--linked` for needs-eval) | self-contained ES module (~120 KB, runs on Node 20+ / Bun / Deno 2) |\n| Julia | `julia --project=shen-julia shen-julia/bin/ratatoskr-build.jl \u003cdir\u003e \u003coutdir\u003e [--sysimage]` | artifact project; with `--sysimage` a per-program sysimage (~266 MB, ~0.15 s warm startup), else a lib-mode `.jl` (~4 s, no sysimage). The shaken kernel+user defuns are baked as module methods (same AOT technique as shen-julia's own fast boot). |\n| Chez Scheme | `builders/scheme/build.sh \u003cdir\u003e \u003coutdir\u003e` (this repo; `SHEN_SCHEME=\u003ccheckout\u003e`) | self-contained Scheme program dir + `run` launcher (`chez --script`). The shaken kernel+user are compiled with shen-scheme's own `kl-\u003escheme`; overridden kernel fns (`pr`, `shen.char-stoutput?`, dict ops, …) come from shen-scheme's `overrides.scm`, exactly as its own build does. |\n| Swift | `builders/swift/build.sh \u003cdir\u003e \u003coutdir\u003e` (this repo; `SHEN_SWIFT=\u003ccheckout\u003e`) | slice + `run` launcher driving the shen-swift tree-walking interpreter in `--shaken` mode. shen-swift is an *interpreter*, so there is nothing to code-generate (like LuaJIT/Julia it references its runtime); the artifact is the KL slice and the win is boot speed — a ~200-line shaken kernel vs the full ~2500-line kernel. |\n\n**Builder contract**: load `kernel.kl`'s defuns, call `(shen.initialise)`\n(41.2 consolidates all global initialisation there), then run each user\nfile's forms in manifest order — user files contain defuns *and* toplevel\nexpressions that must execute in source order.\n\n## How the shake works\n\nThe kernel call graph (1129 defuns, 5619 edges) is built once by walking\nevery defun body for call-position symbols and cached as plain text\n(`KLambda/callgraph-41.2.shen`). Per shake, a pure worklist reachability\npass runs from the seed set `{shen.initialise} ∪ symbols(user KL)`. See\n`docs/reachability.md` for why this replaced Yggdrasil 1.0's O(N³)\nWarshall transitive closure, and why fancier algorithms lose on this graph.\n\nSeveral kernel \"tables masquerading as code\" (the arity table, the package\nexternal-symbols registry, `*special*`, type-signature keys, lambda-form\neta-entries) are treated as data, not calls; lambda-form entries are\nadditionally filtered to the footprint at write time.\n\n**Eval-stripping**: when the user KL never mentions an eval-capable entry\npoint (`eval`, `eval-kl`, `load`, `tc`, `read`, `input+`, …), the shake\nadditionally drops the `*macros*` registration and replaces\n`shen.f-error`'s interactive track-prompt with a plain `simple-error`,\nletting the macro expander, typechecker, reader and `eval` fall away.\nStripped programs shake to ~100 kernel defuns (~66 KB of KL) and the\nmanifest reports `needs-eval=false`; eval-capable programs keep the full\nmachinery (~561 defuns). Detection over-approximates safely — a stray\nsymbol named `eval` keeps the machinery.\n\nThe eval-capable path is exercised end-to-end by `tests/metaeval.shen`\n(builds expressions as data — a list, a runtime `define`, a string — and\nevaluates them) on all five targets. Each port embeds or links its own\nKL compiler for runtime `eval-kl`: the Lisp builder stages shen-cl's\nprecompiled `compiled/compiler.lsp` when the manifest says\n`needs-eval=true`, and ShenScript requires `--linked` (self-contained\nmode refuses eval-capable manifests).\n\n## Gotchas (hard-won)\n\n- Shen's `read-file` is **not a data reader**: it applies the currying\n  transform to paren applications and turns `[a b c]` into cons ASTs.\n  `.kl` files survive because the symbol walk doesn't care about tree\n  shape; anything else (like the call-graph cache) must be written and\n  parsed as plain text.\n- 41.2's stlib is lazily materialised: `mapc`, `filter`,\n  `remove-duplicates`, `copy-file` don't exist in port runtimes.\n  `ratatoskr.shen` carries its own `rat.*` versions.\n- Compiled KL carries explicit property-table arguments — e.g. the\n  external-symbols registration is a 5-element `put` node, not 4.\n- **Stage 1 runs on all seven ports** (verified 2026-06-12 for `fib` and\n  `prolog` on the first five; shen-julia and shen-swift verified 2026-06-19:\n  byte-identical `kernel.kl` + both manifests against the shen-cl reference,\n  user KL identical modulo gensym numbering). Getting there took one fix per\n  non-shen-cl host, since the user program's KL comes from the host's\n  `bootstrap` (shen→KL) compiler and each had a way of emitting non-portable KL\n  (shen-julia and shen-swift were the exceptions — both matched byte-for-byte\n  with no portability fix):\n  - **shen-cl** — reference host, fastest (~0.06 s): `shen eval -q -l ratatoskr.shen -e '(ratatoskr.shake [\"prog.shen\"] \"out\")'`\n  - **shen-lua** — `bin/shen ratatoskr.shen -e '(ratatoskr.shake ...)'`. Its native engine compiled `prolog?` to port-local `shen.lua-run-query*` hooks; that expansion is now gated to skip the dynamic extent of `bootstrap`, so compiled `.kl` carries the kernel's portable CPS expansion.\n  - **shen-go** — `shen eval -q -l ratatoskr.shen -e '(ratatoskr.shake ...)'`. Gained the standard launcher CLI (`extension-launcher.kl`); the stock binary previously had no `-l`/`-e` and fell straight into the REPL.\n  - **shen-rust** — `shen-rust eval -l ratatoskr.shen -e '(ratatoskr.shake ...)'`. Gained the same launcher CLI (on a 1 GB-stack thread for the deep call-graph walk); also fixed `open/2` to honour the `in`/`out` direction symbol so the KL writers truncate-for-write.\n  - **ShenScript** — `node bin/shen.js eval -l ratatoskr.shen -e '(ratatoskr.shake ...)'`. The async `read-byte`/file streams left EOF as an unsettled promise, so `read-file-as-bytelist` looped forever (the 50-min hang); file streams are now synchronous and the shake finishes in ~25 s.\n  - **shen-julia** — `shen-julia/bin/shen eval -l ratatoskr.shen -e '(ratatoskr.shake ...)'` (omit `-q`: like shen-lua/shen-rust, `*hush*` would otherwise silence the `pr` writes; a host-side `pr` override makes `*hush*` gate only stdout). Pre-create the output dir (the shake doesn't `mkdir`). Produced byte-identical `kernel.kl` + manifests on the first try — no portability fix needed.\n  - **shen-swift** — `shen-swift/.build/release/shen-swift eval -q -l ratatoskr.shen -e '(ratatoskr.shake ...)'`. Tree-walking KLambda interpreter (iOS-capable), drives the standard `extension-launcher.kl` CLI. A host-side `pr` override gates `*hush*` to stdout only (file streams always write), so `-q` is safe. Produced byte-identical `kernel.kl` + manifests against the shen-cl reference on the first try — no portability fix needed.\n  - **`*hush*` caveat**: `-q` sets `*hush*`, and on **shen-lua and\n    shen-rust** that silences the `pr` writes to the output files,\n    producing zero-byte artifacts — **omit `-q` on those two**. shen-cl\n    (native `pr` override), shen-go, ShenScript, shen-julia and shen-swift\n    route `pr` to file streams regardless of `*hush*`, so `-q` is harmless\n    there. Dropping `-q` everywhere is the safe default; it only adds a\n    load-echo line to stdout, not to the artifacts.\n\n## Tests\n\n`tests/{hello,fib,prolog,metaeval}.shen` are the four fixtures; expected\noutputs `hello from shaken shen`, `fib 20 = 6765`,\n`mary likes chocolate: true`, and three lines of `eval ...: 42`\n(metaeval is the eval-capable fixture: `needs-eval=true`, ~568 kernel\ndefuns).\nEvery stage-1 change should be verified through at least one stage-2\nbuilder (the Lua one is fastest).\n\nThe Lisp builder is verified on SBCL, GNU CLISP and ECL (`LISP_IMPL=`).\nCCL is unsupported: no native Apple Silicon build exists. Implementation\nnotes that cost real debugging: shen-cl's native `pr` override is\n`#+(or ccl sbcl)`, so other implementations need the optional stream\nprimitives (`shen.write-string` etc. — the driver installs portable\nfallbacks when missing); and streams captured in a saved image are dead\non restart under CLISP, so the image toplevel rebinds\n`*stoutput*`/`*stinput*` at startup. ECL cannot dump images at all — the\ndriver compiles each module to an object file and links a real\nexecutable via `c:build-program`, with boot replayed at program startup.\n\n## Name and lineage\n\nThis project was previously published as \"Yggdrasil 2.0\". It was renamed\nto Ratatoskr to leave the Yggdrasil name to Dr. Tarver's original work,\nof which this is an independent continuation — same idea, retargeted and\nrebuilt for the 41.2 kernel. If you need to relate the two: Ratatoskr ≈\nYggdrasil 2.0.\n\n[shen-cl]: https://github.com/Shen-Language/shen-cl\n[showboat]: https://github.com/simonw/showboat\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpyrex41%2Fratatoskr","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpyrex41%2Fratatoskr","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpyrex41%2Fratatoskr/lists"}