{"id":51019724,"url":"https://github.com/pyrex41/shen-rust","last_synced_at":"2026-06-21T15:30:32.755Z","repository":{"id":363726265,"uuid":"1256186641","full_name":"pyrex41/shen-rust","owner":"pyrex41","description":"Shen language port in Rust — 134/134 kernel conformance; tree-walker + AOT-compiled kernel + bytecode VM + opt-in GC; first-class AWS Cedar authorization integration.","archived":false,"fork":false,"pushed_at":"2026-06-21T04:20:12.000Z","size":2373,"stargazers_count":2,"open_issues_count":1,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-21T05:21:54.882Z","etag":null,"topics":["authorization","bytecode-vm","cedar","compiler","functional-programming","garbage-collector","interpreter","klambda","programming-language","rust","shen","theorem-prover"],"latest_commit_sha":null,"homepage":null,"language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/pyrex41.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-06-01T14:41:10.000Z","updated_at":"2026-06-21T04:03:56.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/pyrex41/shen-rust","commit_stats":null,"previous_names":["pyrex41/shen-rust"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/pyrex41/shen-rust","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pyrex41%2Fshen-rust","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pyrex41%2Fshen-rust/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pyrex41%2Fshen-rust/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pyrex41%2Fshen-rust/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/pyrex41","download_url":"https://codeload.github.com/pyrex41/shen-rust/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pyrex41%2Fshen-rust/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34616509,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-21T02:00:05.568Z","response_time":54,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["authorization","bytecode-vm","cedar","compiler","functional-programming","garbage-collector","interpreter","klambda","programming-language","rust","shen","theorem-prover"],"created_at":"2026-06-21T15:30:31.922Z","updated_at":"2026-06-21T15:30:32.740Z","avatar_url":"https://github.com/pyrex41.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# shen-rust\n\n[![CI](https://github.com/pyrex41/shen-rust/actions/workflows/ci.yml/badge.svg)](https://github.com/pyrex41/shen-rust/actions/workflows/ci.yml)\n\nA port of the [Shen](https://shenlanguage.org/) programming language to Rust,\nwith native [AWS Cedar](https://www.cedarpolicy.com/) authorization integration.\n\nShen is a functional language with an integrated logic engine and an optional,\nvery expressive type system (a sequent-calculus theorem prover). `shen-rust`\nboots the upstream **ShenOSKernel-41.2** and passes its full conformance suite —\n**134 / 134 kernel tests**, in every execution mode.\n\nThe name follows the Shen-port convention (`shen-cl`, `shen-go`, `shen-ocaml`):\nthe engine is `shen-rust`. The name `shen-cedar` is reused for the Shen **+**\nCedar integration that ships in `examples/`.\n\n## Quick start\n\n```sh\n# Build the workspace\ncargo build --release\n\n# REPL\ncargo run --release --bin shen-rust\n\n# Long-running / served mode (enables the bytecode VM — ~2.3× warm)\ncargo run --release --bin shen-rust -- --served\n\n# Run the upstream Shen kernel conformance suite (expect 134/0)\ncargo run --release --bin shen-rust -- --kernel-tests\n\n# Standard Shen launcher protocol (extension-launcher.kl), as in shen-cl:\n#   shen-rust eval [-q] [-l FILE] [-e EXPR] [-s KEY VALUE] [-r]\n#   shen-rust script FILE [ARGS...]\n#   shen-rust repl | --help | --version\n# e.g. host stage 1 of the Ratatoskr tree-shaker from its repo root:\n#   shen-rust eval -l ratatoskr.shen -e '(ratatoskr.shake [\"tests/fib.shen\"] \"out\")'\n# (note: omit -q for workloads that write files through `pr` — upstream\n#  *hush* semantics silence `pr` on every stream, unlike shen-cl whose\n#  native pr override ignores *hush* entirely)\ntarget/release/shen-rust eval -e \"(+ 1 2)\"\n```\n\nRustup users get the pinned toolchain from `rust-toolchain.toml`\nautomatically; a Nix flake (`nix develop`) provides a dev shell.\n\n## Documentation\n\n| Doc | Contents |\n|---|---|\n| [DEMO.md](DEMO.md) | **executable tour** — every code block ran for real; re-check it with [showboat](https://github.com/simonw/showboat) (`showboat verify DEMO.md`) |\n| [STATUS.md](STATUS.md) | current state, milestones, known limitations |\n| [ARCHITECTURE.md](ARCHITECTURE.md) | layering, value representation, execution tiers, Cedar bridge |\n| [PERFORMANCE.md](PERFORMANCE.md) | how the gap to `shen-cl` was closed (17× → ~3×), and what remains |\n| [BENCHMARKS.md](BENCHMARKS.md) | the numbers, with methodology — all reproducible from `scripts/` and `benches/` |\n| `design/` | internal working notes: decision records, handoffs, falsified experiments |\n\n## Execution engine\n\nThe same Shen semantics run on tiers chosen for the workload:\n\n| Tier | When | Notes |\n|---|---|---|\n| **Tree-walker** | default | Allocation-light interpreter over the KL AST. Best for one-shot runs. |\n| **AOT kernel** | build time | The 21 kernel KL files are compiled to Rust ahead of time by `crates/klcompile` — control flow lowered (self-tail-calls → loops; `if`/`let`/`cond` and ~18 primitives inlined). |\n| **Bytecode VM** | `--served` / `SHEN_RUST_VM=1` | Runtime closures (`defun`/`lambda`/`freeze`) compile to bytecode. **~2.3× faster than the tree-walker on warm / served workloads** (load a theory once, serve many requests), where the compile cost amortizes. Not the bare default because a one-shot run can't amortize it. See `scripts/warm-bench.sh`. |\n| **AOT overlay** | opt-in, per `.shen` file | Known `.shen` files are compiled to Rust offline (`scripts/codegen-shen-aot.sh`, same klcompile) and committed; after a normal load (all side effects live) the host swaps the loaded defuns for the native versions via a verified manifest (`Interp::install_overlay_if_match` — source hash + kernel digest, silent fallback on mismatch). **~3.1× over the VM, ~11.7× over the tree-walker on served authz workloads** (`benches/authz_served.rs`). |\n| **Cranelift JIT** | `--features jit`, `SHEN_RUST_JIT=1` | Experimental; native codegen for runtime closures. Wins on compute loops but was falsified for the type-checker's CPS continuations — kept gated/off. See `design/jit-productionization-plan.md`. |\n\nEvery tier is differentially tested against the tree-walker and held at 134/0.\n\n**Memory**: `Value` is a word-sized `Copy` tagged `u64` over a non-moving\nmark-sweep GC heap. Collection is opt-in (`SHEN_RUST_GC=1`): the heap stays\ngrow-only by default (protects one-shot latency), and in GC mode a long-running\nembedding gets a **bounded heap** — collection runs at interpreter safepoints\nwith hybrid roots (precise interpreter tables + a conservative native-stack\nscan, aarch64 macOS/Linux). On a 20k-request served loop:\ngrow-only ≈ 482 MB and climbing; GC ≈ 26 MB flat, wall-time neutral\n(`cargo bench --bench gc_boundedness`, machine-checked).\n\n## Shen + Cedar\n\nCedar is AWS's authorization-policy language. `shen-rust` combines with it on\ntwo levels:\n\n**1. Cedar as first-class Shen values.** The engine embeds the `cedar-policy`\ncrate and exposes ~15 `cedar.*` primitives (`crates/shen-rust/src/cedar/`), so\nShen *programs* can parse, author, evaluate, and validate Cedar policies\ndirectly — Cedar handles travel as ordinary Shen values:\n\n```shen\n(set p (cedar.parse-policy \"permit(principal, action, resource);\"))\n(set ps (cedar.policy-set-add (cedar.empty-policy-set) (value p)))\n(cedar.is-authorized (value ps) (cedar.empty-entities)\n   (cedar.make-request (cedar.make-entity-uid \"User\" \"alice\")\n                       (cedar.make-entity-uid \"Action\" \"read\")\n                       (cedar.make-entity-uid \"Doc\" \"d1\") ()))\n;; =\u003e Allow\n```\n\n**2. Worked integration patterns.** `examples/shen-cedar-authz` shows three ways\nthe engine (in served / VM mode) and Cedar combine on the Rust side, natively in\none process:\n\n```sh\ncargo run -p shen-cedar-authz --example gate      # Cedar gates Shen evaluation\ncargo run -p shen-cedar-authz --example verify    # Shen reasons ABOUT Cedar policies\ncargo run -p shen-cedar-authz --example generate  # Cedar generated FROM a Shen spec\n```\n\n- **gate** — each request `(principal, action, resource, shen-source)` is\n  authorized by Cedar (schema-validated); only on `Allow` does the served VM\n  evaluate the source.\n- **verify** — Shen-authored, hierarchy-aware analysis of a Cedar `PolicySet`:\n  flags dead (shadowed) permits and partial conflicts that Cedar's per-request\n  evaluator won't surface, cross-checked against the live authorizer.\n- **generate** — a committed Shen spec (`spec/authz.shen`) is the source of\n  truth; the Shen engine computes the transitive role-grant closure; the host\n  renders, strict-validates, and enforces the resulting Cedar policy.\n\nSee the crate's [README](examples/shen-cedar-authz/README.md) for details.\n\n## shengen — formal backpressure\n\n`crates/shengen-rust` compiles Shen sequent-calculus specs (`specs/`) into Rust\n**guard types**, so a spec change becomes a compile error rather than silent\ndrift — Shen as a formal-spec language for other code. The specs are also\nre-type-checked by the engine itself on every CI run (`scripts/shen-check.sh`).\n\n## Layout\n\n```\ncrates/shen-rust/           the engine: KL runtime, kernel boot, tree-walker, VM, AOT, JIT\ncrates/klcompile/           build-time KL → Rust AOT compiler for the kernel\ncrates/shengen-rust/        Shen sequent-calc specs → Rust guard types (backpressure)\nbin/shen-rust/              the REPL / CLI (`--served`, `--kernel-tests`)\nexamples/shen-cedar-authz/  Shen + Cedar integration (gate / verify / generate)\nkernel/                     vendored ShenOSKernel-41.2 (klambda + conformance tests)\nspecs/                      backpressure specs in Shen sequent-calculus syntax\ndesign/                     architecture + performance design notes\nscripts/                    gates.sh (CI), benches, cross-port + warm benchmarks\n```\n\n## Performance\n\nThe reference target is the upstream `shen-cl` (SBCL) port. Two metrics, two\nanswers (paired interleaved runs, 2026-06-10, Apple M-series):\n\n**One-shot** (`--kernel-tests`, boot + load + run + exit):\n\n| Config | Wall | vs shen-cl |\n|---|---:|---:|\n| shen-cl (SBCL) | ≈ 1.0 s | 1× |\n| shen-rust, bare | ≈ 3.0 s | **~3.0× off** |\n| shen-rust + tc-cache (`SHEN_RUST_TC_CACHE=\u003cdir\u003e`, warm) | ≈ 1.0 s | **at parity / ahead** |\n\nThe bare gap is structural — the boxed-`Value` + interpreted-dispatch model,\nnot a single hot spot. A 2026-06-10 profiling round stripped the runtime's\nown overheads (split-TLS heap access, direct-mapped call-target interning,\nthin-LTO build) for ~18 % cumulative; what remains is the model, where each\nlocal lever measures ≤ ~8 %. The tc-cache win is typecheck-verdict\nmemoization, not raw speed; it's off by default.\n\n**Served** (long-lived process: load once, serve many evaluations) — the\nfunded direction, where the tiers stack:\n\n| Tier | vs tree-walk loaded | vs VM loaded |\n|---|---:|---:|\n| bytecode VM (`--served`) | ~2.3× | 1× |\n| **AOT overlay** (committed `.shen` → native) | ~5.3–11.7× | **~1.9–3.2×** |\n\nOn served spec code the overlay leaves the interpreter entirely (the\nSBCL-shaped answer for that niche): `benches/authz_served.rs` measures\n~3.1× over the VM-loaded arm. For long-running served processes,\n`SHEN_RUST_GC=1` bounds the heap (see \"Execution engine\" above). Full history\nand the GC / value-representation / JIT ladder live in `PERFORMANCE.md`,\n`BENCHMARKS.md`, and `design/perf-*.md`.\n\n## Development\n\n```sh\n./scripts/gates.sh    # full CI: fmt+clippy, build, test, shen-check, audits, kernel-tests\n```\n\n### Two test tiers (port vs. canonical)\n\nThe suite is split into two distinct tiers, kept separate on purpose:\n\n- **Port-authored tests** — `cargo test --workspace`. Rust unit tests plus the\n  integration suites under `crates/shen-rust/tests/`. These are OURS: they\n  exercise the evaluator, primitives, I/O, error catchability, the\n  reader/eval robustness corpus, the stdlib, the bytecode-VM differential, and\n  the CLI launcher. The port-authored suites that mirror the shen-go test\n  layout, category for category, are:\n\n  | file | covers |\n  | --- | --- |\n  | `cli_launcher.rs` | spawns the release binary: `eval -e`/`-l`, `script`, `--version`/`--help`, piped-EOF clean exit (timeout-guarded), the `-q`/`*hush*` file-write divergence, adversarial input without a backtrace |\n  | `primitives_coverage.rs` | 50+ assertions over the native KL primitives via `Interp::eval` (arithmetic/float, cons/hd/tl, predicates, string ops, intern/value/set, absvector, hash, get-time) |\n  | `io_coverage.rs` | open/close, write→read round trip, read-byte EOF = −1, read-file, get-time |\n  | `error_robustness.rs` | the error-catchability contract on BOTH the tree-walker and the bytecode-VM path |\n  | `reader_fuzz.rs` | a deterministic seeded corpus of malformed input through reader+eval; asserts no panic, only catchable errors (no `rand`, no cargo-fuzz) |\n  | `library.rs` | booted-kernel stdlib functions (reverse/append/map/remove/element?/length/…) |\n\n- **Canonical kernel suite** — the `kernel-tests` gate (`scripts/kernel-tests.sh`).\n  This runs the VENDORED ShenOSKernel conformance suite (`kernel/tests/`,\n  134 tests) end to end through the binary and is NOT modified by the port.\n  It is the conformance bar; the port-authored tier is the regression net\n  around the parts the kernel suite doesn't reach (CLI, adversarial inputs,\n  the VM path, I/O edges).\n\nCoverage of the port-authored tier: `scripts/coverage.sh` (a `cargo-llvm-cov`\nwrapper that skips gracefully if the tool isn't installed).\n\n## License\n\n[BSD-3-Clause](LICENSE) for the port code (© 2026 Reuben Brooks). The vendored\nShen kernel under `kernel/` is © 2010–2022 Mark Tarver and retains its own\nBSD-3-Clause license (`kernel/LICENSE.txt`).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpyrex41%2Fshen-rust","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpyrex41%2Fshen-rust","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpyrex41%2Fshen-rust/lists"}