{"id":51336470,"url":"https://github.com/talgolan/smoke-test-plugin","last_synced_at":"2026-07-02T03:04:36.966Z","repository":{"id":361261378,"uuid":"1253623639","full_name":"talgolan/smoke-test-plugin","owner":"talgolan","description":"Scaffold an executable smoke-test framework into any project. Two slash commands, generic .smokerc config, opinionated authoring guide.","archived":false,"fork":false,"pushed_at":"2026-07-01T16:52:17.000Z","size":247,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-07-01T18:17:45.058Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/talgolan.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-05-29T16:46:57.000Z","updated_at":"2026-07-01T16:52:22.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/talgolan/smoke-test-plugin","commit_stats":null,"previous_names":["talgolan/smoke-test-skill","talgolan/smoke-test-plugin"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/talgolan/smoke-test-plugin","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/talgolan%2Fsmoke-test-plugin","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/talgolan%2Fsmoke-test-plugin/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/talgolan%2Fsmoke-test-plugin/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/talgolan%2Fsmoke-test-plugin/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/talgolan","download_url":"https://codeload.github.com/talgolan/smoke-test-plugin/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/talgolan%2Fsmoke-test-plugin/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35030924,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-02T02:00:06.368Z","response_time":173,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-07-02T03:04:36.119Z","updated_at":"2026-07-02T03:04:36.947Z","avatar_url":"https://github.com/talgolan.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# smoke-test-plugin\n\nA standalone Claude Code plugin that scaffolds an executable, opinionated smoke-test framework into any project.\n\nTwo slash commands, one config file, a shared zsh primitive library, and an authoring guide where every rule has a real failure attached.\n\n---\n\n## Why this exists\n\nSmoke testing — driving a built binary end-to-end against the real environment (real Docker, real ports, real filesystem, real `stty`) — is fundamentally different from unit testing, and most projects either skip it or reinvent it badly.\n\nThis skill is a **packaged version of the smoke-test framework** that grew inside one project (a Docker-managed dev-container CLI) over months of real use. After ~20 manual smoke runs, four major iterations, and the same paste-corruption / sub-shell / cwd / drift bugs surfacing repeatedly, the framework crystallized into something worth extracting:\n\n- A controller (`run.zsh`) that orchestrates **sections** with per-section budget enforcement.\n- A shared library (`lib/`) of zsh primitives — `verify`, `run`, `log`, `pass`, `fail`, `wait_for_port`, plus tmux helpers for any step that needs a real TTY.\n- An **authoring guide** that documents every trap the framework's authors hit, with the originating failure preserved alongside each rule.\n- A **generic configuration surface** (`.smokerc`) so any project — Bun, Rust, Go, Node, Python — can plug in its own SUT binary, repo path, and build command.\n\nThis skill is, deliberately, **the skill we wish we'd had when we started.** Future-you (or a contributor in a new repo) doesn't relearn the bugs we already paid for.\n\n---\n\n## What problem it solves\n\n### 1. Manual paste-and-run smoke tests degrade fast\n\nThe original framework was markdown files of fenced shell blocks the operator copy-pasted into a terminal. Symptoms:\n\n- macOS Terminal flushes paste keystrokes faster than zsh's line editor consumes them — long pastes break (`EOFmkdir` joined, words concatenated, heredocs corrupted).\n- Multi-line setup blocks half-run, smoke continues with broken state.\n- \"Edit `resources/X.conf` using your editor of choice\" prose between fenced blocks gets bypassed; the mutation never lands; downstream verifications fail spuriously.\n- Sub-shell `cd` doesn't persist (`pbpaste | bash` runs in a sub-shell), so step files using relative paths quietly target the wrong cwd.\n\nThis skill replaces all of that with **executable runners**: zsh files driven by a controller, sections in their own sub-shells with absolute paths and idempotent setup, no human in the paste loop.\n\n### 2. Per-section budget enforcement\n\nSmoke runs that hang are worse than smoke runs that fail. A test that loops forever holds CI hostage. Every section runs under `perl -e 'alarm(N); exec @ARGV'`; if it exceeds its budget (default 30 s, override per-step), it's killed and reported as `TIMEOUT` in the summary. The full run still completes; you find out about all failures on one pass.\n\n### 3. Structured logging\n\nEvery primitive (`verify`, `run`, `log`, `pass`, `fail`) tees to a timestamped `$RUN_LOG` plus stdout. The summary table at the end shows pass/fail/timeout per section with durations. No `grep`'ing through unstructured output to find what broke.\n\nFor tmux-spawned processes (`docker run -it`-style steps), pane output also pipes to a per-section log file — so you can debug a process that died in \u003c1 s, before tmux's session even reported.\n\n### 4. Hard-won bug avoidance, codified\n\n`AUTHORING_GUIDE.md` (shipped into every install) is the heart of the skill. Each \"hard rule\" has its origin attached. Sample rules:\n\n- **Use `RUN_OUT` not `out` for caller captures.** `verify` declares `local out` internally; if a step file also uses `out=...`, it gets shadowed during `eval`. (Originating: a §6 conflict-detection check returned PASS even when `RUN_OUT=\"\"` because `out` was empty.)\n- **Stash outside `pdir`.** If a section stashes host state, the stash dir lives at `$pdir.stash` (sibling), not `$pdir/stash` (child). Otherwise `rm -rf $pdir` in Teardown destroys the stash. (Originating: a smoke run destroyed an operator's real `~/.claude/plugins/`.)\n- **Drift / image-rebuild tests rebuild the SUT after editing source resources.** Compile-time-baked resources (Bun's `import ... with { type: \"text\" }`, Go's `embed`, Rust's `include_str!`) don't update when you edit the source file alone.\n\nThe grep gate at the bottom of the guide catches these mechanically before commit.\n\n### 5. Generic by construction\n\n`.smokerc` declares everything project-specific: `SUT_BIN`, `SUT_REPO`, `BUILD_CMD`, optional `PREFLIGHT_TOOLS`, optional hooks (`pre_run`, `post_run`, `reset_cmd`). The `lib/` and step files reference only those abstractions. The same harness drives a Bun-built CLI, a Rust binary, a Go server, or anything else — the framework doesn't care.\n\n### 6. Sections that need a human, or that touch real state\n\nMost sections run unattended in the no-arg `./run.zsh` pass. Some can't — and the framework handles both via `MANUAL_SECTIONS` (a section listed there is excluded from the no-arg run and invoked deliberately with `./run.zsh NN` or `./run.zsh --all`). There are two distinct kinds, authored differently:\n\n- **Operator-paused** — a step a human must perform or eyeball: a GUI action (VS Code Remote-SSH connect), a physical device, a daemon whose control is GUI-only. `pause \"\u003cheadline\u003e\" \"\u003cbody\u003e\"` blocks the run, reads `/dev/tty`, and returns confirm/fail/skip. The operator acts; the script continues.\n- **Auto-driven manual** — fully scripted, no keystrokes, but kept out of the no-arg run because it **mutates shared machine state**: stops and restarts a real daemon (Docker, Apple `container`, a launchd/systemd unit), removes a real image, rebuilds from cold. It self-skips when its precondition (right OS/backend, CLI present) is absent, caps every hang-prone control call (`cap`), asserts the post-condition rather than the launcher's exit code, and restores state at the end.\n\nThe rule of thumb — prefer auto-driven whenever the action is CLI-scriptable; reserve operator-paused for steps no CLI can perform. Full authoring detail (the daemon-control shape, the manual taxonomy, the `cap` helper, the assert-post-condition rule) is in `AUTHORING_GUIDE.md` §14–§15.\n\n---\n\n## How it works\n\n### Three slash commands\n\n- **`/smoke-init`** — scaffolds the framework into a target project (default install path: `docs/superpowers/smoke-tests/`). Interactive prompts collect `SUT_BIN`, `SUT_REPO`, `BUILD_CMD`. Also creates the first runner.\n- **`/smoke-add \u003ctopic\u003e`** — scaffolds an additional runner from the template, in an existing install. Walks up from `$PWD` looking for `.smokerc`; if the walk halts at `.git`, falls back to `\u003cgit-root\u003e/docs/superpowers/smoke-tests/.smokerc`. Pass `--install-path \u003cpath\u003e \u003ctopic\u003e` to bypass discovery for non-default install dirs. Also version-gate-syncs the shared lib if it's behind the skill.\n- **`/smoke-sync`** — refreshes ONLY the shared `lib/` + `AUTHORING_GUIDE.md` in an existing install, with no new runner. Use after a skill update to pull new helpers/rules into a target repo's committed framework. Same discovery as `/smoke-add`; version-gated (never downgrades a lib newer than the skill). This is the decoupled form of the sync `/smoke-add` does as a side effect.\n\n### What lands in the target project\n\n```\n\u003cinstall-path\u003e/                     # default: docs/superpowers/smoke-tests/\n├── .smokerc                        # SUT_BIN, SUT_REPO, BUILD_CMD, hooks\n├── lib/\n│   ├── env.zsh                     # validates .smokerc; provides wait_for_port\n│   ├── log.zsh                     # log/info/warn/err/sect, pass/fail/skip, verify, run\n│   ├── control.zsh                 # poll_until, cap (hard per-command timeout), smoke_keep_on_fail\n│   ├── term-a.zsh                  # tmux pty helpers (term_a_start/wait_port/grep/close)\n│   ├── pause.zsh                   # operator-action prompts (pause / confirm)\n│   ├── history.zsh                 # per-section duration history + adaptive budget/poll\n│   ├── README.md                   # primitives reference\n│   └── .skill-version              # which skill version generated this lib/\n├── AUTHORING_GUIDE.md              # rule catalog + grep gate\n└── \u003ctopic\u003e/                        # one runner\n    ├── run.zsh                     # controller: sections, budgets, summary\n    ├── steps/01-example.zsh        # opinionated template demonstrating the rules\n    └── README.md\n```\n\n### How the runner finds its install dir\n\n`run.zsh` walks up from `${0:A:h}` until it finds `.smokerc`, halting at the first `.git` directory (repo boundary) or at `$HOME`. This is deliberate:\n\n- `.smokerc` is **project-local** and never fights an unrelated parent project's config.\n- A monorepo can host multiple installs (one per sub-package) without crosstalk.\n- The runner refuses with a clear error if no `.smokerc` is found within the boundary.\n\n### Section lifecycle\n\nEach section is a step file `steps/NN-\u003cslug\u003e.zsh`. The controller, for each section in `ALL_SECTIONS`:\n\n1. Checks for a `# BUDGET_SECONDS=N` header (else uses `BUDGET_DEFAULT`, default 30).\n2. Spawns a sub-shell wrapped by `perl -e 'alarm(N); exec'`.\n3. Sources `lib/log.zsh`, `lib/env.zsh`, `lib/control.zsh`, `lib/term-a.zsh`, `lib/pause.zsh` (the top-level controller also sources `lib/history.zsh`).\n4. Sources the step file with `SECTION_SLUG` and `SECTION_NUM` exported.\n5. Records duration + result (`PASS` / `FAIL (rc=N)` / `TIMEOUT (\u003eNs)` / `FAIL-missing`).\n\nAfter all sections, summary table → optional `post_run` hook → exit follows section results (post_run is warn-only).\n\n`./run.zsh` with no args runs every section except those in `MANUAL_SECTIONS`. `./run.zsh NN` runs one section (manual or not); `./run.zsh --all` includes the manual ones. `./run.zsh --list` shows every section with a `[manual]` tag where it applies. See \"Sections that need a human, or that touch real state\" above.\n\n### Configuration\n\n`.smokerc` is a zsh-sourced file with required keys (`SUT_BIN`, `SUT_REPO`, `BUILD_CMD`) and optional knobs (`SMOKE_ROOT`, `PREFLIGHT_TOOLS`, `BUDGET_DEFAULT`, `RUN_LOG_KEEP`). Hooks (`pre_run`, `post_run`, `reset_cmd`) are zsh functions; the runner calls them only if defined (`typeset -f` test) — runners must work without them.\n\n`lib/env.zsh` validates that the required keys are both set AND non-empty, and that `$SUT_REPO` is a directory and `$SUT_BIN` is executable. On failure it prints `BUILD_CMD` so the operator knows how to rebuild.\n\n---\n\n## Design decisions and tradeoffs\n\n### Files committed into target repo\n\nThe skill **scaffolds**, it doesn't proxy. After `/smoke-init`, the target repo owns its `lib/`, `AUTHORING_GUIDE.md`, `.smokerc`, and runners. They live in version control. CI / other developers / different machines all have the framework without needing the skill installed.\n\nTradeoff: bug fixes in the skill don't auto-propagate. Re-run `/smoke-init --force` to overwrite (with a sibling backup at `\u003cpath\u003e.backup-\u003cts\u003e/`) — preserves `.smokerc` and existing runners.\n\n### tmux, not osascript or `pbpaste|bash`\n\nSteps that drive the SUT through `stty` or `docker run -it` need a real TTY. `pbpaste | bash` runs in a non-TTY sub-shell — TTY check fails. `osascript Terminal.app` works for the pty but introduces AppleScript escaping bugs and zombie windows on close.\n\n`tmux new-session -d` gives a real pty, runs detached (no GUI), and `tmux kill-session` is a clean teardown. macOS + Linux portable.\n\n### zsh, not bash\n\nThe framework uses zsh-only features (`${0:A:h}` for absolute script-dir resolution, `${(l:2::0:)arg}` left-pad for section number normalization, `typeset -f` for hook detection, `emulate -L zsh` for safety in step files). Bash compatibility was traded for cleaner code.\n\n### `.smokerc` as bash-source, not JSON/TOML\n\nThe runner is zsh; sourcing `.smokerc` is zero-overhead and supports comments, defaults via `${VAR:-default}`, and inline hook function definitions. JSON/TOML would require parsing, can't carry hook functions, and reads worse for shell folks.\n\n### One opinionated example step file\n\n`steps/01-example.zsh` is the only template. It demonstrates the four most-violated rules inline:\n- Absolute paths.\n- `RUN_OUT` not `out`.\n- `verify` for gates, `log` for info.\n- Idempotent Setup → Steps → Teardown structure.\n\nAuthors copy it as the structural starting point. Comment density is high on purpose.\n\n### Walk-up boundary at `.git` / `$HOME`\n\nIf the runner walked all the way to `/`, it would happily pick up an unrelated parent project's `.smokerc`. Halting at `.git` means each repo's framework is self-contained. `$HOME` is the secondary boundary for non-repo workspaces.\n\n---\n\n## What this skill is NOT\n\n- **Not a unit-testing replacement.** Smoke runs the *binary*, end-to-end, against real infrastructure. Your project's `bun test` / `pytest` / `cargo test` continues unchanged. The two are complementary:\n\n  | | Smoke | Unit |\n  |---|---|---|\n  | Surface | compiled binary | individual functions |\n  | Environment | real (Docker, ports, FS, network) | DI fakes / `tmpdir` / mocks |\n  | Runtime | seconds-to-minutes per section | milliseconds per case |\n  | Failure | binary regression vs real env | logic regression at function boundary |\n\n- **Not a paste-and-run framework.** Markdown smoke tests with fenced shell blocks are unsupported. The framework's first iteration was paste-and-run; the second was executable. The executable form survived; the paste form did not.\n\n- **Not Windows-compatible.** zsh, tmux, perl, lsof, jq required. macOS and Linux only.\n\n- **Not a log-distillation tool.** A predecessor framework shipped a `smoke-distill.sh` to strip noise from paste-and-run transcripts. The executable runner emits structured `$RUN_LOG` directly; no distill stage needed.\n\n- **Not a CI integrator.** No GitHub Actions snippet, no Jenkinsfile generator. The runner is a zsh script that exits 0 on pass, 1 on fail — wire it into whatever CI you use.\n\n- **Not auto-upgrading.** `/smoke-init --force` gives you a one-shot overwrite-with-backup. No per-file diff/merge tooling. (Deferred until a second consumer appears with a real upgrade pain point.)\n\n- **Not opinionated about your project's smoke-test conventions** beyond the framework itself. Where `.smokerc` goes, what `BUILD_CMD` does, what shape your hooks take — your call.\n\n---\n\n## Install\n\nRepo: \u003chttps://github.com/talgolan/smoke-test-plugin\u003e. Published via the\n[`talgolan/claude-plugins`](https://github.com/talgolan/claude-plugins) marketplace.\n\n### Install from GitHub (recommended)\n\nIn Claude Code:\n\n```\n/plugin marketplace add talgolan/claude-plugins\n/plugin install smoke-test-plugin@talgolan\n```\n\nSlash commands `/smoke-init`, `/smoke-add`, and `/smoke-sync` are then available.\n\n### Auto-update\n\nThird-party marketplaces default to **manual update**. Two ways to flip it on:\n\n1. **UI** — `/plugin` → Marketplaces → `talgolan` → Enable auto-update. Updates pull at Claude Code startup.\n2. **Settings** — add to `~/.claude/settings.json`:\n\n   ```json\n   {\n     \"extraKnownMarketplaces\": {\n       \"talgolan\": {\n         \"source\": { \"source\": \"github\", \"repo\": \"talgolan/claude-plugins\" },\n         \"autoUpdate\": true\n       }\n     }\n   }\n   ```\n\nManual update any time:\n\n```\n/plugin marketplace update talgolan\n```\n\nThe skill payload (files copied into target projects) is versioned independently of the skill itself. After a skill update, run `/smoke-sync` in a target project to refresh its `lib/` + `AUTHORING_GUIDE.md` to the new version (version-gated; never downgrades). `/smoke-add \u003ctopic\u003e` does the same sync as a side effect when it scaffolds a runner. For a full re-scaffold (every payload file, including the example runner), `/smoke-init --force` overwrites with a sibling backup.\n\n### Local install (development)\n\nClone and start Claude Code with `--plugin-dir`:\n\n```bash\ngit clone https://github.com/talgolan/smoke-test-plugin ~/active_development/smoke-test-plugin\nclaude --plugin-dir ~/active_development/smoke-test-plugin\n```\n\nAfter editing files in the clone, run `/reload-plugins` inside Claude Code to pick up changes without restarting.\n\n\u003e Don't symlink into `~/.claude/plugins/cache/` — that directory is an internal cache managed by the plugin manager and gets clobbered.\n\n---\n\n## Use\n\nIn your project, in Claude Code:\n\n```\n/smoke-init\n```\n\nAnswer the prompts. The first runner is scaffolded immediately; run it:\n\n```bash\ndocs/superpowers/smoke-tests/\u003ctopic\u003e/run.zsh\n```\n\nAdd more runners (default install path resolves from repo root via the `.git` fallback):\n\n```\n/smoke-add \u003cnew-topic\u003e\n```\n\nFor non-default install paths, pass `--install-path \u003cpath\u003e`:\n\n```\n/smoke-add --install-path \u003cpath\u003e \u003cnew-topic\u003e\n```\n\nAuthor a new section: see the scaffolded `\u003cinstall-path\u003e/AUTHORING_GUIDE.md`.\n\n---\n\n## Requirements\n\n| Tool       | Why |\n|------------|-----|\n| zsh        | Runtime shell for runners and lib helpers. |\n| bash       | shellcheck CI gate uses bash dialect. |\n| tmux       | Real pty for TTY-required SUT commands. |\n| perl       | `alarm()` for budget enforcement. |\n| lsof       | `wait_for_port` LISTEN check. |\n| jq         | Structured assertions in step files. |\n| shellcheck | Developer-side CI gate (not runtime). |\n\nPlus whatever your `BUILD_CMD` needs (Bun, Rust, Go, Node, Python, etc.).\n\n---\n\n## Develop\n\n```bash\ngit clone https://github.com/talgolan/smoke-test-plugin\ncd smoke-test-plugin\nbun install\nbun test          # 60 tests\n```\n\nTests cover scaffold scripts (init/add/force/walk-up/token-substitution/lib-sync), runner behavior (binary check, empty config, preflight, budget timeout), control helpers (`poll_until`, `cap`, evidence preservation), duration history, hooks (pre_run/post_run), and `shellcheck` cleanliness across all shipped zsh files. A fake SUT (`tests/fixtures/fake-sut.zsh`) lets runner tests execute end-to-end without a real binary or Docker dependency.\n\n---\n\n## License\n\nMIT.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftalgolan%2Fsmoke-test-plugin","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftalgolan%2Fsmoke-test-plugin","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftalgolan%2Fsmoke-test-plugin/lists"}