{"id":51033766,"url":"https://github.com/thanos/livebook_test","last_synced_at":"2026-06-22T03:02:03.811Z","repository":{"id":363424995,"uuid":"1263194515","full_name":"thanos/livebook_test","owner":"thanos","description":"Bring mix test-style workflows to Livebook. Test .livemd notebooks locally or in CI/CD by converting them to executable Elixir scripts, validating examples remain working, and supporting local dependency overrides so standalone notebooks can be tested against your current checkout.","archived":false,"fork":false,"pushed_at":"2026-06-17T20:56:04.000Z","size":99,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-17T22:25:13.949Z","etag":null,"topics":["cicd","developer-tools","documentation-testing","elixir","elixir-lang","github-actions","livebook","livemd","mix","mix-task","notebooks","test","testing"],"latest_commit_sha":null,"homepage":"","language":"Elixir","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/thanos.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-06-08T18:02:36.000Z","updated_at":"2026-06-17T20:56:09.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/thanos/livebook_test","commit_stats":null,"previous_names":["thanos/livebook_test"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/thanos/livebook_test","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thanos%2Flivebook_test","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thanos%2Flivebook_test/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thanos%2Flivebook_test/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thanos%2Flivebook_test/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/thanos","download_url":"https://codeload.github.com/thanos/livebook_test/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thanos%2Flivebook_test/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34632723,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-22T02:00:06.391Z","response_time":106,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cicd","developer-tools","documentation-testing","elixir","elixir-lang","github-actions","livebook","livemd","mix","mix-task","notebooks","test","testing"],"created_at":"2026-06-22T03:02:02.386Z","updated_at":"2026-06-22T03:02:03.805Z","avatar_url":"https://github.com/thanos.png","language":"Elixir","funding_links":[],"categories":[],"sub_categories":[],"readme":"# livebook_test\n\n\u003e **mix test for Livebooks** - Keep your Livebook examples honest.\n\n`livebook_test` brings `mix test`-style workflows to Livebook notebooks. It discovers `.livemd` files, converts them to executable Elixir scripts, runs them, and reports failures - locally and in CI/CD.\n\n[![CI](https://github.com/thanos/livebook_test/actions/workflows/ci.yml/badge.svg)](https://github.com/thanos/livebook_test/actions/workflows/ci.yml)\n[![Hex version](https://img.shields.io/hexpm/v/livebook_test.svg)](https://hex.pm/packages/livebook_test)\n[![Hex docs](https://img.shields.io/badge/docs-hexdocs.pm-blue)](https://hexdocs.pm/livebook_test)\n[![License](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)\n[![Coverage Status](https://coveralls.io/repos/github/thanos/livebook_test/badge.svg?branch=main)](https://coveralls.io/github/thanos/livebook_test?branch=main)\n\n## Why?\n\nLivebook notebooks are great for examples, tutorials, and interactive documentation. But they drift:\n\n- A dependency update breaks your example notebook\n- A refactoring breaks the API shown in a tutorial\n- Published notebooks reference old package versions\n\n**livebook_test** catches these problems before your users do.\n\n## Installation\n\nAdd to your Mix project:\n\n```elixir\ndef deps do\n  [\n    {:livebook_test, \"~\u003e 0.1\", only: [:dev, :test], runtime: false}\n  ]\nend\n```\n\nThen fetch dependencies:\n\n```bash\nmix deps.get\n```\n\n## Supported Versions\n\nInstall `livebook_test` in your Mix project as shown above. That is the supported workflow. You do not need to add `livebook` to your `deps` yourself; it is pulled in transitively so notebooks can be exported.\n\n| Component | Supported |\n|-----------|-----------|\n| Elixir | `~\u003e 1.18` (1.18.0 and later) |\n| OTP | 26, 27, 28 |\n| Livebook (transitive) | `~\u003e 0.19.0` |\n\nOn the tested combinations above, `mix deps.get \u0026\u0026 mix compile` should succeed. `mix livebook.test` also runs preflight checks and reports actionable errors when something is unsupported.\n\n**Caveat:** Livebook's upstream docs describe their Hex package primarily as a CLI, not as a library dependency. `livebook_test` relies on that package anyway, and compilation can fail on untested Elixir/OTP pairs or in projects with conflicting deps. If that happens, see [Troubleshooting](#troubleshooting). This is not a reason to avoid installing `livebook_test`.\n\n## Quick Start\n\n```bash\n# Run all discovered notebooks\nmix livebook.test\n\n# Run with verbose output\nmix livebook.test --verbose\n\n# Test against local checkout instead of Hex\nmix livebook.test --mode local\n\n# Run a specific notebook\nmix livebook.test --path examples/basic.livemd\n```\n\n### Example Notebooks\n\n| Notebook | Description |\n|----------|-------------|\n| `examples/basic.livemd` | Simple arithmetic and IO - passes |\n| `examples/mix_install.livemd` | Uses `Mix.install` with Jason - passes |\n| `examples/broken/broken.livemd` | Intentionally failing cells - use to verify failure reporting |\n| `livebooks/local_dep.livemd` | Uses `Mix.install` with local dependency patching |\n\n## Configuration\n\nConfigure in `config/config.exs`:\n\n```elixir\nconfig :livebook_test,\n  paths: [\"livebooks/**/*.livemd\", \"examples/**/*.livemd\"],\n  exclude: [\"**/broken/**/*.livemd\"],\n  dependency_mode: :remote,\n  timeout: 60_000,\n  local_deps: [],\n  verbose: false\n```\n\n### Options\n\n| Option | Default | Description |\n|--------|---------|-------------|\n| `paths` | `[\"livebooks/**/*.livemd\", \"examples/**/*.livemd\"]` | Glob patterns for notebook discovery |\n| `exclude` | `[\"**/broken/**/*.livemd\"]` | Glob patterns to exclude from discovery |\n| `dependency_mode` | `:remote` | `:remote` leaves deps unchanged, `:local` rewrites to path deps |\n| `timeout` | `60_000` | Per-notebook timeout in milliseconds |\n| `local_deps` | `[]` | Keyword list mapping dependency names to local paths |\n| `verbose` | `false` | Show per-notebook details |\n\n## Local Dependency Testing\n\nA core feature: notebooks that use `Mix.install` can be automatically patched to use your local checkout.\n\n### Problem\n\nYour example notebook says:\n\n```elixir\nMix.install([\n  {:my_lib, \"~\u003e 0.5\"}\n])\n```\n\nBut you want CI to test against the current checkout, not the published Hex version.\n\n### Solution\n\n```elixir\n# config/config.exs\nconfig :livebook_test,\n  dependency_mode: :local,\n  local_deps: [\n    my_lib: \".\"\n  ]\n```\n\nNow `{:my_lib, \"~\u003e 0.5\"}` becomes `{:my_lib, path: \"/abs/path/to/project\"}`.\n\nPath-style dependencies are also rewritten in local mode:\n\n```elixir\nMix.install([{:my_lib, path: Path.join(__DIR__, \"..\")}])\n```\n\nOr via CLI:\n\n```bash\nmix livebook.test --mode local\n```\n\n## CI/CD Integration\n\nAdd to your GitHub Actions workflow:\n\n```yaml\n- name: Test Livebooks\n  run: mix livebook.test\n```\n\nWith local deps:\n\n```yaml\n- name: Test Livebooks (local)\n  run: mix livebook.test --mode local\n```\n\nThe task exits with code `0` on success, `1` on failure, and `2` if no notebooks are discovered, perfect for CI gates.\n\n## CLI Options\n\n```\nmix livebook.test [options]\n\nOptions:\n  --path PATTERN    Glob pattern for discovery (repeatable)\n  --exclude PATTERN Glob pattern to exclude from discovery (repeatable)\n  --mode MODE       Dependency mode: local or remote\n  --timeout SECS    Per-notebook timeout in seconds\n  --verbose         Show per-notebook details\n```\n\n## Programmatic API\n\n```elixir\n# Run with defaults\nLivebookTest.run()\n\n# Run with options\nLivebookTest.run(paths: [\"examples/**/*.livemd\"], mode: :local, timeout: 120_000)\n\n# Run and print report, returns exit code\nLivebookTest.run_and_report(verbose: true)\n```\n\n## Pipeline\n\nNotebooks flow through a pipeline:\n\n1. **Discovery** - Find `.livemd` files via glob patterns\n2. **Export** - Convert to `.exs` scripts using `Livebook.live_markdown_to_elixir/1`\n3. **Patch** - Optionally rewrite `Mix.install` deps to local paths\n4. **Run** - Execute each script as an isolated subprocess\n5. **Report** - Summarize results with pass/fail counts and timing\n\n## Example Output\n\n```\n3 notebooks\n3 passed\n0 failed\nTotal time: 2.1s\n\nAll notebooks passed!\n```\n\nWith failures:\n\n```\n3 notebooks\n2 passed\n1 failed\nTotal time: 5.3s\n\nFailed notebooks:\n--------------------\n\n  examples/broken/broken.livemd\n  exit: 1\n    stderr:\n    ** (RuntimeError) Intentional failure for testing\n```\n\n## Architecture\n\n| Module | Responsibility |\n|--------|---------------|\n| `LivebookTest` | Public entry point, orchestration |\n| `LivebookTest.Config` | Configuration resolution |\n| `LivebookTest.Discovery` | Notebook file discovery |\n| `LivebookTest.Exporter` | `.livemd` → `.exs` conversion |\n| `LivebookTest.DependencyPatcher` | Mix.install dependency rewriting |\n| `LivebookTest.Preflight` | Elixir/OTP/Livebook compatibility checks |\n| `LivebookTest.Runner` | Script execution and result collection |\n| `LivebookTest.Report` | Summary formatting and exit codes |\n\n## Kino and Interactive Notebooks\n\nNotebooks with Kino widgets, smart cells, or user inputs often fail when exported to headless scripts. See [Kino limitations](docs/guides/kino_limitations.md) for strategies.\n\n## Troubleshooting\n\n### Livebook fails to compile\n\n```\n** (Mix) Could not compile dependency :livebook\n```\n\nThis means the **transitive** `:livebook` dependency (brought in by `livebook_test`) failed to compile. That does not mean you installed something incorrectly. Livebook pulls in Phoenix, Bandit, and other heavy deps, which can conflict with your project or break on untested Elixir/OTP pairs.\n\n1. Verify you are on Elixir 1.18+ and OTP 26/27/28\n2. Run `mix deps.get \u0026\u0026 mix compile` and inspect the Livebook error\n3. If another dep pulls a conflicting Livebook version, override with `{:livebook, \"~\u003e 0.19.0\", override: true}` in your `deps`\n4. Check for conflicting Phoenix or Bandit versions in your project\n\n### Preflight check failures\n\nWhen `mix livebook.test` fails before running notebooks, read the preflight message. It includes version requirements and next steps.\n\n### Notebook export errors\n\nEnable verbose mode to see which notebook failed export:\n\n```bash\nmix livebook.test --verbose\n```\n\n### Kino-related hangs or failures\n\nSee [Kino limitations](docs/guides/kino_limitations.md). Exclude interactive notebooks from CI with `exclude` patterns.\n\n## Roadmap\n\n| Version | Feature |\n|---------|---------|\n| v0.1.1 | Preflight checks, path dep patching, Kino docs |\n| v0.2.0 | Snapshot testing |\n| v0.3.0 | Parallel execution |\n| v0.4.0 | JUnit output |\n| v0.5.0 | GitHub annotations |\n| v0.6.0 | Notebook metadata and tags |\n| v0.7.0 | Coverage reporting |\n| v0.8.0 | HTML reports |\n| v0.9.0 | Distributed notebook execution |\n| v1.0.0 | Stable public API |\n\n## License\n\nMIT\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthanos%2Flivebook_test","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fthanos%2Flivebook_test","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthanos%2Flivebook_test/lists"}