{"id":50436324,"url":"https://github.com/lostbean/ash_harness","last_synced_at":"2026-05-31T17:01:31.252Z","repository":{"id":358097708,"uuid":"1239316027","full_name":"lostbean/ash_harness","owner":"lostbean","description":null,"archived":false,"fork":false,"pushed_at":"2026-05-15T18:03:43.000Z","size":6071,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-05-15T19:58:40.203Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Elixir","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lostbean.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-05-15T01:21:21.000Z","updated_at":"2026-05-15T18:03:46.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/lostbean/ash_harness","commit_stats":null,"previous_names":["lostbean/ash_harness"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/lostbean/ash_harness","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lostbean%2Fash_harness","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lostbean%2Fash_harness/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lostbean%2Fash_harness/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lostbean%2Fash_harness/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lostbean","download_url":"https://codeload.github.com/lostbean/ash_harness/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lostbean%2Fash_harness/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33739861,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-05-31T02:00:06.040Z","response_time":95,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-05-31T17:01:30.416Z","updated_at":"2026-05-31T17:01:31.235Z","avatar_url":"https://github.com/lostbean.png","language":"Elixir","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n  \u003cimg src=\"ash_harness_banner.png\" alt=\"AshHarness — turn Ash Framework resources into the operating layer for AI agents driven by jido_composer.\" /\u003e\n\u003c/p\u003e\n\n# AshHarness\n\nTurn Ash Framework resources into the operating layer for AI agents\ndriven by `jido_composer`. One source of truth for what the agent can\ndo (Ash actions) and what it's allowed to do (Ash policies).\n\n## Why\n\nIf you've built agents with LangGraph or Swarm, you wrote the tools,\nthe schemas, and the authorization checks separately from your domain\nmodel. They drifted. AshHarness fixes this by deriving the agent's\ntool surface, schema, and gate pipeline from the same Ash resources\nyour application already uses.\n\n## Quickstart\n\nAdd the dependency:\n\n```elixir\ndef deps do\n  [\n    {:ash_harness, \"~\u003e 0.1.2\"}\n  ]\nend\n```\n\nAnnotate an Ash resource with `AshHarness.Resource`:\n\n```elixir\ndefmodule MyApp.Ticket do\n  use Ash.Resource,\n    domain: MyApp.Ticketing,\n    extensions: [AshHarness.Resource]\n\n  agent_annotations do\n    description \"A unit of work in the support queue.\"\n    traversable [:project, :comments]\n    hidden_attributes [:internal_notes]\n    hint :assign, \"Use this to delegate work to a teammate.\"\n  end\n\n  # ... attributes, actions, relationships, policies ...\nend\n```\n\nDeclare an agent:\n\n```elixir\ndefmodule MyApp.TriageAgent do\n  use AshHarness.Agent, domains: [MyApp.Ticketing]\n\n  identity do\n    name \"TriageBot\"\n    description \"Triages incoming support tickets.\"\n    actor \u0026MyApp.bot_actor/0\n    model \"anthropic:claude-sonnet-4-5\"\n  end\n\n  scope do\n    resource MyApp.Ticket do\n      actions [:read, :open_ticket, :assign]\n    end\n  end\n\n  behavior do\n    confirm_before [:assign]\n    auto_execute [:read, :open_ticket]\n  end\n\n  constraints do\n    max_mutations_per_turn 10\n    require_reasoning_for [:assign]\n  end\nend\n```\n\nRun a turn:\n\n```elixir\n{:ok, session} = AshHarness.Harness.new_session(MyApp.TriageAgent)\n{:ok, reply, session} = AshHarness.Harness.run(session, \"Hello\")\n```\n\n## What you get\n\n- **Compile-time tool generation**: one `Jido.Action` module per scoped\n  action; one `Jido.Composer.Skill` per scoped resource. The agent's\n  tool surface is derived from the resource/action definitions, not a\n  separate registration step.\n- **Canonical schema, three renderers**: derive an\n  `%AshHarness.Schema.Canonical{}` once, render to Anthropic / OpenAI /\n  MCP via pure functions.\n- **Gate pipeline**: scope → reasoning → confirmation → budget → policy.\n  Authorization is enforced inside the generated tool, before any Ash\n  mutation, via `Ash.can?` and the resource's policy block.\n- **Confirmation halts**: actions in `confirm_before` surface as Jido\n  `ApprovalRequest`s for the host application to mediate.\n- **Per-turn mutation budget**: counts successful create/update/destroys\n  per turn; reads don't count; failed mutations don't count.\n- **Repair loop**: formats Ash validation errors into LLM-readable\n  feedback; classifies retryable vs. terminal failures.\n- **Delegation**: cross-agent text-only return with anti-corruption\n  boundary (delegate uses its own actor; delegate's records never\n  leak to the caller).\n- **Telemetry**: `[:ash_harness, ...]` events for every gate and\n  every action execution; OTel attribute attachment to active spans.\n- **Eval framework**: declarative scenarios with pass/fail gates\n  (`gate :resource_state`, `gate :invariant`) and diagnostic reports\n  (`report :trajectory`, `report :qualitative`). No composite weighted\n  score — pass/fail is binary (ADR 0002).\n\n## Development gates\n\nThe repo ships three mix aliases for the local quality matrix:\n\n| Command         | What it runs                                                   |\n| --------------- | -------------------------------------------------------------- |\n| `mix qa`        | `format --check-formatted` → `compile --warnings-as-errors` → `test` → `credo --strict`. Fail-fast. Mirrors CI's parent-package job. |\n| `mix qa.full`   | `mix qa` plus `mix dialyzer`. The slow gate; run before pushing or in nightly. |\n| `mix bench`     | τ-bench airline replay (`benchmarks/tau_bench_airline/`). Capability smoke check, not a code-quality gate. |\n\nAll three exit non-zero on the first failing step.\n\n## Testing eval scenarios\n\n`AshHarness.Eval.Runner` drives scenarios end-to-end against the real\nagent. LLM HTTP traffic is recorded/replayed via\n[`req_cassette`](https://hex.pm/packages/req_cassette) so default\n`mix test` is deterministic and never hits the network.\n\nCassettes live at\n`test/cassettes/\u003cmodule-snake-case\u003e/\u003cscenario-snake-case\u003e.json` and\nare committed to source control. The recording mode is controlled by\n`ASH_HARNESS_CASSETTE_MODE`:\n\n| Mode      | Behaviour                                                       |\n| --------- | --------------------------------------------------------------- |\n| (default) | `replay` — fail loudly on missing cassettes                     |\n| `record`  | hit the real LLM and write any missing cassettes                |\n| `bypass`  | hit the real LLM without recording (debug only)                 |\n\nTo re-record a single cassette:\n\n```bash\nrm test/cassettes/my_eval/scenario.json\nASH_HARNESS_CASSETTE_MODE=record mix test test/my_eval_test.exs\n```\n\n## Documentation\n\n- `design/README.md` — architecture, ADRs, layer specs.\n- `docs/coming-from-langgraph-swarm.md` — mental-model map for\n  LangGraph/Swarm users.\n- `docs/coexistence-with-ash-ai.md` — using both libraries together.\n- `benchmarks/tau_bench_airline/` — τ-bench airline-domain port and\n  reproducible results.\n\n## Status\n\nv0.1.2 is the current stable release. v0.1.0 was the bootstrap; v0.1.1\nlanded the supervised SessionAgent, real `mutation_count`, full\ntrajectory log, per-(resource, action) repair caps, and Jido-native\nresume; v0.1.2 closes a design-vs-implementation audit (wires the\ndelegation skill to the LLM, switches gates to structured errors,\nthreads a `:request_id` through all telemetry, fires four `:checked`\npass-events, and aligns design docs with code). The public API is in\n`design/implementation/public-api.md`; stable across v0.1.x patches.\nDynamicAgentNode-based progressive disclosure is deferred to v0.2\n(see Phase 0 notes in\n`openspec/changes/bootstrap-ash-harness-v0-1-0/tasks.md`).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flostbean%2Fash_harness","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flostbean%2Fash_harness","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flostbean%2Fash_harness/lists"}