{"id":50930760,"url":"https://github.com/lukefwalton/answer-engine","last_synced_at":"2026-06-17T04:01:09.661Z","repository":{"id":364204098,"uuid":"1266859762","full_name":"lukefwalton/answer-engine","owner":"lukefwalton","description":"LLM-backed site search that cites its sources or declines plainly. No-leak boundary, eval harness. Markdown → embed → retrieve → cited JSON.","archived":false,"fork":false,"pushed_at":"2026-06-12T04:44:43.000Z","size":64,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-12T05:13:42.575Z","etag":null,"topics":["answer-engine","answerability","archival-data","citations","embeddings","embeddings-similarity","question-answering","rag","rag-chatbot","semantic-search","structured-output"],"latest_commit_sha":null,"homepage":"https://lukefwalton.com","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lukefwalton.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":"NOTICE","maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-06-12T02:41:29.000Z","updated_at":"2026-06-12T04:44:46.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/lukefwalton/answer-engine","commit_stats":null,"previous_names":["lukefwalton/answer-engine"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/lukefwalton/answer-engine","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lukefwalton%2Fanswer-engine","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lukefwalton%2Fanswer-engine/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lukefwalton%2Fanswer-engine/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lukefwalton%2Fanswer-engine/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lukefwalton","download_url":"https://codeload.github.com/lukefwalton/answer-engine/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lukefwalton%2Fanswer-engine/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34433086,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-17T02:00:05.408Z","response_time":127,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["answer-engine","answerability","archival-data","citations","embeddings","embeddings-similarity","question-answering","rag","rag-chatbot","semantic-search","structured-output"],"created_at":"2026-06-17T04:00:53.152Z","updated_at":"2026-06-17T04:01:09.651Z","avatar_url":"https://github.com/lukefwalton.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Answer Engine: An AI that Says \"I Don't Know\"\n\n[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.20676773.svg)](https://doi.org/10.5281/zenodo.20676773)\n[![License](https://img.shields.io/github/license/lukefwalton/answer-engine)](LICENSE)\n[![Release](https://img.shields.io/github/v/release/lukefwalton/answer-engine)](https://github.com/lukefwalton/answer-engine/releases)\n\nA small answer engine that keeps the authorial frame outside the model:\nsources are bounded, private text cannot leak into the prompt, citations are\ngrounded, and refusals are tested.\n\nThis is site-level search that uses an LLM **without being a chatbot**. You\npoint it at a body of work — essays, lyrics, letters, philosophy,\ndocumentation — and it answers one question at a time, with no conversation\nstate, no memory, no persona improvising on your behalf. Each answer is a\none-shot transaction: question in, cited answer or honest refusal out. A\nchatbot that's right most of the time speaks *for* you; an answer engine that\ncites or declines speaks *from* you.\n\nThis repo is the teaching-sized version of the engine behind \"Ask the\nArchive\" on [lukefwalton.com](https://lukefwalton.com). It runs out of the\nbox on a bundled example corpus (by \"Person A\" — a placeholder, not a\nperson), it's small enough to read in one sitting, and the whole design is\nfive ideas. Here they are, in the order the data flows.\n\n**What this is:** a **GitHub example repo** you clone and run locally (`npm\ninstall`, `npm run …`). It is not published to npm — there is no `bin`,\n`main`, or `exports`; you read the source and invoke the CLI scripts, not\n`npm install answer-engine` as a dependency.\n\nIt is deliberately not a framework, hosted app, chatbot UI, or vector-database\nstarter. It is the smallest useful version of the answer contract: what\nevidence may enter the prompt, what must stay out, how citations are grounded,\nand when the system must decline.\n\n**Example content:** everything under `example-content/` is synthetic fiction\nfor the demo, including the first-person notebook entries — written to show\nthe private-layer boundary, not real notes.\n\n## 1. Public records are quotable; private text is not\n\nThe corpus has two layers, and the distinction drives everything downstream\n(`src/corpus.ts`, `src/types.ts`):\n\n- **Records** are published pages — each markdown file becomes a flat,\n  citable record: title, canonical URL, summary, curated themes, full body.\n  The body travels all the way to the model, because you already published it.\n- **Private notes** are material you want *searchable but never quotable* —\n  here, the songwriter's notebook in `example-content/notebook/`. Each note\n  declares the public page it routes to (`about`) and where the moment lives\n  (`locator`). Its text gets embedded, so retrieval can find it. It is never\n  shown to the model.\n\n\u003e In production ([Ask the Archive](https://lukefwalton.com/ask/)), podcast\n\u003e transcripts are part of the public archive: published passages are **records**\n\u003e (retrieved and cited). Unpublished transcript text may be embedded for search\n\u003e but reaches the model only as **routing hints** — where to listen, never what\n\u003e was said. The system must not turn transcripts into uncited private knowledge\n\u003e or persona-voice.\n\u003e This repo uses hand-written notebook entries to show the same boundary\n\u003e without the transcription pipeline.\n\n## 2. Retrieval returns both; assembly strips prose\n\nBoth layers share one embedding space in one versioned index file\n(`artifacts/index.json` — gitignored, because vectors derived from private\ntext are private). Retrieval (`src/retrieve.ts`) scores everything with\nbrute-force cosine plus two conservative boosts — naming a work's title\n(0.30) and using a curated theme verbatim (0.15), because metadata you\nmaintain should outrank raw similarity — and drops anything under a score\nfloor. Weak matches don't get to masquerade as evidence; an empty result is\nwhere \"I don't know\" begins, before any model is involved.\n\nThe result keeps records and notes in **two separate lists**, because what\nhappens next is different for each:\n\n```\n                 ┌── records ────────────────────────────► quotable, citable\ncorpus ─► index ─┤                                         (body travels)\n                 └── private notes ──► retrieval finds\n                     the moment        │\n                                       ▼\n                              assembleEvidence()           src/no-leak.ts\n                                       │  strips the text\n                                       ▼\n                         RoutingHint { hintId, label,\n                                       url, locator }      ◄─ no field for prose\n                                       │\n                     AnswerEvidence = { records, hints } ──► the model\n```\n\n`src/no-leak.ts` is small enough to audit by eye — the only thing\n`toRoutingHint` does is drop the note's text — and it is the whole point:\n`RoutingHint` has **no field for the note's text**, so there is nothing through\nwhich private prose could reach the model: the boundary is the type's *shape*,\nnot a guard somebody remembers to write.\n\n## 3. The model only sees AnswerEvidence\n\nOne Responses API call (`src/answer.ts`), with the policy versioned in code\n(`src/prompt.ts`): records render with their bodies; hints render as label,\nlocator, and URL — `buildUserPrompt` couldn't leak a hint's text if it wanted\nto, because the field doesn't exist. **What does travel is the label and the\nlocator: any frontmatter field that becomes a hint's label or locator reaches\nthe model, so keep titles and locators public-safe.** The body is stripped;\nthose two are not. (Making that boundary structural rather than advisory is\n[`NEXT-STEPS.md`](./NEXT-STEPS.md) A1.) The model is told what a hint *is*: the\nlocation of a relevant private moment, to be routed to, never restated. If no\nevidence cleared the floor at all, the engine returns `not-found` without\nmaking the call — refusal costs nothing.\n\n## 4. Modes are enforced in schema + validator, not vibes\n\nThe answer declares one of four modes, and the modes exactly partition the\ncitation mix — which makes honesty checkable:\n\n| Mode | Citations | Meaning |\n| --- | --- | --- |\n| `supported` | records + hints | claims grounded in the canon, plus where to look further |\n| `partial` | records only | answered from the canon; no private moment bears on it |\n| `related-material` | hints only | \"I can't quote it, but the moment exists — here\" |\n| `not-found` | none, empty answer | \"I don't know,\" plainly |\n\nThree layers enforce this, because the first two are requests and only the\nthird is a guarantee: the JSON schema constrains the shape; `validateAnswer`\nrejects contract violations (a `not-found` with prose, a sourced mode without\nit); then `repairCitationsToEvidence` snaps almost-right citations onto the\nexact retrieved pairs (models mangle URLs more often than they invent\nsources), dedupes, and **re-derives the mode from the final mix** — the model\ncan't claim `supported` while citing nothing but hints. Finally\n`assertCitationsGroundedInEvidence` verifies every citation is the exact\n(id, url) pair of something actually retrieved. An invented source is an\nerror, not a footnote.\n\nOne UI lesson: **retrieved is not cited**.\nRetrieved neighbors are candidates; final citations are evidence. If you build\na web UI around this, render source cards from the final citation list, not\nfrom raw retrieval hits — and render none for `not-found`, even if retrieval\nfound nearby material. Otherwise a refusal can look like it is backed by the\nvery sources the engine declined to use.\n\n## 5. Gold queries are regression tests for answerability\n\n`eval/gold.yaml` is a fixed set of questions with required behavior —\nincluding questions the engine must refuse, and one that must route to the\nnotebook without quoting it. `npm run eval` checks retrieval (one cheap\nbatched embedding call); `-- --full` runs the answer engine and checks modes.\n**Prefer `--ids` or `--from-report` for `--full`** — see [`eval/README.md`](./eval/README.md).\n\nThe rule that makes the eval worth having: **when a query fails, fix the\ncorpus, the scoring, or the prompt — never special-case the question.** We\nlearned that the hard way; [`eval/README.md`](./eval/README.md) tells the\nstory, including a real failing-then-passing walkthrough.\n\n## What this shows, and where it stops\n\nThe strongest objection to this approach is that it works only because the\nframe is easy to own: one archive, one named author, a delimited corpus.\nThe mechanisms here do not depend on that smallness — none of them refers to\ncorpus size. What a bounded demo cannot do, on its own, is prove that holding\nthese surfaces at public, plural, or contested scale is affordable, or that\nsystems where it is genuinely unsettled *whose* frame holds can be made\nanswerable the same way. That is a real limit, and this repo is the bounded\ncase on purpose, not a proof about the unbounded one. The public-scale cost\nquestion is real, but it belongs to the builders of public-scale systems, not\nto a teaching repo note.\n\nIt is worth being exact about *which* limit, because it is narrower than it\nlooks. The gate owns **soundness**: nothing enters an answer that isn't\ngrounded in retrieved evidence or honestly refused. It does not own\n**completeness** — it cannot certify that what was retrieved is what *should*\nhave been. A source that falls below the score floor is simply absent, and a\ngate sees only what reaches it. But absence isn't therefore unowned: the\nscoring, the floor, and the corpus boundary that decide what becomes a\ncandidate are authored constants someone maintains (`src/retrieve.ts`,\n`archive.config.ts`), and the gold set tests recall for the cases it names\n(`eval/gold.yaml`). What stays irreducible is the relevant source no one\nthought to test for — and that is irreducible for any system, since\nanticipating it in full would mean knowing the answer in advance.\n\nWhat the repo does try to show is concrete: that whether a frame is *held* or\njust *inherited* can be settled at control surfaces in running code, not\npromissory labels. The privacy boundary is structural — a type with no field for private prose, not a guard\nsomeone has to remember (`src/no-leak.ts`); modes are re-derived from the\nevidence, not taken from the model's word for it (`src/answer.ts`); refusals\nare regression-tested like any other behavior (`eval/gold.yaml`).\n\nThe [Answerability papers](#related-writing) take up the harder cases — plural\nauthorship, contested frames, systems where *whose* gate applies is itself\nunsettled. This repo is the bounded reference implementation; discussion, issues,\nand PRs that extend, test, or push against those limits are welcome. The bar\nfor new code is the bar the repo sets for itself: least lines that keep the\npromises, boundaries enforced by types or runtime checks, loud failures, and\nno change that makes the eval pass by special-casing a question. Before opening\none, see [`CONTRIBUTING.md`](./CONTRIBUTING.md): it names what is in scope — a\nfailing gold case is the best PR — and what isn't.\n\n---\n\n## Quick start\n\nRequires Node.js 22+ and an OpenAI API key.\n\n```sh\nnpm install\ncp .env.example .env              # add your OPENAI_API_KEY in an editor\n\nnpm run index                                   # embed the example corpus, both layers\nnpm run ask -- \"what does person a think about routine?\"      # → partial, cites the essay\nnpm run ask -- \"how was the bridge in harbor lights written?\" # → related-material, routes to the notebook\nnpm run ask -- \"what does person a think about crypto?\"       # → I don't know.\nnpm run eval                                    # the promises, checked (retrieval)\nnpm run eval -- --from-report latest            # rerun failures only (cheap)\nnpm run eval -- --full --ids q07                # answer engine on one query\n```\n\nThe default models are in `archive.config.ts` (`text-embedding-3-large` +\n`gpt-4o-mini`). Change `answerModel` to any Responses-API model your key\nsupports — the engine adapts (reasoning models get an effort setting, others\nget `temperature: 0`).\n\n## Make it yours\n\n1. Edit `archive.config.ts`: your name, your archive's name, your base URL,\n   where your markdown lives.\n2. Each collection is a directory of `.md`/`.mdx` files. The filename stem is\n   the slug — it becomes part of the record id and the public URL, so name\n   files the way you want your citations to read. Frontmatter the engine\n   reads: `title` (required), `description`/`summary`/`meaning`,\n   `themes`/`keywords`/`topics`, `date`, `draft: true` to skip a file.\n3. Private notes additionally need `about` (the public URL to route to) and\n   `locator` (where the moment lives). One contract to respect: a note's\n   `title` and `locator` ARE public-safe surface — they travel into hints and\n   answers — so write them like captions, not like the note itself. Only the\n   body is private. No private layer? Remove `privateNotesDir` from the\n   config and the engine runs public-only.\n4. Replace `example-content/` with your corpus and rerun `npm run index`.\n5. Rewrite `eval/gold.yaml` for your corpus — keep the refusals.\n\n## Commands\n\n```\nnpm run index       # build/refresh artifacts/index.json (only embeds changes)\nnpm run ask         # ask one question, get a cited answer\nnpm run eval        # gold set, retrieval checks (-- --full for answers; prefer --ids / --from-report)\nnpm test            # offline, deterministic engine tests — no API key\nnpm run typecheck   # tsc --noEmit\n```\n\n## Where to take it\n\nIn the order we'd add them: chunk long documents into overlapping windows so\nretrieval points at passages; more retrieval signals (recency for \"what do\nyou think *now*\", author aliases, per-collection weights); a\ndocument-frequency cap on the theme boost — at four records a verbatim theme\nmatch is signal, but on a large corpus a theme that appears on half the\nrecords boosts nothing and should be discounted; an evidence-selection prune\nbefore synthesis (keep one record per cluster, then the clear winner plus a\nsingle corroborator when it leads the rest by a margin) for when a large\ncorpus makes wide top-k surface correlated neighbors instead of distinct\nsources, which shapes what synthesis *sees*, not what the gate certifies\n(retrieved is still not cited); an HTTP handler\naround `retrieve` + `answerQuestion` with a rate limit, query cap, and cache;\nSQLite or pgvector when the archive outgrows in-memory cosine — the shapes\ndon't change. In production we also keep the wire contract's `not-found`\nempty and let the UI roll plain decline copy at display time, so refusals\nstay honest *and* human.\n\nCode the invariant. Document the scaling pattern. Comment the footgun.\n\nThe empirical companion to this list — the two levers it doesn't name (vector\ndimension and wire format), which only appear once the index crosses a network\nboundary, each gated by the eval rather than by vibes — is in\n[`docs/production-scaling.md`](./docs/production-scaling.md).\n\n## Next steps / open problems\n\n[`NEXT-STEPS.md`](./NEXT-STEPS.md) is the standing record of the **seams we can\nsee** — where the design leaves something to be *owned* rather than structurally\nguaranteed — and the **levers an adopter might pull** that trade quality for\ncost. It is not a roadmap: nothing in it has to be fixed for the engine to keep\nits promises. Each entry is written to be pulled as a ticket, and the\nperformance section is a starter for anyone adapting this to their own system.\nNaming these edges is the program doing what it claims, in the open.\n\n## What stays out\n\nA running deployment grows layers this engine deliberately omits: deterministic\nproduct routes (help, usage, or corpus-count answers that never call a model),\na domain-specific eval guard taxonomy, an ingestion or transcription pipeline,\nand the site's own config. Those are consumer-adapter concerns. They live in\nthe site layer (for \"Ask the Archive,\" the `ask-the-archive/` adapter), not the\nengine, because the value this repo carries is the boundary and the answer\ncontract, not feature parity (`.github/STANDARDS.md` §3, \"What Matters Less\").\nOne line worth holding if you add a deterministic route downstream: it may\nshortcut *delivery*, but it must never be how a gold query passes. A route that\nflips an eval outcome is special-casing the question wearing a hat: the same\nthing §5 forbids, one layer up.\n\n## Citing this software\n\nIf you use or build on this repo, please cite the Zenodo archive (not just\nthe GitHub URL).\n\n- **[`.zenodo.json`](./.zenodo.json)** — metadata for Zenodo's GitHub archive\n  (title, ORCID, related paper DOIs, documentation links). Commit this before\n  each tag; Zenodo reads it from the release snapshot and ignores\n  `CITATION.cff` when it is present.\n- **[`CITATION.cff`](./CITATION.cff)** — GitHub **Cite this repository** UI\n  only.\n\n**Recommended:** cite the [concept DOI](https://doi.org/10.5281/zenodo.20676773)\n— it represents all versions and always resolves to the latest archived release.\n\n| | |\n| --- | --- |\n| DOI | [10.5281/zenodo.20676773](https://doi.org/10.5281/zenodo.20676773) |\n| Code | [github.com/lukefwalton/answer-engine](https://github.com/lukefwalton/answer-engine) |\n| About | [lukefwalton.com/ask/about/](https://lukefwalton.com/ask/about/) |\n\n**Artifact note:** cite [10.5281/zenodo.20710897](https://doi.org/10.5281/zenodo.20710897)\nfor v1.2 of the formal write-up ([`docs/ARTIFACT-NOTE-v1.2.md`](./docs/ARTIFACT-NOTE-v1.2.md)).\nIts concept DOI, [10.5281/zenodo.20686053](https://doi.org/10.5281/zenodo.20686053),\nis separate from the software archive above and resolves to the latest version.\n\nTo pin a specific archived snapshot, pick that release's version DOI on the\n[Zenodo versions page](https://zenodo.org/records/20676773) — no README update\nrequired when a new release lands.\n\n**Cutting a release:** on `main`, run **Actions → release** (patch/minor/major).\nChecked-in metadata must match the latest `v*` tag on the remote (`v2.0.0`\ntoday — the tag already exists). The workflow queues concurrent runs, bumps\nsemver via [`scripts/sync-release-metadata.mjs`](./scripts/sync-release-metadata.mjs),\npushes `main` and the new tag atomically, then creates the GitHub release\nZenodo archives. `CITATION.cff` and `.zenodo.json` both use the concept DOI for\ncitation; Zenodo assigns a version DOI per release on its own.\nIf the workflow pushes refs but GitHub release creation fails, create the release\nmanually from the existing tag in the GitHub UI — **do not re-run** this workflow:\na rerun would bump semver again (e.g. skip `v1.4.0` and cut `v1.4.1`) because\nthe latest tag already advanced.\n\n```bibtex\n@software{walton_answer_engine_2026,\n  author       = {Walton, Luke F.},\n  title        = {Answer Engine: An AI that Says \"I Don't Know\"},\n  year         = {2026},\n  publisher    = {Zenodo},\n  doi          = {10.5281/zenodo.20676773},\n  url          = {https://github.com/lukefwalton/answer-engine}\n}\n```\n\n## Related writing\n\nFormal description of this implementation:\n[`docs/ARTIFACT-NOTE-v1.2.md`](./docs/ARTIFACT-NOTE-v1.2.md) —\n[DOI](https://doi.org/10.5281/zenodo.20710897) (CC BY-NC-ND 4.0).\n\nThis repo is a practical companion to the Answerability papers:\n\n- [The Decision No One Authored](https://lukefwalton.com/writing/the-decision-no-one-authored/) — [DOI](https://doi.org/10.5281/zenodo.20622946)\n- [The Captured Oracle](https://lukefwalton.com/writing/the-captured-oracle/) — [DOI](https://doi.org/10.5281/zenodo.20676328)\n- [The Invariant of Answerability](https://lukefwalton.com/writing/the-invariant-of-answerability/) — [DOI](https://doi.org/10.5281/zenodo.20606493)\n- [Building Answerable AI: Why Automation Needs Owned Error](https://lukefwalton.com/writing/building-answerable-ai/) — [DOI](https://doi.org/10.5281/zenodo.20682307)\n\n## Licenses\n\n| Work | License |\n| --- | --- |\n| [Artifact note](./docs/ARTIFACT-NOTE-v1.2.md) | [CC BY-NC-ND 4.0](https://creativecommons.org/licenses/by-nc-nd/4.0/) |\n| Answerability papers | [CC BY-NC-ND 4.0](https://creativecommons.org/licenses/by-nc-nd/4.0/) |\n| answer-engine (this software) | [Apache-2.0](./LICENSE) |\n\n## Contact\n\nArchived on Zenodo: [10.5281/zenodo.20676773](https://doi.org/10.5281/zenodo.20676773).\n\nBuilt by [Luke F. Walton](https://lukefwalton.com) — contact\n[luke@lukefwalton.com](mailto:luke@lukefwalton.com).\n\nProvided as-is for personal use; no support, warranty, or maintenance is\nimplied. It is a personal project, not written on behalf of any employer.\n\nPRs on this repo are reviewed with\n[Surmado Code Review](https://www.surmado.com/review). Luke F. Walton is\nSurmado’s founder; this is a personal open-source project, not a Surmado product.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flukefwalton%2Fanswer-engine","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flukefwalton%2Fanswer-engine","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flukefwalton%2Fanswer-engine/lists"}