{"id":50041258,"url":"https://github.com/baabakk/llm-ports","last_synced_at":"2026-05-27T05:09:35.394Z","repository":{"id":355960181,"uuid":"1230393366","full_name":"baabakk/llm-ports","owner":"baabakk","description":"TypeScript LLM Abstraction Layer for Multi-Provider AI Systems. Multi-provider routing, fallback chains, USD cost gating, capability factories, tool-use security.","archived":false,"fork":false,"pushed_at":"2026-05-23T11:29:42.000Z","size":2390,"stargazers_count":1,"open_issues_count":6,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-05-23T13:27:24.121Z","etag":null,"topics":["ai","ai-architecture","anthropic","cost-control","generative-ai","llm","llm-abstraction","llm-fallback","llm-routing","multi-provider","multi-provider-ai-support","ollama","openai","software-architecture","typescript","vendor-lock-in"],"latest_commit_sha":null,"homepage":"https://baabakk.github.io/llm-ports/","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/baabakk.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-05-06T00:38:41.000Z","updated_at":"2026-05-23T11:29:45.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/baabakk/llm-ports","commit_stats":null,"previous_names":["baabakk/llm-ports"],"tags_count":46,"template":false,"template_full_name":null,"purl":"pkg:github/baabakk/llm-ports","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/baabakk%2Fllm-ports","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/baabakk%2Fllm-ports/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/baabakk%2Fllm-ports/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/baabakk%2Fllm-ports/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/baabakk","download_url":"https://codeload.github.com/baabakk/llm-ports/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/baabakk%2Fllm-ports/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33546836,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-05-27T02:00:06.184Z","response_time":53,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","ai-architecture","anthropic","cost-control","generative-ai","llm","llm-abstraction","llm-fallback","llm-routing","multi-provider","multi-provider-ai-support","ollama","openai","software-architecture","typescript","vendor-lock-in"],"created_at":"2026-05-21T03:04:19.872Z","updated_at":"2026-05-27T05:09:35.384Z","avatar_url":"https://github.com/baabakk.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# llm-ports: TypeScript LLM Abstraction Layer for Multi-Provider AI Systems\n\nProvider-agnostic LLM architecture for TypeScript.\n\nSwitch providers without changing code.  \nAvoid vendor lock-in.  \nControl cost.  \nReuse prompts as capabilities.\n\nMulti-provider routing • fallback chains • USD cost gating • capability factories • tool-use security • observability\n\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)\n![Status](https://img.shields.io/badge/status-pre--release-orange)\n![TypeScript](https://img.shields.io/badge/TypeScript-first-blue)\n\n---\n\n## The Problem\n\nMost LLM applications break in predictable ways:\n\n- SDK upgrades touch too many files\n- Switching providers requires refactoring\n- Prompt logic is duplicated across features\n- Cost and routing logic are scattered\n- Business logic becomes coupled to provider-specific SDKs\n\nThis is not just an SDK problem.\n\n**It is an architecture problem.**\n\n---\n\n## The Solution\n\n`llm-ports` applies the ports-and-adapters pattern to LLM systems.\n\n\u003e **Only two files in your codebase should know the LLM SDK exists.**\n\nEverything else talks to a typed interface.\n\nInstead of calling models directly, your application uses reusable capabilities:\n\n- classify\n- draft\n- score\n- summarize\n- extract\n- plan\n- analyze\n\nThe LLM stops being a dependency you manage.  \nIt becomes infrastructure you configure.\n\n---\n\n## What You Get\n\n- **Multi-provider LLM routing** across OpenAI, Anthropic, Ollama, Vercel AI SDK, and compatible providers\n- **Fallback chains** when a provider fails or exceeds budget\n- **USD-based cost gating** with hourly, daily, and monthly limits\n- **Reusable prompt capabilities** so prompts are defined once and reused everywhere\n- **Validation recovery** for structured output failures\n- **Tool-use safety primitives** for destructive or confirmation-required actions\n- **Observability hooks** for cost, latency, quality, and outcomes\n- **TypeScript-first API** with full type support\n- **No runtime dependency on LangChain, LlamaIndex, or heavy frameworks**\n\n---\n\n## 60 Second Setup\n\n### 1. Configure providers in `.env`\n\n```env\nLLM_PROVIDER_FAST=anthropic|\u003cmodel\u003e|cost:50/day\nLLM_PROVIDER_SMART=anthropic|\u003cmodel\u003e|cost:200/day\nLLM_TASK_ROUTE_TRIAGE=fast,smart\n```\n\n### 2. Create the port once\n\n```ts\nimport { createRegistryFromEnv } from \"@llm-ports/core\";\nimport { createAnthropicAdapter } from \"@llm-ports/adapter-anthropic\";\n\nexport const llm = createRegistryFromEnv({\n  adapters: {\n    anthropic: createAnthropicAdapter({\n      apiKey: process.env.ANTHROPIC_API_KEY!,\n    }),\n  },\n}).getPort();\n```\n\n### 3. Use it anywhere, with no SDK imports\n\n```ts\nconst result = await llm.generateText({\n  taskType: \"triage\",\n  prompt: \"Classify this email...\",\n});\n```\n\nThe registry:\n\n- selects the right model for the task\n- enforces cost limits\n- falls back through the provider chain on failure\n- records usage, cost, and latency\n\n---\n\n## Capabilities: Reusable LLM Operations\n\nInstead of duplicating prompt logic across files, define a capability once and reuse it.\n\n```ts\nimport { createClassifier } from \"@llm-ports/capabilities\";\nimport { z } from \"zod\";\n\nconst IntentSchema = z.object({\n  intent: z.enum([\"question\", \"request\", \"complaint\", \"feedback\", \"other\"]),\n  urgency: z.enum([\"low\", \"normal\", \"high\"]),\n  reasoning: z.string(),\n});\n\nexport const classifyIntent = createClassifier({\n  port: llm,\n  schema: IntentSchema,\n  schemaName: \"user-intent\",\n  rubric: `\n    question: asking for information\n    request: wants something done\n    complaint: reports a problem\n    feedback: opinion only\n    other: anything else\n  `,\n});\n```\n\nNow call it anywhere:\n\n```ts\nconst result = await classifyIntent({ content: userMessage });\n```\n\nExample output:\n\n```ts\n{\n  intent: \"request\",\n  urgency: \"high\",\n  reasoning: \"The user is asking for a concrete action.\"\n}\n```\n\nWhy this matters:\n\n- Improve a prompt once, and every call site benefits\n- Keep behavior consistent across the system\n- Make debugging and evaluation easier\n- Keep business logic free from provider-specific SDK details\n\n---\n\n## Architecture Overview\n\nBefore:\n\n```text\nApplication code\n  ├─ direct SDK call\n  ├─ direct SDK call\n  ├─ direct SDK call\n  └─ model router leaking SDK types\n```\n\nAfter:\n\n```text\nApplication code\n  ↓\nCapabilities\n  ↓\nLLM Port\n  ↓\nAdapters and Provider Registry\n  ↓\nLLM providers\n```\n\nThe key shift:\n\n\u003e Application code stops calling models directly. It calls capabilities.\n\n---\n\n## Packages\n\n| Package | Purpose |\n|--------|---------|\n| `@llm-ports/core` | Port interfaces, registry, routing, cost gating, validation strategies, content blocks |\n| `@llm-ports/capabilities` | Reusable LLM operation factories |\n| `@llm-ports/adapter-openai` | OpenAI SDK adapter with `baseURL` support for compatible providers |\n| `@llm-ports/adapter-anthropic` | Anthropic SDK adapter |\n| `@llm-ports/adapter-google` | Google Gemini native adapter (@google/genai SDK) — full multimodal, bundled pricing |\n| `@llm-ports/adapter-ollama` | Ollama native adapter with model management |\n| `@llm-ports/adapter-vercel` | Vercel AI SDK adapter for migration and compatibility |\n\n\u003e `@llm-ports/observability` (quality tracking hooks, sinks, deterministic edit-diff helpers) is planned for v0.2.\n\n---\n\n## Examples\n\nSeven runnable examples in [`examples/`](examples/), each its own pnpm workspace package with a README walking through the code:\n\n| Example | What it shows |\n|---|---|\n| [`basic`](examples/basic/) | The smallest possible end-to-end. One adapter, one task type, one `generateText` call. The 60-second-setup demo. |\n| [`multi-provider`](examples/multi-provider/) | Fallback chain (Anthropic primary → OpenAI backup), USD cost gating per provider, capability factory. |\n| [`email-triage`](examples/email-triage/) | The most common production use case, condensed into ~150 lines. Inbound email → classify (intent + urgency + sentiment) → policy gate → draft brand-voiced reply → queue for human review. Capability composition story. |\n| [`streaming-chat`](examples/streaming-chat/) | Express server with three routes: `POST /chat` (one-shot), `POST /chat/stream` (Server-Sent Events), `POST /chat/agent` (tool-augmented). The most common LLM UX patterns in ~30 lines of glue. |\n| [`extract-from-pdf`](examples/extract-from-pdf/) | Document extraction: raw OCR'd invoice text → fully-typed structured object via Zod. Demonstrates `generateStructured`, validation-retry-with-feedback, and the `createExtractor` factory. |\n| [`agent-with-approval`](examples/agent-with-approval/) | Tool-use agent with first-class security primitives. `destructive`, `requiresConfirmation`, `maxOutputBytes` flags + an approval-gate wrapper. The differentiation example. |\n| [`migrate-from-vercel-ai`](examples/migrate-from-vercel-ai/) | Two migration paths for users on Vercel AI SDK: (a) wrap your existing model factories with `@llm-ports/adapter-vercel`, (b) replace `@ai-sdk/*` with native llm-ports adapters. Side-by-side before/after diffs. |\n\nEach example is runnable from the monorepo root:\n\n```bash\npnpm --filter @llm-ports/example-\u003cname\u003e start\n```\n\nSet the relevant API key (`ANTHROPIC_API_KEY`, `OPENAI_API_KEY`) before running. Each example's README documents which keys it needs.\n\n---\n\n## Supported Use Cases\n\nUse `llm-ports` when you need:\n\n- multi-provider LLM routing\n- LLM fallback chains\n- TypeScript LLM abstraction\n- OpenAI and Anthropic provider switching\n- cost control for production LLM applications\n- reusable prompt capabilities\n- structured output validation and recovery\n- tool-use security in agent workflows\n- observability for LLM cost, latency, and quality\n- vendor-neutral AI architecture\n\n---\n\n## When to Use This\n\nUse `llm-ports` if:\n\n- you use 2 or more LLM providers\n- you may switch providers later\n- SDK upgrades have caused multi-file changes\n- prompt logic is duplicated\n- cost control matters\n- you want business logic decoupled from provider SDKs\n\nSkip it if:\n\n- you have 1 or 2 LLM calls\n- you are only prototyping\n- you are intentionally building around one provider-specific feature\n- you want a full agent framework, memory layer, RAG framework, or hosted gateway\n\n---\n\n## Related Tools\n\n| Tool | How `llm-ports` relates |\n|------|--------------------------|\n| Vercel AI SDK | Vercel unifies provider calls. `llm-ports` adds registry, fallback chains, USD cost gating, validation recovery, and capability factories on top. |\n| LiteLLM | LiteLLM is a Python-first HTTP proxy. `llm-ports` is TypeScript and runs in-process with no extra network hop. |\n| Portkey | Portkey is a commercial hosted gateway. `llm-ports` is MIT, in-process, and has no hosted dependency. |\n| LangChain.js | LangChain is a framework. `llm-ports` is a lightweight architecture and control layer. |\n| LlamaIndex.TS | LlamaIndex is retrieval-first. `llm-ports` handles LLM invocation, routing, fallback, and cost control. |\n| Mastra | Mastra is agent-first with built-in memory and workflow primitives. `llm-ports` provides lower-level LLM primitives beneath that layer. |\n\n---\n\n## Known Limitations in Alpha\n\n`llm-ports` is pre-release. The core architecture is stable and the offline regression suite is comprehensive (250+ tests, latency p99 under 1 ms, no doc-rot detected across 110+ snippets). Some adapter and agent paths are still being hardened.\n\nFourteen medium-impact alpha-bake issues ([#1](https://github.com/baabakk/llm-ports/issues/1), [#3](https://github.com/baabakk/llm-ports/issues/3), [#4](https://github.com/baabakk/llm-ports/issues/4), [#5](https://github.com/baabakk/llm-ports/issues/5), [#6](https://github.com/baabakk/llm-ports/issues/6), [#9](https://github.com/baabakk/llm-ports/issues/9), [#12](https://github.com/baabakk/llm-ports/issues/12), [#14](https://github.com/baabakk/llm-ports/issues/14), [#16](https://github.com/baabakk/llm-ports/issues/16), [#19](https://github.com/baabakk/llm-ports/issues/19), [#20](https://github.com/baabakk/llm-ports/issues/20), [#21](https://github.com/baabakk/llm-ports/issues/21), [#24](https://github.com/baabakk/llm-ports/issues/24), [#32](https://github.com/baabakk/llm-ports/issues/32)) shipped in `0.1.0-alpha.1` → `0.1.0-alpha.13` and are now closed. The alpha line completes the v0.1 surface: Gemini multi-turn `runAgent` + native `responseSchema`, runtime model discovery (`LLMPort.listModels()` across 4 adapters + `Registry.checkPricingFreshness()`), `useStrictResponseFormat` on `adapter-openai` for Cerebras strict-JSON, `dangerouslyAllowBrowser` opt-in on openai + anthropic, `reasoningEffort` parameter for o-series / gpt-5-nano / Groq gpt-oss-120b reasoning depth control, capability factories propagating `reasoningEffort` + `signal` + `forceProviderAlias` to the underlying port call, plus an expanded `attemptValidationRepair` pass that catches markdown-wrapped enums, trailing punctuation, stringified-JSON-as-object, and array-with-single-object misreads. The full per-surface inventory lives at the [v0.1 status page](https://baabakk.github.io/llm-ports/v0-1-status).\n\nWhat's still open:\n\n- Some compat-provider models (Groq, Together AI, Fireworks, Clarifai, SambaNova) may require a `pricingOverrides` entry to satisfy the registry's pricing-validation step. Bundled pricing tables cover OpenAI, Anthropic, Google, and Ollama by default. Worked examples for Clarifai's Qwen3.6 35B A3B FP8 and SambaNova's MiniMax-M2.7 are in the [openai adapter docs](https://baabakk.github.io/llm-ports/adapters/openai).\n- Vercel adapter `runAgent` is single-turn only (multi-turn lands in v0.2).\n- Registry walks the chain on **budget gating** AND on **runtime errors** (alpha.7+, default predicate: `ProviderUnavailableError`). Configurable via `runtimeFallback: \"none\" | \"default\" | { shouldFallback }`. Streaming methods walk only on stream-creation failure, not mid-iteration.\n\nIf you hit something not listed here, please [open an issue](https://github.com/baabakk/llm-ports/issues/new/choose) — the bug-report template captures the version + repro shape we need.\n\n---\n\n## Installation\n\n`llm-ports` is in alpha. All 7 packages are at `v0.1.0-alpha.13`. Stable v0.1 lands after a short alpha bake — see the [v0.1 status page](https://baabakk.github.io/llm-ports/v0-1-status) for what's stable today vs still being hardened.\n\n```bash\nnpm install @llm-ports/core\n```\n\nInstall adapters as needed:\n\n```bash\nnpm install @llm-ports/adapter-anthropic\nnpm install @llm-ports/adapter-openai\nnpm install @llm-ports/adapter-google\nnpm install @llm-ports/adapter-ollama\nnpm install @llm-ports/adapter-vercel\nnpm install @llm-ports/capabilities\n```\n\n(All six packages are scoped under `@llm-ports`. They're versioned together via changesets.)\n\nPeer dependency: `zod \u003e=3.24.0 \u003c5`. Bring your own SDKs (`@anthropic-ai/sdk`, `openai`, `ollama`, `ai`).\n\n---\n\n## Documentation\n\nDocumentation site (auto-deployed from `docs/` on every push to `main`):\n\nhttps://baabakk.github.io/llm-ports/\n\nPages:\n\n- Getting Started\n- Concepts: ports, adapters, task routing, cost gating, content blocks, validation strategies\n- Guides: multi-provider routing, local-to-cloud, cost control, custom adapters, observability, security\n- Capabilities: one page per capability\n- Adapters: one page per adapter and feature matrix\n- Migration: from Vercel AI SDK, LangChain.js, and direct provider SDKs\n\n---\n\n## Security\n\nTool use without a threat model is dangerous.\n\n`llm-ports` treats security as a first-class part of the API:\n\n- destructive tool markers\n- confirmation-required actions\n- max output byte limits\n- redaction capability\n- explicit guidance for prompt injection and tool abuse\n\nSee [SECURITY.md](./SECURITY.md).\n\n---\n\n## Contributing\n\nContributions are welcome after the initial v0.1 scaffolding lands.\n\nSee [CONTRIBUTING.md](./CONTRIBUTING.md).\n\n---\n\n## License\n\nMIT. See [LICENSE](./LICENSE).\n\n---\n\n## Status\n\nPre-release.\n\nCurrent target:\n\n- v0.1: core, adapters, cost gating, 7 capability factories\n- v0.2: expanded capabilities and observability package\n- v0.3: additional adapters and markdown skill format evaluation\n\n---\n\n## Follow Releases\n\n`llm-ports` is pre-release. To get notified when v0.1 lands on the `latest` tag (and for every minor release after):\n\n1. Click the **Watch** button at the top of the [GitHub repo](https://github.com/baabakk/llm-ports)\n2. Choose **Custom**\n3. Enable **Releases**\n\nYou'll get an email or notification only when a real version ships. No PR or commit noise.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbaabakk%2Fllm-ports","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbaabakk%2Fllm-ports","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbaabakk%2Fllm-ports/lists"}