{"id":17188258,"url":"https://github.com/jclem/effect-llm","last_synced_at":"2026-02-21T04:03:20.377Z","repository":{"id":251829054,"uuid":"838566838","full_name":"jclem/effect-llm","owner":"jclem","description":"An Effect wrapper around LLM APIs","archived":false,"fork":false,"pushed_at":"2024-11-06T20:10:07.000Z","size":329,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-10-12T16:39:05.033Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jclem.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-08-05T23:07:53.000Z","updated_at":"2024-11-06T20:10:11.000Z","dependencies_parsed_at":"2024-08-06T02:06:57.183Z","dependency_job_id":"03afcee1-abed-4626-ac23-57374f3f023d","html_url":"https://github.com/jclem/effect-llm","commit_stats":null,"previous_names":["jclem/effect-llm"],"tags_count":61,"template":false,"template_full_name":null,"purl":"pkg:github/jclem/effect-llm","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jclem%2Feffect-llm","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jclem%2Feffect-llm/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jclem%2Feffect-llm/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jclem%2Feffect-llm/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jclem","download_url":"https://codeload.github.com/jclem/effect-llm/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jclem%2Feffect-llm/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29672770,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-21T03:11:15.450Z","status":"ssl_error","status_checked_at":"2026-02-21T03:10:34.920Z","response_time":107,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-15T01:08:32.019Z","updated_at":"2026-02-21T04:03:15.363Z","avatar_url":"https://github.com/jclem.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Effect LLM\n\nEffect LLM built with [Effect](https://effect.website) for interacting with\nlarge language model APIs.\n\nThe goal is of the library is to make it as easy as possible to switch between\nvarious API providers (while also providing a means of using provider-specific\nfunctionality where needed).\n\n# Usage\n\n## Basic Usage\n\nTo use this library, you'll initialize a provider (a client for a specific API\nsuch as Anthropic or OpenAI) and use that provider to make LLM API calls via the\nGeneration service.\n\n```typescript\nconst program = Effect.gen(function* () {\n  const apiKey = yield* Config.redacted(\"ANTHROPIC_API_KEY\");\n  const provider = yield* Providers.Anthropic.make();\n\n  const stream = Generation.stream(provider, {\n    apiKey,\n    model: Providers.Anthropic.Model.Claude35Sonnet,\n    maxTokens: 512,\n    events: [\n      new Thread.UserMessage({\n        content: [new TextChunk({ content: \"Hello, I'm Jonathan.\" })],\n      }),\n    ],\n  });\n\n  const responseText = yield* Generation.getContent(stream);\n\n  yield* Console.log(\"The model says:\", responseText);\n});\n\nprogram.pipe(\n  Effect.provide(HttpClient.layer),\n  Effect.provide(BunContext.layer),\n  BunRuntime.runMain,\n);\n```\n\nNote that provider `make` functions generally also receive a subset of the\nstreaming params as an argument, which can be used to provide some defaults:\n\n```typescript\nEffect.gen(function* () {\n  const provider = yield* Providers.Anthropic.make({\n    defaultParameters: {\n      apiKey: yield* Config.redacted(\"ANTHROPIC_API_KEY\"),\n      model: Providers.Anthropic.Model.Claude35Sonnet,\n      maxTokens: 512,\n      system: \"Be cordial.\",\n      additionalParameters: {\n        temperature: 0.5,\n      },\n    },\n  });\n});\n```\n\nNote that `additionalParameters` passed to a generation function will be merged\nwith `additionalParameters` given to `defaultParams`.\n\nGenerally, it is recommended that you set up the provider as a layer so that it\ncan be swapped out with relative ease[^1].\n\n```typescript\nconst apiKey = Config.redacted(\"ANTHROPIC_API_KEY\");\n\nconst program = Effect.gen(function* () {\n  const provider = yield* Generation.Generation;\n\n  const stream = Generation.stream(provider, {\n    apiKey: yield* apiKey,\n    model: Providers.Anthropic.Model.Claude35Sonnet,\n    maxTokens: 512,\n    events: [\n      new Thread.UserMessage({\n        content: [new TextChunk({ content: \"Hello, I'm Jonathan.\" })],\n      }),\n    ],\n  });\n\n  const responseText = yield* Generation.getContent(stream);\n\n  yield* Console.log(\"The model says:\", responseText);\n});\n\nprogram.pipe(\n  Effect.provide(Layer.effect(Generation.Generation, Anthropic.make())),\n  Effect.provide(HttpClient.layer),\n  Effect.provide(BunContext.layer),\n  BunRuntime.runMain,\n);\n```\n\n## Using the Google Provider\n\nIn order to use the Google Provider, you'll need to keep two things in mind:\n\nThe `Google.make` function accepts two parameters. The first are Google-specific\nconfiguration options, and the second are the optional default parameters that\nthe other providers accept, as well:\n\n```typescript\nGoogle.make(\n  {\n    // Required.\n    serviceEndpoint: \"https://us-central1-aiplatform.googleapis.com\",\n  },\n  {\n    system: \"Be courteous.\",\n  },\n);\n```\n\nSecondly, the \"model\" parameter must be the full model path parameter in this format:\n\n```typescript\nconst params = {\n  model: `projects/${projectID}/locations/${locationID}/publishers/${publisher}/models/${modelName}`,\n};\n```\n\n## Tool-Calling\n\nThere are two ways of utilizing LLM tools in this library.\n\n### Using `Generation.stream`\n\nThe `Generation.stream` function accepts `tools` and `toolCall` as\nparameters. When using these parameters, you can expect to see the following\nevents emitted from the stream:\n\n| Name            | Payload Type                                                                  | Description                                                                       |\n| --------------- | ----------------------------------------------------------------------------- | --------------------------------------------------------------------------------- |\n| `ToolCallStart` | `{ readonly id: string; readonly name: string; }`                             | Emitted when a tool call begins, but before its arguments have been streamed      |\n| `ToolCall`      | `{ readonly id: string; readonly name: string; readonly arguments: string; }` | Emitted when a tool call and its arguments have been fully streamed and collected |\n\nNote that when using `Generation.stream`, the tool calls are not validated or\nexecuted, nor are there arguments even parsed.\n\n### Using `Generation.streamTools`\n\nIf instead you would like to have effect-llm parse and _execute_ tool calls for\nyou, use `Generation.streamTools`. This tool accepts the same parameters as\n`Generation.stream`, with the addition of a `maxIterations` parameter used to\nlimit the number of loops that will be executed. When using\n`Generation.streamTools`, the following sequence of events will occur:\n\n1. Send the completion request to the provider\n2. Parse the response\n3. If there are tool calls in the response:\n   1. Parse the arguments\n   2. Append the tool call to the events list\n   3. Call the tool\n   4. Append the tool result to the events list\n   5. Go to (1) with the new events list\n4. OR, If there are no tool calls in the resopnse:\n   1. End the stream\n\nIf the `maxIterations` limit is exceeded, the stream will emit a\n`MaxIterationsError`.\n\nThe stream returned by `Generation.streamTools` emits the same events as\n`Generation.stream` with some additions:\n\n| Name                | Payload Type                                                                  | Description                                                                                   |\n| ------------------- | ----------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------- |\n| `ToolCallStart`     | `{ readonly id: string; readonly name: string; }`                             | Emitted when a tool call begins, but before its arguments have been streamed                  |\n| `ToolCall`          | `{ readonly id: string; readonly name: string; readonly arguments: string; }` | Emitted when a tool call and its arguments have been fully streamed and collected             |\n| `InvalidToolCall`   | `{ readonly id: string; readonly name: string; readonly arguments: string; }` | Emitted when a tool call's arguments are invalid or the tool call is not in the defined tools |\n| `ToolResultSuccess` | `{ readonly id: string; readonly name: string; readonly result: unknown }`    | Emitted when a tool call's arguments are invalid or the tool call is not in the defined tools |\n| `ToolResultError`   | `{ readonly id: string; readonly name: string; readonly result: unknown }`    | Emitted when a tool call's arguments are invalid or the tool call is not in the defined tools |\n\n### Defining Tools\n\nTo define a tool for use with `Generation.streamTools`[^2], use the\n`Generation.defineTool` function:\n\n```typescript\nconst apiKey = Config.redacted(\"ANTHROPIC_API_KEY\");\n\nconst program = Effect.gen(function* () {\n  const provider = yield* Generation.Generation;\n\n  const sayHello = Generation.defineTool(\"sayHello\", {\n    description: \"Say hello to the user\",\n    input: Schema.Struct({ name: Schema.String }),\n    effect: (toolCallID, toolArgs) =\u003e\n      Console.log(`Hello, ${toolArgs.name}`).pipe(Effect.as({ ok: true })),\n  });\n\n  const stream = Generation.streamTools(provider, {\n    apiKey: yield* apiKey,\n    model: Providers.Anthropic.Model.Claude35Sonnet,\n    maxTokens: 512,\n    tools: [sayHello],\n    events: [\n      new Thread.UserMessage({\n        content: [new TextChunk({ content: \"Hello, I'm Jonathan.\" })],\n      }),\n    ],\n  });\n\n  yield* stream.pipe(Stream.runDrain, Effect.scoped);\n});\n\nprogram.pipe(\n  Effect.provide(Layer.effect(Generation.Generation, Anthropic.make())),\n  Effect.provide(HttpClient.layer),\n  Effect.provide(BunContext.layer),\n  BunRuntime.runMain,\n);\n```\n\n#### Error Handling\n\nAny errors that occur during tool execution will _halt_ the stream and yield\na `ToolExecutionError`. In order to handle an error and report it to the\nmodel, you should instead fail the effect with a `ToolError` using the\n`Generation.toolError` function:\n\n```typescript\nconst sayHello = Generation.defineTool(\"sayHello\", {\n  description: \"Say hello to the user\",\n  input: Schema.Struct({ name: Schema.String }),\n  effect: (toolCallID, toolArgs) =\u003e\n    Console.log(`Hello, ${toolArgs.name}`).pipe(\n      Effect.catchAll((err) =\u003e\n        Generation.toolError({\n          message: \"An error occurred while saying hello\",\n          error: err,\n        }),\n      ),\n      Effect.as({ ok: true }),\n    ),\n});\n```\n\nYou can also fail mid-effect, since `Generation.toolError` actually fails the effect:\n\n```typescript\nconst sayHello = Generation.defineTool(\"sayHello\", {\n  description: \"Say hello to the user\",\n  input: Schema.Struct({ name: Schema.String }),\n  effect: (toolCallID, toolArgs) =\u003e\n    Effect.gen(function* () {\n      return yield* Generation.toolError(\"Whoops!\");\n    }),\n});\n```\n\nThe payload passed to `Generation.toolError` can be any value, and it is\nserialized as JSON and sent to the model, which is notified that an error\noccurred.\n\n#### Halting Early\n\nIf you want to halt the iteration loop eraly, you can use the\n`Generation.haltToolLoop` function:\n\n```typescript\nconst sayHello = Generation.defineTool(\"sayHello\", {\n  description: \"Say hello to the user\",\n  input: Schema.Struct({ name: Schema.String }),\n  effect: (toolCallID, toolArgs) =\u003e\n    Effect.gen(function* () {\n      return yield* Generation.haltToolLoop();\n    }),\n});\n```\n\nThis will immediately halt the loop before executing any other tool calls\nreturned by the model in that same loop, and will yield end the stream without\nan error.\n\n[^1]:\n    There are some caveats to this—for example, the `stream` API doesn't require\n    the `maxTokens` parameter, because OpenAI doesn't require it, but the Anthropic\n    API will return a 400 if it's not provided.\n\n[^2]:\n    You can also use `Generation.defineTool` with `Generation.stream`,\n    because currently, it uses the same parameter type, but doesn't actually\n    validate or execute the tool calls.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjclem%2Feffect-llm","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjclem%2Feffect-llm","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjclem%2Feffect-llm/lists"}