https://github.com/jclem/effect-llm
An Effect wrapper around LLM APIs
https://github.com/jclem/effect-llm
Last synced: 4 months ago
JSON representation
An Effect wrapper around LLM APIs
- Host: GitHub
- URL: https://github.com/jclem/effect-llm
- Owner: jclem
- License: mit
- Created: 2024-08-05T23:07:53.000Z (almost 2 years ago)
- Default Branch: main
- Last Pushed: 2024-11-06T20:10:07.000Z (over 1 year ago)
- Last Synced: 2025-10-12T16:39:05.033Z (8 months ago)
- Language: TypeScript
- Size: 321 KB
- Stars: 2
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE.md
Awesome Lists containing this project
README
# Effect LLM
Effect LLM built with [Effect](https://effect.website) for interacting with
large language model APIs.
The goal is of the library is to make it as easy as possible to switch between
various API providers (while also providing a means of using provider-specific
functionality where needed).
# Usage
## Basic Usage
To use this library, you'll initialize a provider (a client for a specific API
such as Anthropic or OpenAI) and use that provider to make LLM API calls via the
Generation service.
```typescript
const program = Effect.gen(function* () {
const apiKey = yield* Config.redacted("ANTHROPIC_API_KEY");
const provider = yield* Providers.Anthropic.make();
const stream = Generation.stream(provider, {
apiKey,
model: Providers.Anthropic.Model.Claude35Sonnet,
maxTokens: 512,
events: [
new Thread.UserMessage({
content: [new TextChunk({ content: "Hello, I'm Jonathan." })],
}),
],
});
const responseText = yield* Generation.getContent(stream);
yield* Console.log("The model says:", responseText);
});
program.pipe(
Effect.provide(HttpClient.layer),
Effect.provide(BunContext.layer),
BunRuntime.runMain,
);
```
Note that provider `make` functions generally also receive a subset of the
streaming params as an argument, which can be used to provide some defaults:
```typescript
Effect.gen(function* () {
const provider = yield* Providers.Anthropic.make({
defaultParameters: {
apiKey: yield* Config.redacted("ANTHROPIC_API_KEY"),
model: Providers.Anthropic.Model.Claude35Sonnet,
maxTokens: 512,
system: "Be cordial.",
additionalParameters: {
temperature: 0.5,
},
},
});
});
```
Note that `additionalParameters` passed to a generation function will be merged
with `additionalParameters` given to `defaultParams`.
Generally, it is recommended that you set up the provider as a layer so that it
can be swapped out with relative ease[^1].
```typescript
const apiKey = Config.redacted("ANTHROPIC_API_KEY");
const program = Effect.gen(function* () {
const provider = yield* Generation.Generation;
const stream = Generation.stream(provider, {
apiKey: yield* apiKey,
model: Providers.Anthropic.Model.Claude35Sonnet,
maxTokens: 512,
events: [
new Thread.UserMessage({
content: [new TextChunk({ content: "Hello, I'm Jonathan." })],
}),
],
});
const responseText = yield* Generation.getContent(stream);
yield* Console.log("The model says:", responseText);
});
program.pipe(
Effect.provide(Layer.effect(Generation.Generation, Anthropic.make())),
Effect.provide(HttpClient.layer),
Effect.provide(BunContext.layer),
BunRuntime.runMain,
);
```
## Using the Google Provider
In order to use the Google Provider, you'll need to keep two things in mind:
The `Google.make` function accepts two parameters. The first are Google-specific
configuration options, and the second are the optional default parameters that
the other providers accept, as well:
```typescript
Google.make(
{
// Required.
serviceEndpoint: "https://us-central1-aiplatform.googleapis.com",
},
{
system: "Be courteous.",
},
);
```
Secondly, the "model" parameter must be the full model path parameter in this format:
```typescript
const params = {
model: `projects/${projectID}/locations/${locationID}/publishers/${publisher}/models/${modelName}`,
};
```
## Tool-Calling
There are two ways of utilizing LLM tools in this library.
### Using `Generation.stream`
The `Generation.stream` function accepts `tools` and `toolCall` as
parameters. When using these parameters, you can expect to see the following
events emitted from the stream:
| Name | Payload Type | Description |
| --------------- | ----------------------------------------------------------------------------- | --------------------------------------------------------------------------------- |
| `ToolCallStart` | `{ readonly id: string; readonly name: string; }` | Emitted when a tool call begins, but before its arguments have been streamed |
| `ToolCall` | `{ readonly id: string; readonly name: string; readonly arguments: string; }` | Emitted when a tool call and its arguments have been fully streamed and collected |
Note that when using `Generation.stream`, the tool calls are not validated or
executed, nor are there arguments even parsed.
### Using `Generation.streamTools`
If instead you would like to have effect-llm parse and _execute_ tool calls for
you, use `Generation.streamTools`. This tool accepts the same parameters as
`Generation.stream`, with the addition of a `maxIterations` parameter used to
limit the number of loops that will be executed. When using
`Generation.streamTools`, the following sequence of events will occur:
1. Send the completion request to the provider
2. Parse the response
3. If there are tool calls in the response:
1. Parse the arguments
2. Append the tool call to the events list
3. Call the tool
4. Append the tool result to the events list
5. Go to (1) with the new events list
4. OR, If there are no tool calls in the resopnse:
1. End the stream
If the `maxIterations` limit is exceeded, the stream will emit a
`MaxIterationsError`.
The stream returned by `Generation.streamTools` emits the same events as
`Generation.stream` with some additions:
| Name | Payload Type | Description |
| ------------------- | ----------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------- |
| `ToolCallStart` | `{ readonly id: string; readonly name: string; }` | Emitted when a tool call begins, but before its arguments have been streamed |
| `ToolCall` | `{ readonly id: string; readonly name: string; readonly arguments: string; }` | Emitted when a tool call and its arguments have been fully streamed and collected |
| `InvalidToolCall` | `{ readonly id: string; readonly name: string; readonly arguments: string; }` | Emitted when a tool call's arguments are invalid or the tool call is not in the defined tools |
| `ToolResultSuccess` | `{ readonly id: string; readonly name: string; readonly result: unknown }` | Emitted when a tool call's arguments are invalid or the tool call is not in the defined tools |
| `ToolResultError` | `{ readonly id: string; readonly name: string; readonly result: unknown }` | Emitted when a tool call's arguments are invalid or the tool call is not in the defined tools |
### Defining Tools
To define a tool for use with `Generation.streamTools`[^2], use the
`Generation.defineTool` function:
```typescript
const apiKey = Config.redacted("ANTHROPIC_API_KEY");
const program = Effect.gen(function* () {
const provider = yield* Generation.Generation;
const sayHello = Generation.defineTool("sayHello", {
description: "Say hello to the user",
input: Schema.Struct({ name: Schema.String }),
effect: (toolCallID, toolArgs) =>
Console.log(`Hello, ${toolArgs.name}`).pipe(Effect.as({ ok: true })),
});
const stream = Generation.streamTools(provider, {
apiKey: yield* apiKey,
model: Providers.Anthropic.Model.Claude35Sonnet,
maxTokens: 512,
tools: [sayHello],
events: [
new Thread.UserMessage({
content: [new TextChunk({ content: "Hello, I'm Jonathan." })],
}),
],
});
yield* stream.pipe(Stream.runDrain, Effect.scoped);
});
program.pipe(
Effect.provide(Layer.effect(Generation.Generation, Anthropic.make())),
Effect.provide(HttpClient.layer),
Effect.provide(BunContext.layer),
BunRuntime.runMain,
);
```
#### Error Handling
Any errors that occur during tool execution will _halt_ the stream and yield
a `ToolExecutionError`. In order to handle an error and report it to the
model, you should instead fail the effect with a `ToolError` using the
`Generation.toolError` function:
```typescript
const sayHello = Generation.defineTool("sayHello", {
description: "Say hello to the user",
input: Schema.Struct({ name: Schema.String }),
effect: (toolCallID, toolArgs) =>
Console.log(`Hello, ${toolArgs.name}`).pipe(
Effect.catchAll((err) =>
Generation.toolError({
message: "An error occurred while saying hello",
error: err,
}),
),
Effect.as({ ok: true }),
),
});
```
You can also fail mid-effect, since `Generation.toolError` actually fails the effect:
```typescript
const sayHello = Generation.defineTool("sayHello", {
description: "Say hello to the user",
input: Schema.Struct({ name: Schema.String }),
effect: (toolCallID, toolArgs) =>
Effect.gen(function* () {
return yield* Generation.toolError("Whoops!");
}),
});
```
The payload passed to `Generation.toolError` can be any value, and it is
serialized as JSON and sent to the model, which is notified that an error
occurred.
#### Halting Early
If you want to halt the iteration loop eraly, you can use the
`Generation.haltToolLoop` function:
```typescript
const sayHello = Generation.defineTool("sayHello", {
description: "Say hello to the user",
input: Schema.Struct({ name: Schema.String }),
effect: (toolCallID, toolArgs) =>
Effect.gen(function* () {
return yield* Generation.haltToolLoop();
}),
});
```
This will immediately halt the loop before executing any other tool calls
returned by the model in that same loop, and will yield end the stream without
an error.
[^1]:
There are some caveats to this—for example, the `stream` API doesn't require
the `maxTokens` parameter, because OpenAI doesn't require it, but the Anthropic
API will return a 400 if it's not provided.
[^2]:
You can also use `Generation.defineTool` with `Generation.stream`,
because currently, it uses the same parameter type, but doesn't actually
validate or execute the tool calls.