https://github.com/langtail/langtail-node
Langtail TypeScript SDK
https://github.com/langtail/langtail-node
llm llmops llms
Last synced: about 1 year ago
JSON representation
Langtail TypeScript SDK
- Host: GitHub
- URL: https://github.com/langtail/langtail-node
- Owner: langtail
- License: mit
- Created: 2024-03-18T13:06:55.000Z (over 2 years ago)
- Default Branch: main
- Last Pushed: 2025-04-08T08:46:37.000Z (about 1 year ago)
- Last Synced: 2025-04-08T09:44:45.286Z (about 1 year ago)
- Topics: llm, llmops, llms
- Language: TypeScript
- Homepage: https://langtail.com
- Size: 593 KB
- Stars: 11
- Watchers: 5
- Forks: 4
- Open Issues: 2
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
Awesome Lists containing this project
README
# Langtail SDK
Typescript SDK for [Langtail](https://langtail.com/).
[](https://github.com/langtail/langtail-node/actions?query=workflow:"CI+check")
[](https://github.com/langtail/langtail-node/releases/)
[](#license)
## Install
```bash
npm i langtail
```
## Usage
### OpenAI chat completion
basic completion without any prompt. This just wraps openAI api and adds a few extra parameters you can use to affect how the request gets logged in langtail.
```ts
import OpenAI from "openai"
import { createOpenAIProxy } from "langtail/openai"
const openai = new OpenAI({
apiKey: "",
})
const lt = createOpenAIProxy(openai)
const rawCompletion = await lt.chat.completions.create({
// Required
messages: [{ role: "system", content: "You are a helpful assistant." }],
model: "gpt-3.5-turbo",
// Optional:
// All OpenAI fields (temperature, top_p, tools,...)
prompt: "",
doNotRecord: false, // false will ensure logs do not contain any info about payloads. You can still see the request in the logs, but you cannot see the variables etc.
metadata: {
"custom-field": "1",
},
})
```
### Deployed prompts
Completion from a deployed prompt can be called with `lt.prompts.invoke`:
```ts
const deployedPromptCompletion = await lt.prompts.invoke({
prompt: "", // required
environment: "staging",
variables: {
about: "cowboy Bebop",
},
}) // results in an openAI ChatCompletion
```
Of course this assumes that you have already deployed your prompt to `staging` environment. If not, you will get an error thrown an error: `Error: Failed to fetch prompt: 404 {"error":"Prompt deployment not found"}`
## LangtailPrompts
In case you only need deployed prompts, you can import just `LangtailPrompts` like this:
```ts
import { LangtailPrompts } from "langtail"
const lt = new LangtailPrompts({
apiKey: "",
})
// usage
const deployedPromptCompletion = await lt.invoke({
prompt: "",
environment: "staging",
variables: {
about: "cowboy Bebop",
},
})
```
You can initialize LangtailPrompts with workspace and project slugs like so:
```ts
import { Langtail } from "langtail"
const lt = new Langtail({
apiKey: "",
workspace: "",
project: "",
})
```
which is necessary if your API key is workspace wide. For a project api key this is not necessary.
## Streaming responses
both chat.prompts.create and prompts.invoke support streaming responses. All you need to enable it is `{ stream: true }` flag like this:
```ts
const deployedPromptCompletion = await lt.prompts.invoke({
prompt: "",
environment: "staging",
stream: true, // changes result to be a streaming OpenAI response
}) // results in an openAI Stream
```
Full API reference is in [API.md](API.md)
We support the same [runtimes as OpenAI](https://github.com/openai/openai-node?tab=readme-ov-file#requirements).
### Proxyless usage
You can avoid langtail API all together by constructing your prompt locally and calling your provider like openAI directly.
let's suppose you have a prompt called `joke-teller` deployed on staging in langtail. You can `get` it's template and all the playground config by calling `get` method like this:
```ts
import { LangtailPrompts } from "langtail"
const lt = new LangtailPrompts({
apiKey: "",
})
const playgroundState = await lt.get({
prompt: "",
environment: "preview",
version: "", // optional
})
```
`get` will return something like this depending on how your prompt configured when it was deployed:
```
{
"chatInput": {
"optionalExtra": "",
},
"state": {
"args": {
"frequency_penalty": 0,
"jsonmode": false,
"max_tokens": 800,
"model": "gpt-3.5-turbo",
"presence_penalty": 0,
"stop": [],
"stream": true,
"temperature": 0.5,
"top_p": 1,
},
"functions": [],
"template": [
{
"content": "I want you to tell me a joke. Topic of the joke: {{topic}}",
"role": "system",
},
],
"tools": [],
"type": "chat",
},
}
```
render your template and builds the final open AI compatible payload:
```ts
import { getOpenAIBody } from "langtail/getOpenAIBody"
const openAiBody = getOpenAIBody(playgroundState, {
stream: true,
variables: {
topic: "iron man",
},
})
```
openAiBody now contains this object:
```js
{
"frequency_penalty": 0,
"max_tokens": 800,
"messages": [
{
"content": "I want you to tell me a joke. Topic of the joke: iron man",
"role": "system",
},
],
"model": "gpt-3.5-turbo",
"presence_penalty": 0,
"temperature": 0.5,
"top_p": 1,
}
```
Notice that your langtail template was replaced with a variable passed in. You can directly call openAI SDK with this object:
```ts
import OpenAI from "openai"
const openai = new OpenAI()
const joke = await openai.chat.completions.create(openAiBody)
```
This way you are still using langtail prompts without exposing potentially sensitive data in your variables.
## Typed inputs
You can override input types to improve IntelliSense for the `prompt`, `environment`, `version` and `variables` when calling a prompt. Use the command `npx langtail generate-types`.
## Vercel AI provider
You can use Langtail with [Vercel AI SDK](https://github.com/vercel/ai).
Import `langtail` from `langtail/vercel-ai` and provide your prompt slug as an argument.
```typescript
import { generateText } from 'ai'
import { langtail } from 'langtail/vercel-ai'
async function main() {
const result = await generateText({
// API key is loaded from env variable LANGTAIL_API_KEY
model: langtail('stock-simple', {
// Optional Langtail options:
variables: { 'ticker': 'TSLA' },
environment: "production",
version: "2",
doNotRecord: false,
metadata: {},
}),
// Optional LLM options:
prompt: 'show me the price',
temperature: 0, // overrides setting in Langtail
})
console.log(result.text)
}
main().catch(console.error);
```
You can also use `aiBridge` from `langtail/vercel-ai` to use already existing Langtail instance:
```typescript
const langtail = new Langtail({ apiKey })
const lt = aiBridge(langtail)
const result = await generateText({
model: lt('stock-simple', {
variables: { 'ticker': 'TSLA' },
}),
prompt: 'show me the price',
})
```
### Using tools from Langtail
If your prompts in Langtail contain tools, you can generate a file containing tool parameters for every prompt deployment in your project. Run `npx langtail generate-tools --out [output_filepath]` to generate the file. For typings of the `tools` helper to work correctly, you also need to [generate types](#typed-inputs).
After the file is generated, you can provide the Langtail tools to AI SDK like this:
```typescript
import { generateText } from 'ai'
import { langtail } from 'langtail/vercel-ai'
import tools from './langtailTools'; // generated langtailTools.ts file
const ltModel = langtail('stock-simple',
{
environment: "production",
version: "3" // pinning the version is recommended
}
);
const result = await generateText({
model: ltModel,
prompt: 'Show me the current price!',
tools: tools(ltModel), // loads all the tools for the specified prompt version
});
```
You can also define custom execute functions for your tools as follows:
```typescript
tools(ltModel, {
get_current_stock_price: {
execute: async ({ ticker }) => {
return ({
ticker,
price: 200 + Math.floor(Math.random() * 50),
});
},
},
})
```
## Stream helpers
The AI streams are delivered as JSON objects, which are split into chunks. This can pose a challenge because JSON objects might be distributed across multiple chunks. We have provide you with helper functions to manage these JSON streams more effectively.
Here's an example:
```ts
import {
chatStreamToRunner,
type ChatCompletionStream,
} from "langtail/stream"
const stream = await fetch(`/api/langtail`, {
method: "POST",
body: JSON.stringify({ messages: localMessages }),
headers: {
"Content-Type": "application/json",
},
}).then((res) => res.body)
// NOTE: await res.body => ReadableStream
const runner = chatStreamToRunner(stream)
runner.on("message", (messageDelta: string) => {
// NOTE: this is a string delta directly from the AI you can put together
console.log(messageDelta)
})
runner.on("chunk", (chunk: ChatCompletionChunk) => {
// NOTE: chunk here is always a proper JSON even with parts of the message
})
```