https://github.com/inference-gateway/typescript-sdk
An SDK written in Typescript for the Inference Gateway
https://github.com/inference-gateway/typescript-sdk
inference-gateway sdk typescript-sdk
Last synced: 3 months ago
JSON representation
An SDK written in Typescript for the Inference Gateway
- Host: GitHub
- URL: https://github.com/inference-gateway/typescript-sdk
- Owner: inference-gateway
- License: mit
- Created: 2025-01-21T07:26:16.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2025-11-24T13:06:12.000Z (7 months ago)
- Last Synced: 2025-11-28T01:09:46.762Z (7 months ago)
- Topics: inference-gateway, sdk, typescript-sdk
- Language: TypeScript
- Homepage: https://github.com/inference-gateway/inference-gateway
- Size: 367 KB
- Stars: 1
- Watchers: 1
- Forks: 0
- Open Issues: 2
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
- Agents: AGENTS.md
Awesome Lists containing this project
README
# Inference Gateway TypeScript SDK
An SDK written in TypeScript for the [Inference Gateway](https://github.com/edenreich/inference-gateway).
- [Inference Gateway TypeScript SDK](#inference-gateway-typescript-sdk)
- [Installation](#installation)
- [Usage](#usage)
- [Creating a Client](#creating-a-client)
- [Listing Models](#listing-models)
- [Listing MCP Tools](#listing-mcp-tools)
- [Creating Chat Completions](#creating-chat-completions)
- [Streaming Chat Completions](#streaming-chat-completions)
- [Tool Calls](#tool-calls)
- [Proxying Requests](#proxying-requests)
- [Health Check](#health-check)
- [Creating a Client with Custom Options](#creating-a-client-with-custom-options)
- [Examples](#examples)
- [Contributing](#contributing)
- [License](#license)
## Installation
Run `npm i @inference-gateway/sdk`.
## Usage
### Creating a Client
```typescript
import { InferenceGatewayClient } from '@inference-gateway/sdk';
// Create a client with default options
const client = new InferenceGatewayClient({
baseURL: 'http://localhost:8080/v1',
apiKey: 'your-api-key', // Optional
});
```
### Listing Models
To list all available models:
```typescript
import { InferenceGatewayClient, Provider } from '@inference-gateway/sdk';
const client = new InferenceGatewayClient({
baseURL: 'http://localhost:8080/v1',
});
try {
// List all models
const models = await client.listModels();
console.log('All models:', models);
// List models from a specific provider
const openaiModels = await client.listModels(Provider.openai);
console.log('OpenAI models:', openaiModels);
} catch (error) {
console.error('Error:', error);
}
```
### Listing MCP Tools
To list available Model Context Protocol (MCP) tools (only available when
EXPOSE_MCP is enabled):
```typescript
import { InferenceGatewayClient } from '@inference-gateway/sdk';
const client = new InferenceGatewayClient({
baseURL: 'http://localhost:8080/v1',
});
try {
const tools = await client.listTools();
console.log('Available MCP tools:', tools.data);
// Each tool has: name, description, server, and optional input_schema
tools.data.forEach((tool) => {
console.log(`Tool: ${tool.name}`);
console.log(`Description: ${tool.description}`);
console.log(`Server: ${tool.server}`);
});
} catch (error) {
console.error('Error:', error);
}
```
### Creating Chat Completions
To generate content using a model:
```typescript
import {
InferenceGatewayClient,
MessageRole,
Provider,
} from '@inference-gateway/sdk';
const client = new InferenceGatewayClient({
baseURL: 'http://localhost:8080/v1',
});
try {
const response = await client.createChatCompletion(
{
model: 'gpt-4o',
messages: [
{
role: MessageRole.System,
content: 'You are a helpful assistant',
},
{
role: MessageRole.User,
content: 'Tell me a joke',
},
],
},
Provider.openai
); // Provider is optional
console.log('Response:', response.choices[0].message.content);
} catch (error) {
console.error('Error:', error);
}
```
### Streaming Chat Completions
To stream content from a model:
```typescript
import {
InferenceGatewayClient,
MessageRole,
Provider,
} from '@inference-gateway/sdk';
const client = new InferenceGatewayClient({
baseURL: 'http://localhost:8080/v1',
});
try {
await client.streamChatCompletion(
{
model: 'llama-3.3-70b-versatile',
messages: [
{
role: MessageRole.User,
content: 'Tell me a story',
},
],
},
{
onOpen: () => console.log('Stream opened'),
onContent: (content) => process.stdout.write(content),
onChunk: (chunk) => console.log('Received chunk:', chunk.id),
onUsageMetrics: (metrics) => console.log('Usage metrics:', metrics),
onFinish: () => console.log('\nStream completed'),
onError: (error) => console.error('Stream error:', error),
},
Provider.groq // Provider is optional
);
} catch (error) {
console.error('Error:', error);
}
```
### Tool Calls
To use tool calls with models that support them:
```typescript
import {
InferenceGatewayClient,
MessageRole,
Provider,
} from '@inference-gateway/sdk';
const client = new InferenceGatewayClient({
baseURL: 'http://localhost:8080/v1',
});
try {
await client.streamChatCompletion(
{
model: 'openai/gpt-4o',
messages: [
{
role: MessageRole.User,
content: "What's the weather in San Francisco?",
},
],
tools: [
{
type: 'function',
function: {
name: 'get_weather',
parameters: {
type: 'object',
properties: {
location: {
type: 'string',
description: 'The city and state, e.g. San Francisco, CA',
},
},
required: ['location'],
},
},
},
],
},
{
onTool: (toolCall) => {
console.log('Tool call:', toolCall.function.name);
console.log('Arguments:', toolCall.function.arguments);
},
onReasoning: (reasoning) => {
console.log('Reasoning:', reasoning);
},
onContent: (content) => {
console.log('Content:', content);
},
onFinish: () => console.log('\nStream completed'),
}
);
} catch (error) {
console.error('Error:', error);
}
```
### Proxying Requests
To proxy requests directly to a provider:
```typescript
import { InferenceGatewayClient, Provider } from '@inference-gateway/sdk';
const client = new InferenceGatewayClient({
baseURL: 'http://localhost:8080',
});
try {
const response = await client.proxy(Provider.openai, 'embeddings', {
method: 'POST',
body: JSON.stringify({
model: 'text-embedding-ada-002',
input: 'Hello world',
}),
});
console.log('Embeddings:', response);
} catch (error) {
console.error('Error:', error);
}
```
### Health Check
To check if the Inference Gateway is running:
```typescript
import { InferenceGatewayClient } from '@inference-gateway/sdk';
const client = new InferenceGatewayClient({
baseURL: 'http://localhost:8080',
});
try {
const isHealthy = await client.healthCheck();
console.log('API is healthy:', isHealthy);
} catch (error) {
console.error('Error:', error);
}
```
### Creating a Client with Custom Options
You can create a new client with custom options using the `withOptions` method:
```typescript
import { InferenceGatewayClient } from '@inference-gateway/sdk';
const client = new InferenceGatewayClient({
baseURL: 'http://localhost:8080/v1',
});
// Create a new client with custom headers
const clientWithHeaders = client.withOptions({
defaultHeaders: {
'X-Custom-Header': 'value',
},
timeout: 60000, // 60 seconds
});
```
### Examples
For more examples, check the [examples directory](./examples).
## Contributing
Please refer to the [CONTRIBUTING.md](CONTRIBUTING.md) file for information
about how to get involved. We welcome issues, questions, and pull requests.
## License
This SDK is distributed under the MIT License, see [LICENSE](LICENSE) for more information.