Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/Portkey-AI/gateway
A Blazing Fast AI Gateway. Route to 100+ LLMs with 1 fast & friendly API.
https://github.com/Portkey-AI/gateway
ai-gateway gateway generative-ai langchain llama-index llmops llms openai prompt-engineering router
Last synced: 7 days ago
JSON representation
A Blazing Fast AI Gateway. Route to 100+ LLMs with 1 fast & friendly API.
- Host: GitHub
- URL: https://github.com/Portkey-AI/gateway
- Owner: Portkey-AI
- License: mit
- Created: 2023-08-23T11:52:47.000Z (10 months ago)
- Default Branch: main
- Last Pushed: 2024-05-29T09:32:48.000Z (18 days ago)
- Last Synced: 2024-05-29T12:44:55.152Z (18 days ago)
- Topics: ai-gateway, gateway, generative-ai, langchain, llama-index, llmops, llms, openai, prompt-engineering, router
- Language: TypeScript
- Homepage: https://portkey.ai/features/ai-gateway
- Size: 54.5 MB
- Stars: 4,836
- Watchers: 33
- Forks: 322
- Open Issues: 60
-
Metadata Files:
- Readme: README.md
- Contributing: .github/CONTRIBUTING.cn.md
- Code of conduct: .github/CODE_OF_CONDUCT.cn.md
- Security: .github/SECURITY.md
- Support: .github/SUPPORT.md
Lists
- awesome - Portkey-AI/gateway - A Blazing Fast AI Gateway. Route to 200+ LLMs with 1 fast & friendly API. (Jupyter Notebook)
- awesome - Portkey-AI/gateway - A Blazing Fast AI Gateway. Route to 200+ LLMs with 1 fast & friendly API. (Jupyter Notebook)
- awesome - Portkey-AI/gateway - A Blazing Fast AI Gateway. Route to 200+ LLMs with 1 fast & friendly API. (Jupyter Notebook)
- awesome-stars - Portkey-AI/gateway - A Blazing Fast AI Gateway. Route to 200+ LLMs with 1 fast & friendly API. (Jupyter Notebook)
- trackawesomelist - Gateway (⭐4.4k) - AI/gateway?style=social) (Recently Updated / [Apr 20, 2024](/content/2024/04/20/README.md))
- awesome-stars - Portkey-AI/gateway - A Blazing Fast AI Gateway. Route to 200+ LLMs with 1 fast & friendly API. (Jupyter Notebook)
- awesome-stars - gateway - AI | 4939 | (Jupyter Notebook)
- awesome-stars - Portkey-AI/gateway - A Blazing Fast AI Gateway. Route to 200+ LLMs with 1 fast & friendly API. (Jupyter Notebook)
- AiTreasureBox - Portkey-AI/gateway - 06-12_4943_3](https://img.shields.io/github/stars/Portkey-AI/gateway.svg)|A Blazing Fast AI Gateway. Route to 100+ LLMs with 1 fast & friendly API.| (Repos)
- Awesome-LLM - AI Gateway - ready with support for caching, fallbacks, retries, timeouts, loadbalancing, and can be edge-deployed for minimum latency. (LLM Deployment)
- awesome-stars - Portkey-AI/gateway - A Blazing Fast AI Gateway. Route to 200+ LLMs with 1 fast & friendly API. (Jupyter Notebook)
- awesome-stars - Portkey-AI/gateway - A Blazing Fast AI Gateway. Route to 200+ LLMs with 1 fast & friendly API. (Jupyter Notebook)
- Awesome-LLM?tab=readme-ov-file - AI Gateway - ready with support for caching, fallbacks, retries, timeouts, loadbalancing, and can be edge-deployed for minimum latency. (LLM Deployment)
- awesome-stars - Portkey-AI/gateway - `★4950` A Blazing Fast AI Gateway. Route to 200+ LLMs with 1 fast & friendly API. (Jupyter Notebook)
- awesome-stars - Portkey-AI/gateway - A Blazing Fast AI Gateway. Route to 100+ LLMs with 1 fast & friendly API. (TypeScript)
- awesome-stars - Portkey-AI/gateway - A Blazing Fast AI Gateway. Route to 200+ LLMs with 1 fast & friendly API. (Jupyter Notebook)
- awesome-stars - Portkey-AI/gateway - A Blazing Fast AI Gateway. Route to 200+ LLMs with 1 fast & friendly API. (Jupyter Notebook)
- awesome-langchain - Gateway - AI/gateway?style=social) (Other LLM Frameworks / Videos Playlists)
- awesome-langchain-zh - Gateway - AI/gateway?style=social): 一个强大的ai网关. (其他LLM框架 / 文章)
- awesome-ai-api-projects - gateway - commit/Portkey-AI/gateway?label=%20) | (AI Agent / Framework)
README
English | 中文
# AI Gateway
#### Reliably route to 200+ LLMs with 1 fast & friendly API
[](./LICENSE)
[](https://portkey.ai/community)
[](https://twitter.com/portkeyai)
[](https://www.npmjs.com/package/@portkey-ai/gateway)
Gateway streamlines requests to 200+ open & closed source models with a unified API. It is also production-ready with support for caching, fallbacks, retries, timeouts, loadbalancing, and can be edge-deployed for minimum latency.
✅ **Blazing fast** (9.9x faster) with a **tiny footprint** (~45kb installed)
✅ **Load balance** across multiple models, providers, and keys
✅ **Fallbacks** make sure your app stays resilient
✅ **Automatic Retries** with exponential fallbacks come by default
✅ **Configurable Request Timeouts** to easily handle unresponsive LLM requests
✅ **Multimodal** to support routing between Vision, TTS, STT, Image Gen, and more models
✅ **Plug-in** middleware as needed
✅ Battle tested over **300B tokens**
✅ **Enterprise-ready** for enhanced security, scale, and custom deployments
## How to Run the Gateway?
1. [Run it Locally](#run-it-locally) for complete control & customization
2. [Hosted by Portkey](#gateway-hosted-by-portkey) for quick setup without infrastructure concerns
3. [Enterprise On-Prem](#gateway-enterprise-version) for advanced features and dedicated support
### Run it Locally
Run the following command in your terminal and it will spin up the Gateway on your local system:
```bash
npx @portkey-ai/gateway
```
Your AI Gateway is now running on http://localhost:8787 🚀
Gateway is also edge-deployment ready. Explore Cloudflare, Docker, AWS etc. deployment [guides here](#deploying-the-ai-gateway).
### Gateway Hosted by Portkey
This same open-source Gateway powers Portkey API that processes **billions of tokens** daily and is in production with companies like Postman, Haptik, Turing, MultiOn, SiteGPT, and more.
Sign up for the free developer plan (10K request/month) [here](https://app.portkey.ai/) or [discuss here](https://calendly.com/rohit-portkey/noam) for enterprise deployments.
## How to Use the Gateway?
### Compatible with OpenAI API & SDK
Gateway is fully compatible with the OpenAI API & SDK, and extends them to call 200+ LLMs and makes them reliable. To use the Gateway through OpenAI, you only need to update your `base_URL` and pass the provider name in headers.
* To use through Portkey, set your `base_URL` to: `https://api.portkey.ai/v1`
* To run locally, set: `http://localhost:8787/v1`
Let's see how we can use the Gateway to make an Anthropic request in OpenAI spec below - the same will follow for all the other providers.
### 
Python
```bash
pip install portkey-ai
```
While instantiating your OpenAI client,
1. Set the `base_URL` to `http://localhost:8787/v1` (or `PORTKEY_GATEWAY_URL` through the Portkey SDK if you're using the hosted version)
2. Pass the provider name in the `default_headers` param (here we are using `createHeaders` method with the Portkey SDK to auto-create the full header)
```python
from openai import OpenAI
from portkey_ai import PORTKEY_GATEWAY_URL, createHeaders
gateway = OpenAI(
api_key="ANTHROPIC_API_KEY",
base_url=PORTKEY_GATEWAY_URL, # Or http://localhost:8787/v1 when running locally
default_headers=createHeaders(
provider="anthropic",
api_key="PORTKEY_API_KEY" # Grab from https://app.portkey.ai # Not needed when running locally
)
)
chat_complete = gateway.chat.completions.create(
model="claude-3-sonnet-20240229",
messages=[{"role": "user", "content": "What's a fractal?"}],
max_tokens=512
)
```
If you want to run the Gateway locally, don't forget to run `npx @portkey-ai/gateway` in your terminal before this! Otherwise just [sign up on Portkey](https://app.portkey.ai/) and keep your Portkey API Key handy.
### 
Node.JS
Works the same as in Python. Add `baseURL` & `defaultHeaders` while instantiating your OpenAI client and pass the relevant provider details.
```bash
npm install portkey-ai
```
```js
import OpenAI from 'openai';
import { PORTKEY_GATEWAY_URL, createHeaders } from 'portkey-ai'
const gateway = new OpenAI({
apiKey: "ANTHROPIC_API_KEY",
baseURL: PORTKEY_GATEWAY_URL, // Or http://localhost:8787/v1 when running locally
defaultHeaders: createHeaders({
provider: "anthropic",
apiKey: "PORTKEY_API_KEY" // Grab from https://app.portkey.ai / Not needed when running locally
})
});
async function main(){
const chatCompletion = await portkey.chat.completions.create({
messages: [{ role: 'user', content: 'Who are you?' }],
model: 'claude-3-sonnet-20240229',
maxTokens:512
});
console.log(chatCompletion.choices[0].message.content);
}
main()
```
### 
REST
In your OpenAI REST request,
1. Change the request URL to `https://api.portkey.ai/v1` (or `http://localhost:8787/v1` if you're hosting locally)
2. Pass an additional `x-portkey-provider` header with the provider's name
3. Change the model's name to `claude-3`
```bash
curl 'http://localhost:8787/v1/chat/completions' \
-H 'x-portkey-provider: anthropic' \
-H "Authorization: Bearer $ANTHROPIC_API_KEY" \
-H 'Content-Type: application/json' \
-d '{ "model": "claude-3-haiku-20240229", "messages": [{"role": "user","content": "Hi"}] }'
```
For other providers, change the `provider` & `model` to their respective values.
## Gateway Cookbooks
### Trending Cookbooks
* [Run Gateway on prompts from Langchain hub](/cookbook/use-cases/run-gateway-on-prompts-from-langchain-hub.md)
* [Use Porkey Gateway with Vercel's AI SDK](/cookbook/integrations/vercel-ai.md)
* [Set up fallback from SDXL to Dall-E-3](/cookbook/getting-started/fallback-from-stable-diffusion-to-dall-e.ipynb)
### Latest Cookbooks
* [Comparing Top 10 LMSYS Models with Portkey](/cookbook/use-cases/LMSYS%20Series/comparing-top10-LMSYS-models-with-Portkey.ipynb)
* [Fallback from OpenAI to Azure OpenAI](/cookbook/getting-started/fallback-from-openai-to-azure.ipynb)
* [Set up automatic retries for failed requests](/cookbook/getting-started/automatic-retries-on-failures.md)
* [Call Llama 3 on Groq](/cookbook/use-cases/llama-3-on-groq.ipynb)
### [More Examples](/examples/)
## Supported Providers
Explpore Gateway integrations with [20+ providers](https://portkey.ai/docs/welcome/integration-guides) and [6+ frameworks](https://portkey.ai/docs/welcome/integration-guides).
| | Provider | Support | Stream |
| -------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------- | ------- | ------ |
| 
| [OpenAI](https://portkey.ai/docs/welcome/integration-guides/openai) | ✅ | ✅ |
| 
| [Azure OpenAI](https://portkey.ai/docs/welcome/integration-guides/azure-openai) | ✅ | ✅ |
| 
| [Anyscale](https://portkey.ai/docs/welcome/integration-guides/anyscale-llama2-mistral-zephyr) | ✅ | ✅ |
| 
| [Google Gemini & Palm](https://portkey.ai/docs/welcome/integration-guides/gemini) | ✅ | ✅ |
| 
| [Anthropic](https://portkey.ai/docs/welcome/integration-guides/anthropic) | ✅ | ✅ |
| 
| [Cohere](https://portkey.ai/docs/welcome/integration-guides/cohere) | ✅ | ✅ |
| 
| [Together AI](https://portkey.ai/docs/welcome/integration-guides/together-ai) | ✅ | ✅ |
| 
| [Perplexity](https://portkey.ai/docs/welcome/integration-guides/perplexity-ai) | ✅ | ✅ |
| 
| [Mistral](https://portkey.ai/docs/welcome/integration-guides/mistral-ai) | ✅ | ✅ |
| 
| [Nomic](https://portkey.ai/docs/welcome/integration-guides/nomic) | ✅ | ✅ |
| 
| [AI21](https://portkey.ai/docs/welcome/integration-guides) | ✅ | ✅ |
| 
| [Stability AI](https://portkey.ai/docs/welcome/integration-guides/stability-ai) | ✅ | ✅ |
| 
| [DeepInfra](https://portkey.ai/docs/welcome/integration-guides) | ✅ | ✅ |
| 
| [Ollama](https://portkey.ai/docs/welcome/integration-guides/ollama) | ✅ | ✅ |
| 
| Novita AI | ✅ | ✅ | `/chat/completions`, `/completions` |
> [View the complete list of 200+ supported models here](https://portkey.ai/docs/welcome/what-is-portkey#ai-providers-supported)
## Reliability Features
Fallbacks
This feature allows you to specify a prioritized list of LLMs. If the primary LLM fails, Portkey will automatically fallback to the next LLM in the list to ensure reliability.
Automatic Retries
AI Gateway can automatically retry failed requests up to 5 times. A backoff strategy spaces out retry attempts to prevent network overload.
Load Balancing
Distribute load effectively across multiple API keys or providers based on custom weights to ensure high availability and optimal performance.
Request Timeouts
Manage unruly LLMs & latencies by setting up granular request timeouts, allowing automatic termination of requests that exceed a specified duration.
#### Reliability features are set by passing a relevant Gateway Config (JSON) with the `x-portkey-config` header or with the `config` param in the SDKs
### Example: Setting up Fallback from OpenAI to Anthropic
#### Write the fallback logic
```json
{
"strategy": { "mode": "fallback" },
"targets": [
{ "provider": "openai", "api_key": "OPENAI_API_KEY" },
{ "provider": "anthropic", "api_key": "ANTHROPIC_API_KEY" }
]
}
```
#### Use it while making your request
Portkey Gateway will automatically trigger Anthropic if the OpenAI request fails:
```REST```
```bash
curl 'http://localhost:8787/v1/chat/completions' \
-H 'x-portkey-provider: google' \
-H 'x-portkey-config: $CONFIG' \
-H "Authorization: Bearer $GOOGLE_AI_STUDIO_KEY" \
-H 'Content-Type: application/json' \
-d '{ "model": "gemini-1.5-pro-latest", "messages": [{"role": "user","content": "Hi"}] }'
```
You can also trigger Fallbacks only on specific status codes by passing an array of status codes with the `on_status_codes` param in `strategy`.
[Read the full Fallback documentation here.](https://portkey.ai/docs/product/ai-gateway-streamline-llm-integrations/fallbacks)
### Example: Loadbalance Requests across 3 Accounts
#### Write the loadbalancer config
```json
{
"strategy": { "mode": "loadbalance" },
"targets": [
{ "provider": "openai", "api_key": "ACCOUNT_1_KEY", "weight": 1 },
{ "provider": "openai", "api_key": "ACCOUNT_2_KEY", "weight": 1 },
{ "provider": "openai", "api_key": "ACCOUNT_3_KEY", "weight": 1 }
]
}
```
#### Pass the config while instantiating OpenAI client
```ts
import OpenAI from 'openai';
import { PORTKEY_GATEWAY_URL, createHeaders } from 'portkey-ai'
const gateway = new OpenAI({
baseURL: PORTKEY_GATEWAY_URL,
defaultHeaders: createHeaders({
apiKey: "PORTKEY_API_KEY",
config: "CONFIG_ID"
})
});
```
[Read the Loadbalancing docs here.](https://portkey.ai/docs/product/ai-gateway-streamline-llm-integrations/load-balancing)
### Automatic Retries
Similarly, you can write a Config that will attempt retries up to 5 times
```json
{
"retry": { "attempts": 5 }
}
```
[Read the full Retries documentation here.](https://portkey.ai/docs/product/ai-gateway-streamline-llm-integrations/automatic-retries)
### Request Timeouts
Here, the request timeout of 10 seconds will be applied to *all* the targets.
```json
{
"strategy": { "mode": "fallback" },
"request_timeout": 10000,
"targets": [
{ "virtual_key": "open-ai-xxx" },
{ "virtual_key": "azure-open-ai-xxx" }
]
}
```
[Read the full Request Timeouts documentation here.](https://portkey.ai/docs/product/ai-gateway-streamline-llm-integrations/request-timeouts)
### Using Gateway Configs
Here's a guide to [use the config object in your request](https://portkey.ai/docs/api-reference/config-object).
## Supported SDKs
| Language | Supported SDKs |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Node.js / JS / TS | [Portkey SDK](https://www.npmjs.com/package/portkey-ai)
[OpenAI SDK](https://www.npmjs.com/package/openai)
[LangchainJS](https://www.npmjs.com/package/langchain)
[LlamaIndex.TS](https://www.npmjs.com/package/llamaindex) |
| Python | [Portkey SDK](https://pypi.org/project/portkey-ai/)
[OpenAI SDK](https://portkey.ai/docs/welcome/integration-guides/openai)
[Langchain](https://portkey.ai/docs/welcome/integration-guides/langchain-python)
[LlamaIndex](https://portkey.ai/docs/welcome/integration-guides/llama-index-python) |
| Go | [go-openai](https://github.com/sashabaranov/go-openai) |
| Java | [openai-java](https://github.com/TheoKanning/openai-java) |
| Rust | [async-openai](https://docs.rs/async-openai/latest/async_openai/) |
| Ruby | [ruby-openai](https://github.com/alexrudall/ruby-openai) |
## Deploying the AI Gateway
[See docs](docs/installation-deployments.md) on installing the AI Gateway locally or deploying it on popular locations.
- Deploy to [App Stack](docs/installation-deployments.md#deploy-to-app-stack)
- Deploy to [Cloudflare Workers](https://github.com/Portkey-AI/gateway/blob/main/docs/installation-deployments.md#deploy-to-cloudflare-workers)
- Deploy using [Docker](https://github.com/Portkey-AI/gateway/blob/main/docs/installation-deployments.md#deploy-using-docker)
- Deploy using [Docker Compose](https://github.com/Portkey-AI/gateway/blob/main/docs/installation-deployments.md#deploy-using-docker-compose)
- Deploy to [Zeabur](https://github.com/Portkey-AI/gateway/blob/main/docs/installation-deployments.md#deploy-to-zeabur)
- Run a [Node.js server](https://github.com/Portkey-AI/gateway/blob/main/docs/installation-deployments.md#run-a-nodejs-server)
## Gateway Enterprise Version
Make your AI app more reliable and forward compatible, while ensuring complete data security and privacy.
✅ Secure Key Management - for role-based access control and tracking
✅ Simple & Semantic Caching - to serve repeat queries faster & save costs
✅ Access Control & Inbound Rules - to control which IPs and Geos can connect to your deployments
✅ PII Redaction - to automatically remove sensitive data from your requests to prevent indavertent exposure
✅ SOC2, ISO, HIPAA, GDPR Compliances - for best security practices
✅ Professional Support - along with feature prioritization
[Schedule a call to discuss enterprise deployments](https://calendly.com/rohit-portkey/noam)
## Contributing
The easiest way to contribute is to pick any issue with the `good first issue` tag 💪. Read the Contributing guidelines [here](/CONTRIBUTING.md).
Bug Report? [File here](https://github.com/Portkey-AI/gateway/issues) | Feature Request? [File here](https://github.com/Portkey-AI/gateway/issues)
## Community
Join our growing community around the world, for help, ideas, and discussions on AI.
- View our official [Blog](https://portkey.ai/blog)
- Chat with us on [Discord](https://portkey.ai/community)
- Follow us on [Twitter](https://twitter.com/PortkeyAI)
- Connect with us on [LinkedIn](https://www.linkedin.com/company/portkey-ai/)
