https://github.com/edenreich/inference-gateway

A unified Gateway for various LLM providers, including local solutions like Ollama, as well as major platforms such as Groq Cloud, Google, Cloudflare.
https://github.com/edenreich/inference-gateway

agnostic api gateway gateway-api golang inference-api infrastructure infrastructure-configuration kubernetes llm llms opensource opensource-project opensource-projects opentelemetry performance proxy proxy-server self-hosted tracing

Last synced: about 7 hours ago
JSON representation

A unified Gateway for various LLM providers, including local solutions like Ollama, as well as major platforms such as Groq Cloud, Google, Cloudflare.

• Host: GitHub
• URL: https://github.com/edenreich/inference-gateway
• Owner: edenreich
• License: mit
• Created: 2025-01-08T23:27:25.000Z (6 days ago)
• Default Branch: main
• Last Pushed: 2025-01-12T23:01:48.000Z (2 days ago)
• Last Synced: 2025-01-12T23:28:26.295Z (2 days ago)
• Topics: agnostic, api, gateway, gateway-api, golang, inference-api, infrastructure, infrastructure-configuration, kubernetes, llm, llms, opensource, opensource-project, opensource-projects, opentelemetry, performance, proxy, proxy-server, self-hosted, tracing
• Language: Go
• Homepage:
• Size: 34.2 KB
• Stars: 2
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
Inference Gateway




  

  

    

  

  

  

    

  

  

  

    

  



The Inference Gateway is a proxy server designed to facilitate access to various language model APIs. It allows users to interact with different language models through a unified interface, simplifying the configuration and the process of sending requests and receiving responses from multiple LLMs, enabling an easy use of Mixture of Experts.

- [Key Features](#key-features)

- [Supported API's](#supported-apis)

- [Configuration](#configuration)

- [Examples](#examples)

- [License](#license)

## Key Features

- 📜 **Open Source**: Available under the MIT License.

- 🚀 **Unified API Access**: Proxy requests to multiple language model APIs, including Groq, OpenAI, Ollama etc.

- ⚙️ **Environment Configuration**: Easily configure API keys and URLs through environment variables.

- 🐳 **Docker Support**: Use Docker and Docker Compose for easy setup and deployment.

- ☸️ **Kubernetes Support**: Ready for deployment in Kubernetes environments.

- 📊 **OpenTelemetry Tracing**: Enable tracing for the server to monitor and analyze performance.

- 🛡️ **Production Ready**: Built with production in mind, with configurable timeouts and TLS support.

- 🌿 **Lightweight**: Includes only essential libraries and runtime, resulting in smaller size binary of ~8.6MB.

- 📉 **Minimal Resource Consumption**: Designed to consume minimal resources and have a lower footprint.

- 📚 **Documentation**: Well documented with examples and guides.

- 🧪 **Tested**: Extensively tested with unit tests and integration tests.

- 🛠️ **Maintained**: Actively maintained and developed.

- 📈 **Scalable**: Easily scalable and can be used in a distributed environment - with HPA in Kubernetes.

- 🔒 **Compliance** and Data Privacy: This project does not collect data or analytics, ensuring compliance and data privacy.

- 🏠 **Self-Hosted**: Can be self-hosted for complete control over the deployment environment.

## Supported API's

- [OpenAI](https://platform.openai.com/)

- [Ollama](https://ollama.com/)

- [Groq Cloud](https://console.groq.com/)

- [Google](https://aistudio.google.com/)

- [Cloudflare](https://www.cloudflare.com/)

## Configuration

The Inference Gateway can be configured using environment variables. The following [environment variables](./Configurations.md) are supported.

## Examples

- Using [Docker Compose](examples/docker-compose/)

- Using [Kubernetes](examples/kubernetes/)

- Using standard [REST endpoints](examples/rest-endpoints/)

## License

This project is licensed under the MIT License.