https://github.com/EinStack/glide

🐦 A open blazing-fast simple model gateway for rapid development of production GenAI apps
https://github.com/EinStack/glide

ai gateway gateway-api genai generative-ai glide go llm llmops ml mlops router

Last synced: 7 months ago
JSON representation

🐦 A open blazing-fast simple model gateway for rapid development of production GenAI apps

Host: GitHub
URL: https://github.com/EinStack/glide
Owner: EinStack
License: apache-2.0
Created: 2023-12-07T00:49:46.000Z (almost 2 years ago)
Default Branch: develop
Last Pushed: 2024-08-12T19:49:05.000Z (about 1 year ago)
Last Synced: 2024-11-01T01:33:31.561Z (about 1 year ago)
Topics: ai, gateway, gateway-api, genai, generative-ai, glide, go, llm, llmops, ml, mlops, router
Language: Go
Homepage: https://docs.einstack.ai/glide/
Size: 3.43 MB
Stars: 129
Watchers: 5
Forks: 20
Open Issues: 68
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
- Code of conduct: CODE_OF_CONDUCT.md
- Citation: CITATION.cff
- Security: SECURITY.md
- Roadmap: ROADMAP.md

Awesome Lists containing this project

awesome-distributed-system-projects - Glide - an open reliable fast LLM/model gateway for rapid development of GenAI apps
awesome-llmops - Glide - Native LLM Routing Engine. Improve LLM app resilience and speed. | ![GitHub Badge](https://img.shields.io/github/stars/einstack/glide.svg?style=flat-square) | (LLMOps / Observability)

README

Glide: Cloud-Native LLM Gateway for Seamless LLMOps

---

**Glide** is your go-to cloud-native LLM gateway, delivering high-performance LLMOps in a lightweight, all-in-one package.

We take all problems of managing and communicating with external providers out of your applications,
so you can dive into tackling your core challenges.

> [!Important]
> Glide is under active development right now 🛠️
>
> Give us a star⭐ to support the project and watch👀 our repositories not to miss any update. Appreciate your interest 🙏

Glide sits between your application and model providers to seamlessly handle various LLMOps tasks like
model failover, caching, key management, etc.

Check out our [documentation](https://docs.einstack.ai/glide/)!

## Features

- **Unified REST API** across providers. Avoid vendor lock-in and changes in your applications when you swap model providers.
- **High availability** and **resiliency** when working with external model providers. Automatic **fallbacks** on provider failures, rate limits, transient errors. Smart retries to reduce communication latency.
- Support **popular LLM providers**.
- **High performance**. Performance is our priority. We want to keep Glide "invisible" for your latency-wise, while providing rich functionality.
- **Production-ready observability** via OpenTelemetry, emit metrics on models health, allows whitebox monitoring (coming soon)
- Straightforward and simple maintenance and configuration, centralized API key control & management & rotation, etc.

### Large Language Models

| Provider | Supported Capabilities |
|-----------------------------------------------------------------------|-------------------------------------------|
| OpenAI | ✅ Chat
✅ Streaming Chat |
| Anthropic | ✅ Chat
🏗️ Streaming Chat (coming soon) |
| Azure OpenAI | ✅ Chat
✅ Streaming Chat |
| AWS Bedrock (Titan) | ✅ Chat |
| Cohere | ✅ Chat
✅ Streaming Chat |
| Google Gemini | 🏗️ Chat (coming soon) |
| OctoML | ✅ Chat |
| Ollama | ✅ Chat |

## Get Started

### Installation

> [!Note]
> Windows users should follow an instruction right from [the demo README file](https://github.com/EinStack/glide-demo) that specifies how to do the steps without the `make` command as Windows doesn't come with it by default.

The easiest way to deploy Glide is to our [demo repository](https://github.com/EinStack/glide-demo.git) and [docker-compose](https://docs.docker.com/compose/).

### 1. Clone the demo repository

```bash
git clone https://github.com/EinStack/glide-demo.git
```

### 2. Init Configs

The demo repository comes with a basic config. Additionally, you need to init your secrets by running:

```bash
make init # from the demo root
```

This will create the `secrets` directory with one `.OPENAI_API_KEY` file that you need to put your key to.

### 3. Start Glide

After that, just use docker compose via this command to start your demo environment:

```bash
make up
```

### 4. Sample API Request to `/chat` endpoint

See [API Reference](https://glide.einstack.ai/api-reference/introduction) for more details.

```json
{
"message":
{
"role": "user",
"content": "Where was it played?"
},
"message_history": [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "Who won the world series in 2020?"},
{"role": "assistant", "content": "The Los Angeles Dodgers won the World Series in 2020."}
]
}
```

### API Docs

Finally, Glide comes with OpenAPI documentation that is accessible via http://127.0.0.1:9099/v1/swagger

That's it 🙌

Use [our documentation](https://docs.einstack.ai/glide/) to further learn about Glide capabilities and configs.

---

Other ways to install Glide are available:

### Homebrew (MacOS)

```bash
brew tap einstack/tap
brew install einstack/tap/glide
```

### Snapcraft (Linux)

[![Get it from the Snap Store](https://snapcraft.io/static/images/badges/en/snap-store-white.svg)](https://snapcraft.io/glide)

```bash
snap install glide
```

To upgrade the already installed package, you just need to run:

```bash
snap refresh glide
```

Detailed instruction on Snapcraft installation for different Linux distos:

- [Arch](https://snapcraft.io/install/glide/arch)
- [CentOS](https://snapcraft.io/install/glide/centos)
- [Debian](https://snapcraft.io/install/glide/debian)
- [elementaryOS](https://snapcraft.io/install/glide/elementary)
- [Fedora](https://snapcraft.io/install/glide/fedora)
- [KDE Neon](https://snapcraft.io/install/glide/kde-neon)
- [Kubuntu](https://snapcraft.io/install/glide/kubuntu)
- [Manjaro](https://snapcraft.io/install/glide/manjaro)
- [Pop! OS](https://snapcraft.io/install/glide/pop)
- [openSUSE](https://snapcraft.io/install/glide/opensuse)
- [RHEL](https://snapcraft.io/install/glide/rhel)
- [Ubuntu](https://snapcraft.io/install/glide/ubuntu)
- [Raspberry Pi](https://snapcraft.io/install/glide/raspbian)

### Docker Images

Glide provides official images in our [GHCR](https://github.com/EinStack/glide/pkgs/container/glide) & [DockerHub](https://hub.docker.com/u/einstack ):

- Alpine 3.19:
```bash
docker pull ghcr.io/einstack/glide:latest-alpine
```

- Ubuntu 22.04 LTS:
```bash
docker pull ghcr.io/einstack/glide:latest-ubuntu
```

- Google Distroless (non-root)
```bash
docker pull ghcr.io/einstack/glide:latest-distroless
```

- RedHat UBI 8.9 Micro
```bash
docker pull ghcr.io/einstack/glide:latest-redhat
```

### Helm Chart

Add the EinStack repository:

```bash
helm repo add einstack https://einstack.github.io/charts
helm repo update
```

Before installing the Helm chart, you need to create a Kubernetes secret with your API keys like:

```bash
kubectl create secret generic api-keys --from-literal=OPENAI_API_KEY=sk-abcdXYZ
```

Then, you need to create a custom values.yaml file to override the secret name like:

```yaml
# save as custom.values.yaml, for example
glide:
apiKeySecret: "api-keys"
```

Finally, you should be able to install Glide's chart via:

```
helm upgrade glide-gateway einstack/glide --values custom.values.yaml --install
```

## SDKs

To let you work with Glide's API with ease, we are going to provide you with SDKs that fits your tech stack:

- Python (coming soon)
- NodeJS (coming soon)

## Core Concepts

### Routers

Routers are a core functionality of Glide. Think of routers as a group of models with some predefined logic. For example, the resilience router allows a user to define a set of backup models should the initial model fail. Another example, would be to leverage the least-latency router to make latency sensitive LLM calls in the most efficient manner.

Detailed info on routers can be found [here](https://docs.einstack.ai/glide/docs/routers).

#### Available Routers

| Router | Description |
|---------------|-----------------|
| Priority | When the target model fails the request is sent to the secondary model. The entire service instance keeps track of the number of failures for a specific model reducing latency upon model failure |
| Least Latency | This router selects the model with the lowest average latency over time. If the least latency model becomes unhealthy, it will pick the second the best, etc. |
| Round Robin | Split traffic equally among specified models. Great for A/B testing. |
| Weighted Round Robin | Split traffic based on weights. For example, 70% of traffic to Model A and 30% of traffic to Model B. |

## Community

- Join [Discord](https://discord.gg/pt53Ej7rrc) for real-time discussion

Open [an issue](https://github.com/EinStack/glide/issues) or start [a discussion](https://github.com/EinStack/glide/discussions)
if there is a feature or an enhancement you'd like to see in Glide.

## Contribute

- Maintainers

- [Roman Hlushko](https://github.com/roma-glushko), Software Engineer, Distributed Systems & MLOps
- [Max Krueger](https://github.com/mkrueger12), Data & ML Engineer

Thanks everyone for already put their effort to make Glide better and more feature-rich:

## License

Apache 2.0

[![FOSSA Status](https://app.fossa.com/api/projects/git%2Bgithub.com%2FEinStack%2Fglide.svg?type=large)](https://app.fossa.com/projects/git%2Bgithub.com%2FEinStack%2Fglide?ref=badge_large)

ecosyste.ms

Data

Tools

Indexes

Applications

Experiments

Awesome

https://github.com/EinStack/glide

Awesome Lists containing this project

README

Glide: Cloud-Native LLM Gateway for Seamless LLMOps