https://github.com/urbanlab/ai-proxy
FOSS llm proxy
https://github.com/urbanlab/ai-proxy
ai-proxy co2-monitor grafana llm ollama
Last synced: 28 days ago
JSON representation
FOSS llm proxy
- Host: GitHub
- URL: https://github.com/urbanlab/ai-proxy
- Owner: urbanlab
- License: mit
- Created: 2025-09-09T09:20:24.000Z (9 months ago)
- Default Branch: main
- Last Pushed: 2026-03-23T10:57:08.000Z (2 months ago)
- Last Synced: 2026-03-24T08:27:28.136Z (2 months ago)
- Topics: ai-proxy, co2-monitor, grafana, llm, ollama
- Language: Python
- Homepage:
- Size: 435 KB
- Stars: 1
- Watchers: 0
- Forks: 0
- Open Issues: 8
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# AI PROXY
AI Proxy is an open-source project that provides a simple way to create a proxy server for LLM models.
Many existing solutions are pseudo-open-source with hidden features behind paywalls. AI Proxy aims to be fully open-source and free to use.
## 🍱 Features
- Monitor requests and responses
- api key model permission management
- Partial Support of openai api endpoint
## 📅 Planned Features
- Rate limiting
- CO2 emission tracking (CodeCarbon API)
- Same model load balancing
## 🚀 Quickstart
**Requirements:**
- Docker
**Copy the example configuration file and edit it to your needs:**
```bash
cp config.example.yaml config.yaml
```
Edit `config.yaml` to set your OpenAI API key and other configurations.
```
global:
model_list:
- model_name: devstral
params:
model: devstral:latest
api_base: http://ollama-service.ollama.svc.cluster.local:11434/v1
drop_params: true
api_key: "no_token"
max_input_tokens: 25000
keys:
- name: "user"
token: "token"
models:
- "devstral"
```
**Run the server:**
```bash
docker-compose up -d
```
The server will be available at `http://localhost:8000`.
And the docs at `http://localhost:8000/docs`.
## 📈 Monitoring
The api exposes prometheus metrics for monitoring.
The prometheus endpoint is available at `http://localhost:8001/metrics`.
exposed metrics:
- request_count
- request_latency
- request_tokens
- response_tokens
## ❤️ Humans.txt
- aangelot
- pipazoul