{"id":27125805,"url":"https://github.com/1lcb/ollama-gateway","last_synced_at":"2025-10-24T23:03:21.317Z","repository":{"id":273089391,"uuid":"918653960","full_name":"1LCB/ollama-gateway","owner":"1LCB","description":"A lightweight HTTP reverse proxy that routes requests to multiple Ollama servers. It includes features like rate limiting, API key validation, security filtering, metrics collection, and hot-reloading of configurations.","archived":false,"fork":false,"pushed_at":"2025-01-18T15:09:44.000Z","size":11,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-01-18T16:22:43.482Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/1LCB.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-01-18T14:06:00.000Z","updated_at":"2025-01-18T15:09:47.000Z","dependencies_parsed_at":"2025-01-18T16:22:54.634Z","dependency_job_id":"1aaca163-4b17-43ff-832a-c8a5ae1a023b","html_url":"https://github.com/1LCB/ollama-gateway","commit_stats":null,"previous_names":["1lcb/ollama-gateway"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/1LCB%2Follama-gateway","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/1LCB%2Follama-gateway/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/1LCB%2Follama-gateway/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/1LCB%2Follama-gateway/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/1LCB","download_url":"https://codeload.github.com/1LCB/ollama-gateway/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247682106,"owners_count":20978607,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-04-07T15:40:14.115Z","updated_at":"2025-10-24T23:03:21.305Z","avatar_url":"https://github.com/1LCB.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Ollama Gateway Documentation\n\n## Overview\n\nThe Ollama Gateway is a lightweight HTTP reverse proxy with advanced features such as rate limiting, security filtering, API key validation, metrics collection, and hot-reloading of configurations. It enables secure and efficient routing to multiple Ollama servers.\n\n---\n\n## Features\n\n1. **Reverse Proxy**: Distributes incoming requests to configured Ollama servers using a round-robin strategy.\n2. **Rate Limiting**: Limits the number of requests per client within a specified time window.\n3. **API Key Authentication**: Verifies requests based on API keys for access control.\n4. **Security Filters**: Supports IP allowlist and denylist configurations.\n5. **Metrics**: Provides Prometheus-compatible metrics for monitoring.\n6. **Hot Configuration Reload**: Allows reloading of the configuration file without restarting the server.\n7. **Key Generation**: Generates new API keys dynamically via an HTTP endpoint.\n\n---\n\n## Configuration\n\nThe gateway configuration is stored in a `config.json` file. Below is an example:\n\n```json\n{\n    \"ollamaAddresses\":[\n        \"http://localhost:11434/\"\n    ],\n    \"loadBalancer\": {\n        \"healthCheckIntervalInSeconds\": 30,\n        \"healthCheckEndpoint\": \"/\",\n        \"healthCheckTimeoutInMilliseconds\": 5000\n    },\n    \"gatewayAddress\": \"0.0.0.0:8080\",\n    \"logging\": true,\n    \"authHeaderName\": \"Authorization\",\n    \"apiKeys\": [\n        \"ahP7DieD7rNpTUa6No--iJpOaXY3TDK8dIjKg0cp-hI\"\n    ],\n    \"rateLimit\": {\n        \"maxRequests\": 10,\n        \"timeWindowSeconds\": 15,\n        \"enabled\": false\n    },\n    \"security\": {\n        \"allowIps\": [],\n        \"denyIps\": []\n    },\n    \"metrics\": {\n        \"enabled\": true,\n        \"endpoint\": \"/metrics\",\n        \"namespace\": \"gateway\"\n    }\n}\n```\n\n### Key Configuration Fields\n\n- **`ollamaAddresses`**: List of Ollama server addresses to forward requests to.\n- **`gatewayAddress`**: The address and port on which the gateway listens.\n- **`logging`**: Enables or disables logging.\n- **`authHeaderName`**: The name of the HTTP header used for API key authentication.\n- **`rateLimit`**: Configures the maximum requests allowed within a time window.\n- **`security`**: Configures IP allowlists and denylists for access control.\n- **`metrics`**: Configures Prometheus metrics endpoint and namespace.\n\n---\n\n## Run using Docker\n\nYou can run the Ollama Gateway easily using Docker, simply execute the following command:\n\n```bash\ndocker run -d \\\n  -v /path/to/config.json:/config.json \\\n  -p 8080:8080 \\\n  1lcb/ollama-gateway:latest\n```\n\n---\n\n## HTTP Endpoints\n\n- **`/`**: Reverse proxy endpoint to forward requests to Ollama servers.\n- **`/config/reload`**: Reloads the configuration file.\n- **`/config/new-key`**: Generates a new API key.\n- **`/metrics`**: Exposes Prometheus metrics (customizable endpoint).\n\n---\n\n## Metrics\n\nThe following Prometheus metrics are available:\n\n1. **`gateway_requests_total`**: Total number of HTTP requests, labeled by method, endpoint, and status.\n2. **`gateway_requests_duration_seconds`**: Latency histogram of HTTP requests, labeled by method and endpoint.\n3. **`gateway_requests_by_ip_total`**: Total number of requests grouped by IP address, endpoint, and status.\n4. **`gateway_requests_success_total`**: Total number of successful HTTP requests, labeled by method and endpoint.\n5. **`gateway_requests_failure_total`**: Total number of failed HTTP requests, labeled by method and endpoint.\n6. **`gateway_apikeys_total`**: Number of active API keys.\n7. **`gateway_ollama_servers_total`**: Number of configured Ollama servers.\n\n---\n\n## Middleware\n\n1. **RateLimitMiddleware**: Enforces rate limiting based on the configuration.\n2. **IPAndAPIKeyMiddleware**: Validates incoming requests using IP filters and API keys.\n3. **MetricsMiddleware**: Collects and exposes metrics for Prometheus.\n\n---\n\n## Example: Using the Ollama Gateway with the Default Library\n\nWith just a few simple changes, you can easily integrate the Ollama Gateway into your existing code. The gateway automatically handles authentication, rate limiting, IP filtering, and more. Here's how:\n\n```python\nfrom ollama import Client\n\n# Set the Ollama Gateway address and authentication headers\nollama_gateway_address = \"http://localhost:8080\"\nheaders = {\n    \"Authorization\": \"ahP7DieD7rNpTUa6No--iJpOaXY3TDK8dIjKg0cp-hI\"\n}\n\n# Create a client pointing to the Ollama Gateway\nclient = Client(\n    host=ollama_gateway_address,\n    headers=headers\n)\n\n# Send a chat message to the model\nstream = client.chat(\n    model=\"llama3.1:8b\",\n    messages=[{\"role\": \"user\", \"content\": \"Hello, how are you today?\"}],\n    stream=True\n)\n\n# Print the response from the model\nfor chunk in stream:\n    print(chunk.message.content, end=\"\", flush=True)\nprint()\n```\n\n\n## Example: Hot Configuration Reload\n\nTo reload the configuration:\n\n1. Update the `config.json` file.\n2. Send a `GET` request to `/config/reload`.\n\nOn success, the gateway updates its settings dynamically without downtime.\n\n---\n\n## Example: Generating a New API Key\n\nSend a `GET` request to `/config/new-key` to generate a new API key. The response will include the newly created key. The generated key will also be added in the `ApiKeys` list from `config.json` file\n\n---\n\n## Implementation Details\n\n### Reverse Proxy Logic\n\nThe gateway uses a round-robin strategy to distribute requests across Ollama servers:\n\n```go\nfunc getAddressRoundRobin() string {\n    mu.Lock()\n    address := cfg.OllamaAddresses[currentIndex]\n    if len(cfg.OllamaAddresses) \u003e 1 {\n        currentIndex = (currentIndex + 1) % len(cfg.OllamaAddresses)\n    }\n    mu.Unlock()\n\n    return address\n}\n```\n\n### Hot Configuration Reload\n\nThe gateway supports hot reloading via the `ReloadConfig` function. It updates metrics and logger settings dynamically:\n\n```go\nfunc ReloadConfig() error {\n    utils.ReloadLogger()\n\n    if err := config.ReloadConfig(); err != nil {\n        return fmt.Errorf(\"failed to reload config: %v\", err)\n    }\n\n\tif config.Metrics.Enabled{\n\t\tmetrics.APIKeysTotal.Set(float64(len(config.APIKeys)))\n\t\tmetrics.OllamaServersTotal.Set(float64(len(config.OllamaAddresses)))\n\t}\n\n    return nil\n}\n```\n\n---\n\n## Conclusion\n\nThe Ollama Gateway provides a robust and lightweight solution for managing traffic to Ollama servers. With its features like API key validation, rate limiting, and Prometheus metrics, it ensures security, scalability, and observability in a simple, configurable manner.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2F1lcb%2Follama-gateway","html_url":"https://awesome.ecosyste.ms/projects/github.com%2F1lcb%2Follama-gateway","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2F1lcb%2Follama-gateway/lists"}