{"id":25379585,"url":"https://github.com/wx-yz/ai-gateway","last_synced_at":"2025-04-09T11:41:36.730Z","repository":{"id":277597988,"uuid":"932489205","full_name":"wx-yz/ai-gateway","owner":"wx-yz","description":"Developer focused AI Gateway","archived":false,"fork":false,"pushed_at":"2025-03-02T04:20:11.000Z","size":162,"stargazers_count":9,"open_issues_count":0,"forks_count":2,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-03-02T04:25:30.579Z","etag":null,"topics":["ai","ai-gateway","artificial-intelligence","docker","llms"],"latest_commit_sha":null,"homepage":"https://ai-gateway.dev","language":"Ballerina","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/wx-yz.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-02-14T01:52:54.000Z","updated_at":"2025-03-02T04:20:14.000Z","dependencies_parsed_at":"2025-03-02T04:23:00.898Z","dependency_job_id":"5e4276ec-9abe-4bc7-b076-3b299ebfd140","html_url":"https://github.com/wx-yz/ai-gateway","commit_stats":null,"previous_names":["wx-yz/ai-proxy","wx-yz/ai-gateway"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wx-yz%2Fai-gateway","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wx-yz%2Fai-gateway/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wx-yz%2Fai-gateway/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wx-yz%2Fai-gateway/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/wx-yz","download_url":"https://codeload.github.com/wx-yz/ai-gateway/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248033030,"owners_count":21036715,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","ai-gateway","artificial-intelligence","docker","llms"],"created_at":"2025-02-15T05:31:17.769Z","updated_at":"2025-04-09T11:41:36.723Z","avatar_url":"https://github.com/wx-yz.png","language":"Ballerina","funding_links":[],"categories":[],"sub_categories":[],"readme":"# AI Gateway\n\nA configurable API gateway for multiple LLM providers (OpenAI, Anthropic, Gemini, Ollama) with built-in analytics, guardrails, and administrative controls.\n\n## Getting Started\n\n1. Create file named `Config.toml` with following content\n```toml\n[openAIConfig]\napiKey = \"Your_API_Key\"\nmodel = \"gpt-4\"\nendpoint = \"https://api.openai.com\"\n```\n2. Run below docker command\n```shell\ndocker run -p \\\n    8080:8080 -p 8081:8081 -p 8082:8082 \\\n    -v $(pwd)/Config.toml:/home/ballerina/Config.toml \\\n    chintana/ai-gateway:v1.2.0\n```\n3. Start sending requests\n```\ncurl -X POST http://localhost:8080/v1/chat/completions \\\n    -H \"Content-Type: application/json\" \\\n    -H \"x-llm-provider: openai\" \\\n    -d '\n{\n  \"messages\": [\n    {\n        \"role\": \"user\", \n        \"content\": \"Solve world hunger\" \n    }\n  ]\n}\n'\n```\n\n## Compatible with OpenAI SDK\n\nUse any OpenAI compatible SDK to talk to the gateway. Following example use official OpenAI Python SDK\n\n1. Install OpenAI official Python SDK\n```shell\n pip install openai\n```\n\n2. Example client. Note that setting the model and api key is enforced by the SDK. However these will be ignored by the gateway and will use whatever model and key configured at the gateway.\n```python\nimport openai\n\nopenai.api_key = '...' # Required by the SDK, AI Gateway will ignore this\n\n# all client options can be configured just like the `OpenAI` instantiation counterpart\nopenai.base_url = \"http://localhost:8080/v1/\"\nopenai.default_headers = {\"x-llm-provider\": \"openai\"}\n\n# Setting the model is enforced by the SDK. AI Gateway will ignore this value\ncompletion = openai.chat.completions.create(\n    model=\"gpt-4o\",\n    messages=[\n        {\n            \"role\": \"user\",\n            \"content\": \"Solve world hunger\",\n        },\n    ],\n)\nprint(completion.choices[0].message.content)\n```\n\n## Feature Highlights\n\n- **Multi-Provider Support**: Route requests to OpenAI, Anthropic, Gemini, Ollama, and Cohere\n- **Automatic Failover**: When 2+ providers are configured, automatically fails over to alternative providers if primary provider fails\n- **Rate Limiting**: Rate limiting policies\n- **OpenAI compatible interface**: Standardized input and output based on OpenAI API inteface\n- **Response Caching**: In-memory cache with configurable TTL for improved performance and reduced API costs\n- **System Prompts**: Inject system prompts into all LLM requests\n- **Response Guardrails**: Configure content filtering and response constraints\n- **Analytics Dashboard**: Monitor usage, tokens, and errors with visual charts\n- **Admin UI**: Configure AI gateway\n- **Administrative Controls**: Configure gateway behavior via admin API\n\n## HTTP API for chat completion\n\nOpenAI compatible request interface\n```shell\ncurl -X POST 'http://localhost:8080/v1/chat/completions' \\\n--header 'x-llm-provider: ollama' \\\n--header 'Content-Type: application/json' \\\n--data '{\n  \"messages\": [{ \n        \"role\": \"user\",\n        \"content\": \"When will we have AGI? In 10 words\" \n      }]\n}\n'\n```\n\nOpenAI API compatible response\n```json\n{\n    \"id\": \"01eff23c-208f-15a8-acdc-f400bba1bc6d\",\n    \"object\": \"chat.completion\",\n    \"created\": 1740352553,\n    \"model\": \"llama3.1:latest\",\n    \"choices\": [\n        {\n            \"index\": 0,\n            \"message\": {\n                \"role\": \"assistant\",\n                \"content\": \"Estimating exact timeline uncertain, but likely within next few decades.\"\n            },\n            \"finish_reason\": \"stop\"\n        }\n    ],\n    \"usage\": {\n        \"prompt_tokens\": 27,\n        \"completion_tokens\": 14,\n        \"total_tokens\": 41\n    }\n}\n```\n\n## gRPC API for chat completion\n\nAn example client in Python is available in `grpc-client` folder\n```python\ndef run():\n    # Create a gRPC channel\n    channel = grpc.insecure_channel('localhost:8082')\n\n    # Create a stub (client)\n    stub = ai_gateway_pb2_grpc.AIGatewayStub(channel)\n\n    # Create a request\n    request = ai_gateway_pb2.ChatCompletionRequest(\n        llm_provider=\"ollama\",\n        messages=[\n            ai_gateway_pb2.Message(\n                role=\"system\",\n                content=\"You are a helpful assistant.\"\n            ),\n            ai_gateway_pb2.Message(\n                role=\"user\",\n                content=\"What is the capital of France?\"\n            )\n        ]\n    )\n\n    try:\n        # Make the call\n        response = stub.ChatCompletion(request)\n\n    # ...\n```\n\n## Switching between LLM providers\n\nUse `x-llm-provider` HTTP header to route to different providers. AI Gateway mask request format differences between providers. Always use OpenAI API compatible request format and the gateway will always respond in OpenAI compatible response\n\n| LLM Provider | Header name    | Header value |\n| ------------ | -------------- | ------------ |\n| OpenAI       | x-llm-provider | openai       |\n| Ollama       | x-llm-provider | ollama       |\n| Anthropic    | x-llm-provider | anthropic    |\n| Gemini       | x-llm-provider | gemini       |\n| Mistral      | x-llm-provider | mistral      |\n| Cohere       | x-llm-provider | cohere       |\n\n## Disable caching for requests\n\nGateway automatiacally enable response caching to improve performance and save costs. The default cache duration is 1 hour. If you specifically wants to disable caching for equests, then send `Cache-Control: no-cache` HTTP header with each request\n\n## Gateway configuration\n\nGateway configuration can be done using either the built-in admin UI or using the REST API\n\n### Using AI Gateway API\n\nUse the following Postman collection to test the AI Gateway admin API. Configure rate limiting, add guardralis, manage caching etc... with the admin API. Also the collection supports calling chat completion APIs for different LLMs.\n\n[\u003cimg src=\"https://run.pstmn.io/button.svg\" alt=\"Run In Postman\" style=\"width: 128px; height: 32px;\"\u003e](https://god.gw.postman.com/run-collection/14185009-b582afdc-3194-4d82-ae06-416946f78eac?action=collection%2Ffork\u0026source=rip_markdown\u0026collection-url=entityId%3D14185009-b582afdc-3194-4d82-ae06-416946f78eac%26entityType%3Dcollection%26workspaceId%3D64e3340f-fb8a-4791-9887-e559c4fdb5b3)\n\n### Admin UI\n\nMain Admin UI display current stats on the server\n\n![1-admin-dashboard](https://github.com/user-attachments/assets/37ea56af-0d6a-4958-804c-4675c338d67e)\n\nConfigure Settings: system prompt, guardrails, and clear cache\n\n![2-settings](https://github.com/user-attachments/assets/28f57888-fa52-4046-97d9-6c8d7fee29c3)\n\nAdd/modify logging config\n\n![3-logging-config](https://github.com/user-attachments/assets/acaa24ec-9427-441a-81d5-645c6cba3d79)\n\nAdd/modify rate limiting policy\n\n![4-rate-limiting](https://github.com/user-attachments/assets/b0645586-f782-46bc-9d5e-a6534345987a)\n\n### Add rate limiting\n\nTwo separate rate limiting policies are supported,\n1. Wildcard based - Use the value \"*.*.*.*\" as the client IP and this will apply to any client\n2. Specific IP based\n\nAdd a wildcard rate limiting policy\n\n```shell\ncurl -X POST 'http://localhost:8081/admin/ratelimit/clients' \\\n--header 'Content-Type: application/json' \\\n--data '{\n    \"clientIP\": \"*.*.*.*\",\n    \"name\": \"wildcard\",\n    \"requestsPerWindow\": 5,\n    \"windowSeconds\": 60\n}'\n```\n\nAdd an IP based rate limiting policy\n\n```shell\ncurl -X POST 'http://localhost:8081/admin/ratelimit/clients' \\\n--header 'Content-Type: application/json' \\\n--data '{\n    \"clientIP\": \"192.168.1.80\",\n    \"name\": \"wildcard\",\n    \"requestsPerWindow\": 5,\n    \"windowSeconds\": 60\n}'\n```\n\nOnce rate limiting is enbaled, following 3 HTTP response headers will be used to announce current limits. These will be added to every HTTP response generated by the gateway\n\n| Header name     | Value  | Description |\n| --------------- | ------ | ----------- |\n| RateLimit-Limit | number | Maximum number of requests allowed in the current policy |\n| RateLimit-Remaining | number | Number of requests that can be sent before rate limit policy is enforced |\n| RateLimit-Reset | number | How many seconds until current rate limit policy is reset\n\nFollowing GET call will return the currently configured rate limiting policy. If the request is empty then rate limiting is disabled\n\n```shell\ncurl -X GET 'http://localhost:8081/admin/ratelimit/clients'\n```\nRespnose\n```json\n{\n    \"*.*.*.*\": {\n        \"clientIP\": \"*.*.*.*\",\n        \"name\": \"wildcard\",\n        \"requestsPerWindow\": 5,\n        \"windowSeconds\": 60\n    },\n    \"192.168.1.80\": {\n        \"clientIP\": \"192.168.1.80\",\n        \"name\": \"host-1\",\n        \"requestsPerWindow\": 10,\n        \"windowSeconds\": 120\n    }\n}\n```\n\n### Automatic failover\n\nWhen 2 or more LLM providers are configured, the gateway will attempt automatic failover if there's no successful response from the provider user has chosen through `x-llm-provider` header.\n\nThe logs will dispaly a trail of failover like below. Here, the user is trying to send the request to Ollama. We have Ollama and OpenAI configured in the gateway.\n\nFirst we can see a failed message. Following logs are formatted for clarity.\n```json\n{\n  \"timestamp\": \"2025-02-24T00:33:51.127868Z\",\n  \"level\": \"WARN\",\n  \"component\": \"failover\",\n  \"message\": \"Primary provider failed\",\n  \"metadata\": {\n    \"requestId\": \"01eff247-0444-1eb0-b153-61183107b722\",\n    \"provider\": \"ollama\",\n    \"error\": \"Something wrong with the connection:{}\"\n  }\n}\n```\nFirst attempt to failover,\n```json\n{\n  \"timestamp\": \"2025-02-24T00:33:51.129457Z\",\n  \"level\": \"INFO\",\n  \"component\": \"failover\",\n  \"message\": \"Attempting failover\",\n  \"metadata\": {\n    \"requestId\": \"01eff247-0444-1eb0-b153-61183107b722\",\n    \"provider\": \"openai\"\n  }\n}\n```\n\n### System Prompt Injection ###\n\nAdmins can use the admin API to inject a system prompt for all out going requests. This will be appended to the system prompt if a user has supplied one in the request\n```shell\ncurl -X POST 'http://localhost:8081/admin/systemprompt' \\\n--header 'Content-Type: application/json' \\\n--data '{\n    \"prompt\": \"respond only in chinese\"\n}'\n```\n\nFollowing GET request will show current system prompt\n```shell\ncurl -X GET 'http://localhost:8081/admin/systemprompt'\n```\n\n### Enforcing guardrails ###\n\nUse the following API call to add guardrails\n```shell\ncurl -X POST 'http://localhost:8081/admin/guardrails' \\\n--header 'Content-Type: application/json' \\\n--data '{\n    \"bannedPhrases\": [\"obscene\", \"words\"],\n    \"minLength\": 0,\n    \"maxLength\": 500000,\n    \"requireDisclaimer\": false\n}'\n```\n\nGet currently configured guardrails\n```shell\ncurl -X GET 'http://localhost:8081/admin/guardrails'\n```\n\n### Cache Management\n\nGateway automatically enbale response caching for requests to save costs and enable responsiveness. Default cache duration is 1 hour. When requests are served from the cache, there will be a respective log printed to the logs.\n\nThe gateway will look for `Cache-Control: no-cache` header and will disable cache lookup for those requests\n\nView current cached contents\n```shell\ncurl -X GET 'http://localhost:8081/admin/cache'\n```\n\nClear cache\n```shell\ncurl -X DELETE 'http://localhost:8081/admin/cache'\n```\n\n### Publish logs to Elastic Search\n\nConfigure following attributes in `Config.toml` to configure log publishing to Elastic Search\n```toml\n[defaultLoggingConfig]\nenableElasticSearch = true\nelasticSearchEndpoint = \"http://localhost:9200\"\nelasticApiKey = \"T2FtMks1VUIzVG...\"\n```\nAfter that at the server start, you should see an index being created in Elastic Search called \"ai-gateway\"\n\n![elastic-search-1](https://github.com/user-attachments/assets/bbd3bd87-36bb-4449-906d-e20e592e6b34)\n\nAll ongoing logs will gets published to this index\n\n![elastic-search-2](https://github.com/user-attachments/assets/cb95071a-fe42-470a-99a2-725ad4e30c5f)\n\n## Configuration reference ##\n\nFollowing is a complete example of all the configuration possible in the main gateway config file. At least one LLM provider config is mandatory\n\nCreate a `Config.toml` file:\n```\n[defaultLoggingConfig]\nenableElasticSearch = false\nelasticSearchEndpoint = \"http://localhost:9200\"\nelasticApiKey = \"\"\nenableSplunk = false\nsplunkEndpoint = \"\"\nenableDatadog = false\ndatadogEndpoint = \"\"\n\n[openAIConfig]\napiKey=\"your-api-key\"\nendpoint=\"https://api.openai.com\"\nmodel=\"gpt-4o\"\n\n[anthropicConfig]\napiKey=\"your-api-key\"\nmodel=\"claude-3-5-sonnet-20241022\"\nendpoint=\"https://api.anthropic.com\"\n\n[geminiConfig]\napiKey=\"your-api-key\"\nmodel=\"gemini-pro\"\nendpoint=\"https://generativelanguage.googleapis.com/v1/models\"\n\n\n[ollamaConfig]\napiKey=\"\"\nmodel=\"llama3.2\"\nendpoint=\"http://localhost:11434\"\n\n[mistralConfig]\napiKey = \"\"\nmodel = \"mistral-small-latest\"\nendpoint = \"https://api.mistral.ai\"\n\n[cohereConfig]\napiKey = \"\"\nmodel = \"command-r-plus-08-2024\"\nendpoint = \"https://api.cohere.com\"\n```\n\n## Development\n\n```bash\n# Build and run the gateway\n% bal run\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwx-yz%2Fai-gateway","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwx-yz%2Fai-gateway","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwx-yz%2Fai-gateway/lists"}