{"id":28209616,"url":"https://github.com/aashari/go-generative-api-router","last_synced_at":"2025-06-10T21:32:08.797Z","repository":{"id":293786473,"uuid":"985124142","full_name":"aashari/go-generative-api-router","owner":"aashari","description":"Go microservice that proxies OpenAI-compatible API calls to multiple LLM vendors (OpenAI, Gemini) using configurable selection strategies. Supports vendor filtering, streaming responses, and tool calling while maintaining transparent request/response handling.","archived":false,"fork":false,"pushed_at":"2025-06-09T11:22:47.000Z","size":555,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-06-09T12:31:25.988Z","etag":null,"topics":["ai","api","api-gateway","api-proxy","api-router","artificial-intelligence","cloud-native","gemini","go","golang","large-language-model","llm","microservice","models","openai","proxy","routing","streaming","tool-calling","vendor-selection"],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/aashari.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-05-17T05:39:52.000Z","updated_at":"2025-06-09T11:22:19.000Z","dependencies_parsed_at":"2025-06-01T11:22:25.738Z","dependency_job_id":"6724d1b2-1911-4513-b99a-b8fff42676c9","html_url":"https://github.com/aashari/go-generative-api-router","commit_stats":null,"previous_names":["aashari/generative-api-router"],"tags_count":22,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aashari%2Fgo-generative-api-router","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aashari%2Fgo-generative-api-router/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aashari%2Fgo-generative-api-router/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aashari%2Fgo-generative-api-router/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/aashari","download_url":"https://codeload.github.com/aashari/go-generative-api-router/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aashari%2Fgo-generative-api-router/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":259154449,"owners_count":22813602,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","api","api-gateway","api-proxy","api-router","artificial-intelligence","cloud-native","gemini","go","golang","large-language-model","llm","microservice","models","openai","proxy","routing","streaming","tool-calling","vendor-selection"],"created_at":"2025-05-17T16:12:53.435Z","updated_at":"2025-06-10T21:32:08.788Z","avatar_url":"https://github.com/aashari.png","language":"Go","readme":"# Generative API Router\n\n[![Go Report Card](https://goreportcard.com/badge/github.com/aashari/go-generative-api-router)](https://goreportcard.com/report/github.com/aashari/go-generative-api-router)\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)\n[![Go Version](https://img.shields.io/github/go-mod/go-version/aashari/go-generative-api-router)](https://github.com/aashari/go-generative-api-router)\n\nA production-ready Go microservice that provides a **unified OpenAI-compatible API** for multiple LLM vendors (OpenAI, Gemini). This transparent proxy router simplifies AI integration by offering a single interface while intelligently distributing requests across multiple vendors and preserving your original model names in responses.\n\n\u003c!-- \n\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"https://raw.githubusercontent.com/aashari/go-generative-api-router/main/docs/assets/architecture-diagram.png\" alt=\"Architecture Diagram\" width=\"800\"\u003e\n\u003c/div\u003e\n--\u003e\n\n## 🏗️ **Architecture Overview**\n\n### **Multi-Vendor OpenAI-Compatible Router**\n\nThis service acts as a **transparent proxy** that provides a unified OpenAI-compatible API interface while routing requests to multiple LLM vendors behind the scenes:\n\n- **OpenAI API Compatibility**: All vendors accessed through OpenAI-compatible endpoints\n- **Transparent Model Handling**: Preserves your original model names in responses\n- **Multi-Vendor Design**: Currently supports **19 credentials** (18 Gemini + 1 OpenAI) with **4 models**\n- **Even Distribution**: Fair selection across **114 vendor-credential-model combinations**\n\n### **Enterprise-Grade Features (2024)**\n\nRecent comprehensive improvements include:\n\n- 🔒 **Security**: AES-GCM encryption for credentials, sensitive data masking in logs\n- 🔄 **Reliability**: Exponential backoff retry logic, circuit breaker pattern implementation\n- 📊 **Monitoring**: Comprehensive health checks with vendor connectivity monitoring\n- ⚡ **Performance**: Production-optimized logging, conditional detail levels\n- 🧹 **Code Quality**: DRY principles, centralized utilities, eliminated code duplication\n\n## Features\n\n- **Multi-Vendor Support**: Routes requests to OpenAI or Gemini using OpenAI API compatibility\n- **Even Distribution Selection**: Fair distribution across all vendor-credential-model combinations\n- **Vendor Filtering**: Supports explicit vendor selection via `?vendor=` query parameter\n- **Transparent Proxy**: Maintains all original request/response data (except for model selection)\n- **Streaming Support**: Properly handles chunked streaming responses for real-time applications\n- **Tool Calling**: Supports function calling/tools for AI agents with proper validation\n- **Enterprise Reliability**: Circuit breakers, retry logic, comprehensive health monitoring\n- **Security**: Encrypted credential storage, sensitive data masking\n- **Modular Design**: Clean separation of concerns with selector, validator, and client components\n- **Configuration Driven**: Easily configure available models and credentials via JSON files\n- **Health Check**: Built-in health check endpoint with service status monitoring\n- **Comprehensive Testing**: Full test coverage with unit tests for all components\n- **🌐 Public Image URL Support**: Automatic downloading and conversion of public image URLs to base64\n- **📄 Advanced File Processing**: Comprehensive document processing supporting PDF, Word, Excel, PowerPoint, ZIP archives, and more via markitdown integration\n\n## Quick Start\n\n### Prerequisites\n\n- Go 1.21 or higher\n- API keys for OpenAI and/or Google Gemini\n- Make (for build automation)\n- Python 3.8+ with markitdown for file processing (automatically installed via setup)\n\n### Installation\n\n1. **Clone the Repository**:\n   ```bash\n   git clone https://github.com/aashari/go-generative-api-router.git\n   cd go-generative-api-router\n   ```\n\n2. **Setup Environment**:\n   ```bash\n   make setup\n   ```\n   This will:\n   - Download Go dependencies\n   - Install development tools\n   - Create `configs/credentials.json` from the example template\n\n3. **Verify Configuration**:\n   ```bash\n   # Check existing configuration (service likely has working credentials)\n   cat configs/credentials.json | jq length \u0026\u0026 echo \"credentials configured\"\n   cat configs/models.json | jq length \u0026\u0026 echo \"models configured\"\n   ```\n\n4. **Configure Credentials** (if needed):\n   Edit `configs/credentials.json` with your API keys:\n   ```json\n   [\n     {\n       \"platform\": \"openai\",\n       \"type\": \"api-key\",\n       \"value\": \"sk-your-openai-key\"\n     },\n     {\n       \"platform\": \"gemini\",\n       \"type\": \"api-key\",\n       \"value\": \"your-gemini-key\"\n     }\n   ]\n   ```\n\n5. **Configure Models** (if needed):\n   Edit `configs/models.json` to define which vendor-model pairs can be selected:\n   ```json\n   [\n     {\n       \"vendor\": \"gemini\",\n       \"model\": \"gemini-2.0-flash\"\n     },\n     {\n       \"vendor\": \"openai\",\n       \"model\": \"gpt-4o\"\n     }\n   ]\n   ```\n\n6. **Run the Service**:\n   ```bash\n   make run\n   ```\n   \n   The service will be available at http://localhost:8082\n\n## Selection Strategy\n\nThe router uses an **Even Distribution Selector** that ensures fair distribution across all vendor-credential-model combinations. This approach provides true fairness where each combination has exactly equal probability of being selected.\n\n### How It Works\n\n1. **Combination Generation**: The system creates a flat list of all valid vendor-credential-model combinations\n2. **Equal Probability**: Each combination gets exactly `1/N` probability where N = total combinations\n3. **Fair Distribution**: Unlike traditional two-stage selection (vendor → model), this ensures no bias toward vendors with fewer models\n\n### Example Distribution\n\nWith the current configuration:\n- **18 Gemini credentials** × **6 models** = 108 combinations\n- **1 OpenAI credential** × **6 models** = 6 combinations\n- **Total**: 114 combinations\n\nEach combination has exactly **1/114 = 0.877%** probability:\n- **Gemini overall**: 108/114 = 94.7%\n- **OpenAI overall**: 6/114 = 5.3%\n\nThis reflects the actual resource availability rather than artificial vendor-level balancing.\n\n### Benefits\n\n- ✅ **True Fairness**: Each credential-model combination has exactly equal probability\n- ✅ **Resource Proportional**: Distribution reflects actual available resources\n- ✅ **Scalable**: Automatically adapts as credentials/models are added/removed\n- ✅ **Transparent**: Clear logging shows selection and total combination count\n- ✅ **No Bias**: Eliminates bias toward vendors with fewer models per credential\n\n### Monitoring Selection\n\nThe service logs each selection decision for transparency:\n\n```\nEven distribution selected combination - Vendor: openai, Model: gpt-4o (from 114 total combinations)\n```\n\nYou can monitor the distribution by checking the server logs to verify fair selection across all combinations.\n\n## Usage\n\n### Basic API Usage\n\n```bash\n# Health check\ncurl http://localhost:8082/health\n\n# List available models\ncurl http://localhost:8082/v1/models\n\n# Chat completion (any model name)\ncurl -X POST http://localhost:8082/v1/chat/completions \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n    \"model\": \"my-preferred-model\",\n    \"messages\": [{\"role\": \"user\", \"content\": \"Hello!\"}]\n  }'\n\n# Process PDF documents\ncurl -X POST http://localhost:8082/v1/chat/completions \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n    \"model\": \"document-analyzer\",\n    \"messages\": [\n      {\n        \"role\": \"user\",\n        \"content\": [\n          {\n            \"type\": \"text\",\n            \"text\": \"Please summarize this research paper:\"\n          },\n          {\n            \"type\": \"file_url\",\n            \"file_url\": {\n              \"url\": \"https://example.com/research-paper.pdf\"\n            }\n          }\n        ]\n      }\n    ]\n  }'\n\n# Force specific vendor\ncurl -X POST \"http://localhost:8082/v1/chat/completions?vendor=openai\" \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n    \"model\": \"my-model\",\n    \"messages\": [{\"role\": \"user\", \"content\": \"Hello from OpenAI!\"}]\n  }'\n```\n\n### Using Example Scripts\n\nExample scripts are provided for common use cases:\n\n```bash\n# Basic usage examples\n./examples/curl/basic.sh\n\n# Streaming examples\n./examples/curl/streaming.sh\n\n# Tool calling examples\n./examples/curl/tools.sh\n```\n\n### Client Libraries\n\nExample implementations are available for multiple languages:\n\n- **Python**: `examples/clients/python/client.py`\n- **Node.js**: `examples/clients/nodejs/client.js`\n- **Go**: `examples/clients/go/client.go`\n\n### Docker Deployment\n\nBuild and run using Docker:\n\n```bash\n# Build and run with Docker Compose\nmake docker-build\nmake docker-run\n\n# Stop the service\nmake docker-stop\n```\n\n## Testing\n\n### Multi-Vendor Testing\n\n**IMPORTANT**: This is a multi-vendor service. Always test both vendors:\n\n```bash\n# Test OpenAI vendor\ncurl -X POST \"http://localhost:8082/v1/chat/completions?vendor=openai\" \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\"model\": \"test-openai\", \"messages\": [{\"role\": \"user\", \"content\": \"Hello\"}]}'\n\n# Test Gemini vendor  \ncurl -X POST \"http://localhost:8082/v1/chat/completions?vendor=gemini\" \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\"model\": \"test-gemini\", \"messages\": [{\"role\": \"user\", \"content\": \"Hello\"}]}'\n\n# Monitor vendor distribution\ngrep \"Even distribution selected combination\" logs/server.log | tail -5\n```\n\n### Development Testing\n\n```bash\n# Run all tests\nmake test\n\n# Run with coverage\nmake test-coverage\n\n# Full CI check\nmake ci-check\n```\n\n## Documentation\n\n### 📚 **Complete Documentation**\n\n- **[Documentation Index](docs/README.md)** - Complete documentation roadmap\n- **[User Guide](docs/user-guide.md)** - API usage and integration guide\n- **[API Reference](docs/api-reference.md)** - Complete API documentation\n- **[Development Guide](docs/development-guide.md)** - Setup and development workflow\n\n### 🔧 **Development Guides**\n\n- **[Contributing Guide](docs/contributing-guide.md)** - How to contribute to the project\n- **[Testing Guide](docs/testing-guide.md)** - Testing strategies and procedures\n- **[Logging Guide](docs/logging-guide.md)** - Comprehensive logging documentation\n- **[Deployment Guide](docs/deployment-guide.md)** - AWS infrastructure and deployment\n\n### 📖 **Cursor AI Context**\n\nFor Cursor AI development, see the comprehensive guides in `.cursor/rules/`:\n- **[Development Guide](.cursor/rules/development_guide.mdc)** - Complete workflow, architecture, Git practices\n- **[Running \u0026 Testing Guide](.cursor/rules/running_and_testing.mdc)** - Setup, testing, debugging\n\n## Architecture\n\n### Core Components\n\n- **Proxy Handler**: Routes requests to selected vendors, handles streaming/non-streaming responses\n- **Vendor Selector**: Implements even distribution selection across vendor-credential-model combinations\n- **Request Validator**: Validates OpenAI-compatible requests, preserves original model names\n- **Response Processor**: Processes vendor responses while maintaining model name transparency\n- **Health Monitor**: Comprehensive health checks with vendor connectivity monitoring\n- **Circuit Breaker**: Reliability pattern implementation for vendor communication\n- **Retry Logic**: Exponential backoff for failed vendor requests\n\n### Key Principles\n\n1. **Transparent Proxy**: Original model names preserved in responses\n2. **Vendor Agnostic**: Unified interface regardless of backend vendor  \n3. **Fair Distribution**: Even probability across all vendor-model combinations\n4. **OpenAI Compatibility**: 100% compatible with OpenAI API format\n5. **Enterprise Reliability**: Circuit breakers, retries, comprehensive monitoring\n\n## Production Deployment\n\nThe service is production-ready with:\n\n- **AWS ECS Deployment**: Containerized deployment on AWS\n- **Load Balancing**: High availability with load balancer integration\n- **Monitoring**: CloudWatch integration and comprehensive logging\n- **Security**: Encrypted credentials, sensitive data masking\n- **Reliability**: Circuit breakers, retry logic, health monitoring\n\nSee [Deployment Guide](docs/deployment-guide.md) for complete deployment instructions.\n\n## Contributing\n\nWe welcome contributions! Please see our [Contributing Guide](docs/contributing-guide.md) for details on:\n\n- Development setup and workflow\n- Code standards and review process\n- Testing requirements\n- Pull request guidelines\n\n## License\n\nThis project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.\n\n## Support\n\n- **Documentation**: [Complete documentation](docs/README.md)\n- **Issues**: [GitHub Issues](https://github.com/aashari/go-generative-api-router/issues)\n- **Discussions**: [GitHub Discussions](https://github.com/aashari/go-generative-api-router/discussions)\n\n---\n\n**Need help?** Check the [documentation](docs/README.md) or open an issue on GitHub.","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faashari%2Fgo-generative-api-router","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Faashari%2Fgo-generative-api-router","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faashari%2Fgo-generative-api-router/lists"}