{"id":31681082,"url":"https://github.com/ziozzang/llm-toolcall-proxy","last_synced_at":"2025-10-08T07:45:46.451Z","repository":{"id":307403589,"uuid":"1029397273","full_name":"ziozzang/llm-toolcall-proxy","owner":"ziozzang","description":"General Tool-calling API Proxy","archived":false,"fork":false,"pushed_at":"2025-07-31T03:27:24.000Z","size":46,"stargazers_count":2,"open_issues_count":0,"forks_count":2,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-07-31T04:28:01.350Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ziozzang.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-07-31T01:54:52.000Z","updated_at":"2025-07-31T04:08:50.000Z","dependencies_parsed_at":"2025-07-31T04:28:03.388Z","dependency_job_id":"2ed1a5ed-131e-44df-8006-e05ad7fe9b2e","html_url":"https://github.com/ziozzang/llm-toolcall-proxy","commit_stats":null,"previous_names":["ziozzang/llm-toolcall-proxy"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/ziozzang/llm-toolcall-proxy","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ziozzang%2Fllm-toolcall-proxy","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ziozzang%2Fllm-toolcall-proxy/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ziozzang%2Fllm-toolcall-proxy/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ziozzang%2Fllm-toolcall-proxy/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ziozzang","download_url":"https://codeload.github.com/ziozzang/llm-toolcall-proxy/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ziozzang%2Fllm-toolcall-proxy/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":278909713,"owners_count":26066887,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-08T02:00:06.501Z","response_time":56,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-10-08T07:45:45.261Z","updated_at":"2025-10-08T07:45:46.444Z","avatar_url":"https://github.com/ziozzang.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# AI Model Tool Call Proxy Server\n\n**🌍 [한국어 README](README_ko.md)**\n\nA Flask-based proxy server that enables seamless integration of AI models with different tool call formats by automatically converting them to OpenAI's standard format. Perfect for using models like GLM with OpenAI-compatible clients.\n\n## ✨ Features\n\n- **🔄 Automatic Tool Call Conversion**: Converts model-specific tool call formats (like GLM's `\u003ctool_call\u003e` syntax) to OpenAI's standard format\n- **⚡ Streaming Support**: Full support for both streaming and non-streaming responses\n- **🎯 Model-Specific Handling**: Modular converter system that automatically detects and handles different model formats\n- **🌐 Full OpenAI API Compatibility**: Supports all major endpoints (`/v1/chat/completions`, `/v1/completions`, `/v1/models`, `/v1/embeddings`)\n- **🔧 Configurable**: Environment-based configuration for easy deployment\n- **🧩 Extensible**: Easy to add support for new model formats\n\n## 🚀 Quick Start\n\n### Prerequisites\n\n- Python 3.7+\n- A running AI model server (LM Studio, Ollama, etc.)\n\n### Installation\n\n1. **Clone and setup**:\n   ```bash\n   git clone \u003crepository-url\u003e\n   cd proxy\n   pip install -r requirements.txt\n   ```\n\n2. **Configure your backend** (optional):\n   ```bash\n   # Set environment variables or modify config.py\n   export BACKEND_HOST=localhost\n   export BACKEND_PORT=8888\n   ```\n\n3. **Start the proxy**:\n   ```bash\n   python app.py\n   ```\n\n4. **Use with any OpenAI-compatible client**:\n   ```python\n   from openai import OpenAI\n   \n   client = OpenAI(base_url=\"http://localhost:5000\", api_key=\"your-key\")\n   \n   response = client.chat.completions.create(\n       model=\"glm-4.5-air-hi-mlx\",\n       messages=[{\"role\": \"user\", \"content\": \"Search for information about Python\"}],\n       tools=[{\n           \"type\": \"function\",\n           \"function\": {\n               \"name\": \"search_wikipedia\",\n               \"description\": \"Search Wikipedia\",\n               \"parameters\": {\n                   \"type\": \"object\",\n                   \"properties\": {\"query\": {\"type\": \"string\"}},\n                   \"required\": [\"query\"]\n               }\n           }\n       }]\n   )\n   ```\n\n## 🎯 Supported Models\n\n### Currently Supported\n\n| Model Type | Tool Call Format | Status |\n|------------|------------------|---------|\n| **GLM Models** | `\u003ctool_call\u003e` syntax | ✅ Full Support |\n| **OpenAI Models** | Standard format | ✅ Pass-through |\n| **Claude Models** | `\u003cinvoke\u003e` syntax | ✅ Example Implementation |\n\n### GLM Format Example\n\n**Input (GLM format):**\n```\nI'll search for that information.\n\u003ctool_call\u003efetch_wikipedia_content\n\u003carg_key\u003esearch_query\u003c/arg_key\u003e\n\u003carg_value\u003ePython programming\u003c/arg_value\u003e\n\u003c/tool_call\u003e\n```\n\n**Output (OpenAI format):**\n```json\n{\n  \"tool_calls\": [{\n    \"id\": \"123456789\",\n    \"type\": \"function\",\n    \"function\": {\n      \"name\": \"fetch_wikipedia_content\",\n      \"arguments\": \"{\\\"search_query\\\": \\\"Python programming\\\"}\"\n    }\n  }],\n  \"finish_reason\": \"tool_calls\"\n}\n```\n\n## 🔧 Configuration\n\n### Environment Variables\n\n| Variable | Default | Description |\n|----------|---------|-------------|\n| `BACKEND_HOST` | `localhost` | Backend server hostname |\n| `BACKEND_PORT` | `8888` | Backend server port |\n| `BACKEND_PROTOCOL` | `http` | Backend protocol |\n| `PROXY_HOST` | `0.0.0.0` | Proxy server bind address |\n| `PROXY_PORT` | `5000` | Proxy server port |\n| `REQUEST_TIMEOUT` | `3600` | Regular request timeout (seconds) |\n| `STREAMING_TIMEOUT` | `3600` | Streaming request timeout (seconds, use 'none' to disable) |\n| `ENABLE_TOOL_CALL_CONVERSION` | `true` | Enable/disable tool call conversion |\n| `REMOVE_THINK_TAGS` | `true` | Remove complete `\u003cthink\u003e...\u003c/think\u003e` blocks from responses |\n| `LOG_LEVEL` | `INFO` | Logging level |\n| `FLASK_ENV` | `development` | Environment (development/production/testing) |\n\n### Configuration File\n\nCreate a `.env` file or modify `config.py` directly:\n\n```python\n# config.py\nBACKEND_HOST = 'localhost'\nBACKEND_PORT = 8888\nPROXY_PORT = 5000\nENABLE_TOOL_CALL_CONVERSION = True\nREMOVE_THINK_TAGS = True  # Set to False to preserve \u003cthink\u003e content\n```\n\n### Predefined Backend Configurations\n\n```python\nfrom config import get_backend_config\n\n# Use with LM Studio\nlmstudio_config = get_backend_config('lmstudio')  # localhost:8888\n\n# Use with Ollama\nollama_config = get_backend_config('ollama')      # localhost:11434\n\n# Use with OpenAI API\nopenai_config = get_backend_config('openai')     # api.openai.com:443\n```\n\n## 🏗️ Architecture\n\n### Modular Converter System\n\n```\n┌─────────────────┐    ┌──────────────────┐    ┌─────────────────┐\n│   Client        │───▶│  Proxy Server    │───▶│  Backend Model  │\n│  (OpenAI API)   │    │                  │    │   (GLM/etc.)    │\n└─────────────────┘    └──────────────────┘    └─────────────────┘\n                              │\n                              ▼\n                       ┌──────────────────┐\n                       │ Converter Factory │\n                       │  - GLM Converter  │\n                       │  - OpenAI Converter│\n                       │  - Claude Converter│\n                       │  - Custom Converter│\n                       └──────────────────┘\n```\n\n### Adding New Model Support\n\n1. **Create a converter**:\n   ```python\n   # converters/mymodel.py\n   from .base import ToolCallConverter\n   \n   class MyModelConverter(ToolCallConverter):\n       def can_handle_model(self, model_name: str) -\u003e bool:\n           return 'mymodel' in model_name.lower()\n       \n       def parse_tool_calls(self, content: str) -\u003e List[Dict]:\n           # Your parsing logic here\n           pass\n   ```\n\n2. **Register the converter**:\n   ```python\n   # In factory.py or at runtime\n   from converters.factory import converter_factory\n   converter_factory.register_converter(MyModelConverter())\n   ```\n\n## 📡 API Endpoints\n\n### Chat Completions\n- `POST /v1/chat/completions` - Chat completions with tool call conversion\n- `POST /chat/completions` - Alternative endpoint\n\n### Other OpenAI Compatible Endpoints\n- `GET /v1/models` - List available models\n- `POST /v1/completions` - Text completions\n- `POST /v1/embeddings` - Text embeddings\n- `GET /health` - Health check\n\n### Request Example\n\n```bash\ncurl -X POST http://localhost:5000/v1/chat/completions \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n    \"model\": \"glm-4.5-air-hi-mlx\",\n    \"messages\": [\n      {\"role\": \"user\", \"content\": \"Search for Python tutorials\"}\n    ],\n    \"tools\": [\n      {\n        \"type\": \"function\",\n        \"function\": {\n          \"name\": \"web_search\",\n          \"description\": \"Search the web\",\n          \"parameters\": {\n            \"type\": \"object\",\n            \"properties\": {\n              \"query\": {\"type\": \"string\"}\n            },\n            \"required\": [\"query\"]\n          }\n        }\n      }\n    ],\n    \"stream\": false\n  }'\n```\n\n## 🧪 Testing\n\n### Run All Tests\n\n```bash\n# Test modular converters\npython test_modular_converters.py\n\n# Test tool call conversion\npython test_tool_call_real.py\n\n# Test streaming functionality\npython test_streaming_tools.py\n\n# Test full API compatibility\npython test_full_api.py\n```\n\n### Integration Testing\n\n```bash\n# Start your backend model server (e.g., LM Studio on port 8888)\n# Start the proxy server\npython app.py\n\n# Test with the example client\npython lmstudio-tooluse-test.py\n```\n\n## 🐛 Troubleshooting\n\n### Common Issues\n\n1. **Connection Refused**\n   ```bash\n   # Check if backend server is running\n   curl http://localhost:8888/v1/models\n   \n   # Check proxy server\n   curl http://localhost:5000/health\n   ```\n\n2. **Tool Calls Not Converting**\n   ```bash\n   # Check if conversion is enabled\n   export ENABLE_TOOL_CALL_CONVERSION=true\n   \n   # Check model detection\n   # Make sure your model name matches the converter patterns\n   ```\n\n3. **Import Errors**\n   ```bash\n   # Make sure you're in the correct directory\n   cd /path/to/proxy\n   python app.py\n   ```\n\n### Debug Mode\n\n```bash\nexport FLASK_ENV=development\nexport LOG_LEVEL=DEBUG\npython app.py\n```\n\n## 🤝 Contributing\n\n1. **Fork the repository**\n2. **Create a feature branch**: `git checkout -b feature/amazing-feature`\n3. **Add tests** for your changes\n4. **Commit your changes**: `git commit -m 'Add amazing feature'`\n5. **Push to the branch**: `git push origin feature/amazing-feature`\n6. **Open a Pull Request**\n\n### Adding Model Support\n\nWe welcome contributions for new model formats! Please:\n\n1. Create a converter in `converters/`\n2. Add comprehensive tests\n3. Update documentation\n4. Submit a PR with examples\n\n## 📄 License\n\nThis project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.\n\n## 🧠 Think Tag Processing\n\nThe proxy server can handle GLM model `\u003cthink\u003e` tags for better response formatting:\n\n### Configuration\nSet `REMOVE_THINK_TAGS` environment variable to control behavior:\n\n- `REMOVE_THINK_TAGS=true` (default): Remove complete `\u003cthink\u003e...\u003c/think\u003e` blocks\n- `REMOVE_THINK_TAGS=false`: Preserve `\u003cthink\u003e` content for debugging/transparency\n\n### Examples\n\n**With REMOVE_THINK_TAGS=true (default):**\n```\nInput:  \"I need to analyze this. \u003cthink\u003eLet me think step by step...\u003c/think\u003e Here's my answer.\"\nOutput: \"I need to analyze this.  Here's my answer.\"\n```\n\n**With REMOVE_THINK_TAGS=false:**\n```\nInput:  \"I need to analyze this. \u003cthink\u003eLet me think step by step...\u003c/think\u003e Here's my answer.\"\nOutput: \"I need to analyze this. Let me think step by step... Here's my answer.\"\n```\n\n**Note:** Malformed/orphaned think tags are always cleaned up regardless of the setting:\n- `\u003c/think\u003e` without opening tag → removed\n- `\u003cthink\u003e` without closing tag → removed\n\n## ⏱️ Timeout Configuration\n\nThe proxy server supports different timeout settings for regular and streaming requests:\n\n### Environment Variables\n\n- `REQUEST_TIMEOUT=3600`: Regular request timeout (60 minutes)\n- `STREAMING_TIMEOUT=3600`: Streaming request timeout (60 minutes)\n\n### Disable Streaming Timeout\n\nFor long-running streaming requests, you can disable the timeout:\n\n```bash\nexport STREAMING_TIMEOUT=none\n# or\nexport STREAMING_TIMEOUT=0\n# or \nexport STREAMING_TIMEOUT=false\n```\n\n### Usage Examples\n\n**Short timeout for quick responses:**\n```bash\nexport REQUEST_TIMEOUT=30\nexport STREAMING_TIMEOUT=120\n```\n\n**No timeout for long streaming sessions:**\n```bash\nexport REQUEST_TIMEOUT=3600\nexport STREAMING_TIMEOUT=none\n```\n\n**Note:** Disabling streaming timeout is useful for:\n- Long document generation\n- Complex reasoning tasks\n- Large dataset processing\n- Extended conversations\n\n## 🙏 Acknowledgments\n\n- Built for seamless integration with [LM Studio](https://lmstudio.ai/)\n- Compatible with [OpenAI Python SDK](https://github.com/openai/openai-python)\n- Inspired by the need for universal AI model compatibility\n\n## 📞 Support\n\n- 🐛 **Bug Reports**: [Open an issue](issues)\n- 💡 **Feature Requests**: [Start a discussion](discussions)\n- 📖 **Documentation**: [Wiki](wiki)\n- 🌍 **Korean Documentation**: [README_ko.md](README_ko.md)\n\n---\n\n**Made with ❤️ for the AI community**","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fziozzang%2Fllm-toolcall-proxy","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fziozzang%2Fllm-toolcall-proxy","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fziozzang%2Fllm-toolcall-proxy/lists"}