{"id":21371874,"url":"https://github.com/rahulunair/xpu_tgi","last_synced_at":"2025-03-16T08:43:08.886Z","repository":{"id":263508441,"uuid":"889231177","full_name":"rahulunair/xpu_tgi","owner":"rahulunair","description":"TGI server setup for Intel Data Centre GPUs","archived":false,"fork":false,"pushed_at":"2024-11-26T00:11:01.000Z","size":463,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-01-19T21:48:25.239Z","etag":null,"topics":["intel","intelgpu","llm","llm-inference","tgi","xpu"],"latest_commit_sha":null,"homepage":"","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rahulunair.png","metadata":{"files":{"readme":"Readme.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-11-15T21:38:08.000Z","updated_at":"2024-11-26T00:11:04.000Z","dependencies_parsed_at":"2024-11-18T23:53:46.511Z","dependency_job_id":"85d0a32c-32a2-441f-8994-9f61f50bafc9","html_url":"https://github.com/rahulunair/xpu_tgi","commit_stats":null,"previous_names":["rahulunair/xpu_tgi"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rahulunair%2Fxpu_tgi","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rahulunair%2Fxpu_tgi/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rahulunair%2Fxpu_tgi/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rahulunair%2Fxpu_tgi/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rahulunair","download_url":"https://codeload.github.com/rahulunair/xpu_tgi/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243846982,"owners_count":20357297,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["intel","intelgpu","llm","llm-inference","tgi","xpu"],"created_at":"2024-11-22T08:15:49.021Z","updated_at":"2025-03-16T08:43:08.659Z","avatar_url":"https://github.com/rahulunair.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# TGI Models Collection\n\nWelcome to `xpu_tgi`! 🚀  \n\nA curated collection of Text Generation Inference (TGI) models optimized for Intel XPU, with built-in security and traffic management.\n\n\u003cdiv align=\"center\"\u003e\n\u003cimg src=\"./hi_tgi.jpg\" alt=\"TGI LLM Servers\" width=\"400\"/\u003e\n\u003c/div\u003e\n\n## Quick Start\n\n```bash\n# 1. Generate authentication token\npython utils/generate_token.py\n\n# 2. Start a model\n./start.sh Flan-T5-XXL\n\n# 3. Make a request\ncurl -X POST http://localhost:8000/generate \\\n  -H \"Authorization: Bearer YOUR_TOKEN\" \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\"inputs\": \"What is quantum computing?\", \"parameters\": {\"max_new_tokens\": 50}}'\n```\n\n## Architecture \u0026 Security\n\n```mermaid\nflowchart LR\n    Client([Client])\n    Traefik[Traefik Proxy]\n    Auth[Auth Service]\n    TGI[TGI Service]\n\n    Client --\u003e Traefik\n    Traefik --\u003e Auth\n    Auth --\u003e Traefik\n    Traefik --\u003e TGI\n    TGI --\u003e Traefik\n    Traefik --\u003e Client\n\n    subgraph Internal[\"Internal Network\"]\n        Traefik\n        Auth\n        TGI\n    end\n\n    classDef client fill:#f2d2ff,stroke:#9645b7,stroke-width:2px;\n    classDef proxy fill:#bbdefb,stroke:#1976d2,stroke-width:2px;\n    classDef auth fill:#c8e6c9,stroke:#388e3c,stroke-width:2px;\n    classDef tgi fill:#ffccbc,stroke:#e64a19,stroke-width:2px;\n    classDef network fill:#fff9c4,stroke:#fbc02d,stroke-width:1px;\n\n    class Client client;\n    class Traefik proxy;\n    class Auth auth;\n    class TGI tgi;\n    class Internal network;\n\n```\n\n### Key Features\n- 🔒 Token-based authentication with automatic ban after failed attempts\n- 🚦 Rate limiting (global: 10 req/s, per-IP: 10 req/s)\n- 🛡️ Security headers and IP protection\n- 🔄 Health monitoring and automatic recovery\n- 🚀 Optimized for Intel GPUs\n\n## Available Models\n\n### Long Context Models (\u003e8k tokens)\n- **Phi-3-mini-128k** - 128k context window\n- **Hermes-3-llama3.1** - 8k context window\n\n### Code Generation\n- **CodeLlama-7b** - Specialized for code completion\n- **Phi-3-mini-4k** - Efficient code generation\n\n### General Purpose\n- **Flan-T5-XXL** - Versatile text generation\n- **Flan-UL2** - Advanced language understanding\n- **Hermes-2-pro** - Balanced performance\n- **OpenHermes-Mistral** - Fast inference\n\nEach model includes:\n- Individual configuration (`config/model.env`)\n- Detailed documentation (`README.md`)\n- Optimized parameters for Intel XPU\n\n## Security \u0026 Configuration\n\n### Authentication\n```bash\n# Generate secure token (admin)\npython utils/generate_token.py\n\n# Example output:\n# --------------------------------------------------------------------------------\n# Generated at: 2024-03-22T15:30:45.123456\n# Token: XcAwKq7BSbGSoJCsVhUQ2e6MZ4ZOAH_mRR0HgmMNBQg\n# --------------------------------------------------------------------------------\n```\n\n### Traffic Management\n```yaml\n# Rate Limits\nGlobal: 10 req/s (burst: 25)\nPer-IP: 10 req/s (burst: 25)\n\n# Security Headers\n- XSS Protection\n- Content Type Nosniff\n- Frame Deny\n- HSTS\n```\n\n## API Usage\n\n### Basic Generation\n```bash\ncurl -X POST http://localhost:8000/generate \\\n  -H \"Authorization: Bearer YOUR_TOKEN\" \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n    \"inputs\": \"What is quantum computing?\",\n    \"parameters\": {\"max_new_tokens\": 50}\n  }'\n```\n\n### Advanced Parameters\n```bash\ncurl -X POST http://localhost:8000/generate \\\n  -H \"Authorization: Bearer YOUR_TOKEN\" \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n    \"inputs\": \"Explain AI\",\n    \"parameters\": {\n      \"max_new_tokens\": 100,\n      \"temperature\": 0.7,\n      \"top_p\": 0.95\n    }\n  }'\n```\n\n### Health Monitoring\n```bash\n# System health\ncurl http://localhost:8000/health\n\n# Model status\ncurl http://localhost:8000/v1/models\n```\n\n## Contributing\n\nContributions are welcome! Please read our [Contributing Guidelines](CONTRIBUTING.md) first.\n\n## License Notes\n\nEach model has its own license terms. Please review individual model READMEs before use.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frahulunair%2Fxpu_tgi","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frahulunair%2Fxpu_tgi","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frahulunair%2Fxpu_tgi/lists"}