{"id":26626664,"url":"https://github.com/anongecko/job-discovery-llm-vm","last_synced_at":"2025-07-27T03:11:22.400Z","repository":{"id":279989101,"uuid":"940234615","full_name":"anongecko/job-discovery-llm-vm","owner":"anongecko","description":"Very basic Azure LLM (3) VM setup for a private project.","archived":false,"fork":false,"pushed_at":"2025-02-28T15:51:12.000Z","size":52,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-24T11:19:21.194Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/anongecko.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-02-27T20:41:41.000Z","updated_at":"2025-02-28T15:51:15.000Z","dependencies_parsed_at":"2025-02-28T20:51:42.420Z","dependency_job_id":"a818bf7d-1c86-4d18-a56a-d163dd10ed37","html_url":"https://github.com/anongecko/job-discovery-llm-vm","commit_stats":null,"previous_names":["anongecko/job-discovery-llm-vm"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/anongecko/job-discovery-llm-vm","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/anongecko%2Fjob-discovery-llm-vm","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/anongecko%2Fjob-discovery-llm-vm/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/anongecko%2Fjob-discovery-llm-vm/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/anongecko%2Fjob-discovery-llm-vm/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/anongecko","download_url":"https://codeload.github.com/anongecko/job-discovery-llm-vm/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/anongecko%2Fjob-discovery-llm-vm/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":267293908,"owners_count":24065333,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-27T02:00:11.917Z","response_time":82,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-03-24T11:19:24.083Z","updated_at":"2025-07-27T03:11:22.368Z","avatar_url":"https://github.com/anongecko.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# AI Model Server for Job Discovery\n\n![License](https://img.shields.io/badge/license-MIT-blue.svg)\n![Python](https://img.shields.io/badge/python-3.10+-blue.svg)\n![CUDA](https://img.shields.io/badge/CUDA-11.8+-green.svg)\n\nA high-performance model server designed for AI-powered job discovery systems, optimized for A100 GPUs. This server provides powerful AI capabilities including text generation, embeddings, and classification through a consistent REST API.\n\n## 🚀 Features\n\n### Core Capabilities\n- **Advanced Text Generation**: Using Llama 3.1 70B (Q6_K quantized) for high-quality text generation\n- **Semantic Embeddings**: Fast vector embedding generation with Nomic-Embed-Text-v1\n- **Text Classification**: Efficient categorization using Mistral 7B Instruct v0.2\n- **Streaming Responses**: Event-stream support for real-time text generation\n- **Batched Processing**: Optimized batching for embeddings and classifications\n\n### Performance Features\n- **A100 GPU Optimization**: Configured for NVIDIA A100 80GB GPU\n- **Quantization**: Efficient memory usage through 4-bit and 6-bit model quantization\n- **Concurrent Request Handling**: Process multiple requests simultaneously\n- **Dynamic Memory Management**: Intelligent resource allocation based on workload\n\n### Operational Features\n- **Robust Authentication**: API key management with rate limiting and role-based access\n- **Comprehensive Monitoring**: Detailed metrics for performance analysis\n- **Health Checking**: Automated model health monitoring and maintenance\n- **Docker Deployment**: Production-ready containerization\n- **Auto-scaling**: Horizontal scaling support for high-load environments\n\n## 📊 Architecture\n\nThe Model Server is built on a modern, asynchronous architecture:\n\n```\n                           ┌─────────────────┐\n                           │   Client Apps   │\n                           └────────┬────────┘\n                                    │\n                         ┌──────────▼──────────┐\n┌───────────────────────┤   REST API Gateway   ├───────────────────────┐\n│                        └──────────┬──────────┘                       │\n│                                   │                                  │\n│  ┌──────────────────┐   ┌─────────▼─────────┐   ┌──────────────────┐ │\n│  │   Primary Model  │   │  Authentication   │   │  Rate Limiting   │ │\n│  │  (Llama 3.3 70B) ◄───┤    \u0026 Security     ├───►      Layer      │ │\n│  └────────┬─────────┘   └─────────┬─────────┘   └──────────────────┘ │\n│           │                       │                                  │\n│  ┌────────▼─────────┐   ┌─────────▼─────────┐   ┌──────────────────┐ │\n│  │  Embedding Model │   │    Request Queue  │   │ Classification   │ │\n│  │ (Nomic-Embed-Text)◄───┤   \u0026 Dispatcher   ├───►  Model (Mistral) │ │\n│  └──────────────────┘   └─────────┬─────────┘   └──────────────────┘ │\n│                                   │                                  │\n│                        ┌──────────▼──────────┐                       │\n└───────────────────────┤   Health Monitoring  ├───────────────────────┘\n                         └─────────────────────┘\n```\n\n### Component Overview\n- **API Gateway**: FastAPI-based HTTP interface with input validation\n- **Authentication**: Token-based API key system with role permissions\n- **Model Layer**: Optimized inference engines for text generation, embeddings, and classification\n- **Health Monitoring**: Automatic model maintenance and resource optimization\n\n## 🛠️ Installation\n\n### Prerequisites\n- NVIDIA GPU with CUDA 11.8+ support (A100 recommended)\n- Docker and Docker Compose\n- 200GB+ SSD storage\n- 32GB+ system RAM\n\n### Docker Installation (Recommended)\n\n```bash\n# Clone the repository\ngit clone https://github.com/yourusername/ai-model-server.git\ncd ai-model-server\n\n# Run the setup script\nchmod +x setup.sh\nsudo ./setup.sh\n\n# Start the server\ndocker-compose up -d\n```\n\n### Manual Installation\n\n```bash\n# Clone the repository\ngit clone https://github.com/yourusername/ai-model-server.git\ncd ai-model-server\n\n# Create a virtual environment\npython -m venv venv\nsource venv/bin/activate\n\n# Install dependencies\npip install -r requirements.txt\n\n# Generate an API key\npython -c \"from model_server.auth import generate_new_key; print(generate_new_key('admin', 'admin'))\"\n\n# Start the server\npython -m model_server.main\n```\n\n## ⚙️ Configuration\n\n### Model Configuration\nThe server uses `model_config.json` to configure models:\n\n```json\n{\n  \"primary_model\": {\n    \"model_name\": \"llama-3.1-70b-instruct\",\n    \"model_type\": \"gguf\",\n    \"model_path\": \"models/llama-3.1-70b-instruct.Q6_K.gguf\",\n    \"model_url\": \"https://huggingface.co/TheBloke/Llama-3.1-70B-Instruct-GGUF/resolve/main/llama-3.1-70b-instruct.Q6_K.gguf\",\n    \"device\": \"cuda\",\n    \"n_gpu_layers\": -1,\n    \"n_ctx\": 8192,\n    \"max_tokens\": 4096,\n    \"max_batch_size\": 8,\n    \"tensor_split\": \"0.99,0.01\",\n    \"threads\": 12,\n    \"use_mlock\": true,\n    \"n_batch\": 512,\n    \"stream_output\": true,\n    \"download_if_missing\": true,\n    \"max_concurrent_requests\": 12\n  },\n  ...\n}\n```\n\n### Environment Variables\n- `HOST`: Server host address (default: \"0.0.0.0\")\n- `PORT`: Server port (default: 8080)\n- `WORKERS`: Number of Uvicorn workers (default: 1)\n- `CONFIG_FILE`: Path to model configuration file\n- `DEBUG`: Enable debug mode (\"1\" or \"true\")\n- `DOWNLOAD_MODELS_ON_START`: Download models at startup (\"true\" or \"false\")\n\n## 🔑 Authentication\n\nThe server uses a token-based API key system:\n\n```bash\n# Generate a new API key (run in Python)\nfrom model_server.auth import generate_new_key\nnew_key = generate_new_key(name=\"client1\", role=\"client\", rate_limit=100, expires_in_days=365)\nprint(new_key)\n```\n\nAPI keys can be managed through the `/api-keys` endpoints (admin only):\n- `POST /api-keys/create`: Create a new API key\n- `GET /api-keys/list`: List all API keys\n- `PATCH /api-keys/update/{key}`: Update an API key\n- `DELETE /api-keys/revoke/{key}`: Revoke an API key\n\n## 📝 API Documentation\n\n### Text Generation\n```bash\ncurl -X POST http://localhost:8080/v1/completions \\\n  -H \"Content-Type: application/json\" \\\n  -H \"X-API-Key: YOUR_API_KEY\" \\\n  -d '{\n    \"prompt\": \"Extract the job title from: \\\"Senior Software Engineer at Google\\\"\",\n    \"system_message\": \"You are an expert job data extractor.\",\n    \"max_tokens\": 1024,\n    \"temperature\": 0.1,\n    \"stream\": false\n  }'\n```\n\n### Text Embeddings\n```bash\ncurl -X POST http://localhost:8080/v1/embeddings \\\n  -H \"Content-Type: application/json\" \\\n  -H \"X-API-Key: YOUR_API_KEY\" \\\n  -d '{\n    \"input\": [\"Software Engineer\", \"Data Scientist\"]\n  }'\n```\n\n### Text Classification\n```bash\ncurl -X POST http://localhost:8080/v1/classifications \\\n  -H \"Content-Type: application/json\" \\\n  -H \"X-API-Key: YOUR_API_KEY\" \\\n  -d '{\n    \"text\": \"Looking for a Python developer with 5+ years experience\",\n    \"categories\": [\"job_posting\", \"resume\", \"cover_letter\"]\n  }'\n```\n\n### Streaming Text Generation\n```javascript\n// JavaScript example for streaming\nconst eventSource = new EventSource('/v1/completions?api_key=YOUR_API_KEY');\nconst payload = {\n  prompt: \"Write a job description for a Senior AI Engineer\",\n  stream: true,\n  temperature: 0.7\n};\n\nfetch('/v1/completions', {\n  method: 'POST',\n  headers: {\n    'Content-Type': 'application/json',\n    'X-API-Key': 'YOUR_API_KEY'\n  },\n  body: JSON.stringify(payload)\n})\n.then(response =\u003e {\n  const reader = response.body.getReader();\n  // Process stream...\n});\n```\n\n## 📊 Monitoring\n\n### Health Check\n```bash\ncurl http://localhost:8080/health\n```\n\n### Metrics (Requires API Key)\n```bash\ncurl -H \"X-API-Key: YOUR_API_KEY\" http://localhost:8080/metrics\n```\n\nThe server includes Prometheus integration for comprehensive monitoring. Access Grafana dashboards at `http://localhost:3000` when using Docker deployment (default credentials: admin/model-server-admin).\n\n## ⚡ Performance Optimizations\n\n### A100 GPU Configuration\nFor optimal performance on an A100 GPU:\n\n- **VRAM Allocation**: \n  - Primary model: ~30GB \n  - Embedding model: ~2GB \n  - Classification model: ~4GB\n  - Total: ~36GB (with headroom for batching)\n\n- **Worker Configuration**:\n  - For 24 vCPUs, we recommend 8 worker processes\n  - Set `WORKERS=8` in the environment or docker-compose.yml\n\n- **Batch Processing**:\n  - Embeddings support batches of up to 64 inputs\n  - Use batching for processing large datasets\n\n## 🔒 Security Considerations\n\n- **API Key Management**:\n  - Rotate keys regularly\n  - Use role-based permissions\n  - Set appropriate rate limits\n\n- **Network Security**:\n  - Run behind a reverse proxy for TLS\n  - Use firewall rules to restrict access\n  - Consider VPC deployment for production\n\n- **Model Security**:\n  - Input validation prevents prompt injection\n  - Output is sanitized to prevent XSS\n  - Models have safety filters enabled\n\n## 🤝 Contributing\n\nContributions are welcome! Please feel free to submit a Pull Request.\n\n1. Fork the repository\n2. Create your feature branch (`git checkout -b feature/amazing-feature`)\n3. Commit your changes (`git commit -m 'Add some amazing feature'`)\n4. Push to the branch (`git push origin feature/amazing-feature`)\n5. Open a Pull Request\n\n## 📄 License\n\nThis project is licensed under the MIT License - see the LICENSE file for details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fanongecko%2Fjob-discovery-llm-vm","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fanongecko%2Fjob-discovery-llm-vm","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fanongecko%2Fjob-discovery-llm-vm/lists"}