{"id":24966824,"url":"https://github.com/lukehinds/fastllm","last_synced_at":"2025-04-15T22:56:42.750Z","repository":{"id":275363604,"uuid":"925842684","full_name":"lukehinds/fastllm","owner":"lukehinds","description":"FastLLM - Rust based LLM Inference API ","archived":false,"fork":false,"pushed_at":"2025-03-31T13:55:39.000Z","size":155,"stargazers_count":12,"open_issues_count":7,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-15T22:56:34.873Z","etag":null,"topics":["inference","llama","llm","mistral","rust","security","speed"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lukehinds.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-02-01T21:52:56.000Z","updated_at":"2025-03-23T03:57:00.000Z","dependencies_parsed_at":"2025-03-15T04:30:49.527Z","dependency_job_id":null,"html_url":"https://github.com/lukehinds/fastllm","commit_stats":null,"previous_names":["lukehinds/infrs","lukehinds/fastllm"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lukehinds%2Ffastllm","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lukehinds%2Ffastllm/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lukehinds%2Ffastllm/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lukehinds%2Ffastllm/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lukehinds","download_url":"https://codeload.github.com/lukehinds/fastllm/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249167439,"owners_count":21223505,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["inference","llama","llm","mistral","rust","security","speed"],"created_at":"2025-02-03T12:21:28.071Z","updated_at":"2025-04-15T22:56:42.731Z","avatar_url":"https://github.com/lukehinds.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# FastLLM\n\nA Rust inference server providing OpenAI-compatible APIs for local LLM deployment. Run language models directly from HuggingFace with native MacOS Metal, CUDA and CPU support.\n\n## Experimental\n\nThis is a work in progress and the API is not yet stable!\n\n## Key Features\n- Native acceleration on Metal (Apple Silicon) and CUDA\n- Direct model loading from HuggingFace Hub\n- Run various architectures like Mistral, Qwen, TinyLlama\n- Generate embeddings using models like all-MiniLM-L6-v2\n\n## Design Principles\n\nFastLLM adheres to the following core design principles:\n\n**Simple and Modular**\n   - Clean, well-documented code structure\n   - Modular architecture for easy model integration\n   - Trait-based design for flexible model implementations\n   - Automatic architecture detection from model configs\n\n**Zero Config**\n   - Sensible defaults for all features and optimizations\n   - Automatic hardware detection and optimization\n   - Smart fallbacks when optimal settings aren't available\n\n**Easy to Extend**\n   - Clear separation of concerns\n   - Minimal boilerplate for adding new models\n   - Comprehensive test coverage and examples\n   - Detailed documentation for model integration\n\nThe goal is to make it as straightforward as possible to add new models while maintaining high performance by default.\n\n## Supported Models\n\n| Model Family | Supported Architectures | Example Models |\n|------------|-----------------|-------------|\n| **Llama** | LlamaForCausalLM | • TinyLlama-1.1B-Chat\u003cbr\u003e• Any Llama2 derivative |\n| **Mistral** | MistralForCausalLM | • Mistral-7B and derivatives\u003cbr\u003e• Mixtral-8x7B |\n| **Qwen** | Qwen2ForCausalLM\u003cbr\u003eQwen2_5_VLForConditionalGeneration | • Qwen2\u003cbr\u003e• Qwen2.5 |\n| **BERT** | BertModel\u003cbr\u003eRobertaModel\u003cbr\u003eDebertaModel | • all-MiniLM-L6-v2\u003cbr\u003e• Any BERT/RoBERTa/DeBERTa model |\n\n## Quick Start\n\n### Prerequisites\n\n- Rust toolchain ([install from rustup.rs](https://rustup.rs))\n- HuggingFace token (for gated models)\n\n### Installation\n\n```bash\n# Clone the repository\ngit clone https://github.com/yourusername/fastllm.git\ncd fastllm\n\n# Optional: Set HuggingFace token for gated models\nexport HF_TOKEN=\"your_token_here\"\n\n# Build the project (MacOS Metal)\ncargo build --release --features \"metal\"\n\n# Build the project (Linux CUDA)\ncargo build --release --features \"cuda\"\n\n# Build the project (CPU)\ncargo build --release\n```\n\n### Running the Server\n\n```bash\n# Start with default settings\n./target/release/fastllm\n\n# Or specify a model directly\n./target/release/fastllm --model TinyLlama/TinyLlama-1.1B-Chat-v1.0\n```\n\n## 🔧 Configuration\n\nFastLLM can be configured through multiple methods (in order of precedence):\n\n1. **Command Line Arguments**\n   ```bash\n   ./target/release/fastllm --model mistralai/Mistral-7B-v0.1\n   ```\n\n2. **Environment Variables**\n   ```bash\n   export FASTLLM_SERVER__HOST=0.0.0.0\n   export FASTLLM_SERVER__PORT=8080\n   export FASTLLM_MODEL__MODEL_ID=your-model-id\n   ```\n\n3. **Configuration File** (`config.json`)\n   ```json\n   {\n       \"server\": {\n           \"host\": \"127.0.0.1\",\n           \"port\": 3000\n       },\n       \"model\": {\n           \"model_id\": \"TinyLlama/TinyLlama-1.1B-Chat-v1.0\",\n           \"revision\": \"main\"\n       }\n   }\n   ```\n\n## 🔌 API Examples\n\n### Chat Completion\n\n```bash\ncurl http://localhost:3000/v1/chat/completions \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n    \"model\": \"TinyLlama/TinyLlama-1.1B-Chat-v1.0\",\n    \"messages\": [\n      {\"role\": \"system\", \"content\": \"You are a helpful assistant.\"},\n      {\"role\": \"user\", \"content\": \"What is the capital of France?\"}\n    ],\n    \"temperature\": 0.7,\n    \"stream\": true\n  }'\n```\n\n### Text Embeddings\n\n```bash\ncurl http://localhost:3000/v1/embeddings \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n    \"model\": \"sentence-transformers/all-MiniLM-L6-v2\",\n    \"input\": \"The food was delicious and the service was excellent.\"\n  }'\n```\n\n## 🗺️ Roadmap\n\n- [ ] Support for more architectures (DeepSeek, Phi, etc.)\n- [ ] Comprehensive benchmarking suite\n- [ ] Model management API (/v1/models)\n- [ ] Improved caching and optimization\n- [ ] Multi-GPU Inference\n\n## Contributing\n\nContributions are welcome! Feel free to:\n- Open issues for bugs or feature requests\n- Submit pull requests\n- Share benchmarks and performance reports\n\n## License\n\nApache 2.0 - See [LICENSE](LICENSE) for details.\n\n---\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flukehinds%2Ffastllm","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flukehinds%2Ffastllm","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flukehinds%2Ffastllm/lists"}