{"id":31027214,"url":"https://github.com/michael-a-kuykendall/shimmy","last_synced_at":"2025-09-13T18:57:29.778Z","repository":{"id":312170823,"uuid":"1046584285","full_name":"Michael-A-Kuykendall/shimmy","owner":"Michael-A-Kuykendall","description":"⚡ Python-free Rust inference server — OpenAI-API compatible. GGUF + SafeTensors, hot model swap, auto-discovery, single binary. FREE now, FREE forever.","archived":false,"fork":false,"pushed_at":"2025-09-12T14:34:21.000Z","size":7168,"stargazers_count":476,"open_issues_count":3,"forks_count":61,"subscribers_count":10,"default_branch":"main","last_synced_at":"2025-09-12T16:46:39.844Z","etag":null,"topics":["api-server","command-line-tool","developer-tools","gguf","huggingface","huggingface-models","huggingface-transformers","inference-server","llama","llamacpp","llm-inference","local-ai","lora","machine-learning","ollama-api","openai-compatible","rust","rust-crate","transformers"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Michael-A-Kuykendall.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":"CODEOWNERS","security":"SECURITY.md","support":null,"governance":null,"roadmap":"ROADMAP.md","authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null},"funding":{"github":"Michael-A-Kuykendall","ko_fi":"mikekuykendall","open_collective":"shimmy","buy_me_a_coffee":"michaelakuykendall"}},"created_at":"2025-08-28T22:55:46.000Z","updated_at":"2025-09-12T16:14:41.000Z","dependencies_parsed_at":"2025-08-29T04:29:54.561Z","dependency_job_id":"3c3af44b-8c7c-4bd4-8e30-d110d1399450","html_url":"https://github.com/Michael-A-Kuykendall/shimmy","commit_stats":null,"previous_names":["michael-a-kuykendall/shimmy"],"tags_count":8,"template":false,"template_full_name":null,"purl":"pkg:github/Michael-A-Kuykendall/shimmy","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Michael-A-Kuykendall%2Fshimmy","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Michael-A-Kuykendall%2Fshimmy/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Michael-A-Kuykendall%2Fshimmy/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Michael-A-Kuykendall%2Fshimmy/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Michael-A-Kuykendall","download_url":"https://codeload.github.com/Michael-A-Kuykendall/shimmy/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Michael-A-Kuykendall%2Fshimmy/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":275013271,"owners_count":25390481,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-13T02:00:10.085Z","response_time":70,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["api-server","command-line-tool","developer-tools","gguf","huggingface","huggingface-models","huggingface-transformers","inference-server","llama","llamacpp","llm-inference","local-ai","lora","machine-learning","ollama-api","openai-compatible","rust","rust-crate","transformers"],"created_at":"2025-09-13T18:57:28.413Z","updated_at":"2025-09-13T18:57:29.764Z","avatar_url":"https://github.com/Michael-A-Kuykendall.png","language":"Rust","funding_links":["https://github.com/sponsors/Michael-A-Kuykendall","https://ko-fi.com/mikekuykendall","https://opencollective.com/shimmy","https://buymeacoffee.com/michaelakuykendall"],"categories":["Langchain"],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\r\n  \u003cimg src=\"assets/shimmy-logo.png\" alt=\"Shimmy Logo\" width=\"300\" height=\"auto\" /\u003e\r\n  \r\n  # The 5MB Alternative to Ollama\r\n\r\n  [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)\r\n  [![Rust](https://img.shields.io/badge/rust-stable-brightgreen.svg)](https://rustup.rs/)\r\n  [![CI](https://github.com/Michael-A-Kuykendall/shimmy/workflows/CI/badge.svg)](https://github.com/Michael-A-Kuykendall/shimmy/actions)\r\n  [![Tests](https://img.shields.io/badge/Tests-Passing-brightgreen)](https://github.com/Michael-A-Kuykendall/shimmy/actions)\r\n  [![Quality](https://img.shields.io/badge/Quality-Assured-success)](https://github.com/Michael-A-Kuykendall/shimmy/actions)\r\n  [![Sponsor](https://img.shields.io/badge/❤️-Sponsor-ea4aaa?logo=github)](https://github.com/sponsors/Michael-A-Kuykendall)\r\n\u003c/div\u003e\r\n\r\n**Shimmy will be free forever.** No asterisks. No \"free for now.\" No pivot to paid.\r\n\r\n**Fast, reliable local AI inference.** Shimmy provides OpenAI-compatible endpoints for GGUF models with comprehensive testing and automated quality assurance.\r\n\r\n## What is Shimmy?\r\n\r\nShimmy is a **5.1MB single-binary** local inference server that provides OpenAI API-compatible endpoints for GGUF models. It's designed to be the **invisible infrastructure** that just works.\r\n\r\n| Metric | Shimmy | Ollama | \r\n|--------|--------|--------|\r\n| **Binary Size** | 5.1MB 🏆 | 680MB |\r\n| **Startup Time** | \u003c100ms 🏆 | 5-10s |\r\n| **Memory Overhead** | \u003c50MB 🏆 | 200MB+ |\r\n| **OpenAI Compatibility** | 100% 🏆 | Partial |\r\n| **Port Management** | Auto 🏆 | Manual |\r\n| **Configuration** | Zero 🏆 | Manual |\r\n\r\n## 🎯 Perfect for Developers\r\n\r\n- **Privacy**: Your code stays on your machine  \r\n- **Cost**: No per-token pricing, unlimited queries  \r\n- **Speed**: Local inference = sub-second responses  \r\n- **Integration**: Works with VSCode, Cursor, Continue.dev out of the box  \r\n\r\n**BONUS:** First-class LoRA adapter support - from training to production API in 30 seconds.\r\n\r\n## Quick Start (30 seconds)\r\n\r\n### Installation\r\n\r\n```bash\r\n# Install from crates.io (Linux, macOS, Windows)\r\ncargo install shimmy\r\n\r\n# Or download pre-built binary (Windows only)\r\ncurl -L https://github.com/Michael-A-Kuykendall/shimmy/releases/latest/download/shimmy.exe\r\n```\r\n\r\n\u003e **⚠️ Windows Security Notice**: Windows Defender may flag the binary as a false positive. This is common with unsigned Rust executables. **Recommended**: Use `cargo install shimmy` instead, or add an exclusion for shimmy.exe in Windows Defender.\r\n\r\n### Get Models\r\n\r\nShimmy auto-discovers models from:\r\n- **Hugging Face cache**: `~/.cache/huggingface/hub/`\r\n- **Ollama models**: `~/.ollama/models/`\r\n- **Local directory**: `./models/`\r\n- **Environment**: `SHIMMY_BASE_GGUF=path/to/model.gguf`\r\n\r\n```bash\r\n# Download models that work out of the box\r\nhuggingface-cli download microsoft/Phi-3-mini-4k-instruct-gguf --local-dir ./models/\r\nhuggingface-cli download bartowski/Llama-3.2-1B-Instruct-GGUF --local-dir ./models/\r\n```\r\n\r\n### Start Server\r\n\r\n```bash\r\n# Auto-allocates port to avoid conflicts\r\nshimmy serve\r\n\r\n# Or use manual port\r\nshimmy serve --bind 127.0.0.1:11435\r\n```\r\n\r\nPoint your AI tools to the displayed port - VSCode Copilot, Cursor, Continue.dev all work instantly!\r\n\r\n## 📦 Download \u0026 Install\r\n\r\n### Package Managers\r\n- **Rust**: [`cargo install shimmy`](https://crates.io/crates/shimmy)\r\n- **VS Code**: [Shimmy Extension](https://marketplace.visualstudio.com/items?itemName=targetedwebresults.shimmy-vscode)\r\n- **npm**: `npm install -g shimmy-js` *(coming soon)*\r\n- **Python**: `pip install shimmy` *(coming soon)*\r\n\r\n### Direct Downloads\r\n- **GitHub Releases**: [Latest binaries](https://github.com/Michael-A-Kuykendall/shimmy/releases/latest)\r\n- **Docker**: `docker pull shimmy/shimmy:latest` *(coming soon)*\r\n\r\n### 🍎 macOS Support\r\n\r\n**Full compatibility confirmed!** Shimmy works flawlessly on macOS with Metal GPU acceleration.\r\n\r\n```bash\r\n# Install dependencies\r\nbrew install cmake rust\r\n\r\n# Install shimmy\r\ncargo install shimmy\r\n```\r\n\r\n**✅ Verified working:**\r\n- Intel and Apple Silicon Macs\r\n- Metal GPU acceleration (automatic)\r\n- Xcode 17+ compatibility\r\n- All LoRA adapter features\r\n\r\n## Integration Examples\r\n\r\n### VSCode Copilot\r\n```json\r\n{\r\n  \"github.copilot.advanced\": {\r\n    \"serverUrl\": \"http://localhost:11435\"\r\n  }\r\n}\r\n```\r\n\r\n### Continue.dev\r\n```json\r\n{\r\n  \"models\": [{\r\n    \"title\": \"Local Shimmy\",\r\n    \"provider\": \"openai\", \r\n    \"model\": \"your-model-name\",\r\n    \"apiBase\": \"http://localhost:11435/v1\"\r\n  }]\r\n}\r\n```\r\n\r\n### Cursor IDE\r\nWorks out of the box - just point to `http://localhost:11435/v1`\r\n\r\n## Why Shimmy Will Always Be Free\r\n\r\nI built Shimmy because I was tired of 680MB binaries to run a 4GB model.\r\n\r\n**This is my commitment**: Shimmy stays MIT licensed, forever. If you want to support development, [sponsor it](https://github.com/sponsors/Michael-A-Kuykendall). If you don't, just build something cool with it.\r\n\r\n\u003e Shimmy saves you time and money. If it's useful, consider sponsoring for $5/month — less than your Netflix subscription, infinitely more useful.\r\n\r\n## Performance Comparison\r\n\r\n| Tool | Binary Size | Startup Time | Memory Usage | OpenAI API |\r\n|------|-------------|--------------|--------------|------------|\r\n| **Shimmy** | **5.1MB** | **\u003c100ms** | **50MB** | **100%** |\r\n| Ollama | 680MB | 5-10s | 200MB+ | Partial |\r\n| llama.cpp | 89MB | 1-2s | 100MB | None |\r\n\r\n## API Reference\r\n\r\n### Endpoints\r\n- `GET /health` - Health check\r\n- `POST /v1/chat/completions` - OpenAI-compatible chat\r\n- `GET /v1/models` - List available models\r\n- `POST /api/generate` - Shimmy native API\r\n- `GET /ws/generate` - WebSocket streaming\r\n\r\n### CLI Commands\r\n```bash\r\nshimmy serve                    # Start server (auto port allocation)\r\nshimmy serve --bind 127.0.0.1:8080  # Manual port binding\r\nshimmy list                     # Show available models  \r\nshimmy discover                 # Refresh model discovery\r\nshimmy generate --name X --prompt \"Hi\"  # Test generation\r\nshimmy probe model-name         # Verify model loads\r\n```\r\n\r\n## Technical Architecture\r\n\r\n- **Rust + Tokio**: Memory-safe, async performance\r\n- **llama.cpp backend**: Industry-standard GGUF inference\r\n- **OpenAI API compatibility**: Drop-in replacement\r\n- **Dynamic port management**: Zero conflicts, auto-allocation\r\n- **Zero-config auto-discovery**: Just works™\r\n\r\n## Community \u0026 Support\r\n\r\n- **🐛 Bug Reports**: [GitHub Issues](https://github.com/Michael-A-Kuykendall/shimmy/issues)\r\n- **💬 Discussions**: [GitHub Discussions](https://github.com/Michael-A-Kuykendall/shimmy/discussions)\r\n- **📖 Documentation**: [docs/](docs/)\r\n- **💝 Sponsorship**: [GitHub Sponsors](https://github.com/sponsors/Michael-A-Kuykendall)\r\n\r\n### Sponsors\r\n\r\nSee our amazing [sponsors](SPONSORS.md) who make Shimmy possible! 🙏\r\n\r\n**Sponsorship Tiers:**\r\n- **$5/month**: Coffee tier - My eternal gratitude + sponsor badge\r\n- **$25/month**: Bug prioritizer - Priority support + name in SPONSORS.md  \r\n- **$100/month**: Corporate backer - Logo on README + monthly office hours\r\n- **$500/month**: Infrastructure partner - Direct support + roadmap input\r\n\r\n**Companies**: Need invoicing? Email [michaelallenkuykendall@gmail.com](mailto:michaelallenkuykendall@gmail.com)\r\n\r\n## Quality \u0026 Reliability\r\n\r\nShimmy maintains high code quality through comprehensive testing:\r\n\r\n- **Comprehensive test suite** with property-based testing\r\n- **Automated CI/CD pipeline** with quality gates\r\n- **Runtime invariant checking** for critical operations\r\n- **Cross-platform compatibility testing**\r\n\r\nSee our [testing approach](docs/ppt-invariant-testing.md) for technical details.\r\n\r\n---\r\n\r\n## License \u0026 Philosophy\r\n\r\nMIT License - forever and always.\r\n\r\n**Philosophy**: Infrastructure should be invisible. Shimmy is infrastructure.\r\n\r\n**Testing Philosophy**: Reliability through comprehensive validation and property-based testing.\r\n\r\n---\r\n\r\n**Forever maintainer**: Michael A. Kuykendall  \r\n**Promise**: This will never become a paid product  \r\n**Mission**: Making local AI development frictionless\r\n\r\n*\"The best code is code you don't have to think about.\"*  \r\n*\"The best tests are properties you can't break.\"*","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmichael-a-kuykendall%2Fshimmy","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmichael-a-kuykendall%2Fshimmy","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmichael-a-kuykendall%2Fshimmy/lists"}