{"id":29022893,"url":"https://github.com/cyphrriot/npglue","last_synced_at":"2025-07-17T04:32:17.365Z","repository":{"id":300661347,"uuid":"1006358066","full_name":"CyphrRiot/npglue","owner":"CyphrRiot","description":"Intel NPU Support for Goose, Zed, and much more!","archived":false,"fork":false,"pushed_at":"2025-06-23T00:29:19.000Z","size":152,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-06-23T00:32:06.037Z","etag":null,"topics":["ai","goose","npu","ollama","python3","tensorflow","zed"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/CyphrRiot.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-06-22T04:48:09.000Z","updated_at":"2025-06-23T00:29:22.000Z","dependencies_parsed_at":"2025-06-23T00:42:20.597Z","dependency_job_id":null,"html_url":"https://github.com/CyphrRiot/npglue","commit_stats":null,"previous_names":["cyphrriot/npglue"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/CyphrRiot/npglue","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CyphrRiot%2Fnpglue","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CyphrRiot%2Fnpglue/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CyphrRiot%2Fnpglue/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CyphrRiot%2Fnpglue/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/CyphrRiot","download_url":"https://codeload.github.com/CyphrRiot/npglue/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CyphrRiot%2Fnpglue/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":261990350,"owners_count":23241187,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","goose","npu","ollama","python3","tensorflow","zed"],"created_at":"2025-06-26T03:04:05.910Z","updated_at":"2025-06-26T03:04:06.744Z","avatar_url":"https://github.com/CyphrRiot.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ɴᴘɢʟᴜᴇ - Intel NPU Ollama Replacement!\n\n**ɴᴘɢʟᴜᴇ** provides a complete setup for running **multiple AI models** locally using OpenVINO for AI-assisted coding and development with **direct, quality answers**.\n\n![I am a Pickle](Image/pickle_rick.png)\n\n## 🚀 **Quick Start**\n\n```bash\ngit clone https://github.com/CyphrRiot/npglue.git\ncd npglue\n./install\n```\n\nThe installer will ask you to choose your model from **8 options**:\n\n### **OpenVINO Pre-Optimized Models (fastest setup):**\n- **Qwen3-8B-INT8** (~6-8GB) - Best quality for complex tasks\n- **Qwen3-0.6B-FP16** (~1-2GB) - Fast and lightweight  \n- **OpenLlama-7B-INT4** (~4-5GB) - Great balance for coding\n- **OpenLlama-3B-INT4** (~2-3GB) - Lightweight with good performance\n\n### **Community Pre-Optimized Models:**\n- **Llama-3.1-8B-INT4** (~5-6GB) - Latest Llama with excellent coding abilities\n\n### **Convert-on-Install Models:**\n- **Phi-3-Mini-4K** (~4GB) - Microsoft model optimized for NPU\n- **DeepSeek-Coder-6.7B** (~6-7GB) - Specialized coding model, excellent for development  \n- **DeepSeek-Coder-1.3B** (~2GB) - Lightweight coding specialist\n\n## ✅ **What You Get**\n\n- **Multiple AI Models**: 8 models to choose from - Qwen3, Llama, Phi-3, DeepSeek coding specialists\n- **Model Choice**: Pick based on your needs - quality vs speed vs coding specialization\n- **Easy Model Switching**: Use `./switch_model.sh` to change models anytime (no reinstall needed!)  \n- **OpenVINO Optimized**: Fast inference optimized for Intel NPU/GPU hardware\n- **20-30+ tokens/sec**: Fast local inference with memory efficiency\n- **Performance Display**: Every response shows completion time and token rate\n- **Direct Answers**: No rambling - get concise, actionable responses\n- **Zed Compatible**: Works as Ollama provider (no API key hassles!)\n- **Full Ollama API**: Complete compatibility with Ollama ecosystem\n- **Dual API Support**: Both OpenAI and Ollama compatible endpoints\n- **Goose Ready**: Drop-in replacement for OpenAI API\n\n## 🔥 **Ollama GPU vs NPU Performance**\n\n### **Why NPGlue + NPU Beats Traditional GPU Solutions**\n\nMost AI inference solutions (like Ollama) rely on **traditional GPUs or CPUs**, but NPGlue leverages **cutting-edge NPU hardware** that provides significant advantages:\n\n#### **🎯 Performance Comparison:**\n\n| Setup | Hardware Used | Token Speed | Memory Efficiency | Power Usage |\n|-------|---------------|-------------|-------------------|-------------|\n| **Ollama (CPU)** | CPU cores only | 2-8 tok/s | High RAM usage | High power |\n| **Ollama (GPU)** | NVIDIA/AMD GPU | 15-30 tok/s | VRAM limited | Very high power |\n| **NPGlue (NPU)** | Intel/AMD NPU | **20-60 tok/s** | **Optimized** | **Low power** |\n\n#### **🚀 NPU Advantages:**\n\n**1. Purpose-Built for AI:**\n```bash\nTraditional GPU: Designed for graphics, adapted for AI\nNPU: Purpose-built neural processing unit for AI inference\nResult: 2-3x better performance per watt\n```\n\n**2. Memory Efficiency:**\n```bash\nGPU: Requires loading entire model into VRAM (8-24GB limits)\nNPU: Optimized memory access patterns, works with system RAM\nResult: Can run larger models with less memory\n```\n\n**3. Power Efficiency:**\n```bash\nGPU: 150-300W+ power consumption\nNPU: 5-15W power consumption  \nResult: 10-20x more power efficient\n```\n\n**4. Parallel Processing:**\n```bash\nCPU + GPU + NPU: All three can work together\nTraditional: Usually either CPU OR GPU\nResult: Better overall system performance\n```\n\n#### **🔍 Real-World Example:**\n\n**Your Intel Core Ultra 7 256V System:**\n```bash\n# Ollama (CPU-only, no NPU support)\nollama run qwen2.5:7b    # 2-5 tokens/sec, 100% CPU usage\n\n# NPGlue (NPU-accelerated)  \n./start_server.sh        # 20-30 tokens/sec, \u003c20% CPU usage\ncurl -X POST http://localhost:11434/v1/chat/completions \\\n  -d '{\"model\":\"qwen3\",\"messages\":[{\"role\":\"user\",\"content\":\"Hello\"}]}'\n```\n\n**AMD Ryzen AI Max+ 395 System:**\n```bash\n# Ollama (powerful CPU, but still no NPU)\nollama run qwen2.5:7b    # 8-15 tokens/sec\n\n# NPGlue (NPU-accelerated)\n./start_server.sh        # 40-60 tokens/sec\n# Plus can run 70B models that won't fit in GPU VRAM\n```\n\n#### **🎯 When to Choose Each:**\n\n**Choose Ollama if:**\n- ✅ You have a powerful NVIDIA GPU (3080+)\n- ✅ You want the largest model ecosystem  \n- ✅ You don't have NPU hardware\n- ✅ You need specific model formats (GGUF variety)\n\n**Choose NPGlue if:**\n- ✅ You have Intel Core Ultra or AMD Ryzen AI processors (**NPU available**)\n- ✅ You want maximum performance per watt\n- ✅ You prefer purpose-built AI acceleration\n- ✅ You want cutting-edge 2024+ hardware utilization\n- ✅ You need efficient performance on laptops\n\n#### **💡 Hardware Compatibility:**\n\n**NPU Support (NPGlue Advantage):**\n```bash\n✅ Intel Core Ultra (12th gen+)   - Intel NPU\n✅ AMD Ryzen AI (8000 series+)    - AMD XDNA NPU  \n✅ Qualcomm Snapdragon X Elite    - Hexagon NPU\n❌ Older Intel/AMD processors     - No NPU\n```\n\n**GPU Support (Ollama Advantage):**\n```bash\n✅ NVIDIA RTX 20/30/40 series    - CUDA acceleration\n✅ AMD RX 6000/7000 series       - ROCm acceleration  \n✅ Apple M1/M2/M3                - Metal acceleration\n❌ Intel integrated graphics     - Limited support\n```\n\n#### **🔥 The Future is NPU:**\n\nNPGlue positions you at the **forefront of AI hardware evolution**:\n\n- **2024**: NPUs becoming standard in new processors\n- **2025**: Expected 3-5x NPU performance improvements  \n- **2026+**: NPU-first AI software ecosystem\n\n**You're not just running AI faster today - you're using tomorrow's standard technology!**\n\n---\n\n**💡 Bottom Line:** If you have NPU hardware, NPGlue gives you **hardware acceleration that Ollama simply cannot access**, making it the superior choice for performance, efficiency, and future-proofing.\n\n## 🔧 **Requirements**\n\n- **OS**: Linux (Arch/CachyOS recommended)\n- **Memory**: 2GB+ RAM (for 0.6B model) or 8GB+ RAM (for 8B model)\n- **Storage**: 10-15GB free space\n- **CPU**: Intel preferred (excellent OpenVINO optimization)\n- **Shell**: Compatible with bash, zsh, and fish\n- **Hardware acceleration**:\n    - **Best**: Intel NPU (12th gen+ processors) - 20-30 tokens/sec\n    - **Good**: Intel integrated GPU - 5-10 tokens/sec\n    - **Basic**: Any CPU - 2-5 tokens/sec (slower but functional)\n\n## 📊 **Performance Monitoring**\n\nɴᴘɢʟᴜᴇ automatically displays performance metrics with every response:\n\n```\nWhat is the capital of France?\n\nThe capital of France is Paris.\n\n*Completed in 0.85 seconds at 23.2 tokens/sec*\n```\n\n**Benefits:**\n\n- **Real-time feedback** on AI response speed\n- **Performance monitoring** under different loads\n- **Model comparison** when testing different configurations\n- **System optimization** insights for tuning\n\nThis helps you:\n\n- Monitor system performance\n- Compare model variants (8B vs 0.6B)\n- Identify when your system needs optimization\n- Debug slow response issues\n\n**Token Limits:**\n\n- **Respects user preferences** - Request up to 4096 tokens\n- **No artificial caps** - Let the model complete naturally\n- **Smart defaults** - 200 tokens if not specified\n- **Memory aware** - Monitors available RAM during generation\n\n## 🛠️ **Performance Optimization**\n\nɴᴘɢʟᴜᴇ includes built-in tools to diagnose and optimize performance:\n\n### **Model Switching**\n\n```bash\n./switch_model.sh\n```\n\nEasily switch between models based on your needs:\n\n- **8B model**: Maximum quality for complex tasks (needs 8GB+ RAM)\n- **0.6B model**: Speed and efficiency for quick responses (needs 2GB+ RAM)\n\n**Tip**: If you're getting slow performance (under 15 tok/sec), run the diagnostics tool to identify memory pressure or other issues.\n\n### **CPU Performance Management**\n\n```bash\n# Manual CPU optimization\n./boost_cpu.sh           # Set CPU to performance mode\n\n# Manual CPU restoration\n./restore_cpu.sh         # Restore power-saving mode\n\n# Automatic management (recommended)\n./start_server.sh        # Auto-saves/restores CPU settings\n```\n\n**Note**: `start_server.sh` automatically saves your CPU governor settings and restores them when you press `Ctrl+C` or exit the server.\n\n## 📊 **Performance Expectations**\n\n| Model                     | Size   | Memory   | Speed (NPU) | Speed (iGPU) | Speed (CPU) | Best For                              |\n| ------------------------- | ------ | -------- | ----------- | ------------ | ----------- | ------------------------------------- |\n| **Qwen3-8B-INT8**         | ~6-8GB | 8GB+ RAM | 20-30 tok/s | 5-10 tok/s   | 2-5 tok/s   | Complex tasks, detailed explanations |\n| **Qwen3-0.6B-FP16**       | ~1-2GB | 2GB+ RAM | 25-40 tok/s | 8-15 tok/s   | 4-8 tok/s   | Quick answers, simple tasks           |\n| **OpenLlama-7B-INT4**     | ~4-5GB | 6GB+ RAM | 22-35 tok/s | 6-12 tok/s   | 3-6 tok/s   | Balanced coding and general tasks     |\n| **OpenLlama-3B-INT4**     | ~2-3GB | 4GB+ RAM | 30-45 tok/s | 10-18 tok/s  | 5-9 tok/s   | Fast responses, lightweight           |\n| **Llama-3.1-8B-INT4**     | ~5-6GB | 8GB+ RAM | 20-30 tok/s | 5-10 tok/s   | 2-5 tok/s   | Latest Llama, excellent coding        |\n| **Phi-3-Mini-4K**         | ~4GB   | 6GB+ RAM | 25-35 tok/s | 7-14 tok/s   | 3-7 tok/s   | NPU-optimized, Microsoft quality      |\n| **DeepSeek-Coder-6.7B**   | ~6-7GB | 8GB+ RAM | 18-28 tok/s | 4-9 tok/s    | 2-4 tok/s   | Best for coding, development tasks    |\n| **DeepSeek-Coder-1.3B**   | ~2GB   | 3GB+ RAM | 35-50 tok/s | 12-20 tok/s  | 6-10 tok/s  | Fast coding assistant, lightweight    |\n\n## 🛠️ **What the Installer Does**\n\n### **System Setup:**\n\n- ✅ Checks system requirements (RAM, disk space)\n- ✅ Installs system dependencies (Python, OpenVINO drivers, etc.)\n- ✅ Creates clean Python virtual environment\n- ✅ Installs **CPU-only** AI packages (OpenVINO 2024.x, transformers, PyTorch-CPU)\n\n### **Model Setup:**\n\n- ✅ **Interactive model choice**: Pick Qwen3-8B-INT8 or Qwen3-0.6B-FP16\n- ✅ Downloads your chosen optimized OpenVINO model\n- ✅ Memory-safe verification (no crashes during setup)\n- ✅ CPU performance optimization\n\n### **Configuration Instructions:**\n\n- ✅ **Safe Goose setup**: Checks for existing config, won't overwrite\n- ✅ **Zed integration**: Exact settings for assistant configuration\n- ✅ **Testing steps**: How to verify everything works properly\n\n## 🦆 **Goose Integration (Safe)**\n\nThe installer provides **safe configuration** that won't overwrite existing settings:\n\n**If you DON'T have Goose configured:**\n\n```bash\nmkdir -p ~/.config/goose\ncp goose_config_example.yaml ~/.config/goose/config.yaml\n# No API key needed! Uses Ollama provider which is simpler.\n```\n\n**If you HAVE existing Goose config, just add:**\n\n```yaml\nGOOSE_PROVIDER: ollama\nGOOSE_MODEL: qwen3\nOLLAMA_HOST: http://localhost:11434\n```\n\n**Why Ollama provider?** ɴᴘɢʟᴜᴇ supports both OpenAI and Ollama APIs, but Goose's Ollama provider doesn't require API key setup - much simpler!\n\n## ⚡ **Zed Integration (WORKING!)**\n\n**ɴᴘɢʟᴜᴇi works as an Ollama provider** (no API key hassles!):\n\n```json\n{\n    \"language_models\": {\n        \"ollama\": {\n            \"api_url\": \"http://localhost:11434\",\n            \"available_models\": [\n                {\n                    \"name\": \"qwen3\",\n                    \"display_name\": \"Qwen3 Local\",\n                    \"max_tokens\": 4096,\n                    \"supports_tools\": true\n                }\n            ]\n        }\n    },\n    \"agent\": {\n        \"default_model\": {\n            \"provider\": \"ollama\",\n            \"model\": \"qwen3\"\n        }\n    }\n}\n```\n\n**Why this works:** Zed's OpenAI provider is finicky about API keys, but the Ollama provider \"just works\"!\n\n## 🧪 **Testing Your Installation**\n\nAfter running `./install`, test with:\n\n```bash\n# Start the server\n./start_server.sh\n\n# Test health\ncurl http://localhost:11434/health\n\n# Test Ollama API (for Zed)\ncurl http://localhost:11434/api/tags\n\n# Test OpenAI API (for Goose)\ncurl http://localhost:11434/v1/models\n\n# Run full model test\n```\n\n## 🔌 **API Endpoints**\n\nɴᴘɢʟᴜᴇ provides **complete API compatibility** with both OpenAI and Ollama:\n\n**OpenAI API** (for Goose):\n\n- `GET /v1/models` - List models\n- `POST /v1/chat/completions` - Chat completions\n- `GET /health` - Health check\n\n**Ollama API** (for Zed):\n\n- `GET /api/tags` - List models\n- `POST /api/chat` - Chat completions\n- `POST /api/generate` - Text generation\n- `POST /api/show` - Model details\n- `GET /api/version` - Version info\n- `POST /api/pull` - Model management (returns success for local models)\n\n**Utilities:**\n\n- `GET /models` - System information\n- `POST /unload` - Unload model from memory\n\n## 🌍 **Works With**\n\n- **Goose**: AI development assistant\n- **Zed**: Modern code editor\n- **Cursor**: AI-powered IDE\n- **Continue.dev**: VS Code extension\n- **Any OpenAI-compatible client**\n\n## 📁 **Project Structure**\n\n```\nnpglue/\n├── install                    # 🌟 Beautiful one-command installer\n├── start_server.sh            # Start the AI server (auto CPU cleanup on exit)\n├── server_production.py       # FastAPI server with dual API compatibility\n├── boost_cpu.sh               # CPU performance optimization\n├── restore_cpu.sh             # 🔄 Restore CPU to power-saving mode\n├── switch_model.sh            # 🔄 Easy model switching utility\n├── goose_config_example.yaml  # Safe Goose configuration template\n├── README.md                  # This documentation\n├── LICENSE                    # License file\n└── models/                    # Downloaded models (created by installer)\n    ├── qwen3-8b-int8/         # High quality model (8GB)\n    └── qwen3-0.6b-fp16/       # Fast model (1-2GB)\n```\n\n## 🎯 **Why Choose ɴᴘɢʟᴜᴇ?**\n\n- **One Command Setup**: `./install` does everything beautifully\n- **Model Choice**: Choose between quality (8B) or speed (0.6B)\n- **Memory Safe**: Won't crash during installation or use\n- **Configuration Safe**: Won't overwrite your existing tool settings\n- **Expert Optimized**: Uses official OpenVINO optimized models\n- **Direct Answers**: No rambling - designed for practical Q\u0026A\n- **Clear Instructions**: Tells you exactly what to do next\n- **Local Privacy**: No data sent to external APIs\n- **Fast Performance**: Optimized for Intel hardware\n- **Production Ready**: Proper error handling and monitoring\n\n## 🔧 **Advanced Usage**\n\n### **API Endpoints:**\n\n- **Chat**: `http://localhost:11434/v1/chat/completions` (OpenAI compatible)\n- **Health**: `http://localhost:11434/health`\n- **Docs**: `http://localhost:11434/docs`\n\n### **Environment Control:**\n\n```bash\n# Activate environment manually\nsource npglue-env/bin/activate\n\n# Check available devices\npython -c \"import openvino; print(openvino.Core().available_devices)\"\n```\n\n## 🚀 **Recent Improvements**\n\n- ✅ **NPU vs GPU Comparison**: Detailed analysis of why NPGlue + NPU beats traditional GPU solutions\n- ✅ **8 Model Choices**: Added OpenLlama, Phi-3, DeepSeek, and Llama-3.1 models to installer\n- ✅ **Enhanced Model Switching**: Easy utility to switch between models (`switch_model.sh`)\n- ✅ **Optional CPU Performance**: Installer now asks before enabling performance mode (no automatic changes)\n- ✅ **Robust Dependencies**: Better handling of protobuf, sentencepiece, and model-specific requirements\n- ✅ **Smart Chat Templates**: Automatic handling for different model families (Qwen, Phi-3, DeepSeek)\n- ✅ **Complete Ollama API**: Added `/api/show`, `/api/version`, `/api/pull` endpoints (no more 404s!)\n- ✅ **Memory Optimization**: Automatic detection and fixes for memory pressure issues\n- ✅ **Flexible Token Limits**: Respects user preferences up to 4096 tokens (no more artificial caps!)\n- ✅ **Performance Display**: All responses now show \"Completed in X.XX seconds at X.X tokens/sec\"\n- ✅ **CPU-Only Install**: No NVIDIA dependencies on Intel systems\n- ✅ **Dual API Support**: Both OpenAI AND Ollama compatible endpoints\n- ✅ **Zed Integration Fixed**: Works as Ollama provider (no API key issues!)\n- ✅ **Safe configuration**: Protects existing Goose/Zed settings\n- ✅ **Simplified installer**: One beautiful command does everything\n- ✅ **Expert models**: Official OpenVINO optimized versions\n\n---\n\n**ɴᴘɢʟᴜᴇ: One command to local AI coding bliss!** 🚀\n\n_Get the power of Qwen3's direct, practical responses running locally on your machine in minutes._\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcyphrriot%2Fnpglue","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcyphrriot%2Fnpglue","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcyphrriot%2Fnpglue/lists"}