{"id":31642437,"url":"https://github.com/tameronline/neuroserve","last_synced_at":"2025-10-07T03:59:30.915Z","repository":{"id":313723896,"uuid":"1052300669","full_name":"TamerOnLine/NeuroServe","owner":"TamerOnLine","description":null,"archived":false,"fork":false,"pushed_at":"2025-09-08T02:53:30.000Z","size":30,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-09-08T04:28:03.767Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/TamerOnLine.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-09-07T20:07:54.000Z","updated_at":"2025-09-08T02:53:33.000Z","dependencies_parsed_at":"2025-09-08T04:28:05.140Z","dependency_job_id":"af3ea5fa-7b04-4597-ae21-8327cac30777","html_url":"https://github.com/TamerOnLine/NeuroServe","commit_stats":null,"previous_names":["tameronline/neuroserve"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/TamerOnLine/NeuroServe","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TamerOnLine%2FNeuroServe","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TamerOnLine%2FNeuroServe/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TamerOnLine%2FNeuroServe/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TamerOnLine%2FNeuroServe/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/TamerOnLine","download_url":"https://codeload.github.com/TamerOnLine/NeuroServe/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TamerOnLine%2FNeuroServe/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":278717438,"owners_count":26033542,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-07T02:00:06.786Z","response_time":59,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-10-07T03:59:27.557Z","updated_at":"2025-10-07T03:59:30.910Z","avatar_url":"https://github.com/TamerOnLine.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# 🧠 NeuroServe – GPU/CPU FastAPI AI Server\n\n[![Python](https://img.shields.io/badge/python-3.12+-blue.svg)](https://www.python.org)\n[![FastAPI](https://img.shields.io/badge/FastAPI-Framework-green)](https://fastapi.tiangolo.com)\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)\n[![CI (Linux)](https://github.com/TamerOnLine/NeuroServe/actions/workflows/ci-linux.yml/badge.svg)](https://github.com/TamerOnLine/NeuroServe/actions/workflows/ci-linux.yml)\n[![CI (macOS)](https://github.com/TamerOnLine/NeuroServe/actions/workflows/ci-macos.yml/badge.svg)](https://github.com/TamerOnLine/NeuroServe/actions/workflows/ci-macos.yml)\n[![CI (Windows)](https://github.com/TamerOnLine/NeuroServe/actions/workflows/ci-windows.yml/badge.svg)](https://github.com/TamerOnLine/NeuroServe/actions/workflows/ci-windows.yml)\n[![GPU CI (Windows self-hosted)](https://github.com/TamerOnLine/NeuroServe/actions/workflows/gpu-ci.yml/badge.svg)](https://github.com/TamerOnLine/NeuroServe/actions/workflows/gpu-ci.yml)\n\n\nA lightweight **REST API server** powered by **FastAPI** \u0026 **PyTorch**.  \nIt runs seamlessly on **GPU (CUDA)** if available, and safely falls back to **CPU**.  \n\n🚀 Includes a demo model, performance probes, a **web control panel**, and an extendable **Plugin System** for real AI models.\n\n---\n\n## ✨ Features\n- ✅ Auto-selects **GPU or CPU** at runtime (`DEVICE` in `.env`).\n- ✅ Clean **FastAPI** endpoints with Swagger UI (`/docs`) and ReDoc (`/redoc`).\n- ✅ Interactive **Control Panel** (`/ui`) \u0026 **Model Size Calculator** (`/tools/model-size`).\n- ✅ Example **TinyNet** model + benchmarks (`/matmul`, `/infer`).\n- ✅ Extensible **Plugin System** with ready-to-use AI models.\n- ✅ Works on **Windows/Linux/macOS** (CPU) \u0026 **CUDA** (NVIDIA GPUs).\n\n---\n\n## 📂 Project Structure\n```\napp/\n  main.py           # FastAPI app \u0026 endpoints\n  runtime.py        # device selection, CUDA info, warmup\n  toy_model.py      # TinyNet demo model\n  plugins/          # Modular plugins (bart, clip, resnet18, ner, etc.)\n  templates/        # HTML templates for UI\nscripts/\n  install_torch.py  # Auto-installs correct PyTorch (CPU/GPU)\n  test_api.py       # Quick client to test endpoints\nrequirements.txt\nREADME.md\n```\n\n---\n\n## ⚡ Quickstart\n\n### 1. Setup Environment\n**Windows (PowerShell)**:\n```powershell\npy -3.12 -m venv .venv\n.\\.venv\\Scripts\\Activate.ps1\n```\n\n**Linux/macOS**:\n```bash\npython3.12 -m venv .venv\nsource .venv/bin/activate\n```\n\n### 2. Install Dependencies\n```bash\npip install -r requirements.txt\npython -m scripts.install_torch\n```\n\n### 3. Run the Server\n```bash\nuvicorn app.main:app --reload --host 0.0.0.0 --port 8000\n```\n\nOpen in browser:\n- **Swagger UI** → http://localhost:8000/docs  \n- **Control Panel** → http://localhost:8000/ui  \n- **Plugin Console** → http://localhost:8000/plugins/ui  \n- **Infer Client** → http://localhost:8000/infer-client  \n\n---\n\n## ⚙️ Configuration (`.env`)\n```ini\n# Prefer the first CUDA GPU if available\nDEVICE=cuda:0\n\n# Force CPU mode (override)\n# DEVICE=cpu\n\n# Warmup matrix size (for benchmarks)\nWARMUP_MATMUL_SIZE=1024\n```\n\n---\n\n## 🔌 API Endpoints\n\n| Method | Path                | Description                      |\n|--------|---------------------|----------------------------------|\n| GET    | `/health`           | Health check                     |\n| GET    | `/cuda`             | CUDA / device info               |\n| GET    | `/env`              | Environment summary              |\n| GET    | `/env/full`         | Full environment + GPU list      |\n| GET    | `/env/system`       | OS/CPU/RAM system info           |\n| POST   | `/matmul`           | Matrix multiply benchmark        |\n| POST   | `/infer`            | TinyNet inference                |\n| POST   | `/inference`        | Generic plugin inference         |\n| GET    | `/plugins`          | List available plugins           |\n| GET    | `/ui`               | Interactive control panel        |\n| GET    | `/tools/model-size` | Model size calculator (UI)       |\n\n---\n\n## 🧩 Plugins\nNeuroServe uses a **modular plugin system** under `app/plugins/`.\n\nAvailable plugins:\n- **bart** → Text summarization (`facebook/bart-large-cnn`)\n- **clip** → Text ↔ Image embeddings \u0026 similarity\n- **distilbert** → Sentiment classification\n- **resnet18** → Image classification (ImageNet)\n- **ner** → Named Entity Recognition\n- **tinyllama** → Lightweight text generation\n- **translator / translator_m2m** → Translation\n- **pdf_reader** → Extract text from PDFs\n- **dummy** → Ping test\n- **dichfoto_proxy** → Forwarding proxy\n\n🔧 **Build your own Plugin (5 min)**\n```python\nfrom app.plugins.base import AIPlugin\n\nclass Plugin(AIPlugin):\n    tasks = [\"hello\"]\n\n    def load(self):\n        print(\"[plugin] hello ready\")\n\n    def infer(self, payload):\n        name = payload.get(\"name\", \"world\")\n        return {\"task\": \"hello\", \"message\": f\"Hello, {name}!\"}\n```\nAdd it in `app/plugins/hello/` with `manifest.json`.\n\n---\n\n## 🖥️ Example Requests\n\n**Matrix Multiply**\n```bash\ncurl -X POST http://localhost:8000/matmul   -H \"Content-Type: application/json\"   -d '{\"n\": 2048}'\n```\n\n**BART Summarization**\n```bash\ncurl -X POST http://localhost:8000/inference   -H \"Content-Type: application/json\"   -d '{\"provider\":\"bart\",\"task\":\"summarize\",\"text\":\"Deep learning is a subfield...\"}'\n```\n\n**NER Entity Extraction**\n```bash\ncurl -X POST http://localhost:8000/inference   -H \"Content-Type: application/json\"   -d '{\"provider\":\"ner\",\"task\":\"extract-entities\",\"text\":\"Barack Obama was born in Hawaii.\"}'\n```\n\n---\n\n## 🛠️ Troubleshooting\n- **Torch import error** → Ensure Python 3.12 + rerun `python -m scripts.install_torch`.\n- **No GPU detected** → Falls back to CPU. Force with `DEVICE=cpu`.\n- **CUDA mismatch** → Reinstall torch with matching CUDA runtime.\n- **Out of Memory (OOM)** → Reduce `max_length` or use CPU mode.\n- **Windows Execution Policy** →  \n  ```powershell\n  Set-ExecutionPolicy -Scope CurrentUser RemoteSigned\n  ```\n\n---\n\n## 📍 Roadmap\n- [ ] File upload endpoints (images/text).\n- [ ] Docker images (CPU \u0026 GPU).\n- [ ] Extended plugin demo UI.\n- [ ] CI/CD with pre-commit hooks \u0026 GitHub Actions.\n- [x] Real AI plugins (BART, CLIP, ResNet, DistilBERT, NER, etc.).\n\n---\n\n## 📜 License\nMIT © 2025 [TamerOnLine](https://github.com/TamerOnLine)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftameronline%2Fneuroserve","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftameronline%2Fneuroserve","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftameronline%2Fneuroserve/lists"}