{"id":31505663,"url":"https://github.com/liebemama/repo-fastapi","last_synced_at":"2026-04-05T21:01:50.924Z","repository":{"id":314661177,"uuid":"1056322979","full_name":"liebemama/repo-fastapi","owner":"liebemama","description":"GPU-ready FastAPI AI inference server with plugin system, supporting CUDA, ROCm, CPU, and macOS MPS.","archived":false,"fork":false,"pushed_at":"2025-09-13T21:25:06.000Z","size":79,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-09-13T23:31:36.332Z","etag":null,"topics":["ai-server","cuda","fastapi","gpu","inference","mps","plugins","pytorch","rocm"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/liebemama.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null},"funding":{"github":["TamerOnLine"]}},"created_at":"2025-09-13T21:01:34.000Z","updated_at":"2025-09-13T21:43:24.000Z","dependencies_parsed_at":"2025-09-13T23:31:39.068Z","dependency_job_id":"49960acb-074a-4352-964f-6c6a2ad02526","html_url":"https://github.com/liebemama/repo-fastapi","commit_stats":null,"previous_names":["liebemama/repo-fastapi"],"tags_count":null,"template":true,"template_full_name":"TamerOnLine/repo-fastapi","purl":"pkg:github/liebemama/repo-fastapi","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/liebemama%2Frepo-fastapi","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/liebemama%2Frepo-fastapi/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/liebemama%2Frepo-fastapi/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/liebemama%2Frepo-fastapi/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/liebemama","download_url":"https://codeload.github.com/liebemama/repo-fastapi/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/liebemama%2Frepo-fastapi/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":278063119,"owners_count":25923594,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-02T02:00:08.890Z","response_time":67,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai-server","cuda","fastapi","gpu","inference","mps","plugins","pytorch","rocm"],"created_at":"2025-10-02T20:09:42.619Z","updated_at":"2025-10-02T20:09:46.889Z","avatar_url":"https://github.com/liebemama.png","language":"Python","funding_links":["https://github.com/sponsors/TamerOnLine","https://paypal.me/tameronline"],"categories":[],"sub_categories":[],"readme":"# 🚀 NeuroServe — GPU-Ready FastAPI AI Server\n\n## 📊 Project Status\n\n| Category      | Badges |\n|---------------|--------|\n| **Languages** | ![Python](https://img.shields.io/badge/Python-3.12%2B-3776AB?logo=python\u0026logoColor=white) ![HTML5](https://img.shields.io/badge/HTML5-E34F26?logo=html5\u0026logoColor=white) ![CSS3](https://img.shields.io/badge/CSS3-1572B6?logo=css3\u0026logoColor=white) |\n| **Framework** | ![FastAPI](https://img.shields.io/badge/FastAPI-0.116.x-009688?logo=fastapi\u0026logoColor=white) |\n| **ML / GPU**  | ![PyTorch](https://img.shields.io/badge/PyTorch-2.6.x-EE4C2C?logo=pytorch\u0026logoColor=white) ![CUDA Ready](https://img.shields.io/badge/CUDA-Ready-76B900?logo=nvidia\u0026logoColor=white) |\n| **CI**        | [![Ubuntu CI](https://github.com/TamerOnLine/repo-fastapi/actions/workflows/ci-ubuntu.yml/badge.svg)](https://github.com/TamerOnLine/repo-fastapi/actions/workflows/ci-ubuntu.yml) [![Windows CI](https://github.com/TamerOnLine/repo-fastapi/actions/workflows/ci-windows.yml/badge.svg)](https://github.com/TamerOnLine/repo-fastapi/actions/workflows/ci-windows.yml) [![Windows GPU CI](https://github.com/TamerOnLine/repo-fastapi/actions/workflows/ci-gpu.yml/badge.svg)](https://github.com/TamerOnLine/repo-fastapi/actions/workflows/ci-gpu.yml) [![macOS CI](https://github.com/TamerOnLine/repo-fastapi/actions/workflows/ci-macos.yml/badge.svg)](https://github.com/TamerOnLine/repo-fastapi/actions/workflows/ci-macos.yml) |\n| **Code Style**| ![Ruff](https://img.shields.io/badge/style-Ruff-000?logo=ruff\u0026logoColor=white) |\n| **Tests**     | ![Tests](https://img.shields.io/badge/tests-passing-brightgreen) |\n| **Docs**      | [![Docs](https://img.shields.io/badge/docs-available-blue)](docs/API.md) |\n| **OS**        | ![Ubuntu](https://img.shields.io/badge/Ubuntu-E95420?logo=ubuntu\u0026logoColor=white) ![Windows](https://img.shields.io/badge/Windows-0078D6?logo=windows\u0026logoColor=white) ![macOS](https://img.shields.io/badge/macOS-000000?logo=apple\u0026logoColor=white) |\n| **Version**   | [![GitHub release](https://img.shields.io/github/v/release/TamerOnLine/repo-fastapi)](https://github.com/TamerOnLine/repo-fastapi/releases) |\n| **License**   | ![License](https://img.shields.io/badge/License-MIT-green) |\n| **Support**   | [![Sponsor](https://img.shields.io/badge/Sponsor-💖-pink)](https://paypal.me/tameronline) |\n| **GitHub**    | [![Stars](https://img.shields.io/github/stars/TamerOnLine/repo-fastapi?style=social)](https://github.com/TamerOnLine/repo-fastapi/stargazers) [![Forks](https://img.shields.io/github/forks/TamerOnLine/repo-fastapi?style=social)](https://github.co)\n\n\n---\n\n## 📖 Overview\n\n**NeuroServe** is an **AI Inference Server** built on **FastAPI**, designed to run seamlessly on **GPU (CUDA/ROCm)**, **CPU**, and **macOS MPS**.\nIt provides ready-to-use REST APIs, a modular **plugin system**, runtime utilities, and a consistent unified response format — making it the perfect foundation for AI-powered services.\n\n---\n\n## Quick Setup\n 🔧 Virtualenv quick guide: see **[docs/README_venv.md](docs/README_venv.md)**.\n\n---\n\n## 📚 API Documentation\nDetailed API reference and usage examples are available here:\n➡️ [API Documentation](docs/API.md)\n\n---\n\n## ✨ Key Features\n\n- 🌐 **REST APIs out-of-the-box** with Swagger UI (`/docs`) \u0026 ReDoc (`/redoc`).\n- ⚡ **PyTorch integration** with automatic device selection (`cuda`, `cpu`, `mps`, `rocm`).\n- 🔌 **Plugin system** to extend functionality with custom AI models or services.\n- 📊 **Runtime tools** for GPU info, warm-up routines, and environment inspection.\n- 🧠 **Built-in utilities** like a toy model and model size calculator.\n- 🧱 **Unified JSON responses** for predictable API behavior.\n- 🧪 **Cross-platform CI/CD** (Ubuntu, Windows, macOS, Self-hosted GPU).\n\n---\n\n## 📂 Project Structure\n\n```text\nrepo-fastapi/\n├─ app/                             # application package\n│  ├─ core/                         # settings \u0026 configuration\n│  │  └─ config.py                  # app settings (Pydantic v2)\n│  ├─ routes/                       # HTTP API routes\n│  ├─ plugins/                      # extensions / integrations\n│  ├─ workflows/                    # workflow definitions \u0026 orchestrators\n│  └─ templates/                    # Jinja templates (if used)\n├─ docs/                            # documentation \u0026 generated diagrams\n│  ├─ ARCHITECTURE.md               # main architecture report\n│  ├─ architecture.mmd              # Mermaid source (no fences)\n│  ├─ architecture.html             # browser-friendly diagram\n│  ├─ architecture.png              # exported PNG (if mmdc installed)\n│  ├─ runtime.mmd                   # runtime/infra diagram\n│  ├─ imports.mmd                   # Python import graph (if generated)\n│  ├─ endpoints.md                  # discovered API endpoints (if generated)\n│  └─ README_venv.md                # virtualenv quick guide\n├─ tools/                           # project tooling \u0026 scripts\n│  └─ build_workflows_index.py      # builds docs/workflows-overview.md\n├─ tests/                           # test suite\n│  └─ test_run.py                   # smoke test for app startup\n├─ gen_arch.py                      # architecture generator script\n├─ requirements.txt                 # runtime dependencies\n├─ requirements-dev.txt             # dev dependencies (ruff, pre-commit, pytest, ...)\n├─ .pre-commit-config.yaml          # pre-commit hooks configuration\n├─ README.md                        # project overview \u0026 usage\n└─ LICENSE                          # project license\n\n```\n\n---\n\n## 🏗️ Architecture\nFor a deeper look into the internal design, modules, and flow of the system, see:\n➡️ [Architecture Guide](docs/ARCHITECTURE.md)\n\n---\n\n## ⚙️ Installation\n\n### 1. Clone the repository\n```bash\ngit clone https://github.com/USERNAME/gpu-server.git\ncd gpu-server\n```\n\n### 2. Create a virtual environment\n```bash\npython -m venv .venv\n# Linux/macOS\nsource .venv/bin/activate\n# Windows\n.venv\\Scripts\\activate\n```\n\n### 3. Install dependencies\n```bash\npip install -r requirements.txt\n```\n\n### 4. (Optional) Auto-install PyTorch\n```bash\npython -m scripts.install_torch --gpu    # or --cpu / --rocm\n```\n\n---\n\n## 🚀 Running the Server\n\n```bash\nuvicorn app.main:app --reload --host 0.0.0.0 --port 8000\n```\n\nAvailable endpoints:\n- 🏠 **Home** → [http://localhost:8000/](http://localhost:8000/)\n- ❤️ **Health** → [http://localhost:8000/health](http://localhost:8000/health)\n- 📚 **Swagger UI** → [http://localhost:8000/docs](http://localhost:8000/docs)\n- 📘 **ReDoc** → [http://localhost:8000/redoc](http://localhost:8000/redoc)\n- 🧭 **Env Summary** → [http://localhost:8000/env](http://localhost:8000/env)\n- 🔌 **Plugins** → [http://localhost:8000/plugins](http://localhost:8000/plugins)\n\nQuick test:\n```bash\ncurl http://localhost:8000/health\n# {\"status\": \"ok\"}\n```\n\n---\n\n## 🔌 Plugin System\n\nEach plugin lives in `app/plugins/\u003cname\u003e/` and typically includes:\n\n```\nmanifest.json\nplugin.py        # Defines Plugin class inheriting AIPlugin\nREADME.md        # Documentation\n```\n\nAPI Endpoints:\n- `GET /plugins` — list all plugins with metadata.\n- `POST /plugins/{name}/{task}` — execute a task inside a plugin.\n\nExample:\n```python\nfrom app.plugins.base import AIPlugin\n\nclass Plugin(AIPlugin):\n    name = \"my_plugin\"\n    tasks = [\"infer\"]\n\n    def load(self):\n        # Load models/resources once\n        ...\n\n    def infer(self, payload: dict) -\u003e dict:\n        return {\"message\": \"ok\", \"payload\": payload}\n```\n\n---\n\n## Workflow System\nA lightweight orchestration layer to chain plugins into reproducible pipelines (steps → plugin + task + payload).\nAll endpoints are exposed under `/workflow`.\n\n- **Endpoints:** `GET /workflow/ping`, `GET /workflow/presets`, `POST /workflow/run`\n- **System Guide (EN):** [app/workflows/README.md](app/workflows/README.md)\n- **Workflows Index:** [docs/workflows-overview.md](docs/workflows-overview.md)\n\n---\n\n## 🔄 Available Workflows\n\nA full list of available workflows with their versions, tags, and step counts is maintained in the **Workflows Index**.\n\n➡️ [View Workflows Index](docs/workflows-overview.md)\n\n---\n\n## 🧩 Available Plugins\n\nA full list of available plugins with their providers, tasks, and source files is maintained in the **Plugins Index**.\n\n➡️ [View Plugins Index](docs/plugins-overview.md)\n\n---\n\n## 🧪 Development\n\nInstall dev dependencies:\n```bash\npip install -r requirements-dev.txt\npre-commit install\n```\n\nRun tests:\n```bash\npytest\n```\n\nRuff (lint + format check) runs automatically via pre-commit hooks.\n\n---\n\n## 🧹 Code Style\n\nWe enforce a clean and consistent code style using **Ruff** (linter, import sorter, and formatter).\nFor full details on configuration, commands, helper scripts, and CI integration, see:\n\n➡️ [Code Style \u0026 Linting Guide](docs/CODE_STYLE_GUIDE.md)\n\n---\n\n## 📦 Model Management\n\nDownload models in advance:\n```bash\npython -m scripts.prefetch_models\n```\n\nModels are cached in `models_cache/` (see `docs/LICENSES.md` for licenses).\n\n---\n\n## 🏭 Deployment Notes\n\n- Use `uvicorn`/`hypercorn` behind a reverse proxy (e.g., Nginx).\n- Configure environment with `APP_*` variables instead of hardcoding.\n- Enable HTTPS and configure CORS carefully in production.\n\n---\n\n## 📝 Changelog\nA complete history of changes and improvements:\n➡️ [CHANGELOG](docs/CHANGELOG.md)\n\n## 📦 Release Notes\nDetails about the initial release v0.1.0:\n➡️ [Release Notes v0.1.0](docs/RELEASE_NOTES_v0.1.0.md)\n\n---\n\n## 🗺️ Roadmap\n\n- [ ] Add `/cuda` endpoint → return detailed CUDA info.\n- [ ] Add `/warmup` endpoint for GPU readiness.\n- [ ] Provide a **plugin generator CLI**.\n- [ ] Implement API Key / JWT authentication.\n- [ ] Example plugins: translation, summarization, image classification.\n- [ ] Docker support for one-click deployment.\n- [ ] Benchmark suite for model inference speed.\n\n---\n\n## 🤝 Contributing\n\nContributions are welcome!\n- Open **Issues** for bugs or ideas.\n- Submit **Pull Requests** for improvements.\n- Follow style guidelines (Ruff + pre-commit).\n\n---\n\n## 📜 License\nLicensed under the **MIT License** — see [LICENSE](./LICENSE).\n\n### 📜 Model Licenses\nSome AI/ML models are licensed separately — see [Model Licenses](docs/LICENSES.md).\n\n---\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fliebemama%2Frepo-fastapi","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fliebemama%2Frepo-fastapi","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fliebemama%2Frepo-fastapi/lists"}