{"id":35143123,"url":"https://github.com/ramborogers/cyber-inference","last_synced_at":"2026-05-24T00:06:19.231Z","repository":{"id":329831199,"uuid":"1120687074","full_name":"RamboRogers/cyber-inference","owner":"RamboRogers","description":"Cyber-Inference is a web GUI management tool for running OpenAI-compatible inference servers. Built on llama.cpp, it provides automatic model management, dynamic resource allocation, and a beautiful cyberpunk-themed interface designed for edge deployment.","archived":false,"fork":false,"pushed_at":"2026-04-11T00:52:39.000Z","size":4648,"stargazers_count":11,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"master","last_synced_at":"2026-04-11T02:22:59.449Z","etag":null,"topics":["ai","ai-agents","inference-api","inference-engine","llamacpp","metal","nvidia"],"latest_commit_sha":null,"homepage":"https://matthewrogers.org","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/RamboRogers.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":"AGENTS.md","dco":null,"cla":null}},"created_at":"2025-12-21T18:28:36.000Z","updated_at":"2026-04-11T00:52:44.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/RamboRogers/cyber-inference","commit_stats":null,"previous_names":["ramborogers/cyber-inference"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/RamboRogers/cyber-inference","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RamboRogers%2Fcyber-inference","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RamboRogers%2Fcyber-inference/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RamboRogers%2Fcyber-inference/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RamboRogers%2Fcyber-inference/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/RamboRogers","download_url":"https://codeload.github.com/RamboRogers/cyber-inference/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RamboRogers%2Fcyber-inference/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31954738,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-18T00:39:45.007Z","status":"online","status_checked_at":"2026-04-18T02:00:07.018Z","response_time":103,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","ai-agents","inference-api","inference-engine","llamacpp","metal","nvidia"],"created_at":"2025-12-28T12:51:12.922Z","updated_at":"2026-05-24T00:06:19.217Z","avatar_url":"https://github.com/RamboRogers.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Cyber-Inference\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://img.shields.io/badge/Python-3.12+-00ff9f?style=for-the-badge\u0026logo=python\u0026logoColor=00ff9f\" alt=\"Python\"\u003e\n  \u003cimg src=\"https://img.shields.io/badge/License-GPLv3-00ff9f?style=for-the-badge\" alt=\"License\"\u003e\n  \u003cimg src=\"https://img.shields.io/badge/llama.cpp-Powered-00ff9f?style=for-the-badge\" alt=\"llama.cpp\"\u003e\n  \u003cimg src=\"https://img.shields.io/badge/Transformers-Powered-00ff9f?style=for-the-badge\" alt=\"Transformers\"\u003e\n  \u003cimg src=\"https://img.shields.io/badge/whisper.cpp-Powered-00ff9f?style=for-the-badge\" alt=\"whisper.cpp\"\u003e\n  \u003cimg src=\"https://img.shields.io/badge/NVIDIA-Containers-76B900?style=for-the-badge\u0026logo=nvidia\u0026logoColor=white\" alt=\"NVIDIA containers\"\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"cyber-inference.png\" alt=\"Cyber-Inference UI\"\u003e\n  \u003cstrong\u003eEdge inference server management with an OpenAI-compatible API\u003c/strong\u003e\n\u003c/p\u003e\n\nCyber-Inference is a web GUI and API server for running local inference engines behind OpenAI-compatible `/v1` endpoints. It supports:\n- `llama.cpp` for GGUF models\n- `transformers` for full HuggingFace model directories\n- `whisper.cpp` for transcription/translation\n\n## Features\n\n- OpenAI-compatible API (`/v1/chat/completions`, `/v1/completions`, `/v1/embeddings`, `/v1/audio/*`)\n- Model download + registration from HuggingFace, including split GGUF shard sets\n- Automatic MTP speculative decoding for detected GGUF models, with managed llama.cpp upgrade when needed\n- Automatic lazy loading and idle unloading\n- Web dashboard for model and resource management\n- Optional admin auth (JWT)\n- NVIDIA-only published container images for Linux AMD64 and Thor ARM64\n- Native local startup paths for macOS Apple Silicon and non-container development\n\n## Releases\n\n- GitHub Releases are the canonical release surface for Cyber-Inference.\n- `CHANGELOG.md` is the repo release log.\n- Release notes are generated from commits since the previous release plus the core-functions summary in this README.\n- Versioning is patch-by-default. Use `release:minor` or `feat:` to force a minor bump, and `release:major`, `BREAKING CHANGE`, or `type!:` to force a major bump.\n- Container images publish from the GitHub release event and receive immutable versioned tags such as `v0.2.0-linux-amd64` and `v0.2.0-thor-arm64`, floating platform tags, and a multi-arch `latest` tag.\n\n## Inference Engines\n\n| Engine | Model Format | Typical Hardware | Primary Use |\n| --- | --- | --- | --- |\n| `llama.cpp` | GGUF | CPU / Apple Metal / CUDA | Quantized local chat + embeddings |\n| `transformers` | HuggingFace directory (`config.json`, safetensors, tokenizer) | CPU / CUDA / MPS | Full HF model inference |\n| `whisper.cpp` | Whisper GGUF/bin | CPU / Apple Metal / CUDA | Speech transcription and translation |\n\n## Quick Start\n\n### One-shot startup\n\nYou have a NVIDIA Thor/DGX Spark ARM64 host and want to run Cyber-Inference.\n\u003e [!TIP] The latest llama.cpp is built natively on Thor and baked into the container image.\n```bash\ndocker pull ghcr.io/ramborogers/cyber-inference:latest\n\ndocker run -d --name cyber-inference \\\n  --runtime nvidia \\\n  -p 8337:8337 \\\n  -v \"$PWD/data:/app/data\" \\\n  -v \"$PWD/models:/app/models\" \\\n  ghcr.io/ramborogers/cyber-inference:latest\n```\nQuick ⚡️ Update:\n```bash\ndocker pull ghcr.io/ramborogers/cyber-inference:latest\ndocker rm -f cyber-inference\ndocker run -d --name cyber-inference \\\n  --runtime nvidia \\\n  -p 8337:8337 \\\n  -v \"$PWD/data:/app/data\" \\\n  -v \"$PWD/models:/app/models\" \\\n  ghcr.io/ramborogers/cyber-inference:latest\n\n```\n\n### Local development\n```bash\ngit clone https://github.com/ramborogers/cyber-inference.git\ncd cyber-inference\n./start.sh\n```\n\n`start.sh` will:\n1. Ensure `uv` is available.\n2. Validate Python 3.12+.\n3. Detect NVIDIA GPU/CUDA.\n4. Run `uv sync`.\n5. Verify CUDA-enabled PyTorch on NVIDIA machines.\n6. Start `cyber-inference serve` with auto-restart.\n\n### Manual setup\n\n```bash\nuv sync\nuv run cyber-inference init\nuv run cyber-inference serve --reload\n```\n\nOpen the UI at `http://localhost:8337`.\n\n## Model Download\n\n![download.png](download.png)\n\nUse the **Models** page in the UI or the CLI.\n\nCyber-Inference handles GGUF repositories that publish one model across multiple shard files such as\n`Model-00001-of-00003.gguf`, `Model-00002-of-00003.gguf`, and `Model-00003-of-00003.gguf`.\nThe downloader presents the shard set as one logical model choice, downloads any missing shards,\nskips complete shards on repeat runs, and registers one canonical model entry.\n\nMTP-capable speculative GGUF repositories are detected automatically. Repos such as\n`unsloth/Qwen3.6-27B-MTP-GGUF` and `unsloth/Qwen3.6-35B-A3B-MTP-GGUF` default to MTP text mode,\nprefer the balanced `UD-Q4_K_XL` quantization when present, and launch llama.cpp with\n`--parallel 1`, `--flash-attn on`, `--spec-type draft-mtp`, and `--spec-draft-n-max 2`.\nQwen3.6 MTP models also receive `--chat-template-kwargs '{\"preserve_thinking\":true}'`.\nIf a repo also publishes `mmproj` files, MTP takes priority and the projector is not downloaded\nor launched unless explicitly selected; disable MTP in the model settings to use vision/projector mode.\n\n## API Usage\n\n### Python (OpenAI SDK)\n\n```python\nfrom openai import OpenAI\n\nclient = OpenAI(base_url=\"http://localhost:8337/v1\", api_key=\"not-needed\")\n\nresp = client.chat.completions.create(\n    model=\"Qwen3-4B-Q4_K_M\",\n    messages=[{\"role\": \"user\", \"content\": \"hello\"}],\n)\nprint(resp.choices[0].message.content)\n```\n\n### cURL\n\n```bash\ncurl http://localhost:8337/v1/chat/completions \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n    \"model\": \"Qwen3-4B-Q4_K_M\",\n    \"messages\": [{\"role\": \"user\", \"content\": \"hello\"}]\n  }'\n```\n\n## Docker\n\nCyber-Inference publishes NVIDIA-only container images for the two supported deployment targets:\n\n| Target | Image |\n| --- | --- |\n| Multi-arch default | `ghcr.io/ramborogers/cyber-inference:latest` |\n| Linux AMD64 NVIDIA hosts | `ghcr.io/ramborogers/cyber-inference:linux-amd64` |\n| Thor / DGX Spark ARM64 NVIDIA hosts | `ghcr.io/ramborogers/cyber-inference:thor-arm64` |\n\nBoth images expect durable host directories mounted into the container:\n\n- `./data` → `/app/data` for the database and logs\n- `./models` → `/app/models` for downloaded model files\n\nThe Linux AMD64 image does not ship bundled native inference servers; on demand, Cyber-Inference\ninstalls compatible `llama-server` and `whisper-server` binaries into `/app/bin`, so the container\nneeds outbound network access on first boot and first transcription use. The Thor image bakes in the\ncurrent native CUDA `llama.cpp` build produced on `thor.lab` during the publish workflow and an\nisolated native CUDA `whisper.cpp` runtime exposed through `/app/bin/whisper-server`, then\nsmoke-tests those binaries with the NVIDIA runtime enabled after the image is built. `transformers`\nremains Python-managed in both images and relies on CUDA-capable PyTorch rather than a staged server\nbinary.\n\nDocker is **not** the recommended path for macOS Apple Silicon MPS. On macOS, use the native local\nstartup flow above so Metal/MPS support is available directly from the host.\n\n### Linux AMD64 NVIDIA\n\n```bash\nmkdir -p data models\n\ndocker pull ghcr.io/ramborogers/cyber-inference:linux-amd64\n\ndocker run -d --name cyber-inference \\\n  --gpus all \\\n  -p 8337:8337 \\\n  -v \"$PWD/data:/app/data\" \\\n  -v \"$PWD/models:/app/models\" \\\n  ghcr.io/ramborogers/cyber-inference:linux-amd64\n```\n\n### Thor / DGX Spark ARM64 NVIDIA\n\n```bash\nmkdir -p data models\n\ndocker pull ghcr.io/ramborogers/cyber-inference:latest\n\ndocker run -d --name cyber-inference \\\n  --runtime nvidia \\\n  -p 8337:8337 \\\n  -v \"$PWD/data:/app/data\" \\\n  -v \"$PWD/models:/app/models\" \\\n  ghcr.io/ramborogers/cyber-inference:latest\n```\n\n### Upgrade while preserving local state\n\nUse the same host `data` and `models` directories when replacing a container. Use `latest` for the\nnormal multi-arch release tag, or pick a platform tag explicitly (`linux-amd64` or `thor-arm64`):\n\n```bash\nTARGET_TAG=latest  # or linux-amd64 / thor-arm64\n\ndocker stop cyber-inference\ndocker rm cyber-inference\ndocker pull \"ghcr.io/ramborogers/cyber-inference:${TARGET_TAG}\"\n\ndocker run -d --name cyber-inference \\\n  --gpus all \\\n  -p 8337:8337 \\\n  -v \"$PWD/data:/app/data\" \\\n  -v \"$PWD/models:/app/models\" \\\n  \"ghcr.io/ramborogers/cyber-inference:${TARGET_TAG}\"\n```\n\nFor Thor hosts that require the NVIDIA runtime flag instead of `--gpus all`, replace that line with:\n\n```bash\n  --runtime nvidia \\\n```\n\n## Configuration\n\n![config.png](config.png)\n\nEnvironment variables use the `CYBER_INFERENCE_` prefix, or you can just use the UI.\n\n| Variable | Default | Description |\n| --- | --- | --- |\n| `CYBER_INFERENCE_HOST` | `0.0.0.0` | API bind host |\n| `CYBER_INFERENCE_PORT` | `8337` | API bind port |\n| `CYBER_INFERENCE_DATA_DIR` | `./data` | Database + logs directory |\n| `CYBER_INFERENCE_MODELS_DIR` | `./models` | Model storage directory |\n| `CYBER_INFERENCE_DEFAULT_CONTEXT_SIZE` | `8192` | Default context for llama.cpp |\n| `CYBER_INFERENCE_MAX_CONTEXT_SIZE` | `32768` | Max allowed context |\n| `CYBER_INFERENCE_MODEL_IDLE_TIMEOUT` | `300` | Idle unload timeout in seconds |\n| `CYBER_INFERENCE_MODEL_LOAD_TIMEOUT` | `300` | Startup readiness timeout in seconds |\n| `CYBER_INFERENCE_PRE_MODEL_LOAD_COMMAND_ENABLED` | `false` | Run host command before model startup |\n| `CYBER_INFERENCE_PRE_MODEL_LOAD_COMMAND` | `sudo sysctl -w vm.drop_caches=3` | Host command for pre-model load preparation |\n| `CYBER_INFERENCE_PRE_MODEL_LOAD_COMMAND_TIMEOUT` | `15` | Pre-load command timeout in seconds |\n| `CYBER_INFERENCE_MAX_LOADED_MODELS` | `1` | Max simultaneously loaded models |\n| `CYBER_INFERENCE_MAX_MEMORY_PERCENT` | `80` | Memory pressure threshold |\n| `CYBER_INFERENCE_LLAMA_GPU_LAYERS` | `-1` | llama.cpp GPU layer setting |\n| `CYBER_INFERENCE_LLAMA_MTP_AUTO_ENABLE` | `true` | Auto-enable MTP for detected speculative GGUF models |\n| `CYBER_INFERENCE_LLAMA_MTP_DEFAULT_DRAFT_N_MAX` | `2` | Default llama.cpp MTP draft token count |\n| `CYBER_INFERENCE_ADMIN_PASSWORD` | unset | Enables admin auth when set |\n| `CYBER_INFERENCE_HF_TOKEN` | unset | HuggingFace token for private repos |\n\nLarge models can take several minutes before their backend server reports ready. The\n`Model Load Timeout (seconds)` admin setting controls how long Cyber-Inference waits during model\nstartup before treating the load as failed. If startup times out, the launched backend process is\nterminated and its port is released.\n\nThor/DGX Spark operators can enable `Run pre-model load command` in Admin Settings to clear Linux\ndisk/page cache before loading very large models. The default command is\n`sudo sysctl -w vm.drop_caches=3`; configure passwordless sudo or run Cyber-Inference in a service\ncontext that can execute the command without an interactive prompt. In this first release, public\n`/v1` API lazy-loads skip the pre-load command; use the Admin UI load action when the host needs the\ncache-clear hook before startup.\n\n## Admin Endpoints\n\n- `GET /admin/status`\n- `GET /admin/resources`\n- `GET /admin/models`\n- `GET /admin/models/repo-files`\n- `POST /admin/models/download`\n- `POST /admin/models/download-transformers`\n- `POST /admin/models/{model}/load`\n- `POST /admin/models/{model}/unload`\n- `DELETE /admin/models/{model}`\n- `GET /admin/config`\n- `PUT /admin/config/{key}`\n\n\n## License\nGPL-3.0\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Framborogers%2Fcyber-inference","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Framborogers%2Fcyber-inference","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Framborogers%2Fcyber-inference/lists"}