{"id":49555127,"url":"https://github.com/shanevcantwell/llauncher","last_synced_at":"2026-05-03T01:55:49.444Z","repository":{"id":349450047,"uuid":"1201897991","full_name":"shanevcantwell/llauncher","owner":"shanevcantwell","description":"llama-server model endpoint manager with JiT swapping with human UI, MCP and pi extension support.","archived":false,"fork":false,"pushed_at":"2026-05-02T14:40:57.000Z","size":764,"stargazers_count":0,"open_issues_count":18,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-05-03T01:55:04.957Z","etag":null,"topics":["agentic-ai","agentic-workflow","ai","llama-cpp","llama-server","llm","mcp","mcp-server","pi-harness","python","server-admin"],"latest_commit_sha":null,"homepage":"http://reflectiveattention.ai","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/shanevcantwell.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-04-05T10:11:37.000Z","updated_at":"2026-05-02T14:41:00.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/shanevcantwell/llauncher","commit_stats":null,"previous_names":["shanevcantwell/llauncher"],"tags_count":3,"template":false,"template_full_name":null,"purl":"pkg:github/shanevcantwell/llauncher","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shanevcantwell%2Fllauncher","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shanevcantwell%2Fllauncher/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shanevcantwell%2Fllauncher/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shanevcantwell%2Fllauncher/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/shanevcantwell","download_url":"https://codeload.github.com/shanevcantwell/llauncher/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shanevcantwell%2Fllauncher/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32555839,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-03T00:31:16.350Z","status":"ssl_error","status_checked_at":"2026-05-03T00:31:15.546Z","response_time":132,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["agentic-ai","agentic-workflow","ai","llama-cpp","llama-server","llm","mcp","mcp-server","pi-harness","python","server-admin"],"created_at":"2026-05-03T01:55:48.873Z","updated_at":"2026-05-03T01:55:49.439Z","avatar_url":"https://github.com/shanevcantwell.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# llauncher\n\nAn MCP-first launcher and management tool for llama.cpp `llama-server` instances. Designed for both programmatic control via LLMs and human operators via a web UI.\n\n## Features\n\n### MCP Server\nFull programmatic control for LLM agents and automation:\n- **List models** with current status (running/stopped)\n- **Start/stop servers** with validation and audit logging\n- **Manage configurations** - add, update, remove model configs\n- **Get server logs** for debugging and monitoring\n- **Validate configurations** before applying changes\n\n### Streamlit UI\nWeb-based dashboard for human operators:\n- **Dashboard**: Overview of all models with quick Start/Stop buttons\n- **Manager**: Add new models or edit existing configurations\n- **Running**: View live logs from active servers with Stop controls\n\n### Configuration\n- **Config Persistence**: Store configurations in `~/.llauncher/config.json` (single source of truth)\n- **Validation**: Model paths verified, port conflicts detected, blacklists enforced\n\n## Installation\n\n```bash\n# Clone the repository\ngit clone https://github.com/shanevcantwell/llauncher\ncd llauncher\n\n# Install in development mode (with UI)\npip install -e \".[ui]\"\n\n# Optional: Install test dependencies\npip install -e \".[test]\"\n```\n\n### Windows Notes\n\nIf you see warnings like `WARNING: Ignoring invalid distribution ~` during install:\n\n```bat\n# Clean up corrupted site-packages and reinstall\ncd github\\llauncher\nrmdir /s /q .venv\npython -m venv .venv\n\\.venv\\Scripts\\activate\npip install -e \".[ui]\"\n```\n\n## Quick Start\n\nUse the runner scripts for easiest setup:\n\n**Linux/macOS:**\n```bash\n./run.sh install   # Set up virtual environment and install\n./run.sh ui        # Start dashboard (auto-starts agent)\n./run.sh agent     # Start agent in foreground\n./run.sh stop      # Stop running agent\n```\n\n**Windows:**\n```cmd\nrun.bat install    # Set up virtual environment and install\nrun.bat ui         # Start dashboard (auto-starts agent)\nrun.bat agent      # Start agent in foreground\nrun.bat stop       # Stop running agent\n```\n\n## Usage\n\n### MCP Server\n\nStart the MCP server:\n\n```bash\nllauncher-mcp\n```\n\nOr configure in your MCP client (e.g., Claude Code):\n\n```json\n{\n  \"mcpServers\": {\n    \"llauncher\": {\n      \"command\": \"llauncher-mcp\",\n      \"args\": []\n    }\n  }\n}\n```\n\n### Available MCP Tools\n\n| Tool | Description |\n|------|-------------|\n| `list_models` | List all configured models with current status (running/stopped) |\n| `get_model_config` | Get full configuration details for a specific model |\n| `start_server` | Start a llama-server instance for a model (with validation) |\n| `stop_server` | Stop a running server by port number |\n| `swap_server` | Atomically swap models on a port with rollback guarantee |\n| `server_status` | Get status summary of all running servers |\n| `get_server_logs` | Fetch recent log lines from a running server |\n| `update_model_config` | Update an existing model's configuration |\n| `validate_config` | Validate a configuration without applying it |\n| `add_model` | Add a new model configuration to the store |\n| `remove_model` | Remove a model configuration (blocks if running) |\n\n### Streamlit UI\n\nStart the UI using the runner script (recommended):\n\n**Linux/macOS:**\n```bash\n./run.sh ui\n```\n\n**Windows:**\n```cmd\nrun.bat ui\n```\n\nThe UI automatically starts a local agent if one isn't running. You can also start the agent separately with `./run.sh agent` or `run.bat agent`.\n\n#### Dashboard Tab\n- Grid view of all configured models with status indicators (🟢 Running / ⚫ Stopped)\n- Quick **Start** and **Stop** buttons for each model\n- **Edit** button redirects to Manager for configuration changes\n- Links to API docs when server is running\n\n#### Manager Tab\n- **List Models**: View all models with expandable details (port, model path, GPU layers)\n- **Add New Model**: Form to create new configurations with validation\n- **Edit Model**: Pre-populated form to modify existing configurations\n- **Delete Model**: Remove configurations (blocked if server is running)\n\n#### Running Tab\n- List of currently running servers with uptime\n- Live log streaming for each server\n- Stop button for each running instance\n\n### CLI\n\nllauncher provides an MCP server and Streamlit UI for model management. Use the runner scripts to start services:\n\n```bash\n./run.sh mcp    # Start MCP server\n./run.sh ui     # Start Streamlit dashboard\n```\n\n## Configuration\n\nCreate model configurations directly in `~/.llauncher/config.json`. Configs can be managed via the UI or MCP tools.\n\nExample config entry:\n\n\n```json\n{\n  \"mistral\": {\n    \"name\": \"mistral\",\n    \"model_path\": \"/path/to/model.gguf\",\n    \"mmproj_path\": null,\n    \"default_port\": 8081,\n    \"n_gpu_layers\": 255,\n    \"ctx_size\": 131072,\n    \"threads\": 8,\n    \"threads_batch\": 8,\n    \"ubatch_size\": 512,\n    \"batch_size\": null,\n    \"flash_attn\": \"on\",\n    \"no_mmap\": false,\n    \"cache_type_k\": \"f32\",\n    \"cache_type_v\": \"f32\",\n    \"n_cpu_moe\": null,\n    \"parallel\": 1,\n    \"temperature\": null,\n    \"top_k\": null,\n    \"top_p\": null,\n    \"min_p\": null,\n    \"repeat_penalty\": null,\n    \"reverse_prompt\": null,\n    \"mlock\": false,\n    \"extra_args\": \"\"\n  }\n}\n```\n\n## Change Management\n\nllauncher includes validation rules to prevent problematic actions:\n\n- **Port conflicts**: Prevents starting models on ports already in use\n- **Blacklisted ports**: Default blacklist includes port 8080 (commonly used by other services)\n- **Model whitelists**: Optionally restrict which models can be started\n- **Caller blacklists**: Restrict which callers (UI, MCP, etc.) can perform actions\n\n## Project Structure\n\n```\nllauncher/\n├── pyproject.toml\n├── llauncher/\n│   ├── __init__.py\n│   ├── __main__.py\n│   ├── agent/             # HTTP agent for multi-node management\n│   │   ├── config.py\n│   │   ├── routing.py\n│   │   └── server.py\n│   ├── core/\n│   │   ├── config.py      # Config persistence\n│   │   ├── process.py     # Process management\n│   │   └── settings.py    # Global settings\n│   ├── mcp/\n│   │   ├── server.py      # MCP server\n│   │   └── tools/         # Tool implementations\n│   ├── models/\n│   │   └── config.py      # Pydantic models\n│   ├── remote/            # Multi-node support\n│   │   ├── node.py\n│   │   ├── registry.py\n│   │   └── state.py\n│   ├── state.py           # StateManager\n│   └── ui/\n│       ├── app.py         # Streamlit app\n│       └── tabs/          # UI components\n```\n\n## Testing\n\nRun the test suite:\n\n```bash\npytest\n# or with coverage\npytest --cov=llauncher --cov-report=term-missing\n```\n\nTest files are in `tests/`:\n- `tests/unit/`: Unit tests for models, config, and process\n- `tests/integration/`: Integration tests for state management\n\n## Multi-Node Management (Remote)\n\nllauncher supports managing llama-server instances across multiple machines (Windows and Linux) on a local network from a single dashboard.\n\n### Architecture\n\nEach managed node runs a lightweight **agent** that exposes an HTTP API. The \"head\" dashboard connects to these agents over the LAN:\n\n```\n┌─────────────────────────────────────┐\n│         HEAD DASHBOARD              │\n│  - Streamlit UI with node selector  │\n│  - Connects to all agents via HTTP  │\n└─────────────┬───────────────────────┘\n              │ LAN (port 8765)\n    ┌─────────┼─────────┐\n    ▼         ▼         ▼\n┌────────┐ ┌────────┐ ┌────────┐\n│ Agent  │ │ Agent  │ │ Agent  │\n│ Linux  │ │Windows │ │ Linux  │\n│ :8765  │ │ :8765  │ │ :8765  │\n└────────┘ └────────┘ └────────┘\n```\n\n### Deployment\n\n#### 1. Install on Each Node\n\nOn every machine you want to manage (including the head):\n\n**Linux/macOS:**\n```bash\ngit clone https://github.com/shanevcantwell/llauncher\ncd llauncher\n./run.sh install\n```\n\n**Windows:**\n```cmd\ngit clone https://github.com/shanevcantwell/llauncher\ncd llauncher\nrun.bat install\n```\n\n#### 2. Start the Agent on Each Node\n\n**Using runner scripts (recommended):**\n\n**Linux/macOS:**\n```bash\n./run.sh agent     # Foreground\n./run.sh agent-bg  # Background\n./run.sh stop      # Stop agent\n```\n\n**Windows:**\n```cmd\nrun.bat agent      # Foreground\nrun.bat agent-bg   # Background\nrun.bat stop       # Stop agent\n```\n\n**With custom configuration:**\n```bash\n# Linux/macOS\nLAUNCHER_AGENT_PORT=9000 LAUNCHER_AGENT_NODE_NAME=\"my-server\" ./run.sh agent\n\n# Windows (PowerShell)\n$env:LAUNCHER_AGENT_PORT=\"9000\"\n$env:LAUNCHER_AGENT_NODE_NAME=\"my-server\"\nrun.bat agent\n```\n\n**Environment Variables:**\n- `LAUNCHER_AGENT_HOST`: Host to bind to (default: `0.0.0.0`)\n- `LAUNCHER_AGENT_PORT`: Port to listen on (default: `8765`)\n- `LAUNCHER_AGENT_NODE_NAME`: Friendly name for the node\n\n#### 3. Start the Dashboard on the Head Machine\n\n**Linux/macOS:**\n```bash\n./run.sh ui\n```\n\n**Windows:**\n```cmd\nrun.bat ui\n```\n\nThe dashboard will automatically:\n1. Show a loading screen while initializing\n2. Start a local agent if one isn't running\n3. Register itself as the \"local\" node\n\n#### 4. Add Remote Nodes\n\nIn the dashboard:\n1. Go to the **Nodes** tab\n2. Click **➕ Add New Node**\n3. Enter:\n   - **Node Name**: Friendly name (e.g., `linux-box`, `windows-server`)\n   - **Host**: IP address or hostname (e.g., `192.168.1.100`)\n   - **Port**: Agent port (default: `8765`)\n4. Click **🔍 Test Connection** to verify\n5. Click **➕ Add Node** to register\n\n### Network Configuration\n\n#### Firewall Rules\n\nEnsure port 8765 is open on managed nodes:\n\n**Linux (ufw):**\n```bash\nsudo ufw allow 8765/tcp\n```\n\n**Linux (firewalld):**\n```bash\nsudo firewall-cmd --permanent --add-port=8765/tcp\nsudo firewall-cmd --reload\n```\n\n**Windows (PowerShell):**\n```powershell\nNew-NetFirewallRule -DisplayName \"llauncher Agent\" -Direction Inbound -LocalPort 8765 -Protocol TCP -Action Allow\n```\n\n#### Security Notes\n\n- **Trusted LAN Only**: Agents run without authentication by default. Only expose them on trusted networks.\n- **Bind to Specific Interface**: Use `LAUNCHER_AGENT_HOST` to bind to a specific IP instead of `0.0.0.0`.\n- **Firewall**: Restrict port 8765 to your LAN subnet.\n\n### Usage\n\n#### Dashboard Tab\n\n- **Node Selector** (sidebar): Filter view by specific node or \"All Nodes\"\n- **Running Servers**: Shows all active servers with node badges\n- **Models**: Lists all configured models grouped by node\n- **Start/Stop**: Control servers on any node\n\n#### Nodes Tab\n\n- **Registered Nodes**: List of all connected nodes with status\n- **Test Connection**: Verify agent connectivity\n- **Remove Node**: Unregister a node from the dashboard\n\n### Troubleshooting\n\n#### \"Connection Failed\" when adding node\n\n1. Verify agent is running on the remote node:\n   ```bash\n   curl http://\u003cnode-ip\u003e:8765/health\n   ```\n\n2. Check firewall rules on the remote node\n\n3. Verify the agent is binding to the correct interface:\n   ```bash\n   # Should show 0.0.0.0:8765 or your LAN IP\n   netstat -tlnp | grep 8765\n   ```\n\n#### Agent won't start\n\n1. Check if port 8765 is already in use:\n   ```bash\n   lsof -i :8765\n   # or\n   netstat -tlnp | grep 8765\n   ```\n\n2. Use a different port:\n   ```bash\n   LAUNCHER_AGENT_PORT=9000 llauncher-agent\n   ```\n\n#### Can't connect from Windows to Linux (or vice versa)\n\n1. Verify network connectivity:\n   ```bash\n   ping \u003cremote-node-ip\u003e\n   ```\n\n2. Check that the agent is not binding to localhost only:\n   - Look for `0.0.0.0:8765` in agent startup logs\n   - If it shows `127.0.0.1:8765`, set `LAUNCHER_AGENT_HOST=0.0.0.0`\n\n### API Documentation\n\nWhen an agent is running, visit `http://\u003cnode-ip\u003e:8765/docs` for interactive API documentation.\n\n### License\n\nMIT\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshanevcantwell%2Fllauncher","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fshanevcantwell%2Fllauncher","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshanevcantwell%2Fllauncher/lists"}