{"id":49767014,"url":"https://github.com/retospect/slullama","last_synced_at":"2026-05-11T10:57:04.440Z","repository":{"id":346866284,"uuid":"1181218456","full_name":"retospect/slullama","owner":"retospect","description":"Shared Ollama gateway over Slurm — wake a GPU node on demand","archived":false,"fork":false,"pushed_at":"2026-04-24T23:45:43.000Z","size":1369,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-05-11T10:57:04.221Z","etag":null,"topics":["gpu","llm","model-management","ollama","python"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/retospect.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-03-13T21:57:35.000Z","updated_at":"2026-04-24T23:45:41.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/retospect/slullama","commit_stats":null,"previous_names":["retospect/slullama"],"tags_count":4,"template":false,"template_full_name":null,"purl":"pkg:github/retospect/slullama","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/retospect%2Fslullama","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/retospect%2Fslullama/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/retospect%2Fslullama/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/retospect%2Fslullama/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/retospect","download_url":"https://codeload.github.com/retospect/slullama/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/retospect%2Fslullama/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32891966,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-10T13:40:02.631Z","status":"online","status_checked_at":"2026-05-11T02:00:05.975Z","response_time":120,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["gpu","llm","model-management","ollama","python"],"created_at":"2026-05-11T10:57:03.294Z","updated_at":"2026-05-11T10:57:04.432Z","avatar_url":"https://github.com/retospect.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# slullama\n\n**Shared Ollama gateway over Slurm** — wake a GPU node on demand.\n\nA slumbering llama that wakes on demand. One package, two roles: a **server\ndaemon** on the Slurm head node and a **client library** on team members'\nlaptops. The first API request submits a Slurm job, opens an SSH tunnel to\nthe compute node, and proxies standard Ollama API traffic. When nobody has\nmade a request for a configurable idle period the job is torn down\nautomatically. The next request wakes it all back up.\n\nMultiuser by design — the whole team shares one GPU node and one Ollama\nprocess. Authentication is via a shared bearer token.\n\n---\n\n## Architecture\n\n```\nlaptop (any team member)       head node                  compute node (GPU)\n  │                              │                              │\n  │──SSH tunnel (port 11434)───▶│                              │\n  │                              │──SSH tunnel (port 19434)───▶│\n  │                              │   aiohttp reverse proxy     │   ollama serve\n  │                              │   (listens 0.0.0.0:11435)   │   (0.0.0.0:11434)\n  │                              │   + SlurmManager             │\n  │                              │   + idle watchdog            │\n  │                              │                              │\n```\n\n### Request flow\n\n1. Client code (litellm, `SlulamaClient`, or raw curl) hits\n   `localhost:11434` on the laptop.\n2. The SSH tunnel forwards the request to the head node proxy on port 11435.\n3. The proxy checks the bearer token (`Authorization: Bearer \u003ctoken\u003e`).\n4. If no Slurm job is running, the proxy:\n   a. Renders the sbatch template and submits it (`sbatch`).\n   b. Polls `scontrol show job` until the job reaches `RUNNING`.\n   c. Opens an SSH tunnel from the head node to the compute node's Ollama\n      port.\n   d. Polls Ollama's `/api/tags` health endpoint through the tunnel until\n      healthy.\n5. The proxy forwards the request to Ollama (streaming supported).\n6. On every proxied request the Slurm job timeout is extended (or the\n   last-activity timestamp is recorded, depending on keep-alive strategy).\n7. A background watchdog checks idle time every 30 s. If idle ≥\n   `idle_timeout` minutes it cancels the job and closes tunnels.\n8. The next request goes back to step 4.\n\n### Two SSH hops\n\nBoth hops are managed automatically:\n\n- **Laptop → head node**: managed by `ClientTunnel` (client side). Sync or\n  async. Opened lazily on first request, closed on process exit / context\n  manager exit.\n- **Head node → compute node**: managed by `SshTunnel` (server side). Opened\n  after the Slurm job reaches RUNNING, closed on idle teardown.\n\nCompute nodes are typically not reachable from outside the cluster, which is\nwhy both hops are necessary.\n\n---\n\n## Quick start\n\n### 1. Head node (server)\n\n```bash\npip install slullama          # or: uv pip install slullama\n\nmkdir -p ~/.config/slullama\n```\n\nCreate `~/.config/slullama/config.toml`:\n\n```toml\n[server]\nport = 11435                  # proxy listens here\ntoken = \"your-shared-secret\"  # bearer token for auth\nlog_dir = \"/tmp/slullama\"     # sbatch scripts + job logs\n\n[slurm]\npartition = \"gpu\"\ngres = \"gpu:1\"\nmem = \"\"                      # optional, e.g. \"64G\"\ntime = \"4:00:00\"              # initial job time limit\nidle_timeout = 30             # minutes of inactivity before teardown\nkeep_alive = \"extend\"         # \"extend\" or \"cancel\" (see below)\nextra_args = []               # extra #SBATCH lines, e.g. [\"--exclusive\"]\n\n[ollama]\nport = 11434                  # port ollama listens on inside the compute node\nbinary = \"ollama\"             # path to ollama binary (must be on compute node)\nmodels_dir = \"\"               # OLLAMA_MODELS dir; empty = ollama default\npre_pull = []                 # models to pull on cold start, e.g. [\"qwen3.5:9b\"]\n\n# If the binary is NOT on a shared filesystem, copy it per job:\n# copy_binary = true\n# copy_source = \"/shared/bin/ollama\"\n# cleanup_binary = true       # rm the copy when the job ends\n```\n\nStart the daemon (foreground; use tmux/screen/systemd for persistence):\n\n```bash\nslullama serve                       # uses ~/.config/slullama/config.toml\nslullama serve --config /etc/slullama.toml   # explicit path\nslullama serve --port 9999 --partition gpu-large  # CLI overrides\nslullama serve -v                    # debug logging\n```\n\n### 2. Laptop (client)\n\n```bash\npip install slullama                 # base client\npip install \"slullama[litellm]\"      # with litellm integration\n```\n\nCreate `~/.config/slullama/config.toml` (client section only is fine):\n\n```toml\n[client]\nhost = \"youruser@headnode\"    # SSH destination\nserver_port = 11435           # must match server's port\ntoken = \"your-shared-secret\"  # must match server's token\nlocal_port = 11434            # local Ollama-compatible endpoint\n```\n\n#### Option A — litellm (recommended)\n\n```python\nimport slullama   # auto-registers the \"slullama/\" provider on import\nimport litellm\n\n# Synchronous\nresp = litellm.completion(\n    model=\"slullama/qwen3.5:9b\",\n    messages=[{\"role\": \"user\", \"content\": \"hello\"}],\n)\nprint(resp.choices[0].message.content)\n\n# Streaming\nfor chunk in litellm.completion(\n    model=\"slullama/qwen3.5:9b\",\n    messages=[{\"role\": \"user\", \"content\": \"hello\"}],\n    stream=True,\n):\n    print(chunk.choices[0].delta.content, end=\"\")\n```\n\nUnder the hood this opens the SSH tunnel on the first call (lazily), proxies\nthrough the head node, and returns standard litellm `ModelResponse` objects.\n\n#### Option B — Python client (async)\n\n```python\nfrom slullama import SlulamaClient\n\nasync with SlulamaClient(host=\"user@headnode\") as client:\n    # Ollama-compatible chat\n    resp = await client.chat(\"qwen3.5:9b\", messages=[\n        {\"role\": \"user\", \"content\": \"What is electrochemistry?\"},\n    ])\n    print(resp[\"message\"][\"content\"])\n\n    # List models\n    tags = await client.tags()\n    print(tags)\n\n    # Server status (job state, idle time, etc.)\n    status = await client.status()\n    print(status)\n\n    # Raw Ollama URL for any tool that speaks Ollama\n    print(client.ollama_url)   # http://localhost:11434\n```\n\n#### Option C — Python client (singleton, no context manager)\n\n```python\nfrom slullama import SlulamaClient\n\nclient = SlulamaClient.get_default()   # lazy singleton\nclient.connect_sync()                  # opens SSH tunnel (blocking)\nprint(client.ollama_url)               # http://localhost:11434\n# Tunnel stays open until process exits (atexit hook).\n```\n\n#### Option D — CLI tunnel (foreground)\n\n```bash\nslullama connect user@headnode\n# Tunnel open: localhost:11434 → headnode:11435\n# Press Ctrl+C to close.\n\n# Now any Ollama-speaking tool works:\ncurl http://localhost:11434/api/tags\n```\n\n### 3. Check status\n\n```bash\n# From the head node (no tunnel needed):\nslullama status\n\n# From a laptop (uses config or args):\nslullama status headnode --port 11435 --token your-shared-secret\n```\n\nReturns JSON with job state, node, idle time, request count, uptime, etc.\n\n---\n\n## Full configuration reference\n\nConfig file: `~/.config/slullama/config.toml`\nOverride path: `--config \u003cpath\u003e` or `SLULLAMA_CONFIG=\u003cpath\u003e`\n\n### `[server]` — head node daemon\n\n| Key | Type | Default | Description |\n|---|---|---|---|\n| `port` | int | `11435` | Port the proxy listens on |\n| `token` | str | `\"\"` | Bearer token for auth (empty = no auth) |\n| `log_dir` | str | `\"/tmp/slullama\"` | Directory for sbatch scripts and job logs |\n| `job_template` | str | `\"\"` | Path to custom sbatch template (empty = built-in) |\n\n### `[slurm]` — Slurm job parameters\n\n| Key | Type | Default | Description |\n|---|---|---|---|\n| `partition` | str | `\"gpu\"` | Slurm partition |\n| `gres` | str | `\"gpu:1\"` | Generic resources |\n| `mem` | str | `\"\"` | Memory (e.g. `\"64G\"`); empty = cluster default |\n| `time` | str | `\"4:00:00\"` | Initial job time limit |\n| `idle_timeout` | int | `30` | Minutes of inactivity before teardown |\n| `keep_alive` | str | `\"extend\"` | `\"extend\"` or `\"cancel\"` (see below) |\n| `extra_args` | list | `[]` | Extra `#SBATCH` directives (e.g. `[\"--exclusive\"]`) |\n\n### `[ollama]` — Ollama on the compute node\n\n| Key | Type | Default | Description |\n|---|---|---|---|\n| `port` | int | `11434` | Port ollama listens on |\n| `binary` | str | `\"ollama\"` | Path to ollama binary |\n| `models_dir` | str | `\"\"` | `OLLAMA_MODELS` dir; empty = ollama default |\n| `copy_binary` | bool | `false` | Copy binary to compute node per job |\n| `copy_source` | str | `\"\"` | Source path for copy |\n| `cleanup_binary` | bool | `false` | Delete the copy when the job ends |\n| `pre_pull` | list | `[]` | Models to pull after ollama starts |\n\n### `[client]` — laptop / workstation\n\n| Key | Type | Default | Description |\n|---|---|---|---|\n| `host` | str | `\"\"` | SSH destination (`user@headnode`) |\n| `server_port` | int | `11435` | Proxy port on head node |\n| `token` | str | `\"\"` | Bearer token |\n| `local_port` | int | `11434` | Local port (appears as Ollama to tools) |\n\n### Environment variable overrides\n\nEnvironment variables override the config file. Useful for CI, containers,\nor quick one-offs.\n\n| Variable | Overrides |\n|---|---|\n| `SLULLAMA_CONFIG` | Config file path |\n| `SLULLAMA_TOKEN` | `server.token` and `client.token` |\n| `SLULLAMA_HOST` | `client.host` |\n| `SLULLAMA_SERVER_PORT` | `server.port` and `client.server_port` |\n| `SLULLAMA_LOCAL_PORT` | `client.local_port` |\n| `SLULLAMA_PARTITION` | `slurm.partition` |\n| `SLULLAMA_GRES` | `slurm.gres` |\n| `SLULLAMA_IDLE_TIMEOUT` | `slurm.idle_timeout` |\n| `SLULLAMA_OLLAMA_BINARY` | `ollama.binary` |\n| `SLULLAMA_OLLAMA_PORT` | `ollama.port` |\n\n---\n\n## Keep-alive strategies\n\n| Strategy | How it works | When to use |\n|---|---|---|\n| `extend` (default) | On each request, runs `scontrol update JobId=X TimeLimit=+Nmin` to push the deadline forward. | Clusters that allow job time extensions. |\n| `cancel` | Submits the job with the full `time` limit. On idle timeout, runs `scancel`. Next request resubmits. | Clusters that restrict `scontrol update`. |\n\nSet via `keep_alive` in `[slurm]` config or try both on your cluster — the\nserver logs a warning if `scontrol update` fails.\n\n---\n\n## Custom job template\n\nThe default sbatch template is in `slullama/template.py`. To use your own:\n\n```toml\n[server]\njob_template = \"/path/to/my_template.sh\"\n```\n\nTemplates use Python `string.Template` syntax (`${variable_name}`).\nAvailable variables:\n\n| Variable | Value |\n|---|---|\n| `${partition}` | Slurm partition |\n| `${gres}` | GRES string |\n| `${time}` | Time limit |\n| `${log_dir}` | Log directory |\n| `${extra_sbatch}` | Extra `#SBATCH` lines (rendered from `mem` + `extra_args`) |\n| `${copy_commands}` | Binary copy commands (or comment if disabled) |\n| `${ollama_port}` | Ollama listen port |\n| `${models_env}` | `export OLLAMA_MODELS=...` (or comment if not set) |\n| `${cleanup_commands}` | Binary cleanup commands (or comment if disabled) |\n| `${ollama_binary}` | Path to ollama binary (may be the copied path) |\n| `${pull_commands}` | `ollama pull` commands (or comment if empty) |\n\nUse `$$` for literal `$` in bash (e.g. `$$SLURM_JOB_ID`, `$$!`).\n\n---\n\n## Module map\n\n```\nsrc/slullama/\n├── __init__.py              # Public API: SlulamaClient, Config, auto-registers litellm\n├── config.py                # Dataclasses: Config, ServerConfig, ClientConfig, SlurmConfig, OllamaConfig\n│                            #   Config.load() reads TOML + env overrides\n├── template.py              # DEFAULT_TEMPLATE + render_template(ServerConfig) → str\n├── cli.py                   # CLI: slullama serve | connect | status\n├── litellm_provider.py      # litellm CustomLLM subclass, registers \"slullama/\" prefix\n├── server/\n│   ├── slurm.py             # SlurmManager: submit(), query(), extend_time(), cancel(), wait_for_running()\n│   ├── tunnel.py            # SshTunnel: head node → compute node SSH port-forward\n│   └── proxy.py             # OllamaProxy: aiohttp reverse proxy, auth, idle watchdog, /slullama/status\n└── client/\n    ├── tunnel.py            # ClientTunnel: laptop → head node SSH port-forward (sync + async)\n    └── client.py            # SlulamaClient: high-level client, context manager, singleton\n```\n\n### Key classes\n\n- **`SlurmManager`** (`server/slurm.py`): Wraps `sbatch`, `scontrol`,\n  `scancel` as async subprocess calls. Tracks job ID and node. Parses\n  `scontrol show job` output for state, node, time left.\n\n- **`SshTunnel`** (`server/tunnel.py`): Async SSH port-forward via\n  `ssh -N -L`. Used on the head node to reach the compute node.\n\n- **`OllamaProxy`** (`server/proxy.py`): aiohttp web app that:\n  - Serves `/slullama/status` (GET) — JSON status endpoint.\n  - Proxies everything else (`/{path:.*}`) to Ollama on the compute node.\n  - Manages the full lifecycle: job submission → tunnel → health poll →\n    proxy → idle watchdog → teardown.\n  - `_boot_lock` ensures only one concurrent boot sequence.\n  - Idle watchdog runs every 30 s, tears down if idle ≥ timeout.\n\n- **`ClientTunnel`** (`client/tunnel.py`): SSH port-forward from laptop to\n  head node. Has both sync (`open_sync`/`close_sync` for litellm/atexit)\n  and async (`open`/`close`) APIs.\n\n- **`SlulamaClient`** (`client/client.py`): High-level client.\n  - Context manager: `async with SlulamaClient(...) as c:`\n  - Lazy singleton: `SlulamaClient.get_default()` for litellm provider use.\n  - Methods: `chat()`, `tags()`, `status()`.\n  - Property: `ollama_url` — the local URL that speaks Ollama.\n\n- **`SlulamaLLM`** (`litellm_provider.py`): litellm `CustomLLM` subclass\n  with `completion()` and `streaming()`. Registered into\n  `litellm.custom_provider_map` on `import slullama`.\n\n---\n\n## Integration with acatome-lambic\n\nThe existing `LlmClient` in acatome-lambic supports `provider=\"ollama\"` with\na configurable `ollama_url`. Two integration paths:\n\n1. **Via litellm**: Set `provider=\"slullama\"` and `model=\"qwen3.5:9b\"` in\n   `LlmConfig`. Since acatome-lambic already uses litellm as a backend, this\n   works if slullama is installed with `[litellm]` extra.\n\n2. **Direct**: Run `slullama connect` in a terminal to create a local tunnel,\n   then use `provider=\"ollama\"` with `ollama_url=\"http://localhost:11434\"`.\n   No code changes needed in acatome-lambic.\n\n---\n\n## Known limitations / future work\n\n- **Cold start latency**: First request after idle waits for Slurm job\n  allocation + Ollama startup (30–120 s depending on cluster). The proxy\n  blocks the request until ready; it does not return 503.\n- **Single GPU node**: Currently manages one Slurm job. Multiple concurrent\n  jobs / load balancing is out of scope for v1.\n- **No model routing**: All requests go to the same Ollama instance. Model\n  loading/unloading is handled by Ollama natively.\n- **`scontrol update` permissions**: Some clusters don't allow users to\n  extend job time. Use `keep_alive = \"cancel\"` as a workaround.\n- **No HTTPS**: Tunnel traffic is encrypted by SSH; the HTTP proxy itself is\n  plaintext. Fine for a cluster environment.\n- **Ollama binary distribution**: If no shared filesystem, the binary must\n  be copied per job (`copy_binary = true`). This adds cold start time.\n\n---\n\n## CLI reference\n\n```\nslullama serve [--port N] [--token T] [--partition P] [--gres G]\n               [--idle-timeout M] [--config PATH] [-v]\n\n    Start the proxy daemon on the head node.\n\nslullama connect [user@headnode] [--port N] [--token T]\n                 [--local-port N] [--config PATH] [-v]\n\n    Open an SSH tunnel to the head node (foreground, Ctrl+C to close).\n\nslullama status [headnode] [--port N] [--token T] [--config PATH]\n\n    Query the server's /slullama/status endpoint and print JSON.\n```\n\n---\n\n## License\n\nMIT\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fretospect%2Fslullama","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fretospect%2Fslullama","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fretospect%2Fslullama/lists"}