{"id":44843683,"url":"https://github.com/flyingfathead/rust-linuxgsm-watchdog","last_synced_at":"2026-02-17T04:02:09.998Z","repository":{"id":337698283,"uuid":"1154745787","full_name":"FlyingFathead/rust-linuxgsm-watchdog","owner":"FlyingFathead","description":null,"archived":false,"fork":false,"pushed_at":"2026-02-10T23:19:52.000Z","size":108,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-02-10T23:36:01.703Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/FlyingFathead.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-02-10T18:22:45.000Z","updated_at":"2026-02-10T23:19:56.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/FlyingFathead/rust-linuxgsm-watchdog","commit_stats":null,"previous_names":["flyingfathead/rust-linuxgsm-watchdog"],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/FlyingFathead/rust-linuxgsm-watchdog","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FlyingFathead%2Frust-linuxgsm-watchdog","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FlyingFathead%2Frust-linuxgsm-watchdog/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FlyingFathead%2Frust-linuxgsm-watchdog/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FlyingFathead%2Frust-linuxgsm-watchdog/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/FlyingFathead","download_url":"https://codeload.github.com/FlyingFathead/rust-linuxgsm-watchdog/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FlyingFathead%2Frust-linuxgsm-watchdog/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29532928,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-17T03:01:11.216Z","status":"ssl_error","status_checked_at":"2026-02-17T03:00:31.803Z","response_time":100,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-02-17T04:02:08.002Z","updated_at":"2026-02-17T04:02:09.992Z","avatar_url":"https://github.com/FlyingFathead.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# rust-linuxgsm-watchdog\n\nA watchdog for **[Rust (the game)](https://rust.facepunch.com/), i.e. for dedicated servers managed by LinuxGSM** to keep your server up, running and up to date in a more automated way than what [LinuxGSM](https://linuxgsm.com/) offers by default.\n\nThis program is stdlib-only by default. If you enable WebRCON features (tests / SmoothRestarter bridge), it uses `websocket-client`. It polls server health and, if the server is *confirmed down*, runs a recovery sequence, i.e.:\n\n1) `./rustserver update`  \n2) `./rustserver mu` (Oxide update via LinuxGSM mods)  \n3) `./rustserver restart`\n\nThis is meant to complement workflows like uMod’s **[Smooth Restarter](https://umod.org/plugins/smooth-restarter)** that can *stop the server gracefully* but don’t handle **Steam-end server update + mod updates + restart** on their own.\n\nThe Rust Watchdog currently supports server status/restart/update alerts via the [Telegram Bot API](https://core.telegram.org/bots).\n\n---\n\n## Why this exists\n\n- Rust receives constant updates from [Facepunch](https://rust.facepunch.com/) -- so keeping the server current with minimal downtime matters.\n- LinuxGSM already knows how to do the boring-but-correct sequence: **update server + update mods + restart**.\n- But LinuxGSM does not automatically run that sequence when the server goes down due to external reasons (crashes, plugin actions, etc).\n- Many “restart schedulers” can only stop the server. Coordinating **stop/update/mu/restart** reliably on LinuxGSM is a separate problem.\n- LinuxGSM runs Rust inside **tmux**. If you try to run recovery from inside `screen`/`tmux`, you’ll get tmuxception and everything gets stupid.\n\nSo the watchdog is designed to run **outside** `screen`/`tmux` (ideally via `systemd`).\n\n---\n\n## What \"health\" means here\n\nHealth is decided by simple signals (no log parsing, no fragile regex soup):\n\n- **Process identity check (strong):**\n  - `pgrep -af RustDedicated` must show `+server.identity \u003cidentity\u003e`\n- **TCP connect check (medium):**\n  - TCP connect to the configured RCON port (default `127.0.0.1:28016`) to verify the port is reachable (not full WebRCON auth)\n\nIf any RUNNING signal passes, the watchdog reports `RUNNING`.\n\nIf RUNNING signals fail repeatedly for `down_confirmations` checks, it becomes “confirmed down” and recovery starts.\n\nOptional (disabled by default): `./rustserver details` parsing exists for debugging, but it can hang or be slow.\n\n---\n\n## Requirements / assumptions\n\n- Python 3.9+ (uses `zoneinfo`; install `tzdata` on minimal hosts if your timezone DB is missing)\n- A working LinuxGSM Rust install where `server_dir` contains an executable `./rustserver`\n\nOptional (only needed for WebRCON features like `--test-rcon-say` and the SmoothRestarter bridge):\n- `websocket-client` (install via `requirements.txt`, or `pip install websocket-client`) \n\n---\n\n## Files\n\n- `rust_watchdog.py` -- the watchdog\n- `rust_watchdog.json` -- config (merged over defaults)\n- `rust-watchdog.service` -- example systemd unit\n\n---\n\n## Config\n\nExample `rust_watchdog.json`:\n\n```json\n{\n  \"server_dir\": \"/home/rustserver\",\n  \"identity\": \"rustserver\",\n\n  \"pause_file\": \"/home/rustserver/rust-linuxgsm-watchdog/.watchdog_pause\",\n  \"dry_run\": false,\n\n  \"interval_seconds\": 10,\n  \"cooldown_seconds\": 120,\n  \"down_confirmations\": 2,\n\n  \"check_process_identity\": true,\n\n  \"check_tcp_rcon\": true,\n  \"rcon_host\": \"127.0.0.1\",\n  \"rcon_port\": 28016,\n  \"tcp_timeout\": 2.0,\n\n  \"check_lgsm_details\": false,\n  \"details_timeout\": 20,\n\n  \"recovery_steps\": [\"update\", \"mu\", \"restart\"],\n  \"timeouts\": { \"update\": 1800, \"mu\": 900, \"restart\": 600 }\n}\n```\n\nNotes:\n\n* `enable_server_update`: if false, skip the `update` step even if it’s listed in `recovery_steps`.\n* `enable_mods_update`: if false, skip the `mu` step even if it’s listed in `recovery_steps`.\n* `pause_file`: if this file exists, the watchdog pauses (no checks, no recovery).\n* `dry_run`: logs what it *would* do, but never runs recovery steps.\n* `down_confirmations`: prevents one bad poll from causing a recovery.\n* `timeouts`: per-step hard limits so SteamCMD slowness doesn’t hang the watchdog forever.\n\n---\n\n## Usage\n\nFirst, clone the repo i.e. with:\n\n```bash\ncd \u0026\u0026\ngit clone https://github.com/FlyingFathead/rust-linuxgsm-watchdog \u0026\u0026\ncd rust-linuxgsm-watchdog\n\n# stdlib-only mode (no WebRCON features) -- nothing to install\n\n# OPTIONAL: enable WebRCON features (tests + SmoothRestarter bridge)\npython3 -m venv .venv\n./.venv/bin/python -m pip install -U pip\n./.venv/bin/python -m pip install -r requirements.txt\n```\n\n**(Option B to install the websocket if the venv isn't working out for you):**\n\nOn Ubuntu/Debian tree Linux systems:\n\n```bash\nsudo apt update\nsudo apt install -y python3-websocket || sudo apt install -y python3-websocket-client\n```\n\nOn Fedora/RHEL:\n\n```bash\nsudo dnf install -y python3-websocket-client\n```\n\n### One-shot (manual test)\n\nRun one loop iteration and exit:\n\n```bash\n./rust_watchdog.py --config ./rust_watchdog.json --once\n```\n\n### Long-running\n\n```bash\n./rust_watchdog.py --config ./rust_watchdog.json\n```\n\nDo **not** run it inside `screen`/`tmux` if you want it to actually recover (LinuxGSM will tmuxception).\n\n### WebRCON test helpers\n\nSend a chat broadcast via WebRCON:\n\n```bash\n./rust_watchdog.py --config ./rust_watchdog.json --test-rcon-say \"hello from watchdog\"\n```\n\nSend an arbitrary WebRCON command:\n\n```bash\n./rust_watchdog.py --config ./rust_watchdog.json --test-rcon-cmd \"status\"\n```\n\n---\n\n## systemd setup (recommended)\n\nCopy the unit file (**make sure to edit your necessary changes first!**):\n\n```bash\nsudo cp ./rust-watchdog.service /etc/systemd/system/rust-watchdog.service\nsudo systemctl daemon-reload\nsudo systemctl enable --now rust-watchdog.service\n```\n\nCheck logs:\n\n```bash\nsudo systemctl status --no-pager -l rust-watchdog.service\njournalctl -u rust-watchdog.service -f\n```\n\n### After editing the script or JSON\n\nRestart the service:\n\n```bash\nsudo systemctl restart rust-watchdog.service\n```\n\n---\n\n## Troubleshooting\n\n### \"tmuxception\"\n\nYou’re running recovery from inside `screen` or another multiplexer. Run the watchdog via `systemd` (or a plain shell) instead.\n\n### Lock file complaints\n\nThe watchdog uses a lock to prevent multiple instances.\n\nIf you see a lock complaint, it will mention your configured lockfile path, e.g.:\n\n* `Lock exists at /home/rustserver/rust-linuxgsm-watchdog/data/lock/rust_watchdog.lock`\n\nCheck if it’s actually running:\n\n```bash\npgrep -af rust_watchdog.py\n```\n\nIf nothing is running and the lock is stale:\n\n```bash\nrm -f /home/rustserver/rust-linuxgsm-watchdog/data/lock/rust_watchdog.lock\nsudo systemctl restart rust-watchdog.service\n```\n\n### Timeouts / hanging updates\n\nBump `timeouts.update` / `timeouts.mu` if SteamCMD is slow, or keep them strict if you prefer fail-fast + retry later.\n\n---\n\n## Optional: SmoothRestarter bridge (graceful restarts)\n\nIf you use uMod’s **[Smooth Restarter](https://umod.org/plugins/smooth-restarter)** for player-visible countdown/UI, the watchdog can act as a bridge **while the server is RUNNING**:\n\n1. Watchdog periodically runs `./rustserver check-update` (or `./rustserver cu`) via LinuxGSM.\n2. If an update is detected, watchdog **always broadcasts**:\n   - `update_watch_announce_message` (default: \"Update detected -- restart incoming.\")\n3. Then it chooses one of two paths:\n\n### Path A -- SmoothRestarter countdown (preferred)\n\nIf SmoothRestarter bridging is enabled and usable, watchdog sends (via **Rust WebRCON**) the configured command:\n- `smoothrestarter_console_cmd` (default: `srestart restart {delay}`)\n\nEven when using SmoothRestarter’s own countdown/UI, watchdog also sends **one** informational line using:\n- `update_watch_countdown_template` (example: \"Time until server update and restart: {seconds} seconds.\")\n\nAnd it also sends the final fallback message once:\n- `update_watch_final_message` (default: \"Server is restarting, come back in a few minutes!\")\n\nSmoothRestarter then performs the graceful shutdown. Once the server is down, LinuxGSM restart/update happens on the next normal watchdog recovery cycle.\n\n### Path B -- No SmoothRestarter (or bridge failed)\n\nIf SmoothRestarter is disabled OR the bridge fails at runtime, watchdog does a crude countdown itself:\n- broadcasts `update_watch_countdown_template` every `update_watch_no_sr_tick_seconds`\n- for `update_watch_no_sr_countdown_seconds` total\n- then broadcasts `update_watch_final_message`\n- then runs the immediate sequence:\n  `./rustserver stop` -\u003e `./rustserver update` -\u003e `./rustserver mu` -\u003e `./rustserver restart`\n\n### What “SR check” means in this project\n\nThere are three different ideas people confuse:\n\n- **Bridge enabled:** `enable_smoothrestarter_bridge=true` (note: bridge only triggers if `enable_update_watch=true`)\n- **SmoothRestarter installed:** plugin file exists:\n  `{server_dir}/serverfiles/oxide/plugins/SmoothRestarter.cs`\n- **Bridge usable right now:** `websocket-client` is available and WebRCON autodetect works (find `+rcon.ip/+rcon.port/+rcon.password` from the RustDedicated cmdline for this identity), and the RCON send succeeds.\n\nIf “usable” fails, watchdog logs why and falls back to Path B.\n\nEnable in `rust_watchdog.json`:\n\n```json\n{\n  \"enable_update_watch\": true,\n  \"update_check_interval_seconds\": 600,\n  \"update_check_timeout\": 60,\n\n  \"enable_smoothrestarter_bridge\": true,\n  \"smoothrestarter_restart_delay_seconds\": 300,\n  \"smoothrestarter_console_cmd\": \"srestart restart {delay}\",\n\n  \"update_watch_announce_message\": \"Update detected -- restart incoming.\",\n  \"update_watch_countdown_template\": \"Time until server update and restart: {seconds} seconds.\",\n  \"update_watch_final_message\": \"Server is restarting, come back in a few minutes!\",\n\n  \"update_watch_no_sr_countdown_seconds\": 30,\n  \"update_watch_no_sr_tick_seconds\": 10,\n\n  \"restart_request_cooldown_seconds\": 3600\n}\n```\n\n### SmoothRestarter file locations (defaults + overrides)\n\nBy default, under a standard LinuxGSM layout, watchdog expects:\n\n* `{server_dir}/serverfiles/oxide/plugins/SmoothRestarter.cs`\n* `{server_dir}/serverfiles/oxide/config/SmoothRestarter.json`\n\nThe watchdog treats the **plugin file** as the “installed” signal.\nThe config file may be missing on first run and that’s OK (it will log a note).\n\nIf your layout is custom, override paths in `rust_watchdog.json`:\n\n```json\n{\n  \"smoothrestarter_config_path\": \"\",\n  \"smoothrestarter_plugin_path\": \"\"\n}\n```\n\n* Leave them empty to use defaults.\n* If you set a relative path, it’s resolved relative to `server_dir`.\n* `~` and `$VARS` are expanded.\n\nWhen `enable_smoothrestarter_bridge=true`, the watchdog logs the expected SmoothRestarter paths on startup and prints the download URL if the plugin isn’t installed:\n[https://umod.org/plugins/smooth-restarter](https://umod.org/plugins/smooth-restarter)\n\nNote: the bridge sends commands via Rust WebRCON (requires `websocket-client`).\nRun the watchdog outside tmux/screen (systemd recommended) so recovery isn’t blocked by nested multiplexers.\n\n---\n\n## Telegram alerts setup\n\nThe watchdog can send alert messages via **Telegram Bot API** (outbound HTTPS).\n\n### 1) Create a Telegram bot (get a token)\n\n1. In Telegram, open **@BotFather**\n2. Run:\n\n   * `/newbot`\n   * pick a name + username\n3. BotFather will give you a token that looks like:\n\n   * `123456789:AA...`\n\nKeep that token secret.\n\n### 2) Get your `chat_id` (private chat or group)\n\n#### Option A: Private chat (simplest)\n\n1. Open your new bot in Telegram and press **Start** (or send any message).\n2. On the server, run:\n\n```bash\nexport TELEGRAM_BOT_TOKEN=\"123456789:AA...\"\ncurl -s \"https://api.telegram.org/bot${TELEGRAM_BOT_TOKEN}/getUpdates\" | jq\n```\n\nLook for something like:\n\n* `.result[].message.chat.id`\n\nYou can also extract the latest chat id quickly:\n\n```bash\ncurl -s \"https://api.telegram.org/bot${TELEGRAM_BOT_TOKEN}/getUpdates\" \\\n  | jq '.result[-1].message.chat.id'\n```\n\n#### Option B: Group chat\n\n1. Add the bot to your group.\n2. In the group, send a command so the bot definitely “sees” it (privacy mode won’t block commands):\n\n   * `/start`\n3. Then run the same `getUpdates` command above and read the group `chat.id` (usually a **negative** number).\n\n### 3) Quick “does Telegram even work from this server” test\n\n```bash\nexport TELEGRAM_BOT_TOKEN=\"123456789:AA...\"\nexport TELEGRAM_CHAT_ID=\"123456789\"   # or -1001234567890 for a group\n\ncurl -sS -X POST \"https://api.telegram.org/bot${TELEGRAM_BOT_TOKEN}/sendMessage\" \\\n  -d \"chat_id=${TELEGRAM_CHAT_ID}\" \\\n  --data-urlencode \"text=rust-linuxgsm-watchdog: test message\" \\\n  | jq\n```\n\nIf it returns `\"ok\": true`, you’re good.\n\n### 4) Store secrets safely (recommended)\n\nDon’t hardcode the token in a public config. Use an env file readable only by root:\n\n```bash\nsudo install -m 600 /dev/null /etc/default/rust-watchdog\nsudo nano /etc/default/rust-watchdog\n```\n\nPut:\n\n```bash\nTELEGRAM_BOT_TOKEN=\"123456789:AA...\"\nTELEGRAM_CHAT_ID=\"-1001234567890\"\n```\n\nThen in your `rust-watchdog.service`, add:\n\n```ini\nEnvironmentFile=/etc/default/rust-watchdog\n```\n\nReload + restart:\n\n```bash\nsudo systemctl daemon-reload\nsudo systemctl restart rust-watchdog.service\n```\n\n### 5) Configure the watchdog\n\n**Config key names depend on the version.** If you’re unsure, just search the code:\n\n```bash\ngrep -nRi \"telegram\" rust_watchdog.py\n```\n\nTypical configuration patterns look like one of these:\n\n#### Pattern A (flat keys)\n\n```json\n{\n  \"enable_alerts\": true,\n  \"alerts_backend\": \"telegram\",\n  \"telegram_bot_token\": \"$TELEGRAM_BOT_TOKEN\",\n  \"telegram_chat_id\": \"$TELEGRAM_CHAT_ID\"\n}\n```\n\n#### Pattern B (nested)\n\n```json\n{\n  \"alerts\": {\n    \"enabled\": true,\n    \"backend\": \"telegram\",\n    \"telegram\": {\n      \"bot_token\": \"$TELEGRAM_BOT_TOKEN\",\n      \"chat_id\": \"$TELEGRAM_CHAT_ID\"\n    }\n  }\n}\n```\n\n\u003e Note: This project expands `$VARS` in config strings, so using environment variables keeps secrets out of the JSON.\n\n### 6) Verify alerts end-to-end\n\nRun a one-shot cycle (or whatever minimal run you prefer) and watch logs:\n\n```bash\n./rust_watchdog.py --config ./rust_watchdog.json --once\n# or:\njournalctl -u rust-watchdog.service -f\n```\n\nIf Telegram is misconfigured, you should see a clear error (bad token/chat_id, blocked outbound HTTPS, etc.).\n\n---\n\n### History\n- v0.3.3\n  **Fixed / Added:**\n  - Prevent multiple watchdog instances from running at once (fixes “double processes” / duplicate recovery behavior).\n  - Added alerts support with Telegram backend (dedupe + cooldown; configurable titles/bodies/emoji).\n  - Discord and other API alert backends: sketched out in the code / WIP.\n- v0.3.0\n  **Fixed:**\n  - SmoothRestarter runtime-loaded checks no longer misread unrelated WebRCON frames (serverinfo/chat/keepalive).\n  - Reduced flakiness in RCON-based chat announcements and SR \"ceremony\" tests.\n  - WebRCON receive logic now ignores non-matching frames until deadline; failures are surfaced as a timeout error instead of returning random frames.\n- v0.2.9 - More detailed Smooth Restarter Oxide/Carbon checkup\n- v0.2.8 - Rudimentary checks on [Smooth Restarter](https://umod.org/plugins/smooth-restarter) integrity; more bug fixes\n- v0.2.7 - Small bugfixes\n- v0.2.6 - Implemented a standalone restart timer notification to the server when Smooth Restarter is not available and when we're watching for updates\n  - The watchdog is now calculating a countdown to Facepunch's forced wipe day (by default, the first Thursday of every month at 19:00 GMT); pending restarts over updates are on hold by default that day until we're past the expected update time.\n  - WIP: set wipe levels during forced wipe update-restarts.\n- v0.2.5 - Switched completely to RCON to interact with bridged Oxide plugins like Smooth Restarter\n- v0.2.4 - [Smooth Restarter](https://umod.org/plugins/smooth-restarter) bridge test (`--test-smoothrestarter` and `--test-smoothrestarter-send`)\n- v0.2.3 - initial support for bridging with [Smooth Restarter](https://umod.org/plugins/smooth-restarter)\n- v0.2.2 - server \u0026 plugin updates on restart can now be toggled\n- v0.2.1 - pre-flight checks, interruptible sleep, stop-aware recovery, stop escalation in run_cmd\n- v0.2.0 - stop flag + SIGTERM/SIGINT handler, TCP FAIL counts as DOWN (no “UNKNOWN forever”)\n- v0.1.0 - initial release\n\n---\n\n### About\n\nAs usual, code by [FlyingFathead](https://github.com/FlyingFathead/) with ChaosWhisperer meddling with the steering wheel.\n\nThis repo's official URL: [https://github.com/FlyingFathead/rust-linuxgsm-watchdog](https://github.com/FlyingFathead/rust-linuxgsm-watchdog)\n\n**If you like this repo, remember to give it a star. ;-) Thanks.**","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fflyingfathead%2Frust-linuxgsm-watchdog","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fflyingfathead%2Frust-linuxgsm-watchdog","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fflyingfathead%2Frust-linuxgsm-watchdog/lists"}