https://github.com/embmeals/homelabbackend
Homelab monitoring dashboard — .NET 10, Hangfire, HTMX, Chart.js
https://github.com/embmeals/homelabbackend
dashboard dotnet hangfire homelab htmx infrastructure monitoring razor-pages self-hosted sqlite
Last synced: 2 months ago
JSON representation
Homelab monitoring dashboard — .NET 10, Hangfire, HTMX, Chart.js
- Host: GitHub
- URL: https://github.com/embmeals/homelabbackend
- Owner: embmeals
- Created: 2026-03-31T14:26:57.000Z (3 months ago)
- Default Branch: main
- Last Pushed: 2026-04-06T18:00:36.000Z (3 months ago)
- Last Synced: 2026-04-11T03:40:00.391Z (3 months ago)
- Topics: dashboard, dotnet, hangfire, homelab, htmx, infrastructure, monitoring, razor-pages, self-hosted, sqlite
- Language: C#
- Homepage: https://homelab.emills.net
- Size: 584 KB
- Stars: 0
- Watchers: 0
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Homelab Backend
A .NET 10 application that monitors and manages a multi-machine homelab infrastructure. Built with Hangfire for job scheduling, Razor Pages + HTMX for a live-updating dashboard, and EF Core + SQLite for persistence.
**Live at** [homelab.emills.net](https://homelab.emills.net) (protected by Cloudflare Access)
## What It Does
Monitors a 3-machine homelab (Mac server, Windows PC, Linux laptop) running Plex, Radarr, Sonarr, Prowlarr, qBittorrent, Syncthing, and 12+ Docker containers. Replaces a collection of shell scripts and a Python monitor with a single, always-on service.
### Public Services
These are the services this app monitors — some are publicly accessible:
| Service | Public URL | Purpose |
|---------|----------------------------------------------------|---------|
| Overseerr | [requests.emills.net](https://requests.emills.net) | Request movies and TV shows |
| Tautulli | [stats.emills.net](https://stats.emills.net) | Plex viewing statistics |
| Immich | [photos.emills.net](https://photos.emills.net) | Photo library |
| This Dashboard | [homelab.emills.net](https://homelab.emills.net) | Monitoring dashboard |
| Portainer | [docker.emills.net](https://docker.emills.net) | Docker management |
*Protected services require email OTP via Cloudflare Access.*
## Media Pipeline
The core of the homelab is a fully automated media pipeline. Here's how a movie goes from request to screen:
```
You request a movie Radarr searches for Prowlarr feeds
through Overseerr → the best quality release → indexer results
(web UI) (Movies) / Sonarr (TV) to Radarr/Sonarr
↓ ↓ ↓
qBittorrent downloads Syncthing transfers Radarr/Sonarr imports,
behind Mullvad VPN → completed files from → renames, and organizes
on the Windows PC PC to Mac over LAN into the media library
↓ ↓ ↓
Plex scans the library Tautulli tracks viewing Bazarr downloads
and streams to any → stats and history → matching subtitles
device on any network automatically
```
**Why each service exists:**
- **Overseerr** — clean request UI so friends/family can request without touching Radarr/Sonarr
- **Radarr/Sonarr** — automates searching, downloading, and organizing movies/TV
- **Prowlarr** — single place to manage indexers, syncs to Radarr + Sonarr
- **qBittorrent** — download client, bound to Mullvad VPN for privacy
- **Syncthing** — real-time file sync between PC (downloads) and Mac (storage)
- **Plex** — media server with hardware transcoding, serves all devices
- **Tautulli** — Plex analytics and viewing history
- **Bazarr** — automated subtitle downloads for all media
**Why the PC downloads instead of the Mac:** The VPN protects download traffic. Keeping it on the PC isolates it from the server — if the VPN drops, only the PC is affected. Syncthing handles the transfer seamlessly.
## Dashboard
The dashboard updates every 30 seconds via HTMX partial swaps — no full page reloads.
- **Service health cards** — response time and status for each service
- **Download Pipeline** — real VPN + qBittorrent status via SSH to the PC
- **Disk usage** — progress bars with warning (85%) and critical (95%) thresholds
- **Torrent monitoring** — active downloads with stall/error detection
- **Syncthing sync state** — file counts, pending transfers, errors
- **Now Playing** — active Plex streams with user, device, and transcode status
- **Media Library** — movie/show/episode counts and recently added titles
- **Alerts banner** — aggregates critical and warning issues across all systems
- **Quick Actions** — one-click Plex scan, restarts, DNS flush, VPN control
- **Recent Events** — deduplicated service incidents in the last 24 hours
- **Response time chart** — 24-hour Chart.js trend graph
- **Uptime page** — 90-day per-service uptime with daily history bars
### Recurring Jobs
| Job | Frequency | What it checks |
|-----|-----------|--------------------------------------------------------------------------------|
| Service Health | 5 min | HTTP ping 7 services, measure response time |
| Torrent Monitor | 5 min | SSH to PC, query qBit for stalled/errored downloads |
| Syncthing | 5 min | Folder sync status via REST API |
| VPN Watchdog | 5 min | SSH to PC, detect Mullvad drop, auto-reconnect with cooldown |
| Remediation | 5 min | Match recent pipeline findings to rules, dispatch known fixes with audit |
| Alert Notifications | 5 min | Route critical findings through the notification router |
| Pipeline Health | 10 min | Drive mounts, qBit save path, arr queues, stale downloads, DNS, Plex libraries |
| Docker Health | 10 min | Colima VM + all 13 container statuses |
| Tunnel Health | 15 min | HTTP check all Cloudflare Tunnel endpoints |
| Request Fulfillment | 15 min | Track Overseerr requests end-to-end, flag any stuck > 48h |
| Disk Space | 30 min | `df` on Saved_Media (5TB) and Backups (1.8TB) |
| Media Library Stats | 6 hours | Movie/show/episode counts from Radarr + Sonarr APIs |
| Digest Flush | Daily 8 AM | Bundle warning-severity notifications into one email |
| Backup Health | Daily 9 AM | SSH to Mac, verify backup age + expected artifacts |
| Data Pruning | Daily 3 AM | Delete health check data older than 30 days across all tables |
| Docker Update Reporter | Daily 3:15 AM | Dashboard note of Watchtower container updates |
| Subtitle Gap | Daily 4 AM | Find missing subtitles in Bazarr, trigger searches |
| Gmail Cleanup | Daily 4:15 AM | Archive old promos/social, apply label rules |
| Immich Maintenance | Daily 4:30 AM | Photo library stats, duplicate detection report |
| Disk Usage Trend | Weekly Sun 3:30 AM | 30-day growth projection, alert if filling within 4 weeks |
| Library Integrity | Weekly Sun 5 AM | Reconcile Radarr/Sonarr against disk — orphans, ghosts, duplicates |
| Stale Media | Weekly Mon 3:45 AM | Flag unwatched media (>180 days), optional auto-unmonitor |
| Monthly SLO Report | Monthly 1st 8 AM | Per-service uptime, disk growth, library growth, remediation stats |
### Quick Actions
| Action | What it does |
|--------|-----------------------------------------------|
| Scan Plex | Trigger library rescan for new files |
| Restart Plex | Force-kill and restart Plex Media Server |
| Restart Radarr | Restart Radarr LaunchAgent |
| Flush DNS | Clear macOS DNS cache |
| Rescan Syncthing | Force folder re-check |
| Clean Downloads | Delete stale files (>24h) from PCDownloads |
| Start Downloads | Connect Mullvad VPN + start qBittorrent on PC |
| Stop VPN | Disconnect Mullvad VPN |
| Refresh Media Stats | Re-query Radarr/Sonarr for library counts |
## Infrastructure
```
┌─────────────────────────────────────────────────────┐
│ Mac (M3 Pro) — Primary Server, always on │
│ │
│ Native: Plex, Radarr, Sonarr, Prowlarr, AdGuard │
│ Docker: Overseerr, Tautulli, Bazarr, Immich, │
│ Portainer, Kometa, Recyclarr, Watchtower │
│ Infra: Cloudflare Tunnel, This Dashboard │
│ Storage: Saved_Media 5TB, Backups 1.8TB │
├─────────────────────────────────────────────────────┤
│ Windows PC (RTX 4060 Ti) — Download Relay │
│ │
│ qBittorrent (VPN-bound), Syncthing, Mullvad VPN │
│ Downloads here, syncs to Mac, never stores media │
├─────────────────────────────────────────────────────┤
│ Dell XPS 13 — On-demand dev/testing │
│ │
│ Ollama (local AI), Docker, Moonlight │
└─────────────────────────────────────────────────────┘
```
### Networking
- All machines on the same LAN subnet
- SSH key auth between all machines (Mac ↔ PC, Mac ↔ Dell, PC ↔ Dell)
- Cloudflare Tunnel exposes services at `*.emills.net` with email OTP protection
- Docker containers on a custom bridge network
## Why Hangfire
Every job in this app used to wrap its `ExecuteAsync` in a catch-all `try-catch` that logged the error and moved on. The job would silently fail, Hangfire would mark it as succeeded, and the bug would go unnoticed.
When we removed the catch-all try-catches and let exceptions propagate, Hangfire immediately surfaced a bug that had been hidden for weeks — `StaleMediaJob` was crashing because Tautulli returns Unix timestamps as JSON numbers, but the code called `GetString()` on them:
Hangfire gives you this for free: failed job visibility, stack traces, automatic retries (2 attempts), and manual requeue. Swallowing exceptions defeats all of it. The rule now: only catch exceptions at system boundaries (SSH, HTTP) for per-item isolation. Never wrap an entire job in a catch-all.
## Notifications
Every job that used to send email directly now routes through an `INotificationRouter` that dispatches by severity:
| Severity | Channels | When to use |
|----------|----------|-------------|
| `Critical` | Email + ntfy push | Paging events — service down, backup missing, VPN won't reconnect |
| `Warning` | Daily digest (one email at 8 AM) | Non-urgent — stale media report, disk trends, request stuck > 48h |
| `Info` | Dashboard only | Informational — successful remediations, Docker update summaries |
This keeps the inbox quiet. Critical events page immediately on the phone via ntfy and arrive as email. Warnings accumulate in the `PendingDigestNotifications` table and flush once daily via `DigestFlushJob`. Info-level events land in logs and dashboard surfaces without touching email at all.
ntfy is self-hosted alongside the other homelab containers and exposed via Cloudflare Tunnel. If `Ntfy.BaseUrl` isn't configured, `NtfyChannel` silently no-ops and critical alerts still arrive via email.
## Tech Stack
- **.NET 10** — ASP.NET Core Minimal APIs + Razor Pages
- **Hangfire** — recurring and fire-and-forget job scheduling (SQLite storage)
- **EF Core** — SQLite with migrations for health check history
- **HTMX** — live dashboard updates via partial view polling
- **Chart.js + Luxon** — response time trend visualization
- **Serilog** — structured logging with rolling file sink
- **xUnit + Moq + FluentAssertions** — unit tests
- **GitHub Actions** — CI on Ubuntu, deploy via self-hosted runner on Mac
## Architecture
```
[Browser] ← HTMX 30s polling → [Razor Pages + Minimal APIs]
↓
[Hangfire Jobs]
↙ ↓ ↘
[HTTP APIs] [SSH] [SQLite]
↓ ↓ ↓ ↓
Plex Radarr Mac PC health.db
Sonarr Prowlarr ↓ ↓ hangfire.db
Overseerr df mullvad
Tautulli docker qbit
Syncthing colima torrents
```
The app runs as a LaunchAgent on the Mac. When on macOS, shell commands execute locally. When running on Windows (dev), they SSH to the Mac. qBittorrent is accessed by SSH-ing to the PC and running curl locally (since qBit is bound to the VPN interface and unreachable over LAN).
### Project layout
```
src/homelabBackend/
├── Program.cs — 25-line composition root
├── Startup/ — extension methods called from Program.cs
│ ├── ConfigurationRegistrations.cs (IOptions bindings)
│ ├── PersistenceRegistrations.cs (DbContext + Hangfire)
│ ├── InfrastructureRegistrations.cs (SSH, HTTP, Email, Notifications)
│ ├── JobRegistrations.cs (Hangfire job DI)
│ ├── DashboardRegistrations.cs (Razor + Hangfire dashboard)
│ ├── StatusEndpoints.cs (read-only /api/*)
│ ├── ActionEndpoints.cs (mutating /api/actions/*)
│ └── RecurringJobSchedule.cs (cron wiring)
├── Jobs/ — one file per Hangfire job + OnDemandActions
├── Infrastructure/ — SSH, HTTP, Email, byte formatting
│ └── Notifications/ — router, channels, severity enum
├── Data/ — EF Core entities + HomelabDbContext
├── Models/
│ ├── Configuration/ — IOptions config types
│ ├── Dtos/ — external API + internal domain types
│ └── ViewModels/ — Razor page models
├── Migrations/ — auto-generated EF migrations
├── Pages/ — Razor dashboard
└── appsettings.json — tracked placeholders; real keys in Development.json
```
## Setup
```bash
# Clone and restore
git clone https://github.com/embmeals/homelabBackend.git
cd homelabBackend
dotnet restore
# Configure (copy and edit with your service URLs/keys)
cp src/homelabBackend/appsettings.json src/homelabBackend/appsettings.Development.json
# Run locally
dotnet run --project src/homelabBackend
# Run tests
dotnet test
# Deploy to Mac
bash scripts/deploy-to-mac.sh
```
## Configuration
All values are configurable via `appsettings.json`:
| Section | What it controls |
|---------|----------------------------------------------------------------------------|
| `HomelabServices` | Service URLs, health endpoints, API keys, per-service alert behavior |
| `QBittorrent` | qBit connection (URL, credentials) |
| `DiskCheck` | Drives to monitor, SSH hosts, warning/critical thresholds |
| `Syncthing` | API URL, key, folder ID |
| `Pipeline` | SSH hosts (Mac + PC), paths, Plex library section IDs |
| `Notifications` | Gmail SMTP credentials + per-alert cooldown minutes |
| `Ntfy` | Self-hosted ntfy base URL, topic, optional auth token |
| `Tautulli` | Base URL + API key for Plex watch-history lookups |
| `Bazarr` | Base URL + API key for subtitle-gap scanning |
| `Immich` | Base URL + API key for photo library stats |
| `Overseerr` | Base URL + API key, stuck-threshold hours, page size |
| `StaleMedia` | Unwatched-threshold days, auto-unmonitor toggle |
| `TunnelHealth` | List of Cloudflare Tunnel endpoints to probe |
| `BackupHealth` | SSH host, backups root path, warning/critical age, expected artifacts list |
| `VpnWatchdog` | Enable toggle, reconnect cooldown minutes |
| `Remediation` | Enable toggle, lookback minutes, per-pattern rules (check → action) |
| `LibraryIntegrity` | SSH host, Movies/TV root paths |
| `GmailCleanup` | IMAP credentials, archive-age, label rules |
| `JobSchedules` | Cron expressions for all recurring jobs |
Secrets go in `appsettings.Development.json` (gitignored). The tracked `appsettings.json` uses placeholder values.
## Deployment
Push to `main` triggers the CI/CD pipeline:
1. **CI** (Ubuntu) — restore, build, test
2. **Deploy** (self-hosted Mac runner) — only runs if CI passes, publishes ARM64 binary, deploys via rsync (preserves config + DB), restarts LaunchAgent