{"id":30304576,"url":"https://github.com/essogbe/zangbeto","last_synced_at":"2025-10-11T12:22:09.866Z","repository":{"id":307375033,"uuid":"1029304859","full_name":"Essogbe/Zangbeto","owner":"Essogbe","description":"Simple python project for periodically monitoring sites in background  and generate report","archived":false,"fork":false,"pushed_at":"2025-08-05T19:44:50.000Z","size":243,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-08-17T07:15:18.662Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Essogbe.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-07-30T20:47:52.000Z","updated_at":"2025-08-05T19:44:53.000Z","dependencies_parsed_at":"2025-07-31T00:34:16.398Z","dependency_job_id":null,"html_url":"https://github.com/Essogbe/Zangbeto","commit_stats":null,"previous_names":["essogbe/zangbeto"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/Essogbe/Zangbeto","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Essogbe%2FZangbeto","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Essogbe%2FZangbeto/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Essogbe%2FZangbeto/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Essogbe%2FZangbeto/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Essogbe","download_url":"https://codeload.github.com/Essogbe/Zangbeto/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Essogbe%2FZangbeto/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279007151,"owners_count":26084247,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-11T02:00:06.511Z","response_time":55,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-08-17T07:10:08.672Z","updated_at":"2025-10-11T12:22:09.847Z","avatar_url":"https://github.com/Essogbe.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Zangbéto – Night Guardian\n\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"https://encrypted-tbn0.gstatic.com/images?q=tbn:ANd9GcREvNSSk-Op-0QgGDqch5TZujeZ2Ys-q6hZaw\u0026s\" alt=\"Zangbéto – Night Guardian\" width=\"600\"/\u003e\n\u003c/p\u003e\n\n[Zangbéto](https://en.wikipedia.org/wiki/Zangbeto) is a lightweight and local website monitoring tool. It periodically checks the availability of static and dynamic pages, generates detailed HTML reports with **site-by-site analysis**, and sends intelligent notifications when incidents occur or connectivity is restored.\n\n## How It Works\n\nZangbéto operates as a **recursive web crawler** that monitors your websites' health automatically. Here's what happens during each monitoring cycle:\n\n1. **Connectivity Check**: Before each monitoring cycle, verifies internet connectivity using multiple reliable endpoints to avoid false alerts during network outages\n\n2. **Site Discovery**: Starting from your configured URLs (in `sites.txt`), Zangbéto explores each website by following internal links up to a configurable depth (default: 2 levels)\n\n3. **Health Checks**: For every discovered page, it performs HTTP requests measuring:\n   - Response time (how fast your site responds)\n   - Status codes (200 OK, 404 Not Found, 500 errors, etc.)\n   - Availability (can the page be reached?)\n\n4. **Data Storage**: All results are stored in a local SQLite database with timestamps and connectivity status, creating a comprehensive historical record\n\n5. **Enhanced Report Generation**: After each check, generates an interactive HTML report featuring:\n   - **Site-by-site analysis** with individual success rates and metrics\n   - **Global overview** with total success/failure statistics\n   - **Status code distribution per domain** (pie charts)\n   - **Historical trends per site** showing evolution over time\n   - **Incident tracking** with recent failures per domain\n   - **12-hour trend analysis** (up/down over time)\n   - **Detailed page-by-page results** grouped by domain\n\n6. **Smart Notifications**: \n   - **Instant alerts** for site failures with detailed error information\n   - **Connectivity alerts** when internet connection is lost/restored\n   - **Scheduled reports** (every 12 hours by default) with clickable notifications\n   - **Multi-channel support**: system notifications, email, Slack, Telegram\n\n**Example scenario**: You add `https://mycompany.com` to your sites list. Zangbéto will check the homepage, then follow links to `/about`, `/contact`, `/products`, etc., testing each page every 30 minutes. The enhanced report shows each domain separately with individual success rates, recent incidents, and trends. If connectivity is lost, you'll be notified and monitoring will pause until restored.\n\n## 📦 Tech Stack\n\n* **Language**: Python 3+\n* **HTTP**: `requests`\n* **Parsing**: `BeautifulSoup4`\n* **Scheduling**: `schedule` (portability)\n* **Database**: SQLite (`sqlite3`)\n* **Templates**: Jinja2\n* **Charts**: Plotly.js (via CDN)\n* **Notifications**: Multi-channel via `notify.py` (system, email, Slack, Telegram)\n* **Linux Supervision**: systemd (unit + timer)\n\n**Note**: `notify-send` is available on Linux desktops (default on most distributions)\n\n### Installing notify-send\n\n**Ubuntu**\n```bash\nsudo apt install libnotify-bin\n```\n\n**Fedora**\n```bash\nsudo dnf install libnotify\n```\n\n**Arch Linux**\n```bash\nsudo pacman -Sy libnotify\n```\n\n## Project Structure\n\n```\nZangbeto/\n├── main.py                          # Main script and CLI with advanced options\n├── crawler.py                       # Web exploration, HTTP tests, enhanced reporting\n├── notify.py                        # Multi-channel notification manager\n├── templates/\n│   ├── report_template.html         # Legacy template (simple)\n│   └── enhanced_report_template.html # New enhanced template (site-by-site)\n├── history.db                       # SQLite database for check history\n├── requirements.txt                 # Python dependencies\n├── setup.sh                         # Automated setup and systemd configuration\n└── README.md                        # Documentation and usage guide\n```\n\n## Quick Start Guide\n\n1. **Clone and setup**:\n   ```bash\n   git clone https://github.com/yourusername/zangbeto\n   cd zangbeto\n   python3 -m venv venv\n   source venv/bin/activate\n   pip install -r requirements.txt\n   ```\n\n2. **Configure sites** - Create `sites.txt`:\n   ```bash\n   echo \"https://your-website.com\" \u003e sites.txt\n   echo \"https://your-api.com\" \u003e\u003e sites.txt\n   ```\n\n3. **Test the setup**:\n   ```bash\n   # Quick test - single check with enhanced report\n   python main.py --one-shot\n   ```\n\n4. **Start monitoring**:\n   ```bash\n   # Continuous monitoring with enhanced reporting\n   python main.py\n   ```\n\nThat's it! Open the generated `rapport.html` in your browser to see the enhanced monitoring results with site-by-site analysis.\n\n## Configuration\n\n### Sites Configuration\nCreate a `sites.txt` file with URLs to monitor (one per line):\n```\nhttps://example.com\nhttps://mycompany.org\nhttps://api.myservice.net\n```\n\n### CLI Arguments\n\n**Basic Options:**\n* `-f`, `--frequency`: Check interval in minutes (default: 30)\n* `-o`, `--output`: HTML report file path (default: `rapport.html`)\n* `-i`, `--interval`: Interval in hours for complete report notifications (default: 12)\n\n**Execution Modes:**\n* `-c`, `--count`: Run a specific number of check cycles then exit (default: infinite)\n* `--one-shot`: Run a single check and exit (equivalent to `--count 1`)\n\n**Connectivity Options:**\n* `--skip-connectivity`: Skip internet connectivity checks (use with caution)\n* `--connectivity-wait`: Minutes to wait for connectivity restoration (default: 2)\n\n**Historical Analysis:**\n* `--history`: Enable historical data retrieval mode\n* `--start`: Start date for historical data (ISO format or predefined: today, yesterday, last_7d, last_30d, etc.)\n* `--end`: End date for historical data (ISO format, optional)\n\n### Environment Variables (Optional)\nFor extended notifications, create a `.env` file:\n```bash\n# Email notifications\nSMTP_HOST=smtp.gmail.com\nSMTP_PORT=587\nSMTP_USER=your-email@gmail.com\nSMTP_PASSWORD=your-app-password\n\n# Slack notifications\nSLACK_WEBHOOK_URL=https://hooks.slack.com/services/...\n\n# Telegram notifications\nTELEGRAM_BOT_TOKEN=123456789:ABC-DEF...\nTELEGRAM_CHAT_ID=123456789\n```\n\n## Usage\n\n### Continuous Monitoring \n```bash\n# Start continuous monitoring with enhanced reports\npython main.py\n\n# Custom frequency - check every 10 minutes\npython main.py --frequency 10\n\n# Custom report interval - send reports every 6 hours\npython main.py --interval 6\n\n# Monitor with custom connectivity settings\npython main.py --connectivity-wait 5\n```\n\n### Limited Monitoring (Testing)\n```bash\n# Quick test - single check with enhanced report\npython main.py --one-shot\n\n# Run exactly 5 check cycles then exit\npython main.py --count 5\n\n# Test with custom settings and enhanced reporting\npython main.py --one-shot --frequency 1 --output enhanced_test.html\n```\n\n### Historical Analysis\n```bash\n# Generate report for the last 7 days\npython main.py --history --start last_7d\n\n# Generate report for a specific period\npython main.py --history --start 2023-10-01T00:00:00 --end 2023-10-07T23:59:59\n\n# Generate report for predefined periods\npython main.py --history --start today\npython main.py --history --start yesterday\npython main.py --history --start last_30d\n```\n\n### Output Examples\n* **Enhanced Reports**: Interactive HTML files with site-by-site analysis, trends, and incident tracking\n* **Smart Notifications**: \n  - Instant alerts for site failures with detailed error info\n  - Connectivity status notifications (lost/restored)\n  - Clickable report notifications that open in browser\n* **Console**: Real-time progress with connectivity status and detailed logging\n\n**Example console output:**\n```\n2025-01-30 10:30:15 - INFO - Starting check job #1 of 3\n2025-01-30 10:30:16 - INFO - Internet connectivity confirmed via https://www.google.com\n2025-01-30 10:30:18 - INFO - Loaded 2 sites to monitor\n2025-01-30 10:30:22 - INFO - Site exploration completed: 8 pages found\n2025-01-30 10:30:23 - INFO - Analyzed 2 sites with 8 unique URLs\n2025-01-30 10:30:23 - INFO - All sites are healthy\n2025-01-30 10:30:24 - INFO - Enhanced HTML report generated successfully\n```\n\n## 🔧 systemd Integration (Linux)\n\nFor background execution with automatic startup and management:\n\n**Automated Setup:**\n```bash\nchmod +x setup.sh\nsudo ./setup.sh\n```\n\n**Manual Setup:**\n1. **Create service files**:\n   ```bash\n   sudo cp systemd/zangbeto.service /etc/systemd/system/\n   sudo cp systemd/zangbeto.timer /etc/systemd/system/\n   ```\n\n2. **Enable and start**:\n   ```bash\n   sudo systemctl daemon-reload\n   sudo systemctl enable zangbeto.timer\n   sudo systemctl start zangbeto.timer\n   ```\n\n3. **Monitor and manage**:\n   ```bash\n   # Check timer status\n   systemctl list-timers --all | grep zangbeto\n   \n   # View logs with connectivity info\n   journalctl -u zangbeto.service -f\n   \n   # Manual run\n   sudo systemctl start zangbeto.service\n   ```\n\n## Key Features \u0026 Improvements\n\n###  Enhanced Reporting\n- **Site-by-site analysis** with individual success rates and response times\n- **Visual availability badges** (high/medium/low availability)\n- **Status code distribution per domain** with color-coded pie charts\n- **Historical trend analysis per site** showing evolution over time\n- **Recent incident tracking** with detailed error information\n- **Responsive design** optimized for desktop and mobile viewing\n\n###  Smart Connectivity Handling\n- **Pre-check connectivity verification** using multiple reliable endpoints\n- **Automatic waiting** for connectivity restoration (configurable timeout)\n- **Connectivity status notifications** (lost/restored alerts)\n- **Database logging** of connectivity issues for analysis\n- **Graceful degradation** during network outages\n\n###  Historical Analysis\n- **Flexible date range queries** with predefined periods (today, yesterday, last_7d, etc.)\n- **Historical report generation** with custom time ranges\n- **Trend analysis** showing site performance over time\n- **Incident correlation** across different time periods\n\n###  Advanced Notifications\n- **Multi-channel notification system** (system, email, Slack, Telegram)\n- **Intelligent failure alerts** with detailed error information\n- **Connectivity status notifications** for network issues\n- **Clickable report notifications** that open reports in browser\n- **Asynchronous notification handling** to avoid blocking main process\n\n###  Enhanced Data Management\n- **Connectivity status tracking** in database schema\n- **Improved data organization** with proper foreign key relationships\n- **Automatic database migration** for existing installations\n- **Comprehensive logging** with debug information\n\n## What About Testing APIs (REST APIs for Now)?\n\nThat's a great question because nowadays, most modern web applications are split into two main components: the backend (API) and the frontend.\n\nWhen it comes to monitoring or testing APIs in production, there are generally two main approaches:\n\n1. **Direct API testing**, where requests are made directly to the live endpoints.  \n   This method carries risks such as:\n   - Modifying production data\n   - Triggering authentication or authorization issues  \n   While manageable, it can raise security and data integrity concerns.\n\n2. **Internal health-check endpoints**, where one or more special routes are created specifically for monitoring purposes.  \n   These endpoints:\n   - Perform internal checks on other routes or logic\n   - Return a summary or detailed result about system health  \n\nThe second approach is often safer and cleaner:\n- No side effects on production data\n- Fewer security concerns\n- Easier to automate and monitor\n\nFuture versions of this project aim to support both website and API monitoring, including health-check mechanisms for REST APIs.\n\n## Alternatives\n\nYeah, there are plenty of alternatives out there. For medium to large teams, robust monitoring and alerting tools like **[Grafana](https://github.com/grafana/grafana)**, **[Prometheus](https://github.com/prometheus/prometheus)**, and others are already widely used—and they do their job very well.\n\nThis project is not meant to be a drop-in replacement for those tools (just look at how small it is, haha). Instead, it's a lightweight solution I built to quickly check if my websites are down, even quietly in the background.\n\nThat said, I'm always open to better suggestions and contributions.\n\n## 📝 Roadmap (v2+)\n\n**Core Improvements:**\n* ✅ **Enhanced site-by-site reporting** (v1.1)\n* ✅ **Smart connectivity handling** (v1.1)\n* ✅ **Historical data analysis** (v1.1)\n* ✅ **Multi-channel notifications** (v1.1)\n* Unit tests and CI/CD pipeline\n* Asynchronous crawler for better performance\n* Dynamic pages support (Playwright/Selenium)\n* Advanced retry logic and timeout handling\n\n**Reporting \u0026 Visualization:**\n* ✅ **Improved UI with site-by-site analysis** (v1.1)\n* ✅ **Incident tracking and trends** (v1.1)\n* Automatic PDF export\n* Real-time web dashboard (Flask/React)\n* Custom alert thresholds and rules\n* SLA tracking and uptime calculations\n* Custom templates definition and usage\n\n**Notifications \u0026 Integrations:**\n* ✅ **Multi-channel support (Email, Slack, Telegram)** (v1.1)\n* ✅ **Smart connectivity notifications** (v1.1)\n* Other channels support (SMS, Discord, Teams)\n* Webhook support for custom integrations\n* Alert escalation policies\n* Maintenance windows and alert suppression\n\n**Other Features:**\n* Multi-user authentication and role management\n* API monitoring with JSON/XML validation\n* Protected sites monitoring (login flows)\n\n**Platform \u0026 Deployment:**\n* Windows support and native installers\n\n**Data Management:**\n* ✅ **Enhanced database schema with connectivity tracking** (v1.1)\n* ✅ **Flexible historical data queries** (v1.1)\n* Advanced retention policies and data purging \n* Performance metrics and trends analysis\n\n---\n\n*Zangbéto – Version 1.1.0*","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fessogbe%2Fzangbeto","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fessogbe%2Fzangbeto","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fessogbe%2Fzangbeto/lists"}