{"id":31956300,"url":"https://github.com/catsec/sitemonitor","last_synced_at":"2026-07-11T21:32:12.727Z","repository":{"id":315394067,"uuid":"1059300190","full_name":"catsec/sitemonitor","owner":"catsec","description":null,"archived":false,"fork":false,"pushed_at":"2025-09-18T11:03:13.000Z","size":14,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-09-18T11:37:31.623Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/catsec.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-09-18T09:04:42.000Z","updated_at":"2025-09-18T11:03:17.000Z","dependencies_parsed_at":"2025-09-18T20:33:24.287Z","dependency_job_id":null,"html_url":"https://github.com/catsec/sitemonitor","commit_stats":null,"previous_names":["catsec/sitemonitor"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/catsec/sitemonitor","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/catsec%2Fsitemonitor","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/catsec%2Fsitemonitor/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/catsec%2Fsitemonitor/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/catsec%2Fsitemonitor/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/catsec","download_url":"https://codeload.github.com/catsec/sitemonitor/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/catsec%2Fsitemonitor/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279019146,"owners_count":26086685,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-14T02:00:06.444Z","response_time":60,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-10-14T14:48:22.624Z","updated_at":"2026-07-11T21:32:12.703Z","avatar_url":"https://github.com/catsec.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Site Monitor\r\n\r\nA robust Python-based website monitoring service that watches for specific content patterns on websites and sends push notifications when found. Supports monitoring multiple URLs for multiple search terms simultaneously with intelligent duplicate prevention.\r\n\r\n## Features\r\n\r\n- **Multi-Site Monitoring**: Monitor multiple websites simultaneously\r\n- **Multi-Term Search**: Search for multiple content patterns per site\r\n- **Smart Text Matching**: Intelligent text normalization handles various formats\r\n- **Duplicate Prevention**: Never sends the same notification twice\r\n- **Parallel Processing**: Concurrent URL checking for optimal performance\r\n- **Push Notifications**: Real-time alerts via Pushover API\r\n- **Docker Ready**: Containerized deployment with health checks\r\n- **Security First**: URL validation with SSRF protection\r\n- **Resource Limits**: Built-in memory and content size protections\r\n\r\n## Quick Start\r\n\r\n### Prerequisites\r\n\r\n- Docker and Docker Compose\r\n- Pushover account ([sign up here](https://pushover.net/))\r\n\r\n### Installation\r\n\r\n1. Clone this repository\r\n2. Create your Pushover application and get your tokens\r\n3. Configure environment variables in `docker-compose.yml`\r\n4. Run the service\r\n\r\n```bash\r\n# Clone and enter directory\r\ngit clone github.com/catsec/sitemonitor\r\ncd sitemonitor\r\n\r\n# Edit docker-compose.yml with your settings\r\nnano docker-compose.yml\r\n\r\n# Start the service\r\ndocker-compose up -d --build\r\n\r\n# View logs\r\ndocker-compose logs -f\r\n```\r\n\r\n## Configuration\r\n\r\n### Required Environment Variables\r\n\r\n| Variable | Description | Example |\r\n|----------|-------------|---------|\r\n| `PUSHOVER_TOKEN` | Your Pushover application token | `abcdef123456...` |\r\n| `PUSHOVER_USER` | Your Pushover user key | `uvwxyz789012...` |\r\n| `MONITOR_URL` | Website(s) to monitor (comma-separated) | `https://store1.com,https://store2.com` |\r\n| `SEARCH_TEXT` | Search term(s) to find (comma-separated) | `iPhone 15,PlayStation 5` |\r\n\r\n### Optional Environment Variables\r\n\r\n| Variable | Default | Description |\r\n|----------|---------|-------------|\r\n| `CHECK_INTERVAL` | `300` | Check frequency in seconds |\r\n| `NOTIFICATION_TITLE` | `Content Found!` | Custom notification title |\r\n| `NOTIFICATION_PRIORITY` | `1` | Pushover priority (0-2) |\r\n| `NOTIFICATION_SOUND` | `magic` | Pushover notification sound |\r\n| `AUTO_STOP_ON_FOUND` | `true` | Stop after finding all items |\r\n| `USER_AGENT` | Browser string | Custom browser user agent |\r\n| `CUSTOM_HEADERS` | None | Additional HTTP headers (JSON) |\r\n| `DEBUG` | `false` | Enable debug logging |\r\n\r\n## Usage Examples\r\n\r\n### Monitor Single Site for One Product\r\n\r\n```yaml\r\nenvironment:\r\n  - MONITOR_URL=https://store.com/products/\r\n  - SEARCH_TEXT=iPhone 15 Pro\r\n```\r\n\r\n### Monitor Multiple Sites for Multiple Products\r\n\r\n```yaml\r\nenvironment:\r\n  - MONITOR_URL=https://store1.com,https://store2.com,https://store3.com\r\n  - SEARCH_TEXT=iPhone 15 Pro,PlayStation 5,RTX 4090\r\n```\r\n\r\nThis creates a 3×3 = 9 combination matrix. The service will:\r\n- Send individual notifications as each combination is found\r\n- Track progress (e.g., \"Found 5/9 combinations\")\r\n- Continue until all combinations are found\r\n- Send final completion summary\r\n\r\n### Custom Notification Settings\r\n\r\n```yaml\r\nenvironment:\r\n  - NOTIFICATION_TITLE=Product Alert!\r\n  - NOTIFICATION_PRIORITY=2\r\n  - NOTIFICATION_SOUND=siren\r\n  - AUTO_STOP_ON_FOUND=false  # Keep monitoring even after finding items\r\n```\r\n\r\n### Advanced HTTP Configuration\r\n\r\n```yaml\r\nenvironment:\r\n  - USER_AGENT=MyBot/1.0\r\n  - CUSTOM_HEADERS={\"Authorization\": \"Bearer token\", \"X-API-Key\": \"12345\"}\r\n```\r\n\r\n## How It Works\r\n\r\n### Smart Text Matching\r\n\r\nThe service uses intelligent text normalization that:\r\n- Converts text to lowercase\r\n- Removes punctuation and extra spaces\r\n- Handles various formats (e.g., \"iPhone-15-Pro\" matches \"iphone 15 pro\")\r\n- No complex regex knowledge required\r\n\r\n### Comprehensive Content Analysis\r\n\r\nSearches across all page content including:\r\n- Visible page text\r\n- Meta tags and page titles\r\n- Image alt text and titles\r\n- Link text and URLs\r\n- Data attributes\r\n- Form elements\r\n\r\n### Duplicate Prevention\r\n\r\n- Tracks every URL + search term combination\r\n- Prevents sending the same notification multiple times\r\n- Thread-safe for concurrent processing\r\n- Automatic retry if notification sending fails\r\n\r\n## Monitoring and Logs\r\n\r\n### View Live Logs\r\n\r\n```bash\r\ndocker-compose logs -f\r\n```\r\n\r\n### Check Service Status\r\n\r\n```bash\r\ndocker-compose ps\r\n```\r\n\r\n### Monitor Resource Usage\r\n\r\n```bash\r\ndocker stats sitemonitor\r\n```\r\n\r\n## Security Features\r\n\r\n- **SSRF Protection**: Blocks private network access\r\n- **Input Validation**: Sanitizes all user inputs\r\n- **Size Limits**: Prevents memory exhaustion\r\n- **Safe Parsing**: Validates JSON headers\r\n- **Non-Root Execution**: Container runs as unprivileged user\r\n\r\n## Troubleshooting\r\n\r\n### Service Won't Start\r\n\r\n1. Check your Pushover credentials are correct\r\n2. Verify URLs are accessible and valid\r\n3. Check logs: `docker-compose logs`\r\n\r\n### No Notifications Received\r\n\r\n1. Test Pushover credentials manually\r\n2. Enable debug logging: `DEBUG=true`\r\n3. Check if search terms exist on target pages\r\n4. Verify notification settings\r\n\r\n### High Memory Usage\r\n\r\n1. Reduce number of monitored URLs\r\n2. Increase check interval\r\n3. Monitor large sites less frequently\r\n\r\n### Common Error Messages\r\n\r\n| Error | Solution |\r\n|-------|----------|\r\n| `Invalid or unsafe URL` | Check URL format and accessibility |\r\n| `Configuration validation failed` | Verify required environment variables |\r\n| `Failed to fetch page` | Check network connectivity and URL validity |\r\n| `PUSHOVER_TOKEN environment variable is required` | Set Pushover credentials |\r\n\r\n## Development\r\n\r\n### Local Development\r\n\r\n```bash\r\n# Install dependencies\r\npip install -r requirements.txt\r\n\r\n# Set environment variables\r\nexport PUSHOVER_TOKEN=your_token\r\nexport PUSHOVER_USER=your_user\r\nexport MONITOR_URL=https://example.com\r\nexport SEARCH_TEXT=\"Product Name\"\r\n\r\n# Run directly\r\npython monitor.py\r\n```\r\n\r\n### Project Structure\r\n\r\n```\r\nsitemonitor/\r\n├── monitor.py          # Main application\r\n├── requirements.txt    # Python dependencies\r\n├── Dockerfile         # Container definition\r\n├── docker-compose.yml # Service configuration\r\n├── CLAUDE.md          # Development guidance\r\n└── README.md          # This file\r\n```\r\n\r\n## Resource Requirements\r\n\r\n- **Memory**: 256MB (configurable)\r\n- **CPU**: Minimal usage\r\n- **Network**: Outbound HTTPS access required\r\n- **Storage**: ~100MB for logs\r\n\r\n## License\r\n\r\nThis project is provided as-is for monitoring publicly accessible websites. Please respect robots.txt and website terms of service when using this tool.\r\n\r\n## Support\r\n\r\nFor issues and questions:\r\n1. Check the troubleshooting section above\r\n2. Review logs with `DEBUG=true`\r\n3. Verify configuration against examples\r\n\r\n4. Test with simple single-site setup first\r\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcatsec%2Fsitemonitor","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcatsec%2Fsitemonitor","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcatsec%2Fsitemonitor/lists"}