{"id":30420281,"url":"https://github.com/pexmor/urlmonitor","last_synced_at":"2025-08-22T08:17:50.745Z","repository":{"id":309551051,"uuid":"1035833762","full_name":"PexMor/urlmonitor","owner":"PexMor","description":"python","archived":false,"fork":false,"pushed_at":"2025-08-12T13:10:21.000Z","size":14,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-08-12T15:17:32.921Z","etag":null,"topics":["python","tls","tls-certificate","url-monitor"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/PexMor.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-08-11T06:52:38.000Z","updated_at":"2025-08-12T13:10:25.000Z","dependencies_parsed_at":"2025-08-12T15:17:45.494Z","dependency_job_id":"891ba896-8d3c-4585-8141-2efd0bbad1c5","html_url":"https://github.com/PexMor/urlmonitor","commit_stats":null,"previous_names":["pexmor/urlmonitor"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/PexMor/urlmonitor","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PexMor%2Furlmonitor","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PexMor%2Furlmonitor/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PexMor%2Furlmonitor/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PexMor%2Furlmonitor/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/PexMor","download_url":"https://codeload.github.com/PexMor/urlmonitor/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PexMor%2Furlmonitor/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":271606605,"owners_count":24788981,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-22T02:00:08.480Z","response_time":65,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["python","tls","tls-certificate","url-monitor"],"created_at":"2025-08-22T08:17:50.033Z","updated_at":"2025-08-22T08:17:50.733Z","avatar_url":"https://github.com/PexMor.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# URL Monitor\n\nA low-level, dependency-light HTTP/HTTPS monitor that makes real TCP/TLS connections, supports HTTP and SOCKS5 proxies, optional DNS overrides, and custom CA bundles. It logs timing, protocol, cipher, HTTP status, and rich certificate details for HTTPS.\n\n## Key features\n\n- **Direct, HTTP proxy (CONNECT), or SOCKS5** connectivity\n- **TLS details**: protocol, cipher, certificate subject/issuer/SANs/serial/fingerprint\n- **Hostname verification**: internal verifier (wildcard-aware) logged as part of cert info\n- **Configurable intervals** with duration strings (e.g., 750ms, 10s, 2m, 1h)\n- **Run-once mode**: if `--interval` is omitted, runs a single check and exits\n- **Custom CA bundle** support\n- **Structured logging** with JSON, KV, or colorful console output\n- **Log file rotation** in `~/.config/urlmonitor/logs`\n\n## How it works (HTTPS)\n\n1. Resolves and opens a TCP connection either directly, via HTTP proxy (and issues CONNECT), or via SOCKS5.\n2. Establishes a TLS session using SNI set to the URL hostname.\n3. Parses the peer certificate using the `cryptography` library (falls back to pyOpenSSL if needed) and computes SHA-256 fingerprint.\n4. Performs a minimal HTTP request (HEAD/GET) and captures the response status.\n5. Logs a structured record including timings and TLS/cert info.\n\nFor HTTP (non-TLS), a minimal HTTP request is sent directly or to the HTTP proxy (absolute-form URL) and the response status is logged.\n\n## Installation\n\n- Recommended: sync project dependencies\n\n```bash\nuv sync\n```\n\n- Requires Python 3.13+ (per `pyproject.toml`).\n\n## Usage\n\n- Run once (no interval):\n\n```bash\nuv run python urlmonitor.py --url https://example.com --timeout 5s\n```\n\n- Periodic monitoring:\n\n```bash\nuv run python urlmonitor.py --url https://example.com --timeout 5s --interval 60s\n```\n\n- Alternatively, use the helper wrapper script:\n\n```bash\n./murl --url https://example.com --timeout 5s --interval 60s\n```\n\nIf needed, make it executable first: `chmod +x ./murl`.\n\n## Parameters\n\n| Flag                 | Env var                 | Type / Default                             | Description                                                                                                                                                    | Example                                    |\n| -------------------- | ----------------------- | ------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------ |\n| `-c, --config`       | —                       | Path (none)                                | Optional config file (YAML supported if PyYAML installed). Defaults to `~/.config/urlmonitor/config.yaml` if present.                                          | `--config ./config.yaml`                   |\n| positional `url_pos` | `URLMONITOR_URL`        | String (none)                              | URL to check (http or https). If both positional and `--url` are set, `--url` wins.                                                                            | `https://example.com`                      |\n| `--url`              | `URLMONITOR_URL`        | String (none)                              | URL to check (http or https).                                                                                                                                  | `--url https://example.com`                |\n| `--interval`         | `URLMONITOR_INTERVAL`   | Duration (none)                            | Interval between checks. If omitted, runs once and exits.                                                                                                      | `--interval 30s`                           |\n| `--timeout`          | `URLMONITOR_TIMEOUT`    | Duration (`5s`)                            | Socket/TLS timeout.                                                                                                                                            | `--timeout 750ms`                          |\n| `--proxy`            | `URLMONITOR_PROXY`      | String (none)                              | Proxy URL. Supports `http://user:pass@host:port` and `socks5://user:pass@host:port`.                                                                           | `--proxy http://user:pass@127.0.0.1:8080`  |\n| `--method`           | `URLMONITOR_METHOD`     | `HEAD` or `GET` (`GET`)                    | HTTP method used for the probe.                                                                                                                                | `--method GET`                             |\n| `--log-format`       | `URLMONITOR_LOG_FORMAT` | one of `json`, `kv`, `console` (`console`) | Log output format.                                                                                                                                             | `--log-format console`                     |\n| `--log-file`         | `URLMONITOR_LOG_FILE`   | Path (default under data dir)              | Explicit log file path; otherwise logs to `~/.config/urlmonitor/logs/monitor.log`.                                                                             | `--log-file ./monitor.log`                 |\n| `--data-dir`         | `URLMONITOR_DATA_DIR`   | Path (`~/.config/urlmonitor`)              | Base data directory; logs are saved under `\u003cdata-dir\u003e/logs`.                                                                                                   | `--data-dir ~/.config/urlmonitor`          |\n| `--connect-ip`       | `URLMONITOR_CONNECT_IP` | IPv4/IPv6 (none)                           | For direct connections only, connect to this literal IP instead of resolving the URL hostname (SNI still uses the URL hostname). Ignored when a proxy is used. | `--connect-ip 93.184.216.34`               |\n| `--ca-bundle`        | `URLMONITOR_CA_BUNDLE`  | Path (none)                                | Custom CA bundle path for TLS verification.                                                                                                                    | `--ca-bundle /etc/ssl/certs/ca-bundle.crt` |\n\nNotes:\n\n- Durations accept `ms`, `s`, `m`, `h`, `d` (e.g., `500ms`, `2m30s`, `1h`). Bare numbers are treated as seconds.\n- Exit status `2` is used for configuration/validation errors (e.g., bad duration, missing CA bundle path, bad proxy URL).\n\n## Configuration file (YAML)\n\nIf `PyYAML` is installed, you can use a YAML config and invoke with `--config` (or rely on the default path if present). Keys are the long option names:\n\n```yaml\nurl: https://httpbin.org/get\ntimeout: 5s\n# interval omitted -\u003e run once\nmethod: HEAD\nlog-format: kv\ndata-dir: ~/.config/urlmonitor\n# log-file: ./monitor.log\n# proxy: http://user:pass@127.0.0.1:8080\n# connect-ip: 93.184.216.34\n# ca-bundle: /etc/ssl/certs/ca-bundle.crt\n```\n\n## Proxy behavior\n\n- **HTTP proxy**: For HTTPS targets, the monitor sends `CONNECT host:port` first, then negotiates TLS through the tunnel. For HTTP targets, it sends an absolute-form request to the proxy. If `user:pass` is present in the proxy URL, `Proxy-Authorization: Basic …` is added automatically.\n- **SOCKS5**: The monitor connects to the target host:port via the SOCKS5 proxy. Authentication is supported when provided in the URL.\n- `--connect-ip` is ignored when any proxy is configured.\n\n## TLS and certificate logging\n\n- TLS is negotiated with SNI set to the URL hostname.\n- Certificate fields logged: `subject`, `issuer`, `serial_hex`, `not_before`, `not_after`, `sans`, `fingerprint_sha256`.\n- Hostname verification is performed internally (supports single-label wildcards like `*.example.com`) and results are logged as `hostname_ok` and `hostname_error`.\n\n## Logs\n\n- Default log file: `~/.config/urlmonitor/logs/monitor.log` (rotates at ~5 MB, keeps 3 backups).\n- Formats:\n\n  - `kv`: key=value pairs\n  - `json`: structured JSON\n  - `console`: colorful developer-friendly output\n\n- Console output respects `--log-format`. File logs are always JSON regardless of console format.\n\nExample (KV format, HTTPS):\n\n```log\nts='2025-08-08T18:36:19.475234Z' level='info' event='https_check' url='https://httpbin.org/get' host='httpbin.org' port=443 via_proxy=False tls_ms=387 tls_protocol='TLSv1.2' tls_cipher='ECDHE-RSA-AES128-GCM-SHA256' http_status=200 http_status_line='HTTP/1.1 200 OK' tls_sni='httpbin.org' cert={'subject': 'CN=httpbin.org', 'issuer': 'C=US/O=Amazon/CN=Amazon RSA 2048 M03', 'serial_hex': '…', 'not_before': '20250720000000Z', 'not_after': '20260817235959Z', 'sans': ['DNS:httpbin.org', 'DNS:*.httpbin.org'], 'fingerprint_sha256': '…', 'hostname_to_verify': 'httpbin.org', 'hostname_ok': True, 'hostname_error': None}\n```\n\n```json\n{\n  \"url\": \"https://httpbin.org/get\",\n  \"interval\": null,\n  \"interval_s\": null,\n  \"timeout\": \"5s\",\n  \"timeout_s\": 5.0,\n  \"via_proxy\": false,\n  \"proxy_scheme\": null,\n  \"log_format\": \"console\",\n  \"method\": \"GET\",\n  \"data_dir\": \"~/.config/urlmonitor\",\n  \"log_file\": \"~/.config/urlmonitor/logs/monitor.log\",\n  \"config_file\": null,\n  \"connect_ip\": null,\n  \"ca_bundle\": null,\n  \"event\": \"start\",\n  \"level\": \"info\",\n  \"ts\": \"2025-08-12T13:08:16.375530Z\"\n}\n{\n  \"url\": \"https://httpbin.org/get\",\n  \"host\": \"httpbin.org\",\n  \"port\": 443,\n  \"via_proxy\": false,\n  \"proxy_scheme\": null,\n  \"connect_ip\": null,\n  \"tcp_ms\": 43,\n  \"tls_ms\": 388,\n  \"tls_protocol\": \"TLSv1.2\",\n  \"tls_cipher\": \"ECDHE-RSA-AES128-GCM-SHA256\",\n  \"http_status\": 200,\n  \"http_status_line\": \"HTTP/1.1 200 OK\",\n  \"tls_sni\": \"httpbin.org\",\n  \"ca_bundle\": null,\n  \"cert\": {\n    \"subject\": \"commonName=httpbin.org\",\n    \"issuer\": \"countryName=US/organizationName=Amazon/commonName=Amazon RSA 2048 M03\",\n    \"serial_hex\": \"e2558d492728e9c01a8dadedc05d13d\",\n    \"not_before\": \"20250720000000Z\",\n    \"not_after\": \"20260817235959Z\",\n    \"sans\": [\n      \"DNS:httpbin.org\",\n      \"DNS:*.httpbin.org\"\n    ],\n    \"fingerprint_sha256\": \"10E5C3BD42720A3E3AEBE14D0476E7F906449A09F7C689126619AEEC5701F045\",\n    \"hostname_to_verify\": \"httpbin.org\",\n    \"hostname_ok\": true,\n    \"hostname_error\": null\n  },\n  \"elapsed_ms\": 5060,\n  \"event\": \"https_check\",\n  \"level\": \"info\",\n  \"ts\": \"2025-08-12T13:08:21.437434Z\"\n}\n```\n\n## Tips\n\n- For single-run checks, omit `--interval` (the monitor will exit after one request).\n- For periodic monitoring, set `--interval` to your desired frequency.\n- Use `--ca-bundle` when monitoring endpoints with private or custom CAs.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpexmor%2Furlmonitor","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpexmor%2Furlmonitor","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpexmor%2Furlmonitor/lists"}