{"id":13573927,"url":"https://github.com/AssuranceMaladieSec/CertStreamMonitor","last_synced_at":"2025-04-04T13:30:29.608Z","repository":{"id":46948308,"uuid":"119042047","full_name":"AssuranceMaladieSec/CertStreamMonitor","owner":"AssuranceMaladieSec","description":"Monitor certificates generated for specific domain strings and associated, store data into sqlite3 database, alert you when sites come online.","archived":false,"fork":false,"pushed_at":"2024-09-24T15:47:23.000Z","size":145,"stargazers_count":136,"open_issues_count":6,"forks_count":30,"subscribers_count":11,"default_branch":"master","last_synced_at":"2024-11-05T08:37:18.643Z","etag":null,"topics":["certificate-transparency","certstream","threat-intelligence"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/AssuranceMaladieSec.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-01-26T11:04:04.000Z","updated_at":"2024-09-25T15:54:34.000Z","dependencies_parsed_at":"2024-01-16T20:25:29.684Z","dependency_job_id":"97d16230-b35e-4058-acef-a57196fec61e","html_url":"https://github.com/AssuranceMaladieSec/CertStreamMonitor","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AssuranceMaladieSec%2FCertStreamMonitor","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AssuranceMaladieSec%2FCertStreamMonitor/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AssuranceMaladieSec%2FCertStreamMonitor/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AssuranceMaladieSec%2FCertStreamMonitor/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/AssuranceMaladieSec","download_url":"https://codeload.github.com/AssuranceMaladieSec/CertStreamMonitor/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247184853,"owners_count":20897844,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["certificate-transparency","certstream","threat-intelligence"],"created_at":"2024-08-01T15:00:43.671Z","updated_at":"2025-04-04T13:30:29.217Z","avatar_url":"https://github.com/AssuranceMaladieSec.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# CertStreamMonitor\n\nMonitor certificates generated for specific domain strings and associated, store data into sqlite3 database, alert you when sites come online.\n\nCertStreamMonitor architecture relies on 3 scripts :\n\n- `certstreammonitor.py`\n  - this script runs as a daemon.\n  - reading the certstream feed from [CertStream service](https://certstream.calidog.io/) provided by [Calidog Security](https://calidog.io/) (thanks so much for this!), it selects hostnames covered by certificates that match your criteria (SearchKeyWords parameter in conf).\n  - it writes these hostnames along with its certificate relevant informations to the database.\n- `scanhost.py`\n  - this script can be executed as often as you like.\n  - it checks if site corresponding to the hostanme stored in DB is UP ot not.\n  - it collects informations about the sites that are up to DB and to a JSON file.\n- `gethost.py`\n   - Due to [@nbeguier](https://github.com/nbeguier) contribution, the project has also a `gethost.py` script that provides a way for security operators to request the last hostnames which have been detected by `certstreammointor.py` since X seconds.\n- `check_rules.py`\n   - Due to [@nbeguier](https://github.com/nbeguier) contribution, the project has also a `check_rules.py` script that provides a way for security operators to check their own rules, especially SearchKeywords, DetectionThreshold and BlacklistKeywords.\n\n## Features\n- **Monitoring:**\n  - monitor `wss://certstream.calidog.io` CT logs aggregator server with certstream-python (see [certstream-python](https://github.com/CaliDog/certstream-python)), but you can choose, and operate, your own server (see [certstream-server](https://github.com/CaliDog/certstream-server/)).\n  - choose strings you want to monitor in `Subject Alt Names` field of certificates\n- **Storing:**\n  - store hostnames found along with its certificate relevant data into a sqlite3 database\n- **Alerting:**\n  - for each hostname not already flagged as up : check if corresponding site is up or not\n    - if it's up :\n      - collects informations (IP address, AS informations, HTTP code, web page title, abuse email, (optional) google safe browsing status)\n      - write them to a JSON file in the `/alerts` directory (default value) to push forward investigation.\n      - (optional) push them to a destination (through apprise package) such as an email address, a Slack channel or even Twitter account\n      - flags the hostname in the DB as up\n    - if it's not :\n      - the hostname will be checked until the Alert_Monitor_timelapse configuration variable (in days) will be reached\n      - when the limit is reached a string like `Stop checking on 2019-09-27` is included into the StillInvestig column\n\n## Requirements\n- Python 3\n- certstream\n- sqlite3\n- ipwhois\n- PySocks\n- hues\n- websocket-client\n- apprise\n\n## Install\nInstall the requirements\n~~~\n$ pip3 install -r requirements.txt\n~~~\n\n## Configuration file\nYou can find a configuration file example in 'conf' directory.\nConfigurable parameters are:\n- `SearchKeywords`: Keywords to look for (with '|' (or) as separator)\n- `DetectionThreshold`: set the minimum number of detected SearchKeywords in a hostname before writing it to DB. Under this value but above zero, detected hostnames are only written to logfile. Default value: 2.\n- `DBFile`: SQLite3 database file (the path and file will be created if don't exist)\n- `TABLEname`: The name of the database table\n- `LogFile`: The logging file (the path and file will be created if don't exist)\n- `UAfile`: you can provide a User-Agent file to masquerade this value of requests (random change for each request)\n- `Alerts_dir`: you can specify where JSON alert files are written\n   You can use the following strings to add time/date hashed based subdirectories:\n   %%m -\u003e month, %%d -\u003e day, %%Y -\u003e year, %%H -\u003e hour, %%M -\u003e minute.\n   Example: Alerts_dir = ./alerts/%%Y/%%m/%%d\n\nOptional:\n- `Proxy`: allows to give a SOCKS or HTTP proxy to process your scanhost.py's requests (as Tor)\n- `Proxy_*` parameters : allow you to specify HTTP proxy informations (server, port[, user, password]) for CertStreamMonitor.py script to connect to the CT logs aggregator server.\n- `ACTServer`: you can specify the CT logs aggregator server of your choice. By default, it is the server run by Calidog Security.\n- `Safe_Browsing_API_Key`: indicate (if you want) your Google Safe Browsing API key in order to check hostnames that are UP against Google Safe Browsing Lookup API ([How-To get an API key](https://developers.google.com/safe-browsing/v4/get-started) for the Safe Browsing Lookup API).\n- `Notification_Destination`: specify a notification destination as attended by apprise package. Documentation about the format of this parameter is available on the [apprise Github page](https://github.com/caronc/apprise).\n- `BlacklistKeywords`: Keywords to ignore matched hosts (with '|' (or) as separator)\n\n## Usage\n\n### CertStreamMonitor.py\n\n~~~\n$ python3 ./CertStreamMonitor.py -c conf/example.conf\nLooking for these strings: paypal|apple|account|secure|login, detection threshold: 2\nConnection established to CertStream! Listening for events...\n[2018-03-12T11:40:15] cpanel.my-appleid-apple.net (SAN: mail.my-appleid-apple.net,my-appleid-apple.net,webdisk.my-appleid-apple.net,webmail.my-appleid-apple.net,www.my-appleid-apple.net) (Issuer: /C=US/CN=Let's Encrypt Authority X3/O=Let's Encrypt) (Fingerprint: 45:11:51:2D:24:D3:04:6E:DF:49:46:6D:64:56:67:4B:0A:48:8D:93) (StartTime: 2018-03-12T10:39:40)\n[2018-03-12T11:41:19] cpanel.verification-account-apple-now.com (SAN: mail.verification-account-apple-now.com,verification-account-apple-now.com,webdisk.verification-account-apple-now.com,webmail.verification-account-apple-now.com,www.verification-account-apple-now.com) (Issuer: /C=US/CN=Let's Encrypt Authority X3/O=Let's Encrypt) (Fingerprint: 2D:90:F9:F7:83:F6:48:26:EF:C9:72:50:4B:06:FA:36:53:94:3C:8B) (StartTime: 2018-03-12T10:40:49)\n[2018-03-12T11:41:36] login-apple.sytes.net (SAN: ) (Issuer: /C=US/CN=Let's Encrypt Authority X3/O=Let's Encrypt) (Fingerprint: C7:78:2F:08:1E:CC:83:6C:06:EF:77:14:D2:1A:4E:06:A8:B3:F9:77) (StartTime: 2018-03-12T10:41:08)\n[2018-03-12T11:42:26] cpanel.restore-account-apple.com (SAN: mail.restore-account-apple.com,restore-account-apple.com,webdisk.restore-account-apple.com,webmail.restore-account-apple.com,www.restore-account-apple.com) (Issuer: /C=US/CN=Let's Encrypt Authority X3/O=Let's Encrypt) (Fingerprint: F3:CA:B1:C6:DE:4F:05:16:FD:06:F3:FF:29:8A:D3:1F:10:9D:50:1A) (StartTime: 2018-03-12T10:41:59)\n[2018-03-12T11:49:37] securelogin.here.att.thysseankrupp.com (SAN: ) (Issuer: /C=US/CN=Let's Encrypt Authority X3/O=Let's Encrypt) (Fingerprint: 8F:9B:98:8D:5D:9B:03:0B:4F:62:56:40:C1:DE:9A:A4:FB:2D:A3:3E) (StartTime: 2018-03-12T09:22:41)\n...\n~~~\n\n### scanhost.py\n\n~~~\n$ python3 scanhost.py --help\n\n    -h --help\t\tPrint this help\n    -c --config\t\tConfiguration file to use\n    -f --fqdn-dirs      Store JSON files in sub-directories based on the hostname\n~~~\n\n~~~\n$ python3 ./scanhost.py -c conf/example.conf\nTest all domains in DB for Internet Presence:\n*********************************************\n14:30:12 - ERROR -   https://socialparadiseweb.cf.socialparadise.cf - Connection error\n14:32:18 - ERROR -   https://rapportannuel-assurancemaladie.paris - Connection error\n14:32:23 - SUCCESS - HTTP 200 - socialmediaforsocialaction.com\nCreating ./alerts/socialmediaforsocialaction.com.json : {'hostname': 'socialmediaforsocialaction.com', 'http_code': 200, 'cert_serial_number': '89:6C:03:F6:82:57:03:2A:A8:D0:E1:2F:E8:56:0E:32:83:E5:EC:29', 'webpage_title': 'Social Media for Social Action', 'ip_addr': '198.49.23.145', 'asn': '53831', 'asn_cidr': '198.49.23.0/24', 'asn_country_code': 'US', 'asn_description': 'SQUARESPACE - Squarespace, Inc., US', 'asn_abuse_email': 'abuse-network@squarespace.com'}\n14:32:25 - ERROR -   https://social.socialbride.co.za - Connection error\n14:32:34 - SUCCESS - HTTP 503 - assurances-sociales.com\nCreating ./alerts/assurances-sociales.com.json : {'hostname': 'assurances-sociales.com', 'http_code': 503, 'cert_serial_number': '1A:0D:45:D9:05:15:DC:17:6C:9F:9E:47:A5:62:03:D9:25:02:F9:3C', 'webpage_title': 'Accueil', 'ip_addr': '164.132.235.17', 'asn': '16276', 'asn_cidr': '164.132.0.0/16', 'asn_country_code': 'FR', 'asn_description': 'OVH, FR', 'asn_abuse_email': 'lir@ovh.net'}\n~~~\n\n### gethost.py\n\n~~~\n$ python3 gethost.py --help\n\n    -h --help   Print this help\n    -c --config Configuration file to use\n    --since     Since when it displays findings (seconds)\n~~~\n\n~~~\n$ python3 ./gethost.py -c conf/example.conf --since 36000 # 10 hours\nDisplay all domains in DB for Internet Presence:\n************************************************\nsocialparadiseweb.cf.socialparadise.cf None\nrapportannuel-assurancemaladie.paris None\nsocialmediaforsocialaction.com 2019-06-12T15:54:31\nsocial.socialbride.co.za None\n~~~\n\n### check_rules.py\n\n~~~\n$ python3 check_rules.py --help\n\n    -h --help       Print this help\n    -c --config     Configuration file to use\n    -d --domain     Domain name to check\n~~~\n\n~~~\n# No match - Keywords not found.\n$ ./check_rules.py -c conf/example.conf -d www.google.com\nLooking for these strings: paypal|apple|account|secure|login, detection threshold: 2\nNo match - Keywords not found.\n\n# No match - Detection threashold not reached.\n$ ./check_rules.py -c conf/example.conf -d paypal.com\nLooking for these strings: paypal|apple|account|secure|login, detection threshold: 2\nNo match - Detection threashold not reached.\n\n# No match - Blacklisted keywords.\n$ ./check_rules.py -c conf/example.conf -d login-paypal.gouv\nLooking for these strings: paypal|apple|account|secure|login, detection threshold: 2\nNo match - Blacklisted keywords.\n\n# This is a match, detection threashold reached.\n$ ./check_rules.py -c conf/example.conf -d login-paypal.com\nLooking for these strings: paypal|apple|account|secure|login, detection threshold: 2\nThis is a match, detection threashold reached.\n~~~\n\n## Authors\n- Thomas Damonneville ([thomas.damonneville@assurance-maladie.fr](mailto:thomas.damonneville@assurance-maladie.fr))\n- Christophe Brocas ([christophe.brocas@assurance-maladie.fr](mailto:christophe.brocas@assurance-maladie.fr))\n- And of course, a big shout out to [Calidog Security](https://calidog.io/) and its fantastic [CertStream service](https://certstream.calidog.io/) without which nothing would have been possible!\n\n## Contributors\n- [@ant1](https://github.com/ant1): [#3](https://github.com/AssuranceMaladieSec/CertStreamMonitor/pull/3)\n- [@xme](https://github.com/xme): [#8](https://github.com/AssuranceMaladieSec/CertStreamMonitor/pull/8), [#9](https://github.com/AssuranceMaladieSec/CertStreamMonitor/pull/9)\n- [@nbeguier](https://github.com/nbeguier): [#20](https://github.com/AssuranceMaladieSec/CertStreamMonitor/pull/20), [#21](https://github.com/AssuranceMaladieSec/CertStreamMonitor/pull/21), [#28](https://github.com/AssuranceMaladieSec/CertStreamMonitor/pull/28)\n\n## Presentations\n- [SSTIC 2018](https://www.sstic.org/2018/) | June 2018 - C.Brocas, T. Damonneville: *\"Certificate Transparency ou comment un nouveau standard peut aider votre veille sur certaines menaces\"*.  [Slides (fr)](https://www.sstic.org/2018/presentation/certificate_transparency_ou_comment_un_nouveau_standard_peut_aider_votre_analyse_des_menaces/), [full article (fr)](https://www.sstic.org/media/SSTIC2018/SSTIC-actes/certificate_transparency_ou_comment_un_nouveau_sta/SSTIC2018-Article-certificate_transparency_ou_comment_un_nouveau_standard_peut_aider_votre_analyse_des_menaces-broc_AR1OQsw.pdf), [video (fr)](https://static.sstic.org/videos2018/SSTIC_2018-06-13_P04.mp4).\n- [Hack-it-n 2018bis](http://www.hack-it-n.com/event2018bis/) | December 2018 - C. Brocas, T. Damonneville (given by C. Brocas): *\"CertStreamMonitor, use Certificate Transparency to improve your threats detection\"*. [Slides (en)](https://speakerdeck.com/cbrocas/2018bis-hack-it-n-certstreammonitor-use-certificate-transparency-to-improve-your-threats-detection).\n- [Toulouse hacking Convention 2019](https://19.thcon.party/) | March 2019 - C. Brocas, T. Damonneville (given by C. Brocas): *\"Certificate Transparency \u0026 threats detection, 24 months later\"*. [Slides (en)](https://speakerdeck.com/cbrocas/thc19-certificate-transparency-and-threats-detection-24-months-later), [video (fr)](https://www.youtube.com/watch?v=rUOQE-2NG3Y\u0026feature=youtu.be\u0026t=11485).\n\n## License\nGNU GENERAL PUBLIC LICENSE (GPL) Version 3\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FAssuranceMaladieSec%2FCertStreamMonitor","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FAssuranceMaladieSec%2FCertStreamMonitor","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FAssuranceMaladieSec%2FCertStreamMonitor/lists"}