{"id":28054949,"url":"https://github.com/d-Rickyy-b/certstream-server-go","last_synced_at":"2025-05-12T05:02:44.127Z","repository":{"id":55526495,"uuid":"515788633","full_name":"d-Rickyy-b/certstream-server-go","owner":"d-Rickyy-b","description":"This project aims to be a drop-in replacement for the certstream server by Calidog. This tool aggregates, parses, and streams certificate data from multiple certificate transparency logs via websocket connections to the clients.","archived":false,"fork":false,"pushed_at":"2025-04-29T20:33:23.000Z","size":413,"stargazers_count":123,"open_issues_count":14,"forks_count":18,"subscribers_count":8,"default_branch":"master","last_synced_at":"2025-04-29T21:31:15.606Z","etag":null,"topics":["certificate","certificate-transparency","certificates","certstream","go","golang","osint","phishing","reconnaissance","rfc6962","security","tls","tls-certificate","x509"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/d-Rickyy-b.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2022-07-20T01:04:35.000Z","updated_at":"2025-04-26T07:54:42.000Z","dependencies_parsed_at":"2023-11-26T01:25:11.153Z","dependency_job_id":"56eaec6e-7bca-46f1-9da0-3403bd99d4f2","html_url":"https://github.com/d-Rickyy-b/certstream-server-go","commit_stats":{"total_commits":207,"total_committers":1,"mean_commits":207.0,"dds":0.0,"last_synced_commit":"d74fcbc9fa72b3c087e7afdf33d210ac121db0f9"},"previous_names":[],"tags_count":18,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/d-Rickyy-b%2Fcertstream-server-go","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/d-Rickyy-b%2Fcertstream-server-go/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/d-Rickyy-b%2Fcertstream-server-go/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/d-Rickyy-b%2Fcertstream-server-go/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/d-Rickyy-b","download_url":"https://codeload.github.com/d-Rickyy-b/certstream-server-go/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253678866,"owners_count":21946315,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["certificate","certificate-transparency","certificates","certstream","go","golang","osint","phishing","reconnaissance","rfc6962","security","tls","tls-certificate","x509"],"created_at":"2025-05-12T05:02:00.192Z","updated_at":"2025-05-12T05:02:44.042Z","avatar_url":"https://github.com/d-Rickyy-b.png","language":"Go","funding_links":[],"categories":["Go"],"sub_categories":[],"readme":"![certstream-server-go logo](https://github.com/d-Rickyy-b/certstream-server-go/blob/master/docs/img/certstream-server-go_logo.png?raw=true)\n\n# Certstream Server Go\n\n[![build](https://github.com/d-Rickyy-b/certstream-server-go/actions/workflows/release_build.yml/badge.svg)](https://github.com/d-Rickyy-b/certstream-server-go/actions/workflows/release_build.yml)\n[![Docker Image Version (latest semver)](https://img.shields.io/docker/v/0rickyy0/certstream-server-go?label=docker\u0026sort=semver)](https://hub.docker.com/repository/docker/0rickyy0/certstream-server-go)\n[![Go Reference](https://pkg.go.dev/badge/github.com/d-Rickyy-b/certstream-server-go.svg)](https://pkg.go.dev/github.com/d-Rickyy-b/certstream-server-go)\n\nThis project aims to be a drop-in replacement for the [certstream server](https://github.com/CaliDog/certstream-server/) by Calidog. This tool aggregates, parses, and streams certificate data from multiple [certificate transparency logs](https://www.certificate-transparency.org/what-is-ct) via websocket connections to the clients.\n\nEveryone can use this project to analyze newly created TLS certificates as they are issued.\n\n## Motivation\n\nFrom the moment I first found out about the certificate transparency logs, I was absolutely amazed by the great software of [Calidog](https://github.com/CaliDog/), which made the transparency log easier accessible for everyone. \nTheir software \"Certstream\" parses the log and provides it in an easy-to-use format: JSON.\n\nAfter creating my first application that utilized the certstream server, I found that the hosted (demo) version of the server wasn't as reliable as I thought it would be. \nI got disconnects and sometimes other errors. Eventually, the provided server was still only thought to be **a demo**.\n\nI quickly thought about running my own instance of certstream. But I didn't want to install Elixir/Erlang on my server. Sure, I could have used Docker, but on second thought, I was really into the idea of creating an alternative server written in Go.\n\n\"Why Go?\", you might ask. Because it is a great language that compiles to native binaries on all major architectures and OSes. All the cool kids are using it right now.\n\n## Getting started\n\nSetting up an instance of the certstream server is straightforward. You can either download and compile the code yourself, or use one of the [precompiled binaries](https://github.com/d-Rickyy-b/certstream-server-go/releases).\n\nBy default, certstream-server-go will monitor all logs listed in the [Google Log list](https://www.gstatic.com/ct/log_list/v3/log_list.json), which are also included in the Chrome browser. There are more CT logs available than the ones listed there. Google provides [another list with all known CT logs](https://www.gstatic.com/ct/log_list/v3/all_logs_list.json). But not all of them might be relevant to you. Or maybe you are running your own CT log and want to monitor that as well?\n\nYou can define additional logs in the config file. Check out the [sample config file](https://github.com/d-Rickyy-b/certstream-server-go/blob/master/config.sample.yaml)\n\n### Docker\n\nThere's also a prebuilt [Docker image](https://hub.docker.com/repository/docker/0rickyy0/certstream-server-go) available.\nYou can use it by running this command:\n\n`docker run -d -v /path/to/config.yaml:/app/config.yaml -p 8080:8080 0rickyy0/certstream-server-go`\n\n\u003e [!WARNING]  \n\u003e If you don't mount your own config file, the default config (config.sample.yaml) will be used. For more details, check out the [wiki](https://github.com/d-Rickyy-b/certstream-server-go/wiki/Configuration).\n\n## Connecting\n\ncertstream-server-go offers multiple endpoints to connect to.\n\n| Config             | Default         | Function                                                                                  |\n|--------------------|-----------------|-------------------------------------------------------------------------------------------|\n| `full_url`         | `/full-stream`  | Constant stream of new certificates with all details available                            |\n| `lite_url`         | `/`             | Constant stream of new certificates with reduced details (no `as_der` and `chain` fields) |\n| `domains_only_url` | `/domains-only` | Constant stream of domains found in new certificates                                      |\n\nYou can connect to the certstream-server by opening a **websocket connection** to any of the aforementioned endpoints.\nAfter you're connected, certificate information will be streamed to your websocket.\n\nThe server requires you to send a **ping message** at least every 60 seconds (it's recommended to use an interval of 30s for pings). \nIf the server does not receive a ping message for more than this time, it will disconnect you. \nThe server will **not** send out ping messages to your client.\n\nRead more about ping/pong WebSocket messages in the [Mozilla Developer Docs](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API/Writing_WebSocket_servers#pings_and_pongs_the_heartbeat_of_websockets).\n\n### Performance\n\nAt idle (no clients connected), the server uses about **40 MB** of RAM, **14.5 Mbit/s** and **4–10% CPU** (Oracle Free Tier) on average while processing around **250–300 certificates per second**.\n\n### Network considerations\n\nThis tool requires outgoing access to the public internet to connect to the [Google Log list](https://www.gstatic.com/ct/log_list/v3/log_list.json) and the CT logs themselves.\nSo if you happen to this tool in a corporate environment (e.g., behind a proxy/firewall), make sure to allow outgoing connections to gstatic.com and the CT logs you want to connect to.\n\nIf you plan to connect clients to the server from outside your local network, make sure to allow incoming connections to the port you configured in the config file (webserver.listen_port).\n\n### Monitoring\n\n**certstream-server-go** also offers a Prometheus metrics endpoint at `/metrics`. You can use this to monitor the server with Prometheus and Grafana.\nFor an in-depth guide on how to do this, please refer to the [wiki](https://github.com/d-Rickyy-b/certstream-server-go/wiki/Collecting-and-Visualizing-Metrics).\n\n![grafana dashboard](https://user-images.githubusercontent.com/5798157/211434271-4350766d-2942-4fcb-8fda-f131f3f61cea.png)\n\n### Example\n\nTo receive a live example for any of the endpoints, send an HTTP GET request to the endpoints with `/example.json` appended to the endpoint. \nFor example: `/full-stream/example.json`. This shows the lite format of a certificate update.\n\n```json\n{\n    \"data\": {\n        \"cert_index\": 712420366,\n        \"cert_link\": \"https://yeti2022-2.ct.digicert.com/log/ct/v1/get-entries?start=712420366\u0026end=712420366\",\n        \"leaf_cert\": {\n            \"all_domains\": [\n                \"cmslieferhit.e06.k-k.de\"\n            ],\n            \"extensions\": {\n                \"authorityInfoAccess\": \"URI:http://r3.i.lencr.org/, URI:http://r3.o.lencr.org\",\n                \"authorityKeyIdentifier\": \"keyid:14:2e:b3:17:b7:58:56:cb:ae:50:09:40:e6:1f:af:9d:8b:14:c2:c6\",\n                \"basicConstraints\": \"CA:FALSE\",\n                \"keyUsage\": \"Digital Signature, Key Encipherment\",\n                \"subjectAltName\": \"DNS:cmslieferhit.e06.k-k.de\",\n                \"subjectKeyIdentifier\": \"keyid:4e:cb:ae:47:84:a8:92:f7:e7:de:78:d1:00:9e:d9:cc:80:ac:0b:ce\"\n            },\n            \"fingerprint\": \"27:58:3D:01:3D:71:B8:D3:A6:6E:2C:7A:86:3A:E9:1F:DB:F0:1B:5D\",\n            \"sha1\": \"27:58:3D:01:3D:71:B8:D3:A6:6E:2C:7A:86:3A:E9:1F:DB:F0:1B:5D\",\n            \"sha256\": \"57:61:38:C0:3C:03:A3:34:6A:0B:32:89:11:1B:74:AB:8A:DF:A5:02:9F:06:43:E6:F3:0E:69:F3:0E:4E:4E:FC\",\n            \"not_after\": 1667028404,\n            \"not_before\": 1659252405,\n            \"serial_number\": \"0498BDF812FAF923FEBD5EF7B374899FC61A\",\n            \"signature_algorithm\": \"sha256, rsa\",\n            \"subject\": {\n                \"C\": null,\n                \"CN\": \"cmslieferhit.e06.k-k.de\",\n                \"L\": null,\n                \"O\": null,\n                \"OU\": null,\n                \"ST\": null,\n                \"aggregated\": \"/CN=cmslieferhit.e06.k-k.de\",\n                \"email_address\": null\n            },\n            \"issuer\": {\n                \"C\": \"US\",\n                \"CN\": \"R3\",\n                \"L\": null,\n                \"O\": \"Let's Encrypt\",\n                \"OU\": null,\n                \"ST\": null,\n                \"aggregated\": \"/C=US/CN=R3/O=Let's Encrypt\",\n                \"email_address\": null\n            },\n            \"is_ca\": false\n        },\n        \"seen\": 1659301203.904,\n        \"source\": {\n            \"name\": \"DigiCert Yeti2022-2 Log\",\n            \"url\": \"https://yeti2022-2.ct.digicert.com/log\"\n        },\n        \"update_type\": \"PrecertLogEntry\"\n    },\n    \"message_type\": \"certificate_update\"\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fd-Rickyy-b%2Fcertstream-server-go","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fd-Rickyy-b%2Fcertstream-server-go","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fd-Rickyy-b%2Fcertstream-server-go/lists"}