{"id":13588914,"url":"https://github.com/CaliDog/certstream-server","last_synced_at":"2025-04-08T07:31:36.333Z","repository":{"id":31216518,"uuid":"124506514","full_name":"CaliDog/certstream-server","owner":"CaliDog","description":"Certificate Transparency Log aggregation, parsing, and streaming service written in Elixir","archived":false,"fork":false,"pushed_at":"2024-03-26T15:00:51.000Z","size":6980,"stargazers_count":288,"open_issues_count":31,"forks_count":78,"subscribers_count":20,"default_branch":"master","last_synced_at":"2025-04-05T19:35:48.466Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://certstream.calidog.io","language":"Vue","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/CaliDog.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-03-09T07:47:49.000Z","updated_at":"2025-03-26T00:22:51.000Z","dependencies_parsed_at":"2024-04-08T20:21:08.919Z","dependency_job_id":null,"html_url":"https://github.com/CaliDog/certstream-server","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CaliDog%2Fcertstream-server","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CaliDog%2Fcertstream-server/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CaliDog%2Fcertstream-server/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CaliDog%2Fcertstream-server/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/CaliDog","download_url":"https://codeload.github.com/CaliDog/certstream-server/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247796155,"owners_count":20997522,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-01T15:07:01.254Z","updated_at":"2025-04-08T07:31:33.949Z","avatar_url":"https://github.com/CaliDog.png","language":"Vue","readme":"\u003cp align=\"center\"\u003e\n    \u003cimg align=\"center\" src=\"https://user-images.githubusercontent.com/1072598/31840406-1fe37936-b59a-11e7-939a-71d36e584fc9.png\" /\u003e\n    \u003ch3 align=\"center\"\u003eCertStream-Server\u003c/h3\u003e\n    \u003cp align=\"center\"\u003eAggregate and broadcast SSL certs as they're issued live.\u003c/p\u003e\n\u003c/p\u003e\n\n**Certstream-server** is a service written in elixir to aggregate, parse, and stream certificate data from multiple [certificate transparency logs](https://www.certificate-transparency.org/what-is-ct). It leverages the amazing power of elixir/erlang to achieve great network throughput and concurrency with very little resource usage.\n\nThis is a rewrite of the [original version written in python](https://github.com/CaliDog/certstream-server-python), and is much more efficient than the original and currently ships millions of certificates a day on a single Hetzner dedicated server without issue (\\~250TB of data every month!).\n\n## Getting Started\n\nGetting up and running is pretty easy (especially if you use Heroku, as we include a Dockerfile!).\n\nFirst install elixir (assuming you're on OSX, otherwise follow [instructions for your platform](https://elixir-lang.org/install.html))\n\n```\n$ brew install elixir\n```\n\nThen fetch the dependencies\n\n```\n$ mix deps.get\n```\n\nFrom there you can run it\n\n```\n$ mix run --no-halt\n```\n\nAlternatively, you can run it in an iex session (so you can interact with it while it's running) for development\n\n```\n$ iex -S mix\nInteractive Elixir (1.8.2) - press Ctrl+C to exit (type h() ENTER for help)\niex(1)\u003e :observer.start\n```\n\nThis will open up an http/websocket server on port 4000 (override this by setting a `PORT` environment variable). \n\n## Usage\n\nOnce you have the server runing on some port, you can visit the server to be greeted with the index page. \n\nIf you point a websocket client at the root URL (`/`), it'll give you a data structure that doesn't bundle the DER-encoded certificate.\n\nIf you want the certificate data as well, you can point a websocket client to `/full-stream` and it'll also bundle the DER-encoded certificate along with each certificate. \n\n## Internals\n\nThe structure of the application is pretty simple. The dataflow basically looks like this:\n\n```\n               CertificateBuffer\n                      |\nHTTP Watcher \\        |        / Websocket Connection Process\nHTTP Watcher  - ClientManager -  Websocket Connection Process\nHTTP Watcher /                 \\ Websocket Connection Process\n\n```\n### HTTP Watchers\nFirst we spin up 1 process for every entry in the [known CTL list from the CT project](https://www.gstatic.com/ct/log_list/v3/all_logs_list.json), with each of them being responsible for sleeping for 10 seconds and checking to see if the top of the merkle tree has changed. Once a difference has been found, they go out and download the certificate data, parsing it and coercing it to a hashmap structure using [the EasySSL library](https://github.com/CaliDog/EasySSL) and sending it to the ClientManager.\n\n### ClientManager\nThis agent is responsible for brokering communication between the CT watchers and the currently connected websocket clients. Certificates are broadcast to websocket connection processes through an erlang [pobox](https://github.com/ferd/pobox) in order to properly load-shed when a slow client isn't reading certificates fast enough. The ClientManager also sends a copy of every certificate received to the CertificateBuffer.\n\n### CertificateBuffer\nThis agent is responsible for keeping a ring-buffer in memory of the most recently seen 25 certificates, as well as counting the certificates processed by Certstream.\n\n### Websocket Connection Process\nUnder the hood we use the erlang library [Cowboy](https://github.com/ninenines/cowboy) to handle static content serving, the json APIs, and websocket connections. There's nothing too special about them other than they're assigned a paired pobox at the start of every connection.\n\n## HTTP Routes\n\n`/latest.json` - Get the most recent 25 certificates CertStream has seen\n\n`/example.json` - Get the most recent certificate CertStream has seen\n\n## Data Structure\n\n**Certificate Updates**\n\n```\n{\n    \"message_type\": \"certificate_update\",\n    \"data\": {\n        \"update_type\": \"X509LogEntry\",\n        \"leaf_cert\": {\n            \"subject\": {\n                \"aggregated\": \"/CN=e-zigarette-liquid-shop.de\",\n                \"C\": null,\n                \"ST\": null,\n                \"L\": null,\n                \"O\": null,\n                \"OU\": null,\n                \"CN\": \"e-zigarette-liquid-shop.de\"\n            },\n            \"extensions\": {\n                \"keyUsage\": \"Digital Signature, Key Encipherment\",\n                \"extendedKeyUsage\": \"TLS Web Server Authentication, TLS Web Client Authentication\",\n                \"basicConstraints\": \"CA:FALSE\",\n                \"subjectKeyIdentifier\": \"AC:4C:7B:3C:E9:C8:7F:CB:E2:7D:5D:64:F2:25:0C:89:C2:AE:F0:5E\",\n                \"authorityKeyIdentifier\": \"keyid:A8:4A:6A:63:04:7D:DD:BA:E6:D1:39:B7:A6:45:65:EF:F3:A8:EC:A1\\n\",\n                \"authorityInfoAccess\": \"OCSP - URI:http://ocsp.int-x3.letsencrypt.org\\nCA Issuers - URI:http://cert.int-x3.letsencrypt.org/\\n\",\n                \"subjectAltName\": \"DNS:e-zigarette-liquid-shop.de, DNS:www.e-zigarette-liquid-shop.de\",\n                \"certificatePolicies\": \"Policy: 2.23.140.1.2.1\\nPolicy: 1.3.6.1.4.1.44947.1.1.1\\n  CPS: http://cps.letsencrypt.org\\n  User Notice:\\n    Explicit Text: This Certificate may only be relied upon by Relying Parties and only in accordance with the Certificate Policy found at https://letsencrypt.org/repository/\\n\"\n            },\n            \"not_before\": 1508123861.0,\n            \"not_after\": 1515899861.0,\n            \"as_der\": \"::BASE64_DER_CERT::\",\n            \"all_domains\": [\n                \"e-zigarette-liquid-shop.de\",\n                \"www.e-zigarette-liquid-shop.de\"\n            ]\n        },\n        \"chain\": [\n            {\n                \"subject\": {\n                    \"aggregated\": \"/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3\",\n                    \"C\": \"US\",\n                    \"ST\": null,\n                    \"L\": null,\n                    \"O\": \"Let's Encrypt\",\n                    \"OU\": null,\n                    \"CN\": \"Let's Encrypt Authority X3\"\n                },\n                \"extensions\": {\n                    \"basicConstraints\": \"CA:TRUE, pathlen:0\",\n                    \"keyUsage\": \"Digital Signature, Certificate Sign, CRL Sign\",\n                    \"authorityInfoAccess\": \"OCSP - URI:http://isrg.trustid.ocsp.identrust.com\\nCA Issuers - URI:http://apps.identrust.com/roots/dstrootcax3.p7c\\n\",\n                    \"authorityKeyIdentifier\": \"keyid:C4:A7:B1:A4:7B:2C:71:FA:DB:E1:4B:90:75:FF:C4:15:60:85:89:10\\n\",\n                    \"certificatePolicies\": \"Policy: 2.23.140.1.2.1\\nPolicy: 1.3.6.1.4.1.44947.1.1.1\\n  CPS: http://cps.root-x1.letsencrypt.org\\n\",\n                    \"crlDistributionPoints\": \"\\nFull Name:\\n  URI:http://crl.identrust.com/DSTROOTCAX3CRL.crl\\n\",\n                    \"subjectKeyIdentifier\": \"A8:4A:6A:63:04:7D:DD:BA:E6:D1:39:B7:A6:45:65:EF:F3:A8:EC:A1\"\n                },\n                \"not_before\": 1458232846.0,\n                \"not_after\": 1615999246.0,\n                \"as_der\": \"::BASE64_DER_CERT::\"\n            },\n            {\n                \"subject\": {\n                    \"aggregated\": \"/O=Digital Signature Trust Co./CN=DST Root CA X3\",\n                    \"C\": null,\n                    \"ST\": null,\n                    \"L\": null,\n                    \"O\": \"Digital Signature Trust Co.\",\n                    \"OU\": null,\n                    \"CN\": \"DST Root CA X3\"\n                },\n                \"extensions\": {\n                    \"basicConstraints\": \"CA:TRUE\",\n                    \"keyUsage\": \"Certificate Sign, CRL Sign\",\n                    \"subjectKeyIdentifier\": \"C4:A7:B1:A4:7B:2C:71:FA:DB:E1:4B:90:75:FF:C4:15:60:85:89:10\"\n                },\n                \"not_before\": 970348339.0,\n                \"not_after\": 1633010475.0,\n                \"as_der\": \"::BASE64_DER_CERT::\"\n            }\n        ],\n        \"cert_index\": 19587936,\n        \"seen\": 1508483726.8601687,\n        \"source\": {\n            \"url\": \"mammoth.ct.comodo.com\",\n            \"name\": \"Comodo 'Mammoth' CT log\"\n        }\n    }\n}\n```\n","funding_links":[],"categories":["Vue","Security","Hardening"],"sub_categories":["WebServers"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FCaliDog%2Fcertstream-server","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FCaliDog%2Fcertstream-server","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FCaliDog%2Fcertstream-server/lists"}