Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/CaliDog/certstream-server

Certificate Transparency Log aggregation, parsing, and streaming service written in Elixir
https://github.com/CaliDog/certstream-server

Last synced: 2 months ago
JSON representation

Certificate Transparency Log aggregation, parsing, and streaming service written in Elixir

Awesome Lists containing this project

README

        



CertStream-Server


Aggregate and broadcast SSL certs as they're issued live.

**Certstream-server** is a service written in elixir to aggregate, parse, and stream certificate data from multiple [certificate transparency logs](https://www.certificate-transparency.org/what-is-ct). It leverages the amazing power of elixir/erlang to achieve great network throughput and concurrency with very little resource usage.

This is a rewrite of the [original version written in python](https://github.com/CaliDog/certstream-server-python), and is much more efficient than the original and currently ships millions of certificates a day on a single Hetzner dedicated server without issue (\~250TB of data every month!).

## Getting Started

Getting up and running is pretty easy (especially if you use Heroku, as we include a Dockerfile!).

First install elixir (assuming you're on OSX, otherwise follow [instructions for your platform](https://elixir-lang.org/install.html))

```
$ brew install elixir
```

Then fetch the dependencies

```
$ mix deps.get
```

From there you can run it

```
$ mix run --no-halt
```

Alternatively, you can run it in an iex session (so you can interact with it while it's running) for development

```
$ iex -S mix
Interactive Elixir (1.8.2) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> :observer.start
```

This will open up an http/websocket server on port 4000 (override this by setting a `PORT` environment variable).

## Usage

Once you have the server runing on some port, you can visit the server to be greeted with the index page.

If you point a websocket client at the root URL (`/`), it'll give you a data structure that doesn't bundle the DER-encoded certificate.

If you want the certificate data as well, you can point a websocket client to `/full-stream` and it'll also bundle the DER-encoded certificate along with each certificate.

## Internals

The structure of the application is pretty simple. The dataflow basically looks like this:

```
CertificateBuffer
|
HTTP Watcher \ | / Websocket Connection Process
HTTP Watcher - ClientManager - Websocket Connection Process
HTTP Watcher / \ Websocket Connection Process

```
### HTTP Watchers
First we spin up 1 process for every entry in the [known CTL list from the CT project](https://www.gstatic.com/ct/log_list/v3/all_logs_list.json), with each of them being responsible for sleeping for 10 seconds and checking to see if the top of the merkle tree has changed. Once a difference has been found, they go out and download the certificate data, parsing it and coercing it to a hashmap structure using [the EasySSL library](https://github.com/CaliDog/EasySSL) and sending it to the ClientManager.

### ClientManager
This agent is responsible for brokering communication between the CT watchers and the currently connected websocket clients. Certificates are broadcast to websocket connection processes through an erlang [pobox](https://github.com/ferd/pobox) in order to properly load-shed when a slow client isn't reading certificates fast enough. The ClientManager also sends a copy of every certificate received to the CertificateBuffer.

### CertificateBuffer
This agent is responsible for keeping a ring-buffer in memory of the most recently seen 25 certificates, as well as counting the certificates processed by Certstream.

### Websocket Connection Process
Under the hood we use the erlang library [Cowboy](https://github.com/ninenines/cowboy) to handle static content serving, the json APIs, and websocket connections. There's nothing too special about them other than they're assigned a paired pobox at the start of every connection.

## HTTP Routes

`/latest.json` - Get the most recent 25 certificates CertStream has seen

`/example.json` - Get the most recent certificate CertStream has seen

## Data Structure

**Certificate Updates**

```
{
"message_type": "certificate_update",
"data": {
"update_type": "X509LogEntry",
"leaf_cert": {
"subject": {
"aggregated": "/CN=e-zigarette-liquid-shop.de",
"C": null,
"ST": null,
"L": null,
"O": null,
"OU": null,
"CN": "e-zigarette-liquid-shop.de"
},
"extensions": {
"keyUsage": "Digital Signature, Key Encipherment",
"extendedKeyUsage": "TLS Web Server Authentication, TLS Web Client Authentication",
"basicConstraints": "CA:FALSE",
"subjectKeyIdentifier": "AC:4C:7B:3C:E9:C8:7F:CB:E2:7D:5D:64:F2:25:0C:89:C2:AE:F0:5E",
"authorityKeyIdentifier": "keyid:A8:4A:6A:63:04:7D:DD:BA:E6:D1:39:B7:A6:45:65:EF:F3:A8:EC:A1\n",
"authorityInfoAccess": "OCSP - URI:http://ocsp.int-x3.letsencrypt.org\nCA Issuers - URI:http://cert.int-x3.letsencrypt.org/\n",
"subjectAltName": "DNS:e-zigarette-liquid-shop.de, DNS:www.e-zigarette-liquid-shop.de",
"certificatePolicies": "Policy: 2.23.140.1.2.1\nPolicy: 1.3.6.1.4.1.44947.1.1.1\n CPS: http://cps.letsencrypt.org\n User Notice:\n Explicit Text: This Certificate may only be relied upon by Relying Parties and only in accordance with the Certificate Policy found at https://letsencrypt.org/repository/\n"
},
"not_before": 1508123861.0,
"not_after": 1515899861.0,
"as_der": "::BASE64_DER_CERT::",
"all_domains": [
"e-zigarette-liquid-shop.de",
"www.e-zigarette-liquid-shop.de"
]
},
"chain": [
{
"subject": {
"aggregated": "/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3",
"C": "US",
"ST": null,
"L": null,
"O": "Let's Encrypt",
"OU": null,
"CN": "Let's Encrypt Authority X3"
},
"extensions": {
"basicConstraints": "CA:TRUE, pathlen:0",
"keyUsage": "Digital Signature, Certificate Sign, CRL Sign",
"authorityInfoAccess": "OCSP - URI:http://isrg.trustid.ocsp.identrust.com\nCA Issuers - URI:http://apps.identrust.com/roots/dstrootcax3.p7c\n",
"authorityKeyIdentifier": "keyid:C4:A7:B1:A4:7B:2C:71:FA:DB:E1:4B:90:75:FF:C4:15:60:85:89:10\n",
"certificatePolicies": "Policy: 2.23.140.1.2.1\nPolicy: 1.3.6.1.4.1.44947.1.1.1\n CPS: http://cps.root-x1.letsencrypt.org\n",
"crlDistributionPoints": "\nFull Name:\n URI:http://crl.identrust.com/DSTROOTCAX3CRL.crl\n",
"subjectKeyIdentifier": "A8:4A:6A:63:04:7D:DD:BA:E6:D1:39:B7:A6:45:65:EF:F3:A8:EC:A1"
},
"not_before": 1458232846.0,
"not_after": 1615999246.0,
"as_der": "::BASE64_DER_CERT::"
},
{
"subject": {
"aggregated": "/O=Digital Signature Trust Co./CN=DST Root CA X3",
"C": null,
"ST": null,
"L": null,
"O": "Digital Signature Trust Co.",
"OU": null,
"CN": "DST Root CA X3"
},
"extensions": {
"basicConstraints": "CA:TRUE",
"keyUsage": "Certificate Sign, CRL Sign",
"subjectKeyIdentifier": "C4:A7:B1:A4:7B:2C:71:FA:DB:E1:4B:90:75:FF:C4:15:60:85:89:10"
},
"not_before": 970348339.0,
"not_after": 1633010475.0,
"as_der": "::BASE64_DER_CERT::"
}
],
"cert_index": 19587936,
"seen": 1508483726.8601687,
"source": {
"url": "mammoth.ct.comodo.com",
"name": "Comodo 'Mammoth' CT log"
}
}
}
```