{"id":13610330,"url":"https://github.com/EFForg/starttls-backend","last_synced_at":"2025-04-12T22:33:23.301Z","repository":{"id":73623537,"uuid":"131773519","full_name":"EFForg/starttls-backend","owner":"EFForg","description":"STARTTLS Everywhere web backend and checker","archived":false,"fork":false,"pushed_at":"2020-06-01T21:06:55.000Z","size":613,"stargazers_count":18,"open_issues_count":30,"forks_count":6,"subscribers_count":9,"default_branch":"master","last_synced_at":"2025-03-30T22:51:16.592Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://starttls-everywhere.org/","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/EFForg.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-05-01T23:27:09.000Z","updated_at":"2024-09-06T08:42:29.000Z","dependencies_parsed_at":"2023-07-04T08:31:52.501Z","dependency_job_id":null,"html_url":"https://github.com/EFForg/starttls-backend","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EFForg%2Fstarttls-backend","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EFForg%2Fstarttls-backend/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EFForg%2Fstarttls-backend/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EFForg%2Fstarttls-backend/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/EFForg","download_url":"https://codeload.github.com/EFForg/starttls-backend/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248641801,"owners_count":21138270,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-01T19:01:43.722Z","updated_at":"2025-04-12T22:33:22.949Z","avatar_url":"https://github.com/EFForg.png","language":"Go","funding_links":[],"categories":["Go"],"sub_categories":[],"readme":"**Note:** The STARTTLS Everywhere project is not currently being maintained. The information and resources on this repository may be outdated. See [this post](https://www.eff.org/deeplinks/2020/04/winding-down-starttls-everywhere-project-and-future-secure-email) for more information. If you impacted by this news, or rely on the STARTTLS Policy List, you can [read this post](https://www.eff.org/deeplinks/2020/04/technical-deep-dive-winding-down-starttls-policy-list) for a deeper dive.\n\n# STARTTLS Everywhere Backend API\n\n[![Build Status](https://travis-ci.com/EFForg/starttls-backend.svg?branch=master)](https://travis-ci.org/EFForg/starttls-backend)\n[![Coverage Status](https://coveralls.io/repos/github/EFForg/starttls-backend/badge.svg?branch=master)](https://coveralls.io/github/EFForg/starttls-backend?branch=master)\n\nstarttls-backend is the JSON backend for starttls-everywhere.org. It provides endpoints to run security checks against email domains and manage the status of those domain's on EFF's [STARTTLS Everywhere policy list](https://github.com/EFForg/starttls-everywhere).\n\n## Setup\n1. Install `go` and `postgres`.\n2. Download the project and copy the configuration file:\n```\ngo get github.com/EFForg/starttls-backend\ncd $GOPATH/github.com/EFForg/starttls-backend\ncp .env.example .env\ncp .env.test.example .env.test\n```\n3. Edit `.env` and `.env.test` with your postgres credentials and any other changes.\n4. Ensure `postgres` is running, then run `db/scripts/init_tables.sql` in the appropriate postgres DBs in order to initialize your development and test databases.\n5. Build the scanner and start serving requests:\n```\ngo build\n./starttls-backend\n```\n\n### Via Docker\n```\ncp .env.example .env\ncp .env.test.example .env.test\ndocker-compose build\ndocker-compose up\n```\n\nTo automatically on container start, set `DB_MIGRATE=true` in the `.env` file.\n\n## Testing\n\nTest all packages in this repo with\n```\ngo test -v ./...\n```\n\nThe `main` and `db` packages contain integration tests that require a successful connection to the Postgres database. The remaining packages do not require the database to pass tests.\n\n## Configuration\n\n### No-scan domains\nIn case of complaints or abuse, we may not want to continually scan some domains. You can set the environment variable `DOMAIN_BLACKLIST` to point to a file with a list of newline-separated domains. Attempting to scan those domains from the public-facing website will result in error codes.\n\n## Scan API\n\nOur API objects can look a bit complicated! There's lots of information contained in a TLS scan.\nTo request a scan:\n```\nPOST /api/scan\n  { \"domain\": \"example.com\" }\n```\n\nLet's break down exactly what each part of this giant nested response means. All API responses, not just scans, are wrapped in a JSON object, like:\n```\n{\n    status_code: 200,\n    message: \"\",\n    response: \u003cresponse data\u003e\n}\n```\nOr even:\n```\n{\n    status_code: 400,\n    message: \"query parameter domain not specified\",\n    response: {}\n}\n```\nThe status codes always correspond with the HTTP status that is given for the response. `message` provides more context into why your request failed.\n\n### Scan responses\n\nHere's an abbreviated scan response. There's extra information on these objects that help\ndescribe the errors we encountered.\n```\n{\n    domain: \"example.com\",\n    scandata: {\n        status: 0,\n        results: {  // Individual hostname check results\n            \"mx.example.com\": {\n                \"status\": 0,\n                \"checks\": {\n                    \"connectivity\": { \"status\": 0 },\n                    \"certificate\": { \"status\": 0 },\n                    \"starttls\": { \"status\": 0 },\n                    \"version\": { \"status\": 0 },\n                }\n            }\n            \"dummy.example.com\": {\n                \"status\": 3,\n                \"checks\": {\n                    \"connectivity\": {\n                        \"status\": 3,\n                        \"messages\": [ \"Error: Could not establish connection\" ]\n                    },\n                }\n            },\n        },\n        preferred_hostnames: [\"mx.example.com\"], // Hostnames we were able to connect to\n        extra_results: {\"policylist\": { \"status\": 0 }},\n    },\n    timestamp: 0,\n    version: 1,\n}\n```\n\nThe meat of the response is in `scandata`, which is a JSON-ification of the `DomainResult` structure returned from the `checker` package.\n\n### Domain results\n\nHere's a quick synopsis of the fields you see in a domain response:\n\n - `domain`: the domain name that the scan was performed on.\n - `status`: Whether the check succeeded overall, and some more specific common failure types. Types 4-6 are types of test failures that are particularly common.\n    - 0: Success, all TLS tests passed.\n    - 1: Warning, at least one TLS test produced a warning.\n    - 2: Failure, at least one TLS test failed.\n    - 3: Error, something went wrong during the test.\n    - 4: NoSTARTTLS, at least one of your mailboxes did not advertise STARTTLS.\n    - 5: CouldNotConnect, could not connect to any mailbox.\n    - 6: BadHostnameFailure, one of your mailbox's provided certificates didn't match its hostname.\n - `message`: A more detailed description of the failure type.\n - `preferred_hostnames`: A misnomer, but refers to mailboxes that passed the connectivity test.\n - `mta_sts`: result for MTA STS check.\n - `extra_results`: A map of other security checks for this domain.\n - `results`: A map of mailbox hostnames to their individual results.\n - `timestamp`: Timestamp of when the scan was performed.\n - `version`: The scan API's version when it was performed.\n\n### Hostname results\n\nHere's a sample, \n```\n{\n    \"status\": 0,\n    \"checks\": {\n        \"connectivity\": { \"status\": 0 },\n        \"certificate\": {\n            \"status\": 2,\n            \"messages\": [\"Hostname doesn't match any name in certificate\",\n                         \"Certificate root is not trusted\"]\n        },\n        \"starttls\": { \"status\": 0 },\n        \"version\": { \"status\": 0 },\n    }\n}\n```\n\n - `checks`: A result can have a suite of checks. `checks` is a map from a particular check name to its result.\n - `status`: The status of a particular check, or the overall suite. Can be 0 through 3, which are `Success`, `Warning`, `Failure`, `Error`. The overall suite status takes the max status of all the sub-checks.\n - `messages`: If status of a check isn't success, messages is where all warnings and failure messages go.\n\n### What do we scan for?\n\nRight now, these are the checks we perform.\n\n##### Hostname-level scans\nThese scans are performed for every hostname-- that is, we try these things for every MX we find for the given domain.\n\n * *Connectivity*: This one is performed first. It's common for mailservers to use dummy MX records as a spam-prevention tactic, so a hostname that fails to connect doesn't automatically fail the entire TLS scan, unless *no* hostnames succeed in connectivity.\n * *STARTTLS*: The checker first connects to the mailbox and looks for a STARTTLS support banner. Then, we actively try to initiate a STARTTLS session.\n * *Certificate*: The checker checks for certificate validity, which includes (1) chaining to a valid root in Mozilla's CA store, (2) the hostname matching the certificate, and (3) the certificate being not expired.\n * *Version*: The checker checks your mailserver doesn't support obsolete and insecure protocols prior to TLS 1.0.\n\n##### Domain-level scans\nThese scans are performed for the domain itself.\n\n * *MTA-STS* We check to see whether your email domain follows the MTA-STS specification, and that the MTA-STS policy we find is valid.\n * *Policy List* We check to see whether your email domain is on our policy list, or queued to be added.\n\n### Rate-limiting, caching, and no-scan lists\n\nWe rate-limit several endpoints to prevent abuse and reduce load on our servers. By default, scan requests are cached-- if you're consistently updating your servers and want to check to see if it's passing, we recommend waiting a few minutes and re-scanning.\n\nIn case of complaints of abuse, we may not want to continually scan some domains, who can elect to prevent automated scans from this service.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FEFForg%2Fstarttls-backend","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FEFForg%2Fstarttls-backend","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FEFForg%2Fstarttls-backend/lists"}