{"id":19124137,"url":"https://github.com/efforg/observatory","last_synced_at":"2025-07-21T08:04:08.323Z","repository":{"id":137662888,"uuid":"138079665","full_name":"EFForg/observatory","owner":"EFForg","description":null,"archived":false,"fork":false,"pushed_at":"2018-06-20T19:58:15.000Z","size":23524,"stargazers_count":7,"open_issues_count":0,"forks_count":3,"subscribers_count":5,"default_branch":"master","last_synced_at":"2025-05-05T19:16:10.318Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/EFForg.png","metadata":{"files":{"readme":"README","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-06-20T19:57:39.000Z","updated_at":"2024-10-17T02:27:42.000Z","dependencies_parsed_at":null,"dependency_job_id":"014c143d-f737-4900-849e-d3e321430818","html_url":"https://github.com/EFForg/observatory","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/EFForg/observatory","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EFForg%2Fobservatory","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EFForg%2Fobservatory/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EFForg%2Fobservatory/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EFForg%2Fobservatory/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/EFForg","download_url":"https://codeload.github.com/EFForg/observatory/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EFForg%2Fobservatory/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":266263057,"owners_count":23901356,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-09T05:28:14.372Z","updated_at":"2025-07-21T08:04:08.287Z","avatar_url":"https://github.com/EFForg.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"The SSL Observatory is still a work in progress.\n\nWARNING: This code is not suitable for production systems, it is experimental.\nFor example, low rights users can interfere with the scripts this thing uses,\nand database use is done without privilege seperation, tmp file access is\nwilly nilly and we frequently run with privileges we don't need... so be\ncareful.\n\nThis is the complicated codebase of scripts that we used to collect and\nprocess the EFF SSL Observatory dataset.  The most important top-level scripts\nare as follows:\n\nscan/          -- scripts for scanning IPv4 space and recording TCP port 443\n                  responses in .results files (you also get our .results files\n                  via bittorrent from https://www.eff.org/observatory)\n\ndbconnect.py   -- make yourself a MySQL database for the SSL certificates,\n                  then customise this file with your username, password, and\n                  database name\n\nlaunch_parsers -- edit this to set the path where your .results files reside,\n                  then run it to build the raw certs* tables and the valid_certs\n                  table of valid certificates \n\ntransvalid.py  -- once you have a valid_certs table, run this to compute a more\n                  sophisticated notion of which certs are valid in real\n                  browsers; this updates the raw certs* tables\n\nrebuild.sh     -- run this to build all the important tables in the neatest\n                  and most correct way, once transvalidity has been evaluated\n\nThe observatory stores data in MySQL, which you will need to have installed.\nThe mysql program will also need to be in your path. We recommend you set\nthe following values in your ~/.my.cnf file (personal mysql configuration file)\n\nin the [client] section of ~/.my.cnf (template is often installed in \n/etc/my.cnf or /etc/mysql/my.cnf)\n\nuser=YOUR_DB_USERNAME_WITH_ADMIN_RIGHTS\nhost=localhost\npassword=YOUR_DB_PASSWORD\ndatabase=observatory\n\nNote that keeping passwords in your home directory might not be a great idea.\n\nOnce you install MySQL, you will need to establish a username and password as \nwell as perform a \"create database observatory;\" (or whatever name you like)\nso that you will have somewhere to put your data in mysql.\n \nYou will then need to make some configuration adjustments to the observatory's\nscripts in order help them find your data and know where to import it. The \nREADME.schema file has some tips about this, and explains a bit about what you\nwill actually find in the tables once the data is imported.\n\nAfter you have a working import of the data, you may want to explore it with\nqueries, the questions subdirectory contains example sql and python queries.\nUnderstanding these can help you craft your own. The data definately has some\nquirks, it is frequently useful to use limits on simple queries in order to\nget an example of what you are looking for. For example:\n\njesse@floop:~/sslscanner$ mysql\nReading table information for completion of table and column names\nYou can turn off this feature to get a quicker startup with -A\n\nWelcome to the MySQL monitor.  Commands end with ; or \\g.\nYour MySQL connection id is 1350\nServer version: 5.1.41-3ubuntu12.8 (Ubuntu)\n\nType 'help;' or '\\h' for help. Type '\\c' to clear the current input statement.\n\nmysql\u003e select distinct subject from all_certs where moz_valid=\"Yes\" limit 10;\n+-----------------------------------------------------------------------------------------------------------------------\n------------------------------------------------+\n| subject\n                                                |\n+-----------------------------------------------------------------------------------------------------------------------\n------------------------------------------------+\n|  C=US/postalCode=20770, ST=MD, L=Greenbelt/streetAddress=Suite 140/streetAddress=6305 Ivy lane, O=NFRC, OU=web, OU=Sec\nure Link SSL, CN=mail.nfrc.org                  |\n|  C=US, O=Network Solutions L.L.C., CN=Network Solutions Certificate Authority\n                                                |\n|  C=US, ST=UT, L=Salt Lake City, O=The USERTRUST Network, OU=http://www.usertrust.com, CN=UTN-USERFirst-Hardware\n                                                |\n|  O=mail.nscharter.com, OU=Domain Control Validated, CN=mail.nscharter.com\n                                                |\n|  C=US, ST=Arizona, L=Scottsdale, O=GoDaddy.com, Inc., OU=http://certificates.godaddy.com/repository, CN=Go Daddy Secur\ne Certification Authority/serialNumber=07969287 |\n|  C=US, O=The Go Daddy Group, Inc., OU=Go Daddy Class 2 Certification Authority\n                                                |\n|  OU=Domain Control Validated, OU=Hosted by Register.com, OU=PositiveSSL, CN=mail.rdclient.net\n                                                |\n|  C=US, O=Register.com, CN=Register.com CA SSL Services (DV)\n                                                |\n|  C=US, ST=Massachusetts, L=westford, O=Town of westford, CN=ipass.westford-ma.gov\n                                                |\n|  CN=WyattHome.homeserver.com, OU=Domain Control Validated\n                                                |\n+-----------------------------------------------------------------------------------------------------------------------\n------------------------------------------------+\n10 rows in set (0.00 sec)\n\nmysql\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fefforg%2Fobservatory","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fefforg%2Fobservatory","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fefforg%2Fobservatory/lists"}