{"id":13630691,"url":"https://github.com/jthomas/serverless-pwned-passwords","last_synced_at":"2025-04-14T23:37:19.945Z","repository":{"id":140706612,"uuid":"100390998","full_name":"jthomas/serverless-pwned-passwords","owner":"jthomas","description":"Using serverless functions to provide an API for checking potential passwords against an enormous corpus of passwords leaked from security breaches.","archived":false,"fork":false,"pushed_at":"2017-08-15T15:27:07.000Z","size":168,"stargazers_count":113,"open_issues_count":0,"forks_count":4,"subscribers_count":6,"default_branch":"master","last_synced_at":"2025-03-28T11:42:54.713Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jthomas.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2017-08-15T15:22:37.000Z","updated_at":"2024-07-09T11:14:04.000Z","dependencies_parsed_at":"2024-01-14T06:50:34.594Z","dependency_job_id":"68afbf76-7149-4104-8925-5ac104e71bfd","html_url":"https://github.com/jthomas/serverless-pwned-passwords","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jthomas%2Fserverless-pwned-passwords","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jthomas%2Fserverless-pwned-passwords/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jthomas%2Fserverless-pwned-passwords/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jthomas%2Fserverless-pwned-passwords/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jthomas","download_url":"https://codeload.github.com/jthomas/serverless-pwned-passwords/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248981049,"owners_count":21193142,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-01T22:01:55.131Z","updated_at":"2025-04-14T23:37:19.928Z","avatar_url":"https://github.com/jthomas.png","language":"Go","funding_links":[],"categories":["Go"],"sub_categories":[],"readme":"# Serverless Pwned Passwords\n\nServerless Pwned Passwords uses serverless functions to provide an API for checking potential passwords against an [enormous corpus of passwords](https://haveibeenpwned.com/Passwords) leaked from security breaches.\n\n[Bloom filters](https://en.wikipedia.org/wiki/Bloom_filter) generated from the corpus of leaked passwords are used to test for membership. [Go functions](http://jamesthom.as/blog/2017/01/17/openwhisk-and-go/) running on [Apache OpenWhisk](http://openwhisk.incubator.apache.org/) expose an API to check passwords against the list.\n\n💪 *Testing a potential password against 320 million leaked passwords takes milliseconds!* 💪\n\n![Serverless Bloom Filters](serverless_bloom_filters.png)\n\n## usage \n\n*Need an instance of Apache OpenWhisk to use? Sign up for [IBM Bluemix](https://console.bluemix.net/registration) which comes with a [generous free tier](https://console.bluemix.net/openwhisk/learn/pricing).*\n\nHaving installed the action using instructions below, test the service using the [command-line](https://console.bluemix.net/docs/openwhisk/openwhisk_cli.html#cloud-functions-cli). \n\n*`bloom_filters` takes a single parameter (`password`) with the potential password. `found` is returned as true if password test was positive.*\n\nLet's check the most commonly used password of all time…. `password`!\n\n```\n$ wsk action invoke bloom_filters --result --param password password\n{\n    \"found\": true\n}\t\n```\n\nUnsurprisingly, this was found in the leaked password corpus.\n\nWhat about something else? \n\n```\n$ wsk action invoke bloom_filters --result --param password serverless\n{\n    \"found\": false\n}\n```\n\nGreat, `serverless` was not found and I can keep using it… 😉\n\n### web actions\n\nExposing our service as a \"[web action](https://github.com/apache/incubator-openwhisk/blob/master/docs/webactions.md)\" provides a public API endpoint for our serverless function.\n\n```\n$ wsk action update bloom_filters --web true\nok: updated action bloom_filters\n$ wsk action get bloom_filters --url\nok: got action bloom_filters\nhttps://openwhisk.host/api/v1/web/user@email/default/bloom_filters\n```\n\nAllowing us to call it using a normal HTTP request.\n\n```\n$ http get https://openwhisk.host/api/v1/web/user@email/default/bloom_filters.json?password=password\n{\n    \"found\": true\n}\n```\n\n### performance\n\nLooking at the logging output shows performance data for each password check.\n\n```\n$  wsk activation logs 2818b2657a7043918cf4495b7b6083ff\n2017-08-15T15:17:18.833465399Z stderr: checking password: password\n2017-08-15T15:17:18.833497727Z stderr: hash: 5BAA61E4C9B93F3F0682250B6CF8331B7EE68FD8\n2017-08-15T15:17:18.83350474Z  stderr: reading file @ /bloom_filters/5b.dat\n2017-08-15T15:17:18.833509954Z stderr: file contained 2246528 bytes\n2017-08-15T15:17:18.83351495Z  stderr: elapsed time: 1.780073ms\n2017-08-15T15:17:18.833519804Z stderr: decoding bloom filter from 2246528 bytes\n2017-08-15T15:17:18.833524969Z stderr: decoded bloom filter parameters -\u003e m: 17971985 k: 10\n2017-08-15T15:17:18.833530802Z stderr: decoding bloom filter took 5.910131ms\n2017-08-15T15:17:18.833535692Z stderr: found password in bloom filter: true\n```\n\n\n\n## installation\n\nFollow the instructions below to create the bloom filters, build the runtime image and create the OpenWhisk action. \n\n*Building and publishing the action runtime image is an optional step. If you would rather skip this step, use this pre-existing Docker image: https://hub.docker.com/r/jamesthomas/bloom_filters/* \n\n### download pwned passwords (optional)\n\n* Download all the password hash files listed at https://haveibeenpwned.com/Passwords\n* Extract all files using the [7zip](http://www.7-zip.org/) compression tool.\n\n*There is a [Homebrew Formula](http://brewformulas.org/P7zip) for installing the tool on OS X.*\n\n### generate bloom filters (optional)\n\n* Install Bloom Filter library \n\n  ```\n  $ go get -u github.com/willf/bloom\n  ```\n\n* Run Bloom Filter generator program (`generate/generate_filters.go`) with hash file arguments.\n\n  ```\n  $ cd generate\n  $ go run generate_filters.go pwned-passwords-1.0.txt pwned-passwords-update-1.txt pwned-passwords-update-2.txt \n  2017/08/14 17:43:09 Creating Bloom filters with parameters --\u003e n: 17971985 k: 10\n  2017/08/14 17:43:09 Initialising 256 bloom filters...\n  2017/08/14 17:43:09 Bloom filter buckets: 256\n  2017/08/14 17:43:09 Reading hashes from file:  pwned-passwords-update-1.txt\n  2017/08/14 17:43:10 Processed 1000000 hashes...\n  ...\n  2017/08/14 17:43:19 Added XXX hashes to bloom filters\n  2017/08/14 17:43:19 Serialising 256 bloom filters...\n  2017/08/14 17:43:19 bucket: d1 encoded bytes: XXX\n  ...\n  ```\n\n*Generated bloom filters will be serialised to files in the `bloom_filters` directory. Files use the bucket identifier as the file name (`00.dat -\u003e ff.dat`).* \n\n### build runtime image (optional)\n\n*Hosting images on Docker Hub requires registering a (free) account @ https://hub.docker.com/*\n\n* Build the Docker image for the runtime.\n\n  ```\n  $ docker build -t \u003cDOCKERHUB_USER\u003e/bloom_filters .\n  ```\n\n* Push Docker image to Docker Hub.\n\n  ```\n  $ docker push \u003cDOCKERHUB_USER\u003e/bloom_filters \n  ```\n\n### build openwhisk action\n\n* Build Go binary for OpenWhisk runtime.\n\n  ```\n  $ env GOOS=linux GOARCH=amd64 go build -o exec bloom_filters.go\n  ```\n\n* Add binary to zip file.\n\n  ```\n  $ zip action.zip exec\n  ```\n\n### create openwhisk action\n\n* Create OpenWhisk action using binary archive file and Docker image.\n\n  ```\n  $ wsk action create bloom_filters --docker \u003cDOCKERHUB_USER\u003e/bloom_filters action.zip\n  ```\n\n\n\n***Once the action has been created, use the instructions above for testing it out.***\n\n\n\n## customising\n\nBloom filters use two parameters to control the probability of false positives, the size of the bit field (*m*) and the number of hashing functions (*k*). \n\nGiven a desired false positive rate and the number of items to be stored, optimal values for these parameters [can be calculated](https://stackoverflow.com/questions/658439/how-many-hash-functions-does-my-bloom-filter-need). Bloom filter calculators exist online: https://hur.st/bloomfilter\n\n**In this example, ~320 million passwords need to be checked with a false positive rate of 0.001%.** \n\nOptimal Bloom filter parameter values given these conditions are:\n\n- *m:* 4,600,828,022 (548.46MB)\n- *k:* 10\n\nInstantiating a Bloom filter of this size would add unacceptable delays [during \"cold\" invocations](https://read.acloud.guru/does-coding-language-memory-or-package-size-affect-cold-starts-of-aws-lambda-a15e26d12c76). Splitting the password hashes into groups and generating Bloom filters for each group can be used to reduce the Bloom filter size. \n\n**Password hashes are split into 256 buckets using the first two hex characters from the string.** Each Bloom filter needs to match 1.25M password hashes rather than 320M. \n\nBloom filter parameter values given these new conditions are:\n\n- *m:* 17,971,985 (2.14MB).\n- *k:* 10\n\nParameters for m and k are stored in the `generate.go` file.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjthomas%2Fserverless-pwned-passwords","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjthomas%2Fserverless-pwned-passwords","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjthomas%2Fserverless-pwned-passwords/lists"}