{"id":13659175,"url":"https://github.com/philipperemy/tensorflow-1.4-billion-password-analysis","last_synced_at":"2025-04-08T02:41:53.587Z","repository":{"id":40444189,"uuid":"114227052","full_name":"philipperemy/tensorflow-1.4-billion-password-analysis","owner":"philipperemy","description":"Deep Learning model to analyze a large corpus of clear text passwords.","archived":false,"fork":false,"pushed_at":"2021-06-29T21:11:37.000Z","size":126,"stargazers_count":1927,"open_issues_count":9,"forks_count":394,"subscribers_count":104,"default_branch":"master","last_synced_at":"2025-04-01T01:41:41.269Z","etag":null,"topics":["deep-learning","natural-language-processing","tensorflow"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/philipperemy.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null},"funding":{"github":["philipperemy"]}},"created_at":"2017-12-14T09:02:08.000Z","updated_at":"2025-03-17T14:16:02.000Z","dependencies_parsed_at":"2022-07-12T18:02:19.894Z","dependency_job_id":null,"html_url":"https://github.com/philipperemy/tensorflow-1.4-billion-password-analysis","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/philipperemy%2Ftensorflow-1.4-billion-password-analysis","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/philipperemy%2Ftensorflow-1.4-billion-password-analysis/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/philipperemy%2Ftensorflow-1.4-billion-password-analysis/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/philipperemy%2Ftensorflow-1.4-billion-password-analysis/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/philipperemy","download_url":"https://codeload.github.com/philipperemy/tensorflow-1.4-billion-password-analysis/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247767232,"owners_count":20992538,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deep-learning","natural-language-processing","tensorflow"],"created_at":"2024-08-02T05:01:05.911Z","updated_at":"2025-04-08T02:41:53.571Z","avatar_url":"https://github.com/philipperemy.png","language":"Python","funding_links":["https://github.com/sponsors/philipperemy"],"categories":["Python (144)","Python","Web Hacking","Python (1887)","Programming/Comp Sci/SE Things"],"sub_categories":["Enumeration and Reconnaissance Tools","🛰️ Enumeration and Reconnaissance Tools","Password Lists"],"readme":"# 1.4 Billion Text Credentials Analysis (NLP)\n\nUsing deep learning and NLP to analyze a large corpus of clear text passwords.\n\nObjectives:\n- Train a generative model.\n- Understand how people change their passwords over time: hello123 -\u003e h@llo123 -\u003e h@llo!23.\n\nDisclaimer: for research purposes only.\n\n## In the press\n\n- [1.4 Billion Clear Text Credentials Discovered in a Single Database](https://medium.com/4iqdelvedeep/1-4-billion-clear-text-credentials-discovered-in-a-single-database-3131d0a1ae14)\n- [Collection of 1.4 Billion Plain-Text Leaked Passwords Found Circulating Online](https://thehackernews.com/2017/12/data-breach-password-list.html)\n- [Archive of 1.4 BEEELLION credentials in clear text found in dark web archive](https://www.theregister.co.uk/2017/12/12/archive_of_14_beeelion_credentials_in_clear_text_found_in_dark_web_archive/)\n- [Forbes](https://www.forbes.com/sites/leemathews/2017/12/11/billion-hacked-passwords-dark-web/#74a6cf4221f2)\n\n\n\n## Get the data\n\n- Download any Torrent client.\n- Here is a magnet link you can find on [Reddit](https://www.reddit.com/r/privacy/comments/7j1hhz/14_billion_clear_text_credentials_discovered_in_a/):\n  - magnet:?xt=urn:btih:7ffbcd8cee06aba2ce6561688cf68ce2addca0a3\u0026dn=BreachCompilation\u0026tr=udp%3A%2F%2Ftracker.openbittorrent.com%3A80\u0026tr=udp%3A%2F%2Ftracker.leechers-paradise.org%3A6969\u0026tr=udp%3A%2F%2Ftracker.coppersurfer.tk%3A6969\u0026tr=udp%3A%2F%2Fglotorrents.pw%3A6969\u0026tr=udp%3A%2F%2Ftracker.opentrackr.org%3A1337\n- Checksum list is available here: [checklist.chk](checklist.chk)\n- `./count_total.sh` in `BreachCompilation` should display something like 1,400,553,870 rows.\n\n## Get started (processing + deep learning)\n\nProcess the data and run the first deep learning model:\n\n```\n# make sure to install the python deps first. Virtual env are recommended here.\n# virtualenv -p python3 venv3; source venv3/bin/activate; pip install -r requirements.txt\n# Remove \"--max_num_files 100\" to process the whole dataset (few hours and 50GB of free disk space are required.)\n./process_and_train.sh \u003cBreachCompilation path\u003e\n```\n\n## Data (explanation)\n\n```\nINPUT:   BreachCompilation/\n         BreachCompilation is organized as:\n\n         - a/          - folder of emails starting with a\n         - a/a         - file of emails starting with aa\n         - a/b\n         - a/d\n         - ...\n         - z/\n         - ...\n         - z/y\n         - z/z\n\nOUTPUT: - BreachCompilationAnalysis/edit-distance/1.csv\n        - BreachCompilationAnalysis/edit-distance/2.csv\n        - BreachCompilationAnalysis/edit-distance/3.csv\n        [...]\n        \u003e cat 1.csv\n            1 ||| samsung94 ||| samsung94@\n            1 ||| 040384alexej ||| 040384alexey\n            1 ||| HoiHalloDoeii14 ||| hoiHalloDoeii14\n            1 ||| hoiHalloDoeii14 ||| hoiHalloDoeii13\n            1 ||| hoiHalloDoeii13 ||| HoiHalloDoeii13\n            1 ||| 8znachnuu ||| 7znachnuu\n        EXPLANATION: edit-distance/ contains the passwords pairs sorted by edit distances.\n        1.csv contains all pairs with edit distance = 1 (exactly one addition, substitution or deletion).\n        2.csv =\u003e edit distance = 2, and so on.\n\n        - BreachCompilationAnalysis/reduce-passwords-on-similar-emails/99_per_user.json\n        - BreachCompilationAnalysis/reduce-passwords-on-similar-emails/9j_per_user.json\n        - BreachCompilationAnalysis/reduce-passwords-on-similar-emails/9a_per_user.json\n        [...]\n        \u003e cat 96_per_user.json\n        {\n            \"1.0\": [\n            {\n                \"edit_distance\": [\n                    0,\n                    1\n                ],\n                \"email\": \"96-000@mail.ru\",\n                \"password\": [\n                    \"090698d\",\n                    \"090698D\"\n                ]\n            },\n        {\n                \"edit_distance\": [\n                    0,\n                    1\n                ],\n                \"email\": \"96-96.1996@mail.ru\",\n                \"password\": [\n                    \"5555555555q\",\n                    \"5555555555Q\"\n                ]\n         }\n        EXPLANATION: reduce-passwords-on-similar-emails/ contains files sorted by the first 2 letters of\n        the email address. For example 96-000@mail.ru will be located in 96_per_user.json\n        Each file lists all the passwords grouped by user and by edit distance.\n        For example, 96-000@mail.ru had 2 passwords: 090698d and 090698D. The edit distance between them is 1.\n        The edit_distance and the password arrays are of the same length, hence, a first 0 in the edit distance array.\n        Those files are useful to model how users change passwords over time.\n        We can't recover which one was the first password, but a shortest hamiltonian path algorithm is run\n        to detect the most probably password ordering for a user. For example:\n        hello =\u003e hello1 =\u003e hell@1 =\u003e hell@11 is the shortest path.\n        We assume that users are lazy by nature and that they prefer to change their password by the lowest number\n        of characters.\n```\n\nRun the data processing alone:\n\n```\npython3 run_data_processing.py --breach_compilation_folder \u003cBreachCompilation path\u003e --output_folder ~/BreachCompilationAnalysis\n```\n\nIf the dataset is too big for you, you can set `max_num_files` to something between 0 and 2000.\n \n- Make sure you have enough free memory (8GB should be enough).\n- It took 1h30m to run on a Intel(R) Core(TM) i7-6900K CPU @ 3.20GHz (on a single thread).\n- Uncompressed output is around 45G.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fphilipperemy%2Ftensorflow-1.4-billion-password-analysis","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fphilipperemy%2Ftensorflow-1.4-billion-password-analysis","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fphilipperemy%2Ftensorflow-1.4-billion-password-analysis/lists"}