{"id":13580264,"url":"https://github.com/blacklanternsecurity/credshed","last_synced_at":"2025-04-06T02:31:20.392Z","repository":{"id":113810975,"uuid":"175911445","full_name":"blacklanternsecurity/credshed","owner":"blacklanternsecurity","description":"credshed - a scalable database for credential leaks.  Written in Python, it can easily ingest poorly-formatted files or entire directories into a searchable database.  MongoDB is used in the backend.","archived":false,"fork":false,"pushed_at":"2021-05-13T20:24:06.000Z","size":469,"stargazers_count":54,"open_issues_count":3,"forks_count":9,"subscribers_count":7,"default_branch":"mongo-sharded-v2","last_synced_at":"2024-11-05T19:42:07.074Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/blacklanternsecurity.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2019-03-16T01:19:55.000Z","updated_at":"2024-09-16T19:53:10.000Z","dependencies_parsed_at":"2024-01-16T20:29:43.744Z","dependency_job_id":"c388b33d-16c9-4ef1-89f1-0fbfba90d963","html_url":"https://github.com/blacklanternsecurity/credshed","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/blacklanternsecurity%2Fcredshed","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/blacklanternsecurity%2Fcredshed/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/blacklanternsecurity%2Fcredshed/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/blacklanternsecurity%2Fcredshed/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/blacklanternsecurity","download_url":"https://codeload.github.com/blacklanternsecurity/credshed/tar.gz/refs/heads/mongo-sharded-v2","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247425892,"owners_count":20937033,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-01T15:01:49.265Z","updated_at":"2025-04-06T02:31:20.073Z","avatar_url":"https://github.com/blacklanternsecurity.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# c r e d s h e d\nA full-featured solution for injesting, organizing, storing, and querying public credential leaks.  Injests gigantic files or entire directories with ease.  (It will find every email address in your `/etc` directory if you tell it to!)\nIncludes native Pastebin-scraping functionality.\n\n![credshed-gui-screenshot](https://user-images.githubusercontent.com/20261699/67125567-1d0fc400-f1c3-11e9-850c-b3baa620e791.png)\n\nPastebin Scraper Report\n\n## CLI Usage\n~~~\n$ ./credshed-cli.py --help\nusage: credshed-cli.py [-h] [-q QUERY_TYPE] [-i INGEST [INGEST ...]] [-f] [-db] [-s] [-d [SOURCE_ID [SOURCE_ID ...]]] [-dd] [--drop] [--threads THREADS] [--print0] [--limit LIMIT] [-u] [-v] [--debug] [search [search ...]]\n\npositional arguments:\n  search                search term(s)\n\noptional arguments:\n  -h, --help            show this help message and exit\n  -q QUERY_TYPE, --query-type QUERY_TYPE\n                        query type (email, domain, or username)\n  -i INGEST [INGEST ...], --ingest INGEST [INGEST ...]\n                        import files or directories into the database\n  -f, --force-ingest    also ingest files which have already been imported\n  -db, --db-stats       show all imported leaks and DB stats\n  -s, --stdout          when importing, write to stdout instead of database (null-byte delimited, use tr '\\0')\n  -d [SOURCE_ID [SOURCE_ID ...]], --delete-leak [SOURCE_ID [SOURCE_ID ...]]\n                        delete leak(s) from database, e.g. \"1-3,5,7-9\"\n  -dd, --deduplication  deduplicate accounts ahead of time by loading them into memory\n  --drop                delete the entire database D:\n  --threads THREADS     number of threads for import operations\n  --print0              delimit search results by null byte instead of colon\n  --limit LIMIT         limit number of results (default: unlimited)\n  -u, --unattended      auto-detect import fields without user interaction\n  -v, --verbose         show what is happening\n  --debug               display detailed debugging info\n~~~\n\n## Setup\nCredshed uses mongodb to store data.  Setup can be as simple as `docker run -p 27017 mongo`, although I highly recommend using `deploy/docker-compose.yml`, which contains optimizations for heavy write loads.\n1. Create a directory where you'd like to store the data, and chmod it to UID 999 (the default mongodb user)\n~~~\n$ mkdir /data\n$ chown 999:999 /data\n~~~\n2. Edit `deploy/docker-compose.yml` and ensure the volume is pointing to the new data directory you just created.  And for the love of heaven, pick a secure password.\n3. Bring up the mongodb service\n~~~\n$ cd deploy\n$ docker-compose up\n~~~\n4. Place the same password in `credshed.config` and make sure the host and port matches your database configuration\n6. Verify the config is valid by searching the database (it will be empty at this point, that's fine)\n~~~\n$ ./credshed-cli.py test@example.com\n~~~\n7. (Optional) Set the database to start automatically:\n  - First, stop the database\n  - Edit `WorkingDirectory` in `deploy/credshed.service` to match the directory where it is installed\n  - Install, enable, and start the credshed systemd service\n~~~\n$ sudo cp credshed/docker/credshed.service /etc/systemd/system/\n$ sudo systemctl enable credshed.service --now\n# check on its status\n$ journalctl -xefu credshed.service\n~~~\n8. (Optional) If you want to enable logging, create the directory `/var/log/credshed` and make sure whichever user is running `credshed-cli.py` has write access\n\n\n## Example 1: Extract all compressed files\nCredshed will locate every compressed file, determine its format (e.g. zip, gzip, 7zip, etc.), and attempt to extract it.  This includes converting XLSX files to CSV\n~~~\n# It's a good idea to run this a few times until all possible archives are extracted.\n$ ./filestore-cli.py ./filestore-cli.py --extract --delete /mnt/leaks\n~~~\n\n## Example 2: Load files (or directories) into database\nSearch for emails in every file and import into the database.  Credshed will automatically determine the encoding of each file.  It detects and handles emails, hashes, and even SQL statements.\n~~~\n$ ./credshed-cli.py --unattended --ingest /mnt/leaks\n~~~","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fblacklanternsecurity%2Fcredshed","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fblacklanternsecurity%2Fcredshed","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fblacklanternsecurity%2Fcredshed/lists"}