{"id":23739237,"url":"https://github.com/iudicium/pryingdeep","last_synced_at":"2026-01-14T12:47:39.923Z","repository":{"id":208099529,"uuid":"698935684","full_name":"iudicium/pryingdeep","owner":"iudicium","description":"Prying Deep - An OSINT tool to collect intelligence on the dark web.","archived":true,"fork":false,"pushed_at":"2024-09-03T09:13:38.000Z","size":37050,"stargazers_count":530,"open_issues_count":1,"forks_count":43,"subscribers_count":5,"default_branch":"main","last_synced_at":"2025-09-04T14:43:50.557Z","etag":null,"topics":["crawler","darkweb","go","gocolly","golang-osint","onion","osint","osint-tools","pryingdeep","security-tools"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/iudicium.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2023-10-01T12:43:17.000Z","updated_at":"2025-08-30T14:06:34.000Z","dependencies_parsed_at":"2024-01-04T10:38:58.533Z","dependency_job_id":"95d3b329-62f3-4f5d-b1b6-e949e5b77974","html_url":"https://github.com/iudicium/pryingdeep","commit_stats":null,"previous_names":["iudicium/pryingdeep"],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/iudicium/pryingdeep","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/iudicium%2Fpryingdeep","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/iudicium%2Fpryingdeep/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/iudicium%2Fpryingdeep/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/iudicium%2Fpryingdeep/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/iudicium","download_url":"https://codeload.github.com/iudicium/pryingdeep/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/iudicium%2Fpryingdeep/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28420814,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-14T10:47:48.104Z","status":"ssl_error","status_checked_at":"2026-01-14T10:46:19.031Z","response_time":107,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["crawler","darkweb","go","gocolly","golang-osint","onion","osint","osint-tools","pryingdeep","security-tools"],"created_at":"2024-12-31T09:36:40.416Z","updated_at":"2026-01-14T12:47:39.904Z","avatar_url":"https://github.com/iudicium.png","language":"Go","funding_links":[],"categories":["Go","[↑](#-content) 🛠️ Tools"],"sub_categories":[],"readme":"\n\u003c!-- PROJECT LOGO --\u003e\n\u003cbr /\u003e\n\u003cdiv align=\"center\"\u003e\n  \u003ca href=\"https://raw.githubusercontent.com/iudicium/pryingdeep/main/web/static/logo.png\"\u003e\n    \u003cimg src=\"web/static/logo.png\" alt=\"Logo\" width=\"350\" height=\"200\"\u003e\n  \u003c/a\u003e\n\n  \u003ch3 align=\"center\"\u003ePrying Deep\u003c/h3\u003e\n\n  \u003cp align=\"center\"\u003e\n    \u003cb\u003eOSINT intellegence gathering tool for the dark/clear web\n    \u003cbr /\u003e\n    \u003cbr /\u003e\n    \u003ca href=\"https://github.com/iudicium/pryingdeep/issues\"\u003eReport Bug\u003c/a\u003e\n    \u003ca href=\"https://github.com/iudicium/pryingdeep/issues\"\u003eRequest Feature\u003c/a\u003e\n  \u003c/p\u003e\n\u003c/div\u003e\n\n\n\n\u003c!-- TABLE OF CONTENTS --\u003e\n\u003cdetails\u003e\n  \u003csummary\u003eTable of Contents\u003c/summary\u003e\n  \u003col\u003e\n    \u003cli\u003e\n      \u003ca href=\"#about-the-project\"\u003eAbout The Project\u003c/a\u003e\n    \u003c/li\u003e\n    \u003cli\u003e\n      \u003ca href=\"#getting-started\"\u003eGetting Started\u003c/a\u003e\n      \u003cul\u003e\n        \u003cli\u003e\u003ca href=\"#prerequisites\"\u003ePrerequisites\u003c/a\u003e\u003c/li\u003e\n        \u003cli\u003e\u003ca href=\"#manual-installation\"\u003eManual Installation\u003c/a\u003e\u003c/li\u003e\n        \u003cli\u003e\u003ca href=\"#binary-installation\"\u003eBinary Installation\u003c/a\u003e\u003c/li\u003e\n      \u003c/ul\u003e\n    \u003c/li\u003e\n    \u003cli\u003e\n      \u003ca href=\"#docker\"\u003eDocker\u003c/a\u003e\n      \u003cul\u003e\n        \u003cli\u003e\u003ca href=\"#tor\"\u003eTor\u003c/a\u003e\u003c/li\u003e\n      \u003c/ul\u003e\n    \u003c/li\u003e\n    \u003cli\u003e\u003ca href=\"#usage\"\u003eUsage\u003c/a\u003e\u003c/li\u003e\n    \u003cli\u003e\u003ca href=\"#roadmap\"\u003eRoadmap\u003c/a\u003e\u003c/li\u003e\n    \u003cli\u003e\u003ca href=\"#contributing\"\u003eContributing\u003c/a\u003e\u003c/li\u003e\n    \u003cli\u003e\u003ca href=\"#license\"\u003eLicense\u003c/a\u003e\u003c/li\u003e\n    \u003cli\u003e\u003ca href=\"#preview\"\u003ePreview\u003c/a\u003e\u003c/li\u003e\n  \u003c/ol\u003e\n\u003c/details\u003e\n\n\n\n## Translation (Перевод)\n - [Click](./docs/RU_README.MD)\n\n## IMPORTANT\n - This repo is no longer active, sorry guys, I no longer want to keep working on this :)\n\u003c!-- ABOUT THE PROJECT --\u003e\n## About The Project\n\nI've seen multiple projects out there in GitHub, that are crawlers for the deep web,\nbut most of them did not meet *my* standards of OSINT on the deep web.\nSo I decided to create my own deep web OSINT tool.\n\n\nThis tool serves as a reminder that the best practices of OPSEC should always be followed in the deep web.\n\n\u003cu\u003eThe author of this project is not responsible for any possible harm caused by the usage of this tool.\u003c/u\u003e\n\n## What does this tool do?\n\n*Prying Deep crawls dark/clear net platforms and extracts as much intelligence as possible effectively.*\n\n\n---\n\n\u003c!-- GETTING STARTED --\u003e\n## Getting Started\n\n\n### Prerequisites\n\nBefore you can use our OSINT tool, please ensure you have the following dependencies installed:\n\n1. **Docker: (optional)**\n    - You can download and install Docker by following the official installation instructions for your specific operating system:\n    - [Docker Installation Guide](https://docs.docker.com/get-docker/).\n\n2. **Go: (required)**\n    - [Golang Installation Guide](https://go.dev/doc/install).\n\n3. **PostgresSQL: (required if you don't use docker)**\n    - Make sure your `pryingdeep.yaml` environment variables match the environment in `docker-compose.yaml`\n    - [PostgreSQL Installation](https://www.postgresql.org/download/)\n\n\n### Binary Installation\n\n1. Install the binary via:\n\n```Sh\ngo install -v github.com/iudicium/pryingdeep/cmd/pryingdeep@latest\n```\n\n2. Run the following command:\n```sh\npryingdeep install\n```\n3. Adjust the values inside the config folder to your needs.\n\n### Manual Installation\n\n1. Clone the repo: \n   ```sh\n    git clone https://github.com/iudicium/pryingdeep.git\n   ```\n\n2. Adjust the values in the .yaml configuration either through flags or manually.\n   `Database`, `logger`, `tor` all require manual configuration. \u003cbr\u003e\n\n\u003cu\u003eYou *will need to read*\u003c/u\u003e [Colly Docs](https://github.com/gocolly/colly/blob/v1.2.0/colly.go). Also, refer to [Config Explanation](./docs/CONFIG.MD#table-of-contents)\n\n\n3. Build the binary via:\n   - `go build` -\u003e inside the `cmd/pryingdeep` directory\n   - `go build cmd/pryingdeep/pryingdeep.go` -\u003e root directory, binary will also be there.\n\n## Docker\n\nTo start run *pryingdeep* inside a docker container use this command:\n```sh\ndocker-compose up\n```\n\n\n## Config\nRead more each parameter here:\n[config](./docs/CONFIG.MD#table-of-contents)\n\n\n## Tor\nRead more about building and running our tor container here:\n[Tor](./build/package/tor/README.MD#build)\n\n\n\u003c!-- USAGE EXAMPLES --\u003e\n## Usage\n\n```\nPryingdeep specializes in collecting information about dark-web/clearnet websites.\nThis tool was specifically built to extract as much information as possible from a .onion website\n\nUsage:\npryingdeep [command]\n\nAvailable Commands:\ncompletion  Generate the autocompletion script for the specified shell\ncrawl       Start the crawling process\nexport      Export the collected data into a file.\nhelp        Help about any command\ninstall     Installation of config files\n\nFlags:\n-c, --config string   Path to the .yaml configuration. (default \"pryingdeep.yaml\")\n-h, --help            help for pryingdeep\n-z, --save-config     Save chosen options to your .yaml configuration\n-s, --silent          -s to disable logging and run silently\n\n\n```\n\n\n\u003c!-- ROADMAP --\u003e\n## Roadmap\n\n- [x]  Add a user to the \"tor\" docker container so that it doesn't run as root\n- [ ] Acquire a shodan api key for testing the favicon module\n- [ ] Think of a way to acquire IP address of the server\n- [ ] Implement scan command\n- [ ] Implement file identification and search\n- [ ] Move tests into their corresponding packages\n\n\n\u003c!-- CONTRIBUTING --\u003e\n## Contributing\n\n\n1. Fork the Project\n2. Checkout the dev branch (`git checkout dev`)\n3. Add proper documentation to your code.\n4. Use `goimports` to lint your code\n5. Submit a pull requests and add a detailed description of what has been changed.\n\n\n\n\n\u003c!-- LICENSE --\u003e\n## License\n\nDistributed under the GPL-3.0 license. See `LICENSE` for more information.\n\n\n## Video Preview\n\n[![Video Title](https://img.youtube.com/vi/TvRT_lvixI0/maxresdefault.jpg)](https://www.youtube.com/watch?v=TvRT_lvixI0)\n\n---\n\n## Preview\n\n![Preview](./web/static/preview.png)\n![Data Preview](./web/static/data_preview.png)\n![Data Preview 2](./web/static/data_preview2.png)\n\n\n## Support\n\nIf you have found this repository useful and feel generous, you can donate some Monero (XMR) to the following address:\n\n48bEkvkzP3W4SGKSJAkWx2V8s4axCKwpDFf7ZmwBawg5DBSq2imbcZVKNzMriukuPqjCyf2BSax1D3AktiUq5vWk1satWJt\n\nThank you!\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fiudicium%2Fpryingdeep","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fiudicium%2Fpryingdeep","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fiudicium%2Fpryingdeep/lists"}