{"id":19302491,"url":"https://github.com/bitlytwiser/tormonger","last_synced_at":"2026-06-18T15:32:05.241Z","repository":{"id":39673031,"uuid":"244802128","full_name":"BitlyTwiser/torMonger","owner":"BitlyTwiser","description":"Recursive Tor network crawler","archived":false,"fork":false,"pushed_at":"2022-05-27T23:34:40.000Z","size":7415,"stargazers_count":3,"open_issues_count":3,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-02-24T01:51:06.204Z","etag":null,"topics":["crawler","go","golang","tor"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/BitlyTwiser.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-03-04T03:52:39.000Z","updated_at":"2024-09-17T01:49:15.000Z","dependencies_parsed_at":"2022-09-20T07:23:51.349Z","dependency_job_id":null,"html_url":"https://github.com/BitlyTwiser/torMonger","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/BitlyTwiser/torMonger","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BitlyTwiser%2FtorMonger","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BitlyTwiser%2FtorMonger/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BitlyTwiser%2FtorMonger/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BitlyTwiser%2FtorMonger/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/BitlyTwiser","download_url":"https://codeload.github.com/BitlyTwiser/torMonger/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BitlyTwiser%2FtorMonger/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34497231,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-18T02:00:06.871Z","response_time":128,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["crawler","go","golang","tor"],"created_at":"2024-11-09T23:22:23.775Z","updated_at":"2026-06-18T15:32:05.211Z","avatar_url":"https://github.com/BitlyTwiser.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"```\n    ███      ▄██████▄     ▄████████   ▄▄▄▄███▄▄▄▄    ▄██████▄  ███▄▄▄▄      ▄██████▄     ▄████████    ▄████████ \n▀█████████▄ ███    ███   ███    ███ ▄██▀▀▀███▀▀▀██▄ ███    ███ ███▀▀▀██▄   ███    ███   ███    ███   ███    ███ \n   ▀███▀▀██ ███    ███   ███    ███ ███   ███   ███ ███    ███ ███   ███   ███    █▀    ███    █▀    ███    ███ \n    ███   ▀ ███    ███  ▄███▄▄▄▄██▀ ███   ███   ███ ███    ███ ███   ███  ▄███         ▄███▄▄▄      ▄███▄▄▄▄██▀ \n    ███     ███    ███ ▀▀███▀▀▀▀▀   ███   ███   ███ ███    ███ ███   ███ ▀▀███ ████▄  ▀▀███▀▀▀     ▀▀███▀▀▀▀▀   \n    ███     ███    ███ ▀███████████ ███   ███   ███ ███    ███ ███   ███   ███    ███   ███    █▄  ▀███████████ \n    ███     ███    ███   ███    ███ ███   ███   ███ ███    ███ ███   ███   ███    ███   ███    ███   ███    ███ \n   ▄████▀    ▀██████▀    ███    ███  ▀█   ███   █▀   ▀██████▀   ▀█   █▀    ████████▀    ██████████   ███    ███ \n                         ███    ███                                                                  ███    ███ \n// Mainteiner: BitlyTwiser\n// License: MIT\n```\n\n# torMonger\n- A recursive Tor network crawler\n\n## Tor:\n- For this application to function, open does need to have Tor installed and utilize the PSQL instance built via the docker-compose file. \n- for linux (debian/ubuntu, or any distro using apt) ```sudo apt install tor```\n- When starting the application, you can select the given port to match what exists within your torrc configuration file.\n- ```sudo vim /etc/tor/torrc```\n- Then run tor from the cli (or tor browser) and connect\n- ```tor```\n- You can now run torMonger using the port designated in the torrc file.\n- Validate ports and the service using netstat.\n- ```netstat -tulpn```\n- Note: If you execute ```tor``` via the cli, the application initiates with port 9050. This is the default port used with tormonger, however, if you execute tor via the tor web browser, 9150 is used.\n\n## Usage:\n- The torMonger application can be installed by compiling the Go binary and running the installation script.\n- With the use of Go modules you can perform the installation with ```go install``` then compile the binary.\n- ```sudo ./install.sh```\n- This application was tested and ran on Linux. \n- After installation, one can run the command line tool.\n- ```torMonger -url http://zqktlwiuavvvqqt4ybvgvi7tyo4hjl5xgfuvpdf6otjiycgwqbym2qad.onion/wiki/ -t 1```\n- The above onion link is associated with \"The Hidden Wiki\".\n- The ```-threads``` flag spawns the selected number of processes to ingest data.\n\n- When the application is executed one would expect to see the following output:\n![Output](./images/running_example.png)\n\n#### Env Vars:\n- The application takes advantage of a .env file for loading database credentials and values.\n- The .env is included within the project and can be altered to your liking. \n- Just match the .env credentials with that within the docker-compose file to connect to the database.\n\n## Docker Images:\n- All data is initially stored within the PSQL database instance, the install.sh script will create a directory ```/var/tmp/tormonger_data```.\n- Errors logs are within the logs table.\n- Simply just run docker-compose against the provided yaml file to spawn all necessary containers in the environment.\n- ```docker-compose up -d --build```\n- The following command with daemonize the docker processes and rebuild the containers images from scratch\n- The docker image pulls in the ```create_tables.sql``` file which is executed when the image is built. this will create all the tables and foreign key associations for you.\n\n## Data stroage and Example data:\n- The primary column is the tormonger_data column, which stores the base link url and the base64 encoded link hash.\n![tormonger data column example](./images/tormonger_data_column.png)\n- When the application is running succesfully, you will obtain snapshots of the html data from each webpage within the html_data table.\n![html data table](./images/html_table_example.png)\n- The html is stored in the html_data column, which is direct html that can be pulled out and viewed.\n![html data example](./images/html_example.png)\n- subdirectories can be seen in the ```tormonger_data_sub_directories``` table.\n![subdirectories](./images/sub_dirs.png)\n- you can search for subdirs based off of the id for the link you desire and search html data either by subdirectory or by main link id.\n- index's were used on the primary elements of each column for faster lookup times.\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbitlytwiser%2Ftormonger","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbitlytwiser%2Ftormonger","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbitlytwiser%2Ftormonger/lists"}