{"id":20077818,"url":"https://github.com/thejordanprice/p2pspider","last_synced_at":"2025-05-05T22:31:44.456Z","repository":{"id":181287019,"uuid":"108174273","full_name":"thejordanprice/p2pspider","owner":"thejordanprice","description":"Daemon that scrapes the DHT swarm and an express bootstrapped front-end connected to the same mongodb to create an automated magnet db with search.","archived":false,"fork":false,"pushed_at":"2024-07-26T10:55:52.000Z","size":386,"stargazers_count":18,"open_issues_count":0,"forks_count":10,"subscribers_count":4,"default_branch":"master","last_synced_at":"2024-07-27T06:26:17.326Z","etag":null,"topics":["dht","dht-network","nodejs","pm2","self-hosted","spider","torrent","torrents-crawler","website"],"latest_commit_sha":null,"homepage":"https://github.com/thejordanprice/p2pspider","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/thejordanprice.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2017-10-24T19:34:55.000Z","updated_at":"2024-07-27T04:57:44.000Z","dependencies_parsed_at":"2023-07-14T20:07:05.039Z","dependency_job_id":null,"html_url":"https://github.com/thejordanprice/p2pspider","commit_stats":null,"previous_names":["thejordanprice/p2pspider"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thejordanprice%2Fp2pspider","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thejordanprice%2Fp2pspider/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thejordanprice%2Fp2pspider/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thejordanprice%2Fp2pspider/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/thejordanprice","download_url":"https://codeload.github.com/thejordanprice/p2pspider/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224470960,"owners_count":17316710,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dht","dht-network","nodejs","pm2","self-hosted","spider","torrent","torrents-crawler","website"],"created_at":"2024-11-13T15:11:07.688Z","updated_at":"2025-05-05T22:31:44.426Z","avatar_url":"https://github.com/thejordanprice.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# p2pspider - DHT Spider\n\n[![GitHub issues](https://img.shields.io/github/issues/thejordanprice/p2pspider.svg)](https://github.com/thejordanprice/p2pspider/issues)\n[![GitHub stars](https://img.shields.io/github/stars/thejordanprice/p2pspider.svg)](https://github.com/thejordanprice/p2pspider/stargazers)\n[![GitHub forks](https://img.shields.io/github/forks/thejordanprice/p2pspider.svg)](https://github.com/thejordanprice/p2pspider/network)\n[![GitHub license](https://img.shields.io/github/license/thejordanprice/p2pspider.svg)](https://github.com/thejordanprice/p2pspider/blob/master/LICENSE)\n[![Twitter](https://img.shields.io/twitter/url/https/github.com/thejordanprice/p2pspider.svg?style=social)](https://twitter.com/intent/tweet?text=Wow:\u0026url=https%3A%2F%2Fgithub.com%2Fthejordanprice%2Fp2pspider)\n\nA daemon that crawls the BitTorrent DHT network and an Express web application that provides a searchable database of magnet links with real-time updates through WebSockets.\n\n### Intro\n\nDHT Spider can index over 1 million magnets per 24 hours on modest hardware (2GB of RAM and around 2MB/s connection). It's resource-intensive and will use available CPU and RAM, which can be controlled via the 'ecosystem.json' file. On 2GB RAM, it's recommended to use 8 instances of the daemon and 2 of the webserver, all limited at 175MB.\n\n###### Screenshots\n\n![DHT Spider Home Page](docs/screenshots/0.png)\n\n![DHT Spider Search Results](docs/screenshots/1.png)\n\n### Getting Started\n\n```\n# Install dependencies\nnpm install\n\n# Set up configuration\ncp .env.sample .env\n# Edit .env file as needed\n\n# Run the application\nnpm start  # Start the unified application (both crawler and web interface)\n\n# Alternatively, use PM2 for process management\nnpm install -g pm2\nnpm run start:pm2  # Uses the ecosystem.json file\npm2 monit\n```\n\n### Configuration\n\n**You will need to have port 6881 (or your configured port) open to the internet for the DHT crawler to function properly.**\n\nThe application can be configured through the `.env` file:\n\n```\n# Database and server configuration\nREDIS_URI=redis://127.0.0.1:6379\nMONGO_URI=mongodb://127.0.0.1/magnetdb\nSITE_HOSTNAME=http://127.0.0.1:8080\nSITE_NAME=DHT Spider\nSITE_PORT=8080\n\n# Database options: \"mongodb\" or \"sqlite\"\nDB_TYPE=sqlite\n\n# Redis options: \"true\" or \"false\"\nUSE_REDIS=false\n\n# SQLite database file location (only used if DB_TYPE=sqlite)\nSQLITE_PATH=./data/magnet.db\n\n# Elasticsearch options: \"true\" or \"false\"\nUSE_ELASTICSEARCH=false\n\n# Elasticsearch connection\nELASTICSEARCH_NODE=http://localhost:9200\nELASTICSEARCH_INDEX=magnets\n\n# Component control options: \"true\" or \"false\"\nRUN_DAEMON=true\nRUN_WEBSERVER=true\n```\n\nYou can also fine-tune the crawler performance in the daemon.js file:\n\n```javascript\nconst p2p = P2PSpider({\n    nodesMaxSize: 250,\n    maxConnections: 500,\n    timeout: 1000\n});\n```\n\nIt's not recommended to change the `nodesMaxSize` or `maxConnections`, but adjusting the `timeout` may increase indexing speed. Higher timeout values may require more RAM; the maximum recommended value is 5000ms.\n\n#### Component Control\n\nDHT Spider now allows you to run the daemon and webserver components independently:\n\n- **RUN_DAEMON**: Set to \"true\" to run the P2P Spider daemon, or \"false\" to disable it\n- **RUN_WEBSERVER**: Set to \"true\" to run the web server, or \"false\" to disable it\n\nThis flexibility allows you to:\n- Run only the daemon for dedicated crawling\n- Run only the webserver for serving existing data\n- Run both components together (default behavior)\n\nExample usage:\n```bash\n# Run both components (default)\nnode app.js\n\n# Run only the daemon\nRUN_WEBSERVER=false node app.js\n\n# Run only the webserver\nRUN_DAEMON=false node app.js\n```\n\n#### Database and Redis Configuration\n\nDHT Spider supports both MongoDB and SQLite as database options, and Redis usage can be toggled on/off:\n\n- **DB_TYPE**: Choose between \"mongodb\" or \"sqlite\" as your database\n- **USE_REDIS**: Set to \"true\" to use Redis for caching recent infohashes, or \"false\" to disable Redis\n- **SQLITE_PATH**: Path where the SQLite database file will be created (only used when DB_TYPE=sqlite)\n\nSQLite is ideal for smaller deployments with reduced dependencies, while MongoDB is better for large-scale operations. Redis provides caching to prevent duplicate processing of recently seen infohashes.\n\n#### Elasticsearch Configuration\n\nDHT Spider now includes Elasticsearch integration for powerful full-text search capabilities:\n\n- **USE_ELASTICSEARCH**: Set to \"true\" to enable Elasticsearch integration\n- **ELASTICSEARCH_NODE**: URL of your Elasticsearch server (default: http://localhost:9200)\n- **ELASTICSEARCH_INDEX**: Name of the Elasticsearch index to use (default: magnets)\n\nTo bulk index existing data into Elasticsearch, run:\n```bash\nnode utils/bulkIndexToElasticsearch.js\n```\n\nElasticsearch provides significantly improved search performance and relevance, especially for large datasets. When enabled, search queries will use Elasticsearch instead of database queries.\n\n### Features\n\n- Real-time DHT network crawling and magnet link indexing\n- WebSocket-based live updates on the web interface\n- Searchable database of discovered magnet links\n- Statistics page with database information\n- Support for both MongoDB and SQLite databases\n- Elasticsearch integration for powerful full-text search\n- Redis caching for improved performance\n- Responsive web interface with modern design\n\n### Protocols\n\n[bep_0005](http://www.bittorrent.org/beps/bep_0005.html), [bep_0003](http://www.bittorrent.org/beps/bep_0003.html), [bep_0010](http://www.bittorrent.org/beps/bep_0010.html), [bep_0009](http://www.bittorrent.org/beps/bep_0009.html)\n\n### Notes\n\nCluster mode does not work on Windows. On Linux and other UNIX-like operating systems, multiple instances can listen on the same UDP port, which is not possible on Windows due to operating system limitations.\n\n### Notice\n\nPlease don't share the data DHT Spider crawls to the internet. Because sometimes it discovers sensitive/copyrighted/adult material.\n\n### Performance Optimization\n\nTo maximize performance, DHT Spider now includes several optimizations:\n\n#### 1. Redis Caching\nEnable Redis by setting `USE_REDIS=true` in your `.env` file to significantly reduce database load:\n```\n# Redis options: \"true\" or \"false\"\nUSE_REDIS=true\n```\n\n#### 2. Production Mode\nRun the application in production mode for better performance:\n```bash\nnpm run start:prod   # For the web server\nnpm run daemon:prod  # For the DHT crawler\n\n# Or with PM2 (recommended for production)\npm2 start ecosystem.json\n```\n\n#### 3. Optimized PM2 Configuration\nThe included `ecosystem.json` is configured for optimal performance:\n- Web server runs in cluster mode with multiple instances\n- DHT crawler runs in a single instance to avoid duplicate crawling\n- Memory limits prevent excessive resource usage\n\n#### 4. WebSocket Optimizations\nThe WebSocket server includes:\n- Message batching to reduce overhead\n- Client connection health monitoring\n- Throttled broadcasts to prevent excessive updates\n\n#### 5. Elasticsearch Search Optimization\nWhen dealing with large datasets, enable Elasticsearch for improved search performance:\n```\n# Elasticsearch options: \"true\" or \"false\"\nUSE_ELASTICSEARCH=true\n```\n\n#### Monitoring Performance\nMonitor system resources during operation:\n```bash\npm2 monit\n```\n\nIf the application is still slow:\n1. Increase server resources (RAM/CPU)\n2. Use a CDN for static assets\n3. Consider using a dedicated Redis server\n4. Consider using a dedicated Elasticsearch cluster\n5. Scale horizontally with a load balancer\n\n## License\n\nMIT\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthejordanprice%2Fp2pspider","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fthejordanprice%2Fp2pspider","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthejordanprice%2Fp2pspider/lists"}