{"id":37160741,"url":"https://github.com/hktalent/dht","last_synced_at":"2026-01-14T19:07:48.435Z","repository":{"id":57660152,"uuid":"474888561","full_name":"hktalent/dht","owner":"hktalent","description":"BitTorrent DHT Protocol \u0026\u0026 DHT Spider，faster than shiyanhui/dht","archived":false,"fork":true,"pushed_at":"2022-06-01T00:54:25.000Z","size":11373,"stargazers_count":7,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2023-07-28T12:56:35.902Z","etag":null,"topics":["bittorrent","dht","golang","golang-library","protocol","spider","torrent"],"latest_commit_sha":null,"homepage":"http://51pwn.com","language":"Go","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":"shiyanhui/dht","license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/hktalent.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-03-28T07:16:55.000Z","updated_at":"2023-07-07T19:11:39.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/hktalent/dht","commit_stats":null,"previous_names":[],"tags_count":38,"template":null,"template_full_name":null,"purl":"pkg:github/hktalent/dht","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hktalent%2Fdht","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hktalent%2Fdht/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hktalent%2Fdht/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hktalent%2Fdht/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/hktalent","download_url":"https://codeload.github.com/hktalent/dht/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hktalent%2Fdht/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28431210,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-14T18:57:19.464Z","status":"ssl_error","status_checked_at":"2026-01-14T18:52:48.501Z","response_time":107,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bittorrent","dht","golang","golang-library","protocol","spider","torrent"],"created_at":"2026-01-14T19:07:47.633Z","updated_at":"2026-01-14T19:07:48.427Z","avatar_url":"https://github.com/hktalent.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"![](https://raw.githubusercontent.com/hktalent/dht/master/dht.svg)\n\u003cimg width=\"830\" src=\"https://user-images.githubusercontent.com/18223385/161761145-87883c44-9fa6-49f0-b113-5b1f8c37964e.png\"\u003e\n\u003cimg width=\"410\" src=\"https://user-images.githubusercontent.com/18223385/162596289-e7b928fa-6e74-49e7-b663-5738de823149.png\"\u003e\n\u003cimg width=\"410\" alt=\"image\" src=\"https://user-images.githubusercontent.com/18223385/162596922-315c3408-9c39-4e4c-a85c-acd7100ce581.png\"\u003e\n\n## what's the new\n- :white_check_mark: update all depend mod to new\n- :white_check_mark: update to go 1.18\n- :white_check_mark: and config.LocalNodeId\n- :white_check_mark: 45396 DHT tracker server ips，now,fly in at high speed to DHT network\n- :white_check_mark: Rich annotations\n- :white_check_mark: Friendly UML diagram rendering\n- :white_check_mark: china,please use VPN over GWF\n- :white_check_mark: fix Stuttering problem at startup\n- :white_check_mark: fix do one time bug,now to tick 30 Second to do it\n- :white_check_mark: fix public ip changed, cleanAll blackIp to do join\n\n## Introduction\n\nDHT implements the bittorrent DHT protocol in Go. Now it includes:\n\n- [BEP-3 (part)](http://www.bittorrent.org/beps/bep_0003.html)\n- [BEP-5](http://www.bittorrent.org/beps/bep_0005.html)\n- [BEP-9](http://www.bittorrent.org/beps/bep_0009.html)\n- [BEP-10](http://www.bittorrent.org/beps/bep_0010.html)\n\nIt contains two modes, the standard mode and the crawling mode. The standard\nmode follows the BEPs, and you can use it as a standard dht server. The crawling\nmode aims to crawl as more metadata info as possiple. It doesn't follow the\nstandard BEPs protocol. With the crawling mode, you can build another [BTDigg](http://btdigg.org/).\n\n[bthub.io](http://bthub.io) is a BT search engine based on the crawling mode.\n\n## Installation\n```bash\n\ngo get -u github.com/hktalent/dht@latest\n\n```\n\n## Example\n\nBelow is a simple spider. You can move [here](https://github.com/hktalent/dht/blob/master/sample)\nto see more samples.\n\n$ cat $PWD/config/elasticsearch.yml\n```\ncluster.name: my-application\nnode.name: node-1\npath.data: /usr/share/elasticsearch/data\npath.logs: /usr/share/elasticsearch/logs\nnetwork.host: 0.0.0.0\ntransport.host: 0.0.0.0\nnetwork.publish_host: 192.168.0.107\nhttp.port: 9200\ndiscovery.seed_hosts: [ \"192.168.0.112:9300\",\"192.168.0.107:9301\",\"192.168.0.107:9302\", \"192.168.0.107:9300\"]\ncluster.initial_master_nodes: [ \"192.168.0.112:9300\",\"192.168.0.107:9301\",\"192.168.0.107:9302\", \"192.168.0.107:9300\"]\ncluster.routing.allocation.same_shard.host: true\ndiscovery.zen.fd.ping_timeout: 1m\ndiscovery.zen.fd.ping_retries: 5\nhttp.cors.enabled: true\nhttp.cors.allow-origin: \"*\"\nhttp.cors.allow-methods : OPTIONS, HEAD, GET, POST, PUT, DELETE\nhttp.cors.allow-headers : Authorization, X-Requested-With,X-Auth-Token,Content-Type, Content-Length\ntransport.tcp.port: 9300\nhttp.max_content_length: 400mb\nindices.query.bool.max_clause_count: 20000\ncluster.routing.allocation.disk.threshold_enabled: false\n```\n\n```bash\ncd sample/spider\ngo build spider.go\ndocker run --restart=always --ulimit nofile=65536:65536 -e \"discovery.type=single-node\" --net esnet -p 9200:9200 -p 9300:9300 -d --name es -v $PWD/logs:/usr/share/elasticsearch/logs -v $PWD/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml -v $PWD/config/jvm.options:/usr/share/elasticsearch/config/jvm.options  -v $PWD/data:/usr/share/elasticsearch/data  hktalent/elasticsearch:7.16.2\n\n# your Elasticsearch is http://127.0.0.1:9200/dht_index\n./spider -resUrl=\"http://127.0.0.1:9200/dht_index/_doc/\" -address=\":0\"\nopen http://127.0.0.1:9200/dht_index/_search?q=GB%20and%20mp4\u0026pretty=true\nopen http://127.0.0.1:9200/dht_index/_search?q=1080P%20GB%20and%20mp4\u0026pretty=true\nopen http://127.0.0.1:9200/dht_index/_search?q=pentest%20pdf\u0026pretty=true\n```\n\n```go\nimport (\n    \"fmt\"\n    \"github.com/hktalent/dht\"\n)\n\nfunc main() {\n    downloader := dht.NewWire(65535)\n    go func() {\n        // once we got the request result\n        for resp := range downloader.Response() {\n            fmt.Println(resp.InfoHash, resp.MetadataInfo)\n        }\n    }()\n    go downloader.Run()\n\n    config := dht.NewCrawlConfig()\n    config.OnAnnouncePeer = func(infoHash, ip string, port int) {\n        // request to download the metadata info\n        downloader.Request([]byte(infoHash), ip, port)\n    }\n    d := dht.New(config)\n\n    d.Run()\n}\n```\n\n## Download\n\nYou can download the demo compiled binary file [here](https://github.com/hktalent/dht/tags).\n\n## Note\n\n- The default crawl mode configure costs about 300M RAM. Set **MaxNodes**\n  and **BlackListMaxSize** to fit yourself.\n- Now it cant't run in LAN because of NAT.\n\n## TODO\n\n- :white_check_mark: NAT Traversal.\n- :white_check_mark: Implements the full BEP-3.\n- :white_check_mark: Optimization.\n\n## FAQ\n\n#### Why it is slow compared to other spiders ?\n\nWell, maybe there are several reasons.\n\n- DHT aims to implements the standard BitTorrent DHT protocol, not born for crawling the DHT network.\n- NAT Traversal issue. You run the crawler in a local network.\n- It will block ip which looks like bad and a good ip may be mis-judged.\n\n## License\n\nMIT, read more [here](https://github.com/hktalent/dht/blob/master/LICENSE)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhktalent%2Fdht","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhktalent%2Fdht","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhktalent%2Fdht/lists"}