{"id":17931426,"url":"https://github.com/the8472/mldht","last_synced_at":"2025-03-24T05:31:57.482Z","repository":{"id":29142801,"uuid":"32672920","full_name":"the8472/mldht","owner":"the8472","description":"Bittorrent Mainline DHT implementation in java","archived":false,"fork":false,"pushed_at":"2021-08-27T20:58:11.000Z","size":53278,"stargazers_count":134,"open_issues_count":7,"forks_count":45,"subscribers_count":16,"default_branch":"master","last_synced_at":"2023-11-07T10:26:23.664Z","etag":null,"topics":["bittorrent","cli","dht","kademlia","library","p2p"],"latest_commit_sha":null,"homepage":null,"language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mpl-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/the8472.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-03-22T11:31:32.000Z","updated_at":"2023-10-24T07:56:35.000Z","dependencies_parsed_at":"2022-07-10T20:35:24.269Z","dependency_job_id":null,"html_url":"https://github.com/the8472/mldht","commit_stats":null,"previous_names":[],"tags_count":2,"template":null,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/the8472%2Fmldht","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/the8472%2Fmldht/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/the8472%2Fmldht/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/the8472%2Fmldht/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/the8472","download_url":"https://codeload.github.com/the8472/mldht/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":221935653,"owners_count":16904325,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bittorrent","cli","dht","kademlia","library","p2p"],"created_at":"2024-10-28T21:21:46.486Z","updated_at":"2024-10-28T21:21:46.935Z","avatar_url":"https://github.com/the8472.png","language":"Java","funding_links":[],"categories":["网络编程"],"sub_categories":["Spring Cloud框架"],"readme":"# mldht\n\n[![Build Status (travis)](https://travis-ci.org/the8472/mldht.svg?branch=master)](https://travis-ci.org/the8472/mldht)\n[![Build Status (gitlab)](https://gitlab.com/the8472/mldht/badges/master/pipeline.svg)](https://gitlab.com/the8472/mldht/commits/master)\n[![Jitpack maven repo](https://jitpack.io/v/the8472/mldht.svg)](https://jitpack.io/#the8472/mldht)\n[![bintray maven repo](https://img.shields.io/badge/maven-bintray-green.svg?style=flat-square)](https://bintray.com/the8472/maven/mldht)\n\n\n\nA java library and standalone node implementing the Kademlia-based bittorrent mainline DHT, with long-running server-class nodes in mind.\n\nOriginally developed as [DHT plugin](http://azsmrc.sourceforge.net/index.php?action=plugin-mldht) for Azureus/[Vuze](http://dev.vuze.com/)\n\n## Features\n\nImplemented specs:\n\n| Spec | Title | Status |\n|------|-------|--------|\n|[BEP5](http://bittorrent.org/beps/bep_0005.html)|Bittorrent DHT| Yes |\n|[BEP32](http://bittorrent.org/beps/bep_0032.html)|IPv6| Yes |\n|[BEP33](http://bittorrent.org/beps/bep_0033.html)|Scrapes| Yes |\n|[BEP42](http://bittorrent.org/beps/bep_0042.html)|DHT Announce Security| Partial; only the `ip` fields for external address discovery are supported |\n|[BEP9](http://bittorrent.org/beps/bep_0009.html)|Metadata exchange| Partial; only fetching is supported |\n|[libtorrent.org](http://www.libtorrent.org/dht_extensions.html)| Extended `get_peers` response\u003cbr\u003e Forward compatibility\u003cbr\u003e Client identification|Yes|\n|[BEP45](http://bittorrent.org/beps/bep_0045.html)|multi-homing/multi-address mode|Yes|\n|[BEP44](http://bittorrent.org/beps/bep_0044.html)|Arbitrary data storage|Yes|\n|[BEP50](http://bittorrent.org/beps/bep_0050.html)|Pub/Sub|No|\n|[BEP51](http://bittorrent.org/beps/bep_0051.html)|DHT Infohash Indexing|Yes| \n\nAdditional:\n\n- high-performance implementation without compromising correctness, i.e. the node will be a good citizen\n - can process 20k packets per second on a single Xeon core\n- low latency lookups by using adaptive timeouts and a secondary routing table/cache tuned for RTT instead of stability\n- export of passively observed \\\u003cinfohash, ip\\\u003e tuples to redis to survey torrent activity\n- remote CLI for common DHT operations\n- full automatic torrent indexing (active and passive dht indexing + metadata exchange)\n\n## Dependencies\n\n- java ≥ 8\n- maven 3.1 (building)\n\ninstalled via maven:\n\n- ed25519-java\n- junit 4.x (tests)\n\n## build\n\n    git clone https://github.com/the8472/mldht.git .\n    mvn package dependency:copy-dependencies appassembler:assemble\n    # install symlink scripts to ~/bin/ \n    mvn antrun:run@link\n    \n## embedding as library\n\nSee [docs/use-as-library.md](docs/use-as-library.md) for further information.\nMaven repos are linked in the badges.\n\n## run in standalone mode\n\n    mkdir -p work\n    cd work\n    ../bin/mldht-daemon\n    # or manually\n    # java -cp \"../target/*:../target/dependency/*\" the8472.mldht.Launcher \u0026\n    \nthis will create various files in the current working directory\n- `config.xml`, change settings as needed, core settings will be picked up on file modification\n- `shutdown`, touch to cleanly shutdown running process (SIGHUP works too)\n- `*-table.cache`, persisted routing table for the ipv4/6 dhts, respectively\n- `baseID.config`, persisted node ID\n- `logs/*`, various diagnostics and log files\n- `.keys/`, default storage directory for BEP44 private keys. used by the CLI\n\n**Security note:** the shell script launches the JVM with a debug port bound to localhost for easier maintenance, thus allowing arbitrary code execution with the current user's permissions. In a multi-user environment a custom script with debugging disabled should be used    \n\n\n## network configuration\n\n* stateful NATs or firewalls should be put into stateless mode/disable connection tracking and use static forwarding rules for the configured local ports [default: 49001].\u003cbr\u003eOtherwise state table overflows may occur which lead to dropped packets and degraded performance.\n* nat/firewall rules should not assume any particular remote port, as other DHT nodes are free to chose their own.\n* If no publicly routable IPv6 address is available then IPv6 should be disabled\n* If only NATed IPv4 addresses are available then multihoming mode should be disabled\n* The length of network interface send queues should be increased when the DHT node is operated in multihoming mode on a server with many public IPs.\u003cbr\u003eThis is necessary because UDP sends may be silently dropped when the send queue is full and DHT traffic can be very bursty, easily saturating too-small queues\u003cbr\u003eCheck system logs or netstat statistics to see if outgoing packets are dropped.\n* For similar reasons the maximum socket receive buffer size should be set to at least 2MB, which is the amount this implementation will request when configuring its sockets\n\n\n## optional components\n\nSome features are not enabled out of the box because they only require external infrastructure, provide public services or would cause extra traffic.\n\nThey are enabled by adding or uncommenting a `\u003ccomponent\u003e\u003cclassName\u003e...\u003c/className\u003e\u003c/component\u003e` entry to the config.xml \n\n\n* `the8472.mldht.cli.Server` to enable the remote CLI\n* `the8472.mldht.indexing.TorrentDumper` obtains infohashes from incoming traffic, then does all the necessary work to fetch them. can acquire approximately 0.3 torrents per second on a single-homed setup without firewall.\n* `the8472.mldht.indexing.ActiveLookupProvider` raw TCP interface for requesting DHT scrapes on port 36578. just send infohashes in hex, newline separated\n* `the8472.mldht.indexing.OpentrackerLiveSync` implements a lan-local multicast sender for opentracker's IPv4 live sync. for passively observed DHT lookups will be inserted as peers in opentracker instance. opentracker instance can then be used as source for DHT statistics as if it were just another tracker\n* `the8472.mldht.PassiveRedisIndexer` obtains statistics on peers seen on particular infohashes\n\n### remote-cli\n\nlaunch daemon with\n\n```xml\n    \u003ccomponent\u003e\n      \u003cclassName\u003ethe8472.mldht.cli.Server\u003c/className\u003e\n    \u003c/component\u003e\n```\n\nrun CLI client with\n\n```\nbin/mldht-remote-cli help\n# or manually:\n# java -cp \"target/*\" the8472.mldht.cli.Client help\n```\n\navailable commands (subject to change):\n\n```\nHELP                                                 - prints this help\nPING ip port                                         - continuously pings a DHT node with a 1 second interval\nGET hash [salt]                                      - perform a BEP44 get\nPUT -f \u003cinput-path\u003e [-keyfile \u003cpath\u003e] [-salt \u003csalt\u003e]\nPUT \u003cinput\u003e [-keyfile \u003cpath\u003e] [-salt \u003csalt\u003e]         - perform a BEP44 put, specifying a salt or keyfile implies a mutable put, immutable otherwise. data will be read from file or as single argument\nGETTORRENT [infohash...]                             - peer lookup for \u003cinfohash(es)\u003e, then attempt metadata exchange, then write .torrent file(s) to the current working directory\nGETPEERS [infohash...] [-fast]                       - peer lookup for \u003cinfohash(es)\u003e, print ip address/port tuples\nBURST [count]                                        - run a batch of find_node lookups to random target IDs. intended test the attainable throughput for active lookups, subject to internal throttling\n```\n\n\n**Security note:** The CLI Server component listens on localhost, accepting commands without authentication from any user on the system. It is recommended to not use this component in a multi-user environment.\n\n\n\n### redis statistics export\n\n\n```xml\n    \u003ccomponent xsi:type=\"mldht:redisIndexerType\"\u003e\n      \u003cclassName\u003ethe8472.mldht.PassiveRedisIndexer\u003c/className\u003e\n      \u003caddress\u003e127.0.0.1\u003c/address\u003e\u003c!-- additional parameter to allow exporting to other hosts --\u003e\n    \u003c/component\u003e\n```\n\n\n### custom components\n\nSimply implement [\u003ctt\u003eComponent\u003c/tt\u003e](src/the8472/mldht/Component.java) and configure the launcher to include it on startup through the config.xml:\n\t\n```xml\n    \u003ccomponent\u003e\n      \u003cclassName\u003eyour.class.name.Here\u003c/className\u003e\n    \u003c/component\u003e\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthe8472%2Fmldht","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fthe8472%2Fmldht","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthe8472%2Fmldht/lists"}