{"id":17147917,"url":"https://github.com/toby/det","last_synced_at":"2025-04-13T11:40:28.495Z","repository":{"id":57570519,"uuid":"164479167","full_name":"toby/det","owner":"toby","description":"P2P search and discovery on the BitTorrent network","archived":false,"fork":false,"pushed_at":"2019-09-02T21:31:40.000Z","size":116,"stargazers_count":22,"open_issues_count":2,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-03-27T02:43:41.439Z","etag":null,"topics":["bittorrent","bittorrent-client","bittorrent-dht","bittorrent-network","dht","magnet","magnet-link","p2p","p2p-search","torrent-database","trending-torrents"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/toby.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-01-07T19:13:08.000Z","updated_at":"2024-11-11T05:54:11.000Z","dependencies_parsed_at":"2022-09-10T19:10:35.201Z","dependency_job_id":null,"html_url":"https://github.com/toby/det","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/toby%2Fdet","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/toby%2Fdet/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/toby%2Fdet/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/toby%2Fdet/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/toby","download_url":"https://codeload.github.com/toby/det/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248709221,"owners_count":21149150,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bittorrent","bittorrent-client","bittorrent-dht","bittorrent-network","dht","magnet","magnet-link","p2p","p2p-search","torrent-database","trending-torrents"],"created_at":"2024-10-14T21:26:31.300Z","updated_at":"2025-04-13T11:40:28.468Z","avatar_url":"https://github.com/toby.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Detergent\n\nP2P search, discovery and sharing on the BitTorrent network.\n\n## What is it?\n\nDetergent is a new type of BitTorrent client that includes search and discovery\nbacked entirely by P2P. There are no central servers. This stands in contrast\nto traditional BitTorrent indexes like The Pirate Bay. Users run a Detergent\nnode on their own machine and interact with the BitTorrent network locally.\n\n## Installing\n\nDetergent requires `go` 1.11+ to build.\n\n```\ngit clone https://github.com/toby/det.git\ncd det\ngo build\n```\n\n## Usage\n\nDeteregent is still very early in the development process. Not all\nfunctionality is currently available (see *Functional Roadmap*). Mainly, the\ndistributed search and trending features don't exist yet, so `det` requires\nsome time to build up a local database.\n\n### Listening\n\nDetergent uses a command line tool named `det` to build and query the Torrent\ndatabase. The `det` tool has many commands but to get started you have to build\nyour local Torrent database by listening:\n\n`./det listen`\n\nThis will run `det` in listen mode and watch for Torrents that people are\nsharing on the network. You should start to see Torrent file names log to the\nconsole. You can stop listening by typing **Ctrl+C**.\n\n### Querying\n\nThere are a few ways to query the local Torrent database. Searching looks for\nmatching Torrents using a full-text search:\n\n`./det search TERM`\n\nYou can view the most popular Torrents since your first listen (based on\nAnnounces in your DHT) with:\n\n`./det popular`\n\nThere is also a timeline view with most popular Torrents by day:\n\n`./det timeline`\n\nThe query commands can include a `limit` argument to specify the number of\ndesired results:\n\n`./det popular --limit=1000`\n\nOverall system stats can be displayed with:\n\n`./det info`\n\n### Torrenting\n\nWhile primitive in its current state, `det` does offer basic Torrent functionality:\n\n```\n./det download MAGNETURL\n./det resolve MAGNET URL\n```\n\n## Functional Roadmap\n\n- [x] Command line interface\n- [x] Seed Torrents\n- [x] Download Torrents\n- [x] Store and index announces from BitTorrent DHT\n- [x] Resolve and store magnet url Torrent metadata\n- [x] Search Torrent metadata stored on Detergent peer\n- [x] Show popular and trending Torrents\n- [x] Detergent peer discovery\n- [ ] Distributed searching\n- [ ] Web interface\n- [ ] Distributed trending\n- [ ] Content publication\n- [ ] Content curation and promotion\n\n### Command line interface\n\nThe `det` command provides a CLI for accessing Detergent functionality. While\nfunctional, there is work to be done to clean up output and make it easier to\npipe into other commands.\n\n### Seed Torrents\n\nSome Detergent functionality requires the ability to seed files and arbitrary\nbyte slices. This is currently supported from the CLI and in the code. The\n`torrent/storage` interface from [anacrolix/torrent](https://github.com/anacrolix/torrent)\nis implemented for read-only `[]byte` slices in [bytes.go](https://github.com/toby/det/blob/master/bytes.go).\n\n### Download Torrents\n\nAs you can imagine Torrents can be downloaded using `det`. This is provided for\nuser convenience and required for some internal Detergent functionality.\n\n### Store and index announces from BitTorrent DHT\n\nAt the heart of Detergent is the BitTorrent DHT best described by\n[BEP-5](http://www.bittorrent.org/beps/bep_0005.html). At a high level, each\nDetergent peer acts as a functional DHT node that also indexes every Torrent\nannounce that it encounters. Announces are stored individually in a SQLite\ndatabase and provide sorting order for the trending functionality.\n\n### Resolve and store magnet url Torrent metadata\n\nTorrent announces contain an infohash for the Torrent but don't provide any\nmetadata about the files provided by the Torrent (title, file names...). Each\ninfohash discovered on the DHT must be resolved on the Torrent network to\nretrieve user readable information. Detergent has naive parallelization of\nresolving but has much room to improve on performance.\n\n### Search Torrent metadata stored on Detergent peer\n\nTorrent metadata is stored locally in the SQLite database with\n[FTS4](https://www.sqlite.org/fts3.html) full text indexing. Currently only the\ntop level filename is indexed. Soon all files in the Torrent should be indexed.\n\n### Show popular and trending Torrents\n\nBoth `det popular` and `det timeline` provide *very* basic trending analysis.\nThe queries themselves are in [db.sql](https://github.com/toby/det/blob/master/sql/db.sql).\n\n### Detergent peer discovery\n\nDetergent uses an experimental Torrent based peer discovery mechanism. The goal\nis to link Detergent peers together to provide a network for distributed search\nand trending. The process for peer discovery works roughly like this:\n\n1. Detergent peers seed a deterministic `detergent.json` file that contains a protocol string.\n2. Each Detergent peer also shares a deterministic `PEER_ID.json` file containing its Torrent peer id.\n3. Peer IDs and IP addresses of other Torrent clients sharing `detergent.json` are noted.\n4. `PEER_ID.json` files are constructed and seeded for each potential Detergent peer sharing `detergent.json`.\n5. Any `PEER_ID.json` file that can be downloaded should represent a peer that implements the Detergent protocol.\n\nFurther details are available in [discovery.go](https://github.com/toby/det/blob/master/discovery.go).\n\n### Distributed searching\n\nOnce other Detergent peers are known, searches can be propagated outward\nthrough the network with results returned to the querying peer. Detergent peers\nshould search their own local database first. The exact mechanism for\ndistributed search is TBD but will probably include multiple hops through the\npeer network.\n\n### Web interface\n\nDetergent should have a web interface that provides search and trending.\nIdeally media will be streamable in the web view as that's supported by the\nunderlying Torrent library.\n\nIn addition to streaming media, text content including Markdown should be\nrendered and displayed in the client. With the addition of content publication\nand curation, this should provide a powerful platform for distributed\ncommunication.\n\n### Distributed trending\n\nMore challenging than distributed search, trending should include aggregate\ninformation from as many Detergent peers as possible. Challenges include the\noversampling of infohashes \"near\" a node ID per each Detergent client, how to\ndedupe identical announces seen by multiple peers and host of other unforeseen\nissues that will surely arise.\n\n\n### Content publication\n\nDetergent should include a first-class mechanism for authoring and publishing\ntext content in the form of Markdown. BitTorrent has historically been very\nmedia heavy but can just as easily be used to share text content. By focusing\non discovery and search, Detergent should provide a natural platform on which\nto publish Markdown based posts.\n\n### Content curation and promotion\n\nIn addition to trending and search, Detergent should allow users to curate,\norganize and promote content. Using techniques based on\n[key.run](https://github.com/toby/keyrun), infohashes can be organized\ninto many namespaces and sorted by cryptocurrency transaction amounts. By\nleveraging blockchain based namespacing and encoding, there's an opportunity to\nbuild up durable indexes of infohashes that are manually curated.\n\n## Known Limitations\n\n* Top level filenames are the only metadata indexed\n* Long wait for indexes to build up (distributed search will help)\n* Poor CLI output formatting\n* Many other things, this is very early\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftoby%2Fdet","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftoby%2Fdet","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftoby%2Fdet/lists"}