{"id":22902825,"url":"https://github.com/quackscience/duckdb-extension-dht","last_synced_at":"2025-05-08T05:53:02.335Z","repository":{"id":262469333,"uuid":"887255813","full_name":"quackscience/duckdb-extension-dht","owner":"quackscience","description":"WIP DuckDB DHT Client Extension","archived":false,"fork":false,"pushed_at":"2024-11-13T21:07:36.000Z","size":78,"stargazers_count":5,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-05-08T05:52:50.315Z","etag":null,"topics":["bittorrent","community-extension","dht","duckdb","extension"],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/quackscience.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":["lmangani"]}},"created_at":"2024-11-12T12:39:47.000Z","updated_at":"2025-02-20T04:32:26.000Z","dependencies_parsed_at":null,"dependency_job_id":"fec62eb3-dc1b-438a-8b7b-cab444763453","html_url":"https://github.com/quackscience/duckdb-extension-dht","commit_stats":null,"previous_names":["lmangani/duckdb-extension-dht","quackscience/duckdb-extension-dht"],"tags_count":0,"template":false,"template_full_name":"duckdb/extension-template","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/quackscience%2Fduckdb-extension-dht","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/quackscience%2Fduckdb-extension-dht/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/quackscience%2Fduckdb-extension-dht/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/quackscience%2Fduckdb-extension-dht/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/quackscience","download_url":"https://codeload.github.com/quackscience/duckdb-extension-dht/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253009850,"owners_count":21839713,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bittorrent","community-extension","dht","duckdb","extension"],"created_at":"2024-12-14T02:19:25.525Z","updated_at":"2025-05-08T05:53:02.329Z","avatar_url":"https://github.com/quackscience.png","language":"C++","funding_links":["https://github.com/sponsors/lmangani"],"categories":[],"sub_categories":[],"readme":"\u003c!-- \u003cimg src=\"https://github.com/user-attachments/assets/35bfded5-3f21-46b5-91f7-014f5a09fac3\" width=200 /\u003e --\u003e\n\n\u003cimg src=\"https://github.com/user-attachments/assets/46a5c546-7e9b-42c7-87f4-bc8defe674e0\" width=250 /\u003e\n\n\n# DuckDB DHT Client Extension\nThis extension provides **DuckDB** instances access to mainline **DHT**.\u003cbr\u003e\n\n### Usage\n\nStart a local [DHTd](https://github.com/lmangani/dhtd/releases/tag/v0.0.1) node instance\n\n#### DHTd\nDHTd serves as backend for this extension to operate across multiple DuckDB sessions\n\n```\ndocker run -d --name dhtd -p 6881:6881 -v /tmp:/tmp ghcr.io/lmangani/dhtd:main\n```\n\n#### DuckDB\n\nQuery the DHTd node from DuckDB running on the same host\n\n```sql\nD SELECT dht_search('6f84758b0ddd8dc05840bf932a77935d8b5b8b93');\n┌────────────────────────────────────────────────────────┐\n│ dht_search('6f84758b0ddd8dc05840bf932a77935d8b5b8b93') │\n│                        varchar                         │\n├────────────────────────────────────────────────────────┤\n│ Search started. Run dht_results() to get results       │\n└────────────────────────────────────────────────────────┘\n\nD SELECT * FROM dht_results('6f84758b0ddd8dc05840bf932a77935d8b5b8b93');\n┌────────────────────────────────────────┬───────┐\n│                address                 │ port  │\n│                varchar                 │ int32 │\n├────────────────────────────────────────┼───────┤\n│ 2800:2202:4000:73b:b39f:759:2f3b:134   │ 26976 │\n│ 2800:2202:4000:73b:89e3:8001:d02e:f218 │ 26976 │\n│ 2800:2202:4000:73b::98bd               │ 26976 │\n│ 2601:8c0:800:7626:be24:11ff:fe08:b8a3  │ 30165 │\n│ 2a00:ee2:802:5942::10:0                │  7588 │\n└────────────────────────────────────────┴───────┘\n```\n\n```sql\nD SELECT version, node_id, uptime FROM dht_status();\n┌─────────┬──────────────────────┬─────────┐\n│ version │       node_id        │ uptime  │\n│ varchar │       varchar        │ varchar │\n├─────────┼──────────────────────┼─────────┤\n│ 1.0.2   │ 552ba13e8034e1d3ec…  │ 1h1m    │\n├─────────┴──────────────────────┴─────────┤\n│ 1 rows              21 columns (3 shown) │\n└──────────────────────────────────────────┘\n```\n\n\n## Vision\nThis experiment is part of a Proof-of-Concept to distribute queries through functions _(without modifying the query planner)_\n\n```sql\n--- Start an HTTP Socket (insecure) in the background\nSELECT http_serve('0.0.0.0', 8123, ''); \n\n--- Announce your Socket with your hash/token to DHTd\nSELECT dht_announce('somesupersecrettokennobodyknowsabout', 8123);\n\n-- Repeat for multiple peers w/ same hash/token\n\n--- Check for Discovered Peers by hash/token in DHTd (not self)\nSELECT dht_results('somesupersecrettokennobodyknowsabout');\n\n┌─────────────────┬───────┐\n│     address     │ port  │\n│     varchar     │ int32 │\n├─────────────────┼───────┤\n│ xxx.xx.xx.xxx   │  8123 │\n│ yyy.yyy.yy.yyy  │  8123 │\n├─────────────────┴───────┤\n│ 2 rows        2 columns │\n└─────────────────────────┘\n\n--- Create a VIEW for your hash/token peer network\nCREATE OR REPLACE VIEW backends AS\nSELECT (address) as \"who\", (\"http://\" || address || \":\" || port) AS \"url\",\nFROM dht_results('somesupersecrettokennobodyknowsabout');\n\n--- Query the Network (TODO: add authentication)\nSET variable __backends = (SELECT ARRAY_AGG(url) AS urls_array FROM backends);\nSELECT * FROM url_flock('SELECT ''hello'', version()', getvariable('__backends') );\n\n┌─────────┬─────────────┐\n│ 'hello' │ \"version\"() │\n│ varchar │   varchar   │\n├─────────┼─────────────┤\n│ hello   │ v1.1.2      │\n│ hello   │ v1.1.3      │\n└─────────┴─────────────┘\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fquackscience%2Fduckdb-extension-dht","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fquackscience%2Fduckdb-extension-dht","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fquackscience%2Fduckdb-extension-dht/lists"}