https://github.com/quackscience/duckdb-extension-dht
WIP DuckDB DHT Client Extension
https://github.com/quackscience/duckdb-extension-dht
bittorrent community-extension dht duckdb extension
Last synced: 11 months ago
JSON representation
WIP DuckDB DHT Client Extension
- Host: GitHub
- URL: https://github.com/quackscience/duckdb-extension-dht
- Owner: quackscience
- License: mit
- Created: 2024-11-12T12:39:47.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2024-11-13T21:07:36.000Z (over 1 year ago)
- Last Synced: 2025-05-08T05:52:50.315Z (11 months ago)
- Topics: bittorrent, community-extension, dht, duckdb, extension
- Language: C++
- Homepage:
- Size: 76.2 KB
- Stars: 5
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# DuckDB DHT Client Extension
This extension provides **DuckDB** instances access to mainline **DHT**.
### Usage
Start a local [DHTd](https://github.com/lmangani/dhtd/releases/tag/v0.0.1) node instance
#### DHTd
DHTd serves as backend for this extension to operate across multiple DuckDB sessions
```
docker run -d --name dhtd -p 6881:6881 -v /tmp:/tmp ghcr.io/lmangani/dhtd:main
```
#### DuckDB
Query the DHTd node from DuckDB running on the same host
```sql
D SELECT dht_search('6f84758b0ddd8dc05840bf932a77935d8b5b8b93');
┌────────────────────────────────────────────────────────┐
│ dht_search('6f84758b0ddd8dc05840bf932a77935d8b5b8b93') │
│ varchar │
├────────────────────────────────────────────────────────┤
│ Search started. Run dht_results() to get results │
└────────────────────────────────────────────────────────┘
D SELECT * FROM dht_results('6f84758b0ddd8dc05840bf932a77935d8b5b8b93');
┌────────────────────────────────────────┬───────┐
│ address │ port │
│ varchar │ int32 │
├────────────────────────────────────────┼───────┤
│ 2800:2202:4000:73b:b39f:759:2f3b:134 │ 26976 │
│ 2800:2202:4000:73b:89e3:8001:d02e:f218 │ 26976 │
│ 2800:2202:4000:73b::98bd │ 26976 │
│ 2601:8c0:800:7626:be24:11ff:fe08:b8a3 │ 30165 │
│ 2a00:ee2:802:5942::10:0 │ 7588 │
└────────────────────────────────────────┴───────┘
```
```sql
D SELECT version, node_id, uptime FROM dht_status();
┌─────────┬──────────────────────┬─────────┐
│ version │ node_id │ uptime │
│ varchar │ varchar │ varchar │
├─────────┼──────────────────────┼─────────┤
│ 1.0.2 │ 552ba13e8034e1d3ec… │ 1h1m │
├─────────┴──────────────────────┴─────────┤
│ 1 rows 21 columns (3 shown) │
└──────────────────────────────────────────┘
```
## Vision
This experiment is part of a Proof-of-Concept to distribute queries through functions _(without modifying the query planner)_
```sql
--- Start an HTTP Socket (insecure) in the background
SELECT http_serve('0.0.0.0', 8123, '');
--- Announce your Socket with your hash/token to DHTd
SELECT dht_announce('somesupersecrettokennobodyknowsabout', 8123);
-- Repeat for multiple peers w/ same hash/token
--- Check for Discovered Peers by hash/token in DHTd (not self)
SELECT dht_results('somesupersecrettokennobodyknowsabout');
┌─────────────────┬───────┐
│ address │ port │
│ varchar │ int32 │
├─────────────────┼───────┤
│ xxx.xx.xx.xxx │ 8123 │
│ yyy.yyy.yy.yyy │ 8123 │
├─────────────────┴───────┤
│ 2 rows 2 columns │
└─────────────────────────┘
--- Create a VIEW for your hash/token peer network
CREATE OR REPLACE VIEW backends AS
SELECT (address) as "who", ("http://" || address || ":" || port) AS "url",
FROM dht_results('somesupersecrettokennobodyknowsabout');
--- Query the Network (TODO: add authentication)
SET variable __backends = (SELECT ARRAY_AGG(url) AS urls_array FROM backends);
SELECT * FROM url_flock('SELECT ''hello'', version()', getvariable('__backends') );
┌─────────┬─────────────┐
│ 'hello' │ "version"() │
│ varchar │ varchar │
├─────────┼─────────────┤
│ hello │ v1.1.2 │
│ hello │ v1.1.3 │
└─────────┴─────────────┘
```