https://github.com/artie-labs/ducktape
Lightweight REST API for DuckDB with HTTP/2 streaming support.
https://github.com/artie-labs/ducktape
database duckdb http-server streaming
Last synced: 5 months ago
JSON representation
Lightweight REST API for DuckDB with HTTP/2 streaming support.
- Host: GitHub
- URL: https://github.com/artie-labs/ducktape
- Owner: artie-labs
- License: mit
- Created: 2025-11-07T02:26:37.000Z (9 months ago)
- Default Branch: master
- Last Pushed: 2026-03-07T18:14:29.000Z (5 months ago)
- Last Synced: 2026-03-07T23:55:14.345Z (5 months ago)
- Topics: database, duckdb, http-server, streaming
- Language: Go
- Homepage:
- Size: 89.8 KB
- Stars: 49
- Watchers: 0
- Forks: 1
- Open Issues: 2
-
Metadata Files:
- Readme: README.md
- License: LICENSE.txt
Awesome Lists containing this project
README
Ducktape 🦆
Lightweight REST API for DuckDB with HTTP/2 streaming support.
## What is ducktape?
Ducktape is a standalone microservice to:
- **Append**: Append rows directly into DuckDB by streaming NDJSON over HTTP/2.
- **Query**: Fetch rows from DuckDB.
- **Execute**: Run statements within a transaction.
**Why?** DuckDB's Go driver requires CGO, which breaks cross-compilation, complicates CI/CD, and bloats Docker images. Instead of rewriting the build pipelines for [Transfer](https://github.com/artie-labs/transfer), we isolated DuckDB behind a network boundary.
The performance penalty is small—**~90% of native throughput** over the network. Pure Go apps stay portable; ducktape handles the CGO.
A [native Go client](#go-client) library is included.
## How it works
```
your service → stream NDJSON over HTTP/2 → ducktape → DuckDB (local file or MotherDuck)
```
Ducktape uses:
- **HTTP/2 streaming** for high-throughput ingestion
- **DuckDB's Appender API** for fast, type-aware row insertion
- **NDJSON** as a simple, language-agnostic wire format
If your app can produce NDJSON, it can talk to ducktape.
## Performance
| Benchmark | Throughput |
| ------------------------ | ------------ |
| In-process DuckDB append | ~848 MiB/sec |
| Ducktape over HTTP/2 | ~757 MiB/sec |
That's **~90% of native performance**, even across the network. For real-time ingestion workloads, this was fast enough that we didn't need to embed DuckDB at all.
See [BENCHMARKS.md](BENCHMARKS.md) for detailed results.
## Quick start
### Docker
```bash
docker pull artielabs/ducktape:latest
docker run -e DUCKTAPE_LOG="debug" --rm --publish 8080:8080 --volume $PWD:/data artielabs/ducktape:latest
# absolute path in DSN is required when ducktape runs in Docker and writing to local file
curl -X POST 'http://localhost:8080/api/query' \
--header 'X-DuckDB-Connection-String: /data/test.db' \
--header 'Content-Type: application/json' \
--data '{
"Query": "CREATE TABLE test_file (id BIGINT);"
}'
# test.db will be created in your current working directory
```
### Development
```bash
make start
# Or with debug logging
make debug
# Or manually
PORT=8080 DUCKTAPE_LOG=debug go run cmd/main.go
# Health check
curl http://localhost:8080/health
```
Server runs on port 8080 by default.
## API usage
### Execute
Execute one or more SQL statements in a transaction:
```bash
curl -X POST http://localhost:8080/api/execute \
-H "X-DuckDB-Connection-String: duck.db" \
-H "Content-Type: application/json" \
-d '{"statements": [
{"query": "CREATE TABLE users (name TEXT, age INTEGER)"},
{"query": "INSERT INTO users VALUES (?, ?)", "args": ["Alice", 30]},
{"query": "INSERT INTO users VALUES (?, ?)", "args": ["Bob", 25]}
]}'
```
### Query
```bash
curl -X POST http://localhost:8080/api/query \
-H "X-DuckDB-Connection-String: duck.db" \
-H "Content-Type: application/json" \
-d '{"query": "SELECT * FROM users WHERE name = ?", "args": ["Alice"]}'
```
### Append
Streams NDJSON data over HTTP/2. Each line is a `RowMessage` with a `rv` (row values) array. Use the Go client for streaming large datasets.
## Configuration
- `PORT`: Server port (default: `8080`)
- `DUCKTAPE_LOG`: Log level (`debug`, `info`, `warn`, `error`)
## Go client
- Install Go module for client.
```bash
go get github.com/artie-labs/ducktape/api
```
- Usage:
```go
import "github.com/artie-labs/ducktape/api/pkg/ducktape"
client := ducktape.NewClient("http://localhost:8080")
```
- [Client source code](api/pkg/ducktape/client.go)
## License
MIT License. See [LICENSE](https://github.com/artie-labs/ducktape/blob/master/LICENSE.txt) for details.