Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/quackscience/duckdb-extension-httpserver
DuckDB HTTP API Server and Query Interface in a Community Extension
https://github.com/quackscience/duckdb-extension-httpserver
api api-server community-extension duckdb extension http-server olap user-interface
Last synced: about 3 hours ago
JSON representation
DuckDB HTTP API Server and Query Interface in a Community Extension
- Host: GitHub
- URL: https://github.com/quackscience/duckdb-extension-httpserver
- Owner: quackscience
- License: mit
- Created: 2024-10-08T20:25:46.000Z (about 1 month ago)
- Default Branch: main
- Last Pushed: 2024-11-14T20:07:09.000Z (3 days ago)
- Last Synced: 2024-11-14T20:24:51.333Z (3 days ago)
- Topics: api, api-server, community-extension, duckdb, extension, http-server, olap, user-interface
- Language: C
- Homepage: https://duckdb.org/community_extensions/extensions/httpserver.html
- Size: 148 KB
- Stars: 111
- Watchers: 2
- Forks: 1
- Open Issues: 1
-
Metadata Files:
- Readme: docs/README.md
- License: LICENSE
Awesome Lists containing this project
README
# DuckDB HTTP Server Extension
This extension transforms **DuckDB** instances into tiny multi-player **HTTP OLAP API** services.
Supports Authentication _(Basic Auth or X-Token)_ and includes the _play_ SQL user interface.
The extension goal is to replace the functionality currently offered by [quackpipe](https://github.com/metrico/quackpipe)
### Features
- Turn any [DuckDB](https://duckdb.org) instance into an **HTTP OLAP API** Server
- Use the embedded **Play User Interface** to query and visualize data
- Pair with [chsql](https://community-extensions.duckdb.org/extensions/chsql.html) extension for **ClickHouse flavoured SQL**
- Work with local and remote datasets including [MotherDuck](https://motherduck.com) 🐤
- _100% Opensource, ready to use and extend by the Community!_
#### Extension Functions
- `httpserve_start(host, port, auth)`: starts the server using provided parameters
- `httpserve_stop()`: stops the server thread
#### Notes
🛑 Run DuckDB in `-readonly` mode for enhanced security
### 📦 [Installation](https://community-extensions.duckdb.org/extensions/httpserver.html)
```sql
INSTALL httpserver FROM community;
LOAD httpserver;
```
### 🔌 Usage
Start the HTTP server providing the `host`, `port` and `auth` parameters.
> * If you want no authentication, just pass an empty string as parameter.
> * If you want the API run in foreground set `DUCKDB_HTTPSERVER_FOREGROUND=1`
#### Basic Auth
```sql
D SELECT httpserve_start('localhost', 9999, 'user:pass');
┌───────────────────────────────────────────────┐
│ httpserve_start('0.0.0.0', 9999, 'user:pass') │
│ varchar │
├───────────────────────────────────────────────┤
│ HTTP server started on 0.0.0.0:9999 │
└───────────────────────────────────────────────┘
```
```bash
curl -X POST -d "SELECT 'hello', version()" "http://user:pass@localhost:9999/"
```
#### Token Auth
```sql
SELECT httpserve_start('localhost', 9999, 'supersecretkey');
┌───────────────────────────────────────────────┐
│ httpserve_start('0.0.0.0', 9999, 'secretkey') │
│ varchar │
├───────────────────────────────────────────────┤
│ HTTP server started on 0.0.0.0:9999 │
└───────────────────────────────────────────────┘
```
Query your endpoint using the `X-API-Key` token:
```bash
curl -X POST --header "X-API-Key: secretkey" -d "SELECT 'hello', version()" "http://localhost:9999/"
```
You can perform the same action from DuckDB using HTTP `extra_http_headers`:
```sql
D CREATE SECRET extra_http_headers (
TYPE HTTP,
EXTRA_HTTP_HEADERS MAP{
'X-API-Key': 'secret'
}
);
D SELECT * FROM duck_flock('SELECT version()', ['http://localhost:9999']);
┌─────────────┐
│ "version"() │
│ varchar │
├─────────────┤
│ v1.1.1 │
└─────────────┘
```
#### 👉 QUERY UI
Browse to your endpoint and use the built-in quackplay interface _(experimental)_
#### 👉 QUERY API
Query your API endpoint using curl GET/POST requests
```bash
curl -X POST -d "SELECT 'hello', version()" "http://localhost:9999/?default_format=JSONCompact
```
```json
{
"meta": [
{
"name": "'hello'",
"type": "String"
},
{
"name": "\"version\"()",
"type": "String"
}
],
"data": [
[
"hello",
"v1.1.1"
]
],
"rows": 1,
"statistics": {
"elapsed": 0.01,
"rows_read": 1,
"bytes_read": 0
}
}
```
#### 👉 CROSS-OVER EXAMPLES
You can now have DuckDB instances query each other and... _themselves!_
```sql
D LOAD json;
D LOAD httpfs;
D SELECT httpserve_start('0.0.0.0', 9999);
┌─────────────────────────────────────┐
│ httpserve_start('0.0.0.0', 9999) │
│ varchar │
├─────────────────────────────────────┤
│ HTTP server started on 0.0.0.0:9999 │
└─────────────────────────────────────┘
D SELECT * FROM read_json_auto('http://localhost:9999/?q=SELECT version()');
┌─────────────┐
│ "version"() │
│ varchar │
├─────────────┤
│ v1.1.1 │
└─────────────┘
```
#### Flock Macro by @carlopi
Check out this flocking macro from fellow _Italo-Amsterdammer_ @carlopi @ DuckDB Labs
* a DuckDB CLI, running httpserver extension
* a DuckDB from Python, running httpserver extension
* a DuckDB from the Web, querying all 3 DuckDB at the same time
### API Documentation
#### Endpoints Overview
| Endpoint | Methods | Description |
|----------|---------|-------------|
| `/` | GET, POST | Query API endpoint |
| `/ping` | GET | Health check endpoint |
#### Detailed Endpoint Specifications
##### Query API
**Methods:** `GET`, `POST`
**Parameters:**
| Parameter | Description | Supported Values |
|-----------|-------------|-------------------|
| `default_format` | Specifies the output format | `JSONEachRow`, `JSONCompact` |
| `query` | The DuckDB SQL query to execute | Any valid DuckDB SQL query |
##### Notes
- Ensure that your queries are properly formatted and escaped when sending them as part of the request.
- The root endpoint (`/`) supports both GET and POST methods, but POST is recommended for complex queries or when the query length exceeds URL length limitations.
- Always specify the `default_format` parameter to ensure consistent output formatting.
##### :black_joker: Disclaimers
[^1]: DuckDB ® is a trademark of DuckDB Foundation. All rights reserved by their respective owners. [^1]
[^2]: ClickHouse ® is a trademark of ClickHouse Inc. No direct affiliation or endorsement. [^2]
[^3]: Released under the MIT license. See LICENSE for details. All rights reserved by their respective owners. [^3]