https://github.com/quackscience/quackflight
DuckDB API Server with Arrow Flight SQL Airport support and concurrent writes/reads (quackpipe)
https://github.com/quackscience/quackflight
airport api arrow clickhouse duckdb duckdb-server flight flight-sql fuck-trump server
Last synced: 23 days ago
JSON representation
DuckDB API Server with Arrow Flight SQL Airport support and concurrent writes/reads (quackpipe)
- Host: GitHub
- URL: https://github.com/quackscience/quackflight
- Owner: quackscience
- License: mit
- Created: 2024-10-18T22:51:26.000Z (7 months ago)
- Default Branch: main
- Last Pushed: 2025-03-05T09:32:05.000Z (2 months ago)
- Last Synced: 2025-03-28T18:09:35.896Z (about 1 month ago)
- Topics: airport, api, arrow, clickhouse, duckdb, duckdb-server, flight, flight-sql, fuck-trump, server
- Language: Python
- Homepage: https://quackpy.fly.dev
- Size: 193 KB
- Stars: 66
- Watchers: 1
- Forks: 2
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# :baby_chick: [QuackFlight](https://quackpy.fly.dev/?user=default#U0VMRUNUCiAgICB0b3duLAogICAgZGlzdHJpY3QsCiAgICBjb3VudCgpIEFTIGMsCkZST00gcmVhZF9wYXJxdWV0KCdodHRwczovL2RhdGFzZXRzLWRvY3VtZW50YXRpb24uczMuZXUtd2VzdC0zLmFtYXpvbmF3cy5jb20vaG91c2VfcGFycXVldC9ob3VzZV8wLnBhcnF1ZXQnKQpXSEVSRSByZWFkX3BhcnF1ZXQudG93biA9PSAnTE9ORE9OJwpHUk9VUCBCWQogICAgdG93biwKICAgIGRpc3RyaWN0Ck9SREVSIEJZIGMgREVTQwpMSU1JVCAxMA==)
_Serverless OLAP API built on top of DuckDB exposing HTTP/S and Arrow Flight SQL interfaces_
> [!IMPORTANT]
> - Arrow Flight API for modern data clients _(DuckDB Airport)_
> - Easy HTTP API with multiple formats _(JSON,CSV,Parquet)_
> - Unlocked Concurrent inserts and querying on DuckDB
> - Persistent storage using w/ multiuser authentication
> - Native access to any DuckDB Extension & Format
> - Embedded SQL Query Interface for instant usage
### :seedling: Get Started
Run using [docker](https://github.com/quackscience/quackflight/pkgs/container/quackflight) or build from source
```bash
docker pull ghcr.io/quackscience/quackflight:latest
docker run -ti --rm -p 8123:8123 -p 8815:8815 ghcr.io/quackscience/quackflight:latest
```
### 👉 Usage
> See the [Examples](https://github.com/quackscience/quackflight/tree/main/examples) directory for quick starters
> [!NOTE]
> Quackpipe executes queries in `:memory:` unless _authentication_ details are provided for data persistence
#### 🕸️ HTTP API
Execute DuckDB queries using the HTTP POST/GET API _(compatible with the ClickHouse HTTP API)_
```bash
curl -X POST "http://user:persistence@localhost:8123" \
-H "Content-Type: application/json" \
-d 'SELECT version()'
```
#### ✈️ FLIGHT API
Execute DuckDB queries using the _experimental_ Flight GRPC API and [Airport](https://github.com/Query-farm/duckdb-airport-extension)
> [!NOTE]
> Quackpipe executes queries in `:memory:` unless an `authorization` header is provided for data persistence
##### 🎫 Pass Airport Security
```sql
CREATE PERSISTENT SECRET airport_flight (
· type airport,
‣ auth_token 'user:persistence',
· scope 'grpc://localhost:8815'
· );
```
##### 🎫 Take Airport Flights
```sql
D select flight_descriptor, endpoint from airport_list_flights('grpc://127.0.0.1:8815', null);
┌─────────────────────────────────┬────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ flight_descriptor │ endpoint │
│ union(cmd blob, path varchar[]) │ struct(ticket blob, "location" varchar[], expiration_time timestamp, app_metadata blob)[] │
├─────────────────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ show_databases │ [{'ticket': SHOW DATABASES, 'location': [grpc://localhost:8815], 'expiration_time': NULL, 'app_metadata': }] │
│ show_tables │ [{'ticket': SHOW TABLES, 'location': [grpc://localhost:8815], 'expiration_time': NULL, 'app_metadata': }] │
│ show_version │ [{'ticket': SELECT version(), 'location': [grpc://localhost:8815], 'expiration_time': NULL, 'app_metadata': }] │
└─────────────────────────────────┴────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
D select * from airport_take_flight('grpc://localhost:8815/', ['show_version']);
┌─────────────┐
│ "version"() │
│ varchar │
├─────────────┤
│ v1.2.0 │
└─────────────┘
```
##### 🎫 ATTACH Flights Tables
```sql
D --- Attach to Flight Server
D ATTACH 'deltalake' (TYPE AIRPORT, location 'grpc://localhost:8815/');
D --- Create Schema + Table
D CREATE SCHEMA deltalake.test1;
D CREATE TABLE deltalake.test1.people (
name VARCHAR,
love_of_duckdb INT,
tags VARCHAR[]
);
D --- Insert into Flight Table
D INSERT INTO deltalake.test1.people values
('rusty', 5, ['airport', 'datasketches']);
D --- Select from Flight Table
D SELECT * FROM deltalake.test1.people;
┌─────────┬────────────────┬─────────────────────────┐
│ name │ love_of_duckdb │ tags │
│ varchar │ int32 │ varchar[] │
├─────────┼────────────────┼─────────────────────────┤
│ rusty │ 5 │ [airport, datasketches] │
├─────────┴────────────────┴─────────────────────────┤
│ 1 row. 3 columns │
└────────────────────────────────────────────────────┘
```
> Flight Tables can be accessed via HTTP API using the schema name
```sql
USE test1; SELECT * FROM people;
```
##### 🎫 Take Custom Flights w/ Custom Headers + Ticket
```sql
D SELECT * FROM airport_take_flight('grpc://localhost:8815', 'SELECT 1', headers := MAP{'authorization':'user:persistence'} );
┌───────┐
│ 1 │
│ int32 │
├───────┤
│ 1 │
└───────┘
```
##### 🎫 Take Python Flights
```python
from pyarrow.flight import FlightClient, Ticket, FlightCallOptions
import json
import pandas
import tabulate
sql="""SELECT version()"""
flight_ticket = Ticket(sql)
token = (b"authorization", bytes(f"user:persistence".encode('utf-8')))
options = FlightCallOptions(headers=[token])
client = FlightClient(f"grpc://localhost:8815")
reader = client.do_get(flight_ticket, options)
arrow_table = reader.read_all()
# Use pyarrow and pandas to view and analyze data
data_frame = arrow_table.to_pandas()
print(data_frame.to_markdown())
```
```sql
| | "version"() |
|---:|:--------------|
| 0 | v1.2.0 |
```
### 📺 SQL User-Interface
quackflight ships with the DuckDB SQL quack user-interface based on [duck-ui](https://github.com/caioricciuti/duck-ui)
```mermaid
sequenceDiagram
participant Client
participant QuackFlight
participant DuckDB
Client ->> QuackFlight: ListFlights
QuackFlight ->> Client: Return Flights Table
Client ->> QuackFlight: GetFlightInfo
QuackFlight ->> DuckDB: DuckDB Execute
DuckDB ->> QuackFlight: Arrow Results Stream
QuackFlight ->> Client: FlightInfo(ticket)
Client ->> QuackFlight: do_get(ticket)
QuackFlight ->> Client: Stream of Results
```
###### :black_joker: Disclaimers
[^1]: DuckDB ® is a trademark of DuckDB Foundation. All rights reserved by their respective owners. [^1]
[^2]: ClickHouse ® is a trademark of ClickHouse Inc. No direct affiliation or endorsement. [^2]
[^3]: Released under the MIT license. See LICENSE for details. All rights reserved by their respective owners. [^3]
[^4]: Flight implementation inspired by [Duck Takes Flight](https://www.definite.app/blog/duck-takes-flight) [^4]