Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
awesome-duckdb
🦆 A curated list of awesome DuckDB resources
https://github.com/davidgasquez/awesome-duckdb
Last synced: about 21 hours ago
JSON representation
-
Resources
- Official Documentation - Official DuckDB documentation.
- Official Blog - Official DuckDB blog.
- DuckDB Clients - Client APIs for DuckDB.
- DuckDB Documentation PDF - The DuckDB documentation as a single PDF file.
- DuckDB setup - GitHub Action to install DuckDB in CI.
- DuckDB snippets - Collection of snippets curated by MotherDuck.
- DuckDB tldr page - DuckDB's entry in [tldr pages](https://tldr.sh/), available in CLI via the `tldr duckdb` command.
- Compatible DuckDB Extensions for AWS Lambda - Extensions specifically compiled for the AWS Lambda runtime (GLIBC 2.26).
- Observable notebooks - Notebooks using DuckDB on the Observable data visualization platform.
- DuckDB AWS Lambda layer - Run DuckDB in AWS Lambda functions.
- Serverless DuckDB as API - Use DuckDB as API with Amazon API Gateway and AWS Lambda.
- Serverless Parquet Repartitioner - Use DuckDB to repartition data in S3-based Data Lakes.
- duckdb-nf - Example uses of DuckDB with Nextflow.
- DuckDB version manager (`duckman`) - platform installer and version manager for DuckDB.
- docker-duckdb - Docker image for DuckDB CLI.
- Serverless DuckDB over S3 - Running DuckDB over a data lake on S3 using lambda.
- DuckERD CLI
-
Client APIs
-
Tools Powered by DuckDB
-
- MotherDuck - Serverless data warehouse powered by DuckDB.
- Boiling Data - Serverless data analytics overlay on top of S3 Data Lakes.
- Hex Dataframe SQL - Hex's Dataframe SQL cells are powered by DuckDB.
- Mode - Mode uses DuckDB for their in-memory data engine.
- Honeycomb Maps - A browser-based geospatial analysis tool leveraging DuckDB Wasm.
- Bauplan - A serverless data transformation platform for data lakes.
- VulcanSQL - DuckDB can be used as a caching layer or a data connector in VulcanSQL, a Data API framework for data folks to create REST APIs by writing SQL templates.
- Excalichart.com - A fast, free dashboard for exploring your data.
- Malloy - Malloy is an experimental language for describing data relationships and transformations. Malloy connects to BigQuery, Snowflake, Trino, and Postgres, and natively supports DuckDB.
- Huey - Blazing-fast & intuitive pivot tables on .parquet, .csv, .json files and .duckdb tables in the browser based on DuckDB WASM. Open source (MIT). Zero install!
- DatalakeStudio - Load, explore, transform your datasets and expose them via API. Integration with external APIs, S3, PostgreSQL and ChatGPT.
- Spice.ai - A unified SQL query interface and portable runtime to locally materialize (using an embedded DuckDB), accelerate, and query datasets from any database, data warehouse, or data lake.
- Quackpipe - Serverless OLAP API/UI built on top of DuckDB with basic ClickHouse API compatibility and Motherduck support.
- ParadeDB - Postgres for Search and Analytics, powered by DuckDB-embedded-in-Postgres.
- Census - Census's dataset diffing for incremental syncs is powered by DuckDB.
- UniverSQL - An implementation of Snowflake API, enables running queries on Snowflake tables locally with DuckDB without a running warehouse.
- Crunchy Bridge for Analytics - Fully managed DBaaS based in Postgres integrated with DuckDB.
- Honeycomb Maps - A browser-based geospatial analysis tool leveraging DuckDB Wasm.
- Rill Developer - Tool for effortlessly transforming data sets into powerful, opinionated dashboards using SQL.
- Iceburst - The real-time data lake for monitoring & security.
- sqlglot - Python transpiler that translates between 23 different SQL dialects including DuckDB.
- Latitude - Latitude uses DuckDB to power data snapshots. Drop a CSV file and query it with SQL at the speed of light.
- Parquet Explorer - Visual Studio Code extension for exploring Parquet files with SQL, powered by DuckDB.
- DQOps - Data quality platform for data engineers, data quality teams and data operations.
- Ibis Project - A DataFrame API for interacting with DuckDB (and other compute engines).
- yato - The smallest DuckDB SQL orchestrator on Earth.
- Whereabouts - Fast, accurate, open-source geocoding in Python, using DuckDB.
- Phoenix Analytics - Plug and play analytics for Phoenix applications, powered by DuckDB.
- Excalichart.com - A fast, free dashboard for exploring your data.
- Latitude - Latitude uses DuckDB to power data snapshots. Drop a CSV file and query it with SQL at the speed of light.
- Iceburst - The real-time data lake for monitoring & security.
- Definite - Definite pulls all your data into a single place for analytics and dashboards. No engineering or SQL required. Get a managed data warehouse (DuckDB), ELT, data modeling / transformations and BI in a single platform.
- Amphi ETL - Low-code data pipelines for structured and unstructured data. SQL transformations are powered by DuckDB.
-
Web Clients
- QuackDB - Open-source online DuckDB SQL playground and editor.
- SQL Playground - DuckDB-WASM based SQL playground for running queries on local or remote data, being able to show data as tables or visually as graphs.
- Online DuckDB Shell - Online DuckDB shell powered by WebAssembly.
- Sekuel Playground - Query your local parquet, csv, json. Your data will not be sent out of the device you are using.
- CSVFiddle - Free tool to explore and share insights from CSV files using SQL. Import data, write SQL, then instantly share it with anyone.
- Codapi - Embed executable code snippets directly into your product documentation, online course or blog post.
- QuackDB - Open-source online DuckDB SQL playground and editor.
- WhatTheDuck - WhatTheDuck is an open-source web application built on DuckDB. It allows users to upload CSV files, store them in tables, and perform SQL queries on the data.
-
-
SQL Clients and IDE that Support DuckDB
-
- Harlequin - The DuckDB IDE for your terminal. ([GitHub](https://github.com/tconbeer/harlequin)).
-
Web Clients
- VSCode SQLTools (Free) - Free open-source VSCode extension to query and explore your DuckDB databases with latest DuckDB support.
- jOOQ - Type safe querying of DuckDB (and many other RDBMS) from Java. A [transpiler](https://www.jooq.org/translate/) from and to DuckDB is also available.
- SQL DATA LENS - A lightweight, commercial SQL IDE that supports different DBMS, including DuckDB. The focus on performance and special DBMS features.
- qStudio - A free SQL tool specialized for data analysts. It runs on every operating system and allows easy browsing of tables and charting of results.
- DuckDB SQL Tools - Free DuckDB SQL Tools for VS Code IDE. [Premium version available](https://github.com/RandomFractals/pro-data-tools/blob/main/duckdb-tools.md#duckdb-pro-tools) with advanced features.
- DataGrip - Paid SQL IDE by Jetbrains that supports many different database technologies, including DuckDB.
- manifold-sql (DuckDB for Java) - Use native DuckDB SQL of any complexity directly & type-safely in Java source with comprehensive IntelliJ support.
-
-
Integrations
-
Web Clients
- data load tool - DuckDB destination - Extract and load data from APIs to DuckDB using dlt.
- target-duckdb - Load data to DuckDB based on Singer spec.
- Kestra DuckDB plugin - Run queries with DuckDB to schedule data transformations and process automations, and run event-driven anomaly detection pipelines.
- nodbi - NoSQL Database Connector for R, providing a common API across Elasticsearch, CouchDB, MongoDB, SQLite, PostgreSQL, and DuckDB.
-
- Airbyte DuckDB destination - Load data to DuckDB with Airbyte.
-
-
Extensions
-
Web Clients
- `postgres_scanner` - Allows DuckDB to directly read data from a running Postgres instance.
- `spatial` - Enables geospatial processing.
- `sqlite_scanner` - To read and write to SQLite databases.
- ERPL - DuckDB SAP connector using RFC, ODP, or BICS.
- Official Extensions - Official DuckDB extensions.
- `fts` - To support full text search.
- Official Extensions - Official DuckDB extensions.
- `mysql` - To read from and write to MySQL databases.
- `postgres` - To read from and write to PostgreSQL databases.
- `sqlite` - To read from and write to SQLite databases.
- `vss` - Add support for vector similarity search.
- Kùzu - Scan DuckDB tables in Kùzu, an embeddable property graph database management system.
-
- Lance - Integrate Lance (modern columnar data format for ML implemented in Rust) with DuckDB.
-
-
Media
-
Blog Posts
- Modern Data Stack in a Box - Fast, free, and open-source Modern Data Stack deployed on a laptop using the combination of DuckDB, Meltano, dbt, and Apache Superset.
- Exploring StarCraft 2 data with Airflow, DuckDB and Streamlit - Example project using DuckDB to persist API data, but also explains how to use DuckDB as a versatile data manipulation tool in data wrangling scripts.
- How to use DuckDB, Motherduck and Kestra for ETL - How DuckDB can transform data, mask sensitive PII information, detect anomalies in event-driven workflows, and streamline reporting use cases.
- DuckDB vs. MotherDuck — how do they compare - What are key differences between them, and when to choose each of these options.
- Building DuckDB Extensions with Zig and Nix - For Nix users and Zig developers familiar with DuckDB looking to extend it’s capabilities with custom extensions.
- Exploring StarCraft 2 data with Airflow, DuckDB and Streamlit - Example project using DuckDB to persist API data, but also explains how to use DuckDB as a versatile data manipulation tool in data wrangling scripts.
- Exploring StarCraft 2 data with Airflow, DuckDB and Streamlit - Example project using DuckDB to persist API data, but also explains how to use DuckDB as a versatile data manipulation tool in data wrangling scripts.
- Exploring StarCraft 2 data with Airflow, DuckDB and Streamlit - Example project using DuckDB to persist API data, but also explains how to use DuckDB as a versatile data manipulation tool in data wrangling scripts.
- Exploring StarCraft 2 data with Airflow, DuckDB and Streamlit - Example project using DuckDB to persist API data, but also explains how to use DuckDB as a versatile data manipulation tool in data wrangling scripts.
- Putting DuckDB in Postgres to Query Iceberg - How ParadeDB embedded DuckDB in Postgres to achieve fast analytics and Apache Iceberg compatibility from Postgres.
- DuckDB: The Rising Star in the Big Data Landscape
- Exploring StarCraft 2 data with Airflow, DuckDB and Streamlit - Example project using DuckDB to persist API data, but also explains how to use DuckDB as a versatile data manipulation tool in data wrangling scripts.
- Exploring StarCraft 2 data with Airflow, DuckDB and Streamlit - Example project using DuckDB to persist API data, but also explains how to use DuckDB as a versatile data manipulation tool in data wrangling scripts.
- Exploring StarCraft 2 data with Airflow, DuckDB and Streamlit - Example project using DuckDB to persist API data, but also explains how to use DuckDB as a versatile data manipulation tool in data wrangling scripts.
- Exploring StarCraft 2 data with Airflow, DuckDB and Streamlit - Example project using DuckDB to persist API data, but also explains how to use DuckDB as a versatile data manipulation tool in data wrangling scripts.
- How to Make a DuckDB Extension for a Table Function? - How to make a DuckDB extension to fetch data from external sources.
- Exploring StarCraft 2 data with Airflow, DuckDB and Streamlit - Example project using DuckDB to persist API data, but also explains how to use DuckDB as a versatile data manipulation tool in data wrangling scripts.
- Exploring StarCraft 2 data with Airflow, DuckDB and Streamlit - Example project using DuckDB to persist API data, but also explains how to use DuckDB as a versatile data manipulation tool in data wrangling scripts.
- Exploring StarCraft 2 data with Airflow, DuckDB and Streamlit - Example project using DuckDB to persist API data, but also explains how to use DuckDB as a versatile data manipulation tool in data wrangling scripts.
- Exploring StarCraft 2 data with Airflow, DuckDB and Streamlit - Example project using DuckDB to persist API data, but also explains how to use DuckDB as a versatile data manipulation tool in data wrangling scripts.
-
Talks
- In-Process Analytical Data Management with DuckDB @ PyData Amsterdam - Hannes Mühleisen.
- DuckDB: Bringing analytical SQL directly to your Python shell @ EuroPython - Pedro Holanda.
- DuckCon #3 playlist
- DuckDB: Bringing Analytical SQL Directly To Your Python Shell @ FOSDEM - Pedro Holanda.
- State Of The Duck @ DuckCon - Hannes Mühleisen & Mark Raasveldt.
- DuckDB Extensions @ DuckCon - Pedro Holanda & Sam Ansmink.
- Developing Systems in Academia: The Good, the Bad, and the not-so-Ugly Duckling @ CIDR - Hannes Mühleisen.
- DuckDB An Embeddable Analytical Database @ FOSDEM - Hannes Mühleisen.
- DuckDB tutorials playlist by Learn Data with Mark - Mark Needham.
- DuckDB tutorials playlist by MotherDuck - Mehdi Ouazza (MotherDuck).
- Nextflow and database uses: powering data engineering, exploring DuckDB, and beyond - Edmund Miller.
- DuckCon #4 playlist
- State of the Duck @ DuckCon #4 - Hannes Mühleisen and Mark Raasveldt.
- State of the Duck @ DuckCon #3 - Hannes Mühleisen and Mark Raasveldt.
- DuckDB: Bringing Analytical SQL Directly To Your Python Shell @ FOSDEM - Pedro Holanda.
- State of the Duck @ DuckCon #2 - Hannes Mühleisen & Mark Raasveldt.
- DuckDB Extensions @ DuckCon - Pedro Holanda & Sam Ansmink.
- Developing Systems in Academia: The Good, the Bad, and the not-so-Ugly Duckling @ CIDR - Hannes Mühleisen.
- DuckDB An Embeddable Analytical Database @ FOSDEM - Hannes Mühleisen.
- Why should you care about DuckDB? @ Dublin DuckDB meetup - Mihai Bojin.
- Exploring Monte Carlo Simulations With DuckDB @ Dublin DuckDB meetup - James McNeill.
- DuckDB and recommenders : a lightning fast synergy @ Dublin DuckDB meetup - Khalil Muhammad.
- DuckDB: The Power of a Data Warehouse in your Python Process @ PyData Yerevan - Gábor Szárnyas.
- DuckDB keynote @ Data + AI Summit 2023 - Hannes Mühleisen.
- Nextflow and database uses: powering data engineering, exploring DuckDB, and beyond - Edmund Miller.
-
Podcasts
- Software Engineering Daily: DuckDB - Hannes Mühleisen.
- Move Your Database To The Data And Speed Up Your Analytics With DuckDB - Hannes Mühleisen.
- The Personal Data Warehouse - Jordan Tigani.
- Developer Voices: Implementing Hardware-Friendly Databases - Hannes Mühleisen.
- The Geek Narrator: DuckDB Internals - Mark Raasveldt.
- The Analytics Engineering Podcast: The Personal Data Warehouse - Jordan Tigani.
- The Analytics Engineering Podcast: The Personal Data Warehouse - Jordan Tigani.
-
-
Libraries Powered by DuckDB
-
Web Clients
- Mosaic - An extensible framework for linking databases and interactive views.
- pg_lakehouse - PostgreSQL extension embedding DuckDB-in-Postgres for fast on-disk and remote object storage analytics from Postgres. Built as a Foreign Data Wrapper with full query pushdown to DuckDB.
- Fugue - A unified interface for distributed computing. Fugue executes SQL, Python, Pandas, and Polars code on Spark, Dask and Ray without any rewrites.
- PyGWalker - A Pyhon library that turns your dataframe into an interactive UI for data visualization.
-
-
Projects Powered by DuckDB
-
Web Clients
- `endoflife.date` database - Daily dumps of endoflife.date data.
-
-
Chat with this page
Programming Languages
Categories
Sub Categories
Keywords
duckdb
10
sql
8
database
4
olap
4
parquet
3
csv
2
clickhouse
2
golang
2
data-analysis
2
data
2
snowflake
2
data-science
2
s3
2
databricks
2
analytics
2
postgres
2
server
1
aggregations
1
big-data
1
rest-api
1
qryn
1
lambda-functions
1
lambda
1
bm25
1
datalake
1
elasticsearch
1
gigapipe
1
faceting
1
duckdb-engine
1
version-manager
1
versioning
1
docker
1
ruby
1
arrow
1
ffi
1
ffi-bindings
1
rust
1
c-bindings
1
common-lisp
1
lisp
1
artificial-intelligence
1
developers
1
infrastructure
1
machine-learning
1
time-series
1
api
1
clickhouse-server
1
duckdb-api
1
bigquery
1
hive
1