Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

awesome-olap

🧊 A curated list of OLAP databases, data lake tools, columnar engines, and analytics frameworks for data engineers.
https://github.com/samber/awesome-olap

Last synced: 14 days ago
JSON representation

Benchmark

Distributed SQL processing

  • Jepsen - Distributed databases, queues and consensus protocols testing.
  • TPC family benchmarks - Full catalog of TPC benchmarks for big data and analytical databases.
  • ClickBench - De facto OLAP benchmark maintained by ClickHouse; covers 43 analytical queries on a 100GB web analytics dataset and includes results for 50+ engines.
  • TPC-DS - Decision support benchmark modeling a retail data warehouse with 99 complex SQL queries across multiple fact and dimension tables.
  • TPC-H - Business intelligence benchmark with 22 ad-hoc queries on supply-chain data; the most widely cited OLAP benchmark in academic literature.

BI & Visualization

Distributed SQL processing

  • Apache Superset - Open-source business intelligence platform with a rich SQL editor, drag-and-drop chart builder, and support for 40+ data sources.
  • Grafana - Open-source observability and analytics platform for visualizing metrics, logs, and traces; widely used with time-series and OLAP backends.
  • Metabase - Open-source BI tool focused on ease of use, letting non-technical users explore data and build dashboards without SQL.
  • Redash - Open-source query editor and dashboarding tool with broad database connector support.

Brokers and distributed messaging

Codecs, encoding and compression

  • Apache Kafka - Distributed event streaming platform, the de facto standard for high-throughput data pipelines and event-driven architectures.
  • Apache Pulsar - Distributed messaging and streaming platform with multi-tenancy, geo-replication, and a decoupled storage layer.
  • NATS / JetStream - Lightweight cloud-native messaging system; JetStream adds persistence, replay, and streaming semantics.

Communities

Generalist

  • r/dataengineering - The largest Reddit community for data engineering discussions (300k+ members).
  • r/databasedevelopment - Subreddit for database internals, query engines, and storage systems (10k+ members).
  • DataTalks.Club - Global Slack community for data practitioners covering data engineering, ML, and MLOps.
  • Software Internals - Discord community by Phil Eaton focused on database internals, compilers, and distributed systems (9k+ members).
  • Locally Optimistic - Curated Slack community for current and aspiring analytics leaders.
  • Big Data Hebdo - French-language Slack community and podcast covering big data, data engineering, and analytics.

Tool-specific

👤 Contributors

Tool-specific

Data lake

Codecs, encoding and compression

  • Bit packing - Encoding integers using only the bits required, eliminating wasted high-order zeros in columnar data.
  • Brotli - General-purpose lossless compression by Google, offering better ratios than gzip at comparable speed.
  • Deflate - Classic lossless compression combining LZ77 and Huffman coding; the basis of gzip and zlib.
  • Delta - Stores differences between successive values instead of absolutes, ideal for monotonically increasing columns like timestamps.
  • LZ4 - Extremely fast lossless compression algorithm prioritizing throughput over ratio, widely used in real-time pipelines.
  • Snappy - Google's fast lossless codec optimized for speed over compression ratio, default in many Hadoop/Parquet deployments.
  • zstd - Facebook's modern lossless codec delivering high compression ratios at fast speeds; often preferred over gzip in data lakes.
  • Gorilla - Facebook's XOR-based float compression for time-series metrics, achieving 1.37 bytes/value on typical monitoring data.
  • Dictionary + RLE - Replaces repeated values with dictionary codes, then run-length encodes consecutive duplicates; effective for low-cardinality columns.

File formats and serialization

  • Apache Arrow Columnar Format - Columnar format for in-memory Apache Arrow processing.
  • Apache Avro - Row-oriented serialization for data streaming purpose.
  • Apache ORC - Column-oriented serialization for data storage purpose. Part of Hadoop platform.
  • Apache Parquet - Column-oriented serialization for data storage purpose.
  • Apache Thrift - Row-oriented serialization for RPC purpose.
  • Google Protobuf - Row-oriented serialization for RPC purpose.
  • Schema Registry - Centralized repository for validating row-oriented events. Part of Kafka and Confluent platform.
  • Cap’n Proto - Row-oriented serialization with zero-copy access, as fast as mmap.
  • Flatbuffer - Row-oriented serialization with zero-copy access, as fast as mmap.

Metastore

  • AWS Glue - Serverless data integration service with a managed catalog for AWS data assets.
  • Hive Metastore - Component of Hadoop HiveServer2 that can be used standalone as a schema registry for table metadata.
  • Nessie - Git-like versioning catalog for data lakes, enabling branch and merge operations on Iceberg/Delta/Hudi tables.
  • Databricks unity catalog - Unified governance layer for data and AI assets across the Databricks platform.

Object Storage

  • Apache HDFS - Hadoop distributed file system, the original large-scale storage layer for the big data ecosystem.
  • AWS S3 - Highly durable and available object storage service, the dominant cloud storage backend for data lakes.
  • Azure Blob Storage - Microsoft's massively scalable object storage for unstructured data.
  • GCP Cloud Storage - Google's unified object storage service for any amount of data.

Open table formats

Datasets

Distributed SQL processing

  • awesome-public-datasets - Curated list of high-quality public datasets organized by domain.
  • CommonCrawl - Petabyte-scale web crawl dataset updated monthly; used for NLP, analytics, and link graph research.
  • Criteo - 1TB click log dataset from Criteo, a standard benchmark for ad-tech and high-cardinality analytics workloads.
  • Entso-e - European electricity generation and consumption statistics, useful for time-series and energy analytics benchmarks.
  • GitHub Archives - Timestamped record of all public GitHub events; a popular dataset for querying with ClickHouse, BigQuery, and DuckDB.
  • Kaggle - Community-sourced datasets and competitions covering a wide range of domains.
  • NYCTaxi - NYC taxi trip records dating back to 2009; a classic columnar query benchmark with billions of rows.

Durable execution

Distributed SQL processing

  • Temporal - Durable workflow execution platform for building fault-tolerant pipelines and long-running data processes.

ETL, ELT and reverse ETL

Distributed SQL processing

  • Airbyte - Open-source ELT platform with 300+ pre-built connectors for syncing data to your warehouse.
  • Census - Reverse ETL platform for syncing data warehouse data to CRMs, ad tools, and other SaaS.
  • RudderStack - Customer Data Platform providing a pipeline between a tracking plan, event transformation, and destination tools.
  • dbt - SQL-based transformation framework that runs inside your warehouse; the standard tool for the T in ELT.
  • Debezium - Open-source CDC (Change Data Capture) platform that streams row-level changes from databases like PostgreSQL, MySQL, and MongoDB into Kafka and downstream systems.

Events

ClickHouse vs Apache Druid vs Apache Pinot vs StarRocks

  • Databricks Data+AI Summit - The world's largest data, analytics, and AI conference.
  • Snowflake Summit - Annual conference for data and AI practitioners.
  • Confluent Current - The Data Streaming Event focused on Apache Kafka and real-time data streaming.
  • Data Council - Technical conference on data engineering, infrastructure, and analytics.
  • dbt Summit - The world's largest gathering of dbt users and analytics engineering practitioners.
  • Flink Forward - Conference dedicated to real-time stream processing and Apache Flink.
  • Community Over Code - The Apache Software Foundation's official conference (formerly ApacheCon).
  • VLDB - Premier academic conference on Very Large Data Bases.
  • ACM SIGMOD/PODS - Leading international forum for database researchers and practitioners.

Ingestion and querying

Batch processing

  • Apache Spark - Distributed batch processing engine with in-memory computation, supporting SQL, ML, and graph workloads.
  • MapReduce - Programming model for processing large datasets in parallel across a cluster; the foundation of the Hadoop ecosystem.

Distributed SQL processing

  • Apache Spark SQL - Distributed SQL query engine that sit on top of Spark.
  • PrestoDB - Distributed SQL query engine.
  • Trino - Distributed SQL query engine. Fork of PrestoDB.
  • Dremio - SQL lakehouse platform providing a semantic layer and query acceleration on top of data lakes.
  • ksql - SQL interface for Kafka.

In-memory processing

  • Apache Arrow - Low-level in-memory columnar data format with zero-copy access across languages via gRPC/IPC interfaces.
  • DuckDB - In-process SQL OLAP query engine for Parquet, CSV, and JSON files. Built on Apache Arrow.
  • Pandas - Python DataFrame library for data analysis and manipulation, the standard for data science workflows.
  • delta-rs - Standalone DeltaLake driver for Python and Rust. Does not depend on Spark.
  • chDB - Embeddable in-process OLAP engine powered by ClickHouse, callable from Python without a server.
  • Polars - High-performance DataFrame library written in Rust with a lazy query optimizer, significantly faster than Pandas.
  • clickhouse-local - Lightweight CLI version of ClickHouse for running SQL queries against CSV, JSON, Parquet and other files.
  • Delta Standalone - Standalone DeltaLake driver for Java and Scala. Does not depend on Spark.

Stream processing

  • Apache Beam - Unified SDK for cross-language stream and batch processing. Available in Go, Python, Java, Scala and TypeScript.
  • Apache Flink - Stateful stream processing with exactly-once semantics, supporting event time and out-of-order data.
  • Apache Kafka Streams - Lightweight stream processing library embedded in the Kafka client, no separate cluster required.
  • Apache Spark Streaming - Micro-batch stream processing on top of Spark, integrating with the broader Spark ecosystem.
  • Akka Streams - Reactive stream processing library for JVM, built on the actor model.
  • Redpanda Connect (formerly Benthos) - Declarative stream processing toolkit in Go, with a wide connector library.
  • Materialize - Operational data warehouse that incrementally maintains SQL views over streaming data, always-fresh without recomputation.
  • RisingWave - Distributed SQL streaming database (PostgreSQL-compatible) with sub-100ms end-to-end freshness and native Iceberg integration.

📝 License

Tool-specific

OLAP Databases

Hybrid OLAP/OLTP NewSQL (aka HTAP)

  • Citus - PostgreSQL compatible distributed table.
  • TiDB - MySQL compatible SQL database that supports hybrid transactional and analytical processing workloads.
  • CockroachDB - Distributed SQL database with strong consistency, horizontal scaling, and PostgreSQL compatibility for HTAP workloads.
  • YugabyteDB - Distributed SQL database compatible with PostgreSQL and Cassandra APIs, designed for global, cloud-native HTAP applications.

Managed cloud services

  • AWS Redshift - Fully managed petabyte-scale data warehouse on AWS.
  • Azure Synapse Analytics - Unified analytics service combining data integration, warehousing, and big data on Azure.
  • Databricks - Lakehouse platform combining data warehousing and ML, built on Delta Lake and Apache Spark.
  • Firebolt - Cloud-native OLAP warehouse engineered for sub-second query performance at scale.
  • Google BigQuery - Serverless, pay-as-you-go data warehouse with built-in ML and BI capabilities.
  • Snowflake - Cloud data platform with a decoupled storage and compute architecture, supporting multi-cloud deployments.
  • Tinybird - Real-time analytics API platform built on ClickHouse.

Real-time analytics

  • shared-nothing architecture - second response time. DDL, DML and DCL are operated via SQL. These databases also support tiering for long-term cold storage.
  • Apache Doris - MPP analytical database with MySQL-compatible interface, optimized for high-concurrency queries and real-time data ingestion.
  • Apache Druid - Real-time OLAP database optimized for streaming ingestion, time-series analytics, and sub-second queries on high-cardinality data.
  • Apache HBase - Distributed, wide-column NoSQL database on top of HDFS, modeled after Google Bigtable.
  • Apache Pinot - Distributed OLAP datastore for user-facing real-time analytics, designed for low-latency queries at high concurrency.
  • StarRocks - MPP OLAP database with vectorized execution engine, optimized for real-time analytics and high-concurrency workloads.
  • ClickHouse - Column-oriented DBMS for online analytical processing, capable of processing billions of rows per second.

Search engines

  • Elasticsearch - Search and analytics engine based on Apache Lucene.
  • Meilisearch - Open source search engine, that aims to be a ready-to-go solution.
  • OpenSearch - Apache 2.0 fork of Elasticsearch.
  • Quickwit - Search engine on top of object storage, using shared-everything architecture.
  • Typesense - Open-source, typo-tolerant search engine optimized for instant search-as-you-type experiences and developer productivity.

Timeseries

  • Grafana Mimir - Prometheus-compatible TSDB on top of object storage, horizontally scalable.
  • TimeScaleDB - PostgreSQL-compatible TSDB with automatic partitioning and time-series-specific SQL extensions.
  • InfluxDB - Purpose-built time series database optimized for high-write-throughput metrics, events, and IoT data with a SQL-like query language.
  • Prometheus - Pull-based metrics collection and time series database, de facto standard for cloud-native monitoring.
  • VictoriaMetrics - Fast, cost-effective Prometheus-compatible TSDB with low memory and storage footprint.

People to follow

ClickHouse vs Apache Druid vs Apache Pinot vs StarRocks

Readings

Architecture

  • CoW vs MoR - Comparison of Copy-on-Write and Merge-on-Read table strategies in Apache Hudi, with trade-offs for read vs write performance.
  • Kappa architecture - Streaming-only alternative to the Lambda architecture that eliminates the batch layer.
  • Medallion architecture - Bronze/silver/gold data quality layering pattern for incrementally refining data in a lakehouse.
  • Reactive programming - Event-driven programming model based on asynchronous, composable data streams.
  • Lambda architecture - Big data pattern combining a batch layer for accuracy and a speed layer for low-latency results.
  • CQRS (Command Query Responsibility Segregation) - Martin Fowler's guide on separating read and write models to independently scale and optimize each path.
  • Event sourcing - Pattern for persisting application state as an immutable log of events rather than mutable records.
  • DAG - Directed Acyclic Graph — the dependency model used to represent task ordering in data pipeline orchestration.
  • Star schema vs Snowflake schema - Dimensional data modeling patterns for organizing fact and dimension tables in an analytics warehouse.

Blogs to follow

  • Engineering at Meta - Meta's engineering blog covering data infrastructure, distributed systems, and AI at hyperscale.
  • Engineering at Uber - Uber's engineering blog covering data infrastructure, streaming systems, and distributed databases.
  • Engineering at Airbnb - Airbnb's engineering blog on data platform, analytics engineering, and ML infrastructure.
  • Databricks - Engineering blog on lakehouse architecture, Apache Spark, Delta Lake, and MLflow.
  • Engineering at Criteo - Criteo's engineering blog on ad-tech data pipelines, Spark, and large-scale machine learning.
  • Towards Data Science - Community publication covering data science, data engineering, and machine learning.
  • Antithesis - Blog from the autonomous testing platform covering distributed systems correctness, fault injection, and database reliability.

Challenging platforms

  • Datadog event store - How Datadog built Husky, a column-store for ingesting and querying billions of tagged events at scale.
  • Cloudflare logging - How Cloudflare processes 6 million HTTP requests per second using ClickHouse for real-time log analytics.

Consensus

  • Paxos - Lamport's foundational distributed consensus algorithm for agreeing on a value across unreliable nodes.
  • Raft - Understandable distributed consensus algorithm designed as a more accessible alternative to Paxos, used in etcd and CockroachDB.

Data modeling

  • Schema evolution - How Delta Lake enforces and evolves schemas without breaking downstream readers.
  • CDC - Change Data Capture — tracking row-level insertions, updates, and deletes for replication, audit, and event-driven pipelines.

Index

  • Partitioning - Dividing data into logical subsets (by date, region, etc.) to prune irrelevant partitions at query time.
  • Statistics - Column-level statistics in Hive used by the query planner to estimate cardinality and choose optimal join strategies.
  • HyperLogLog - Probabilistic cardinality estimation algorithm used for COUNT(DISTINCT) at scale with sub-1% error.
  • Bitmap index - Compact bitset-based index efficient for low-cardinality columns and multi-predicate AND/OR queries.
  • Dense index - Index with an entry for every row in the table, enabling direct lookups at the cost of index size.
  • Sparse index - Index with entries for only a subset of rows, trading lookup precision for smaller index footprint.
  • Reverse index - Inverted index mapping terms or values back to the rows containing them, foundational to full-text search.
  • N-gram - PostgreSQL trigram-based similarity index for fast fuzzy text matching and LIKE query acceleration.
  • TF-IDF - Term Frequency-Inverse Document Frequency scoring algorithm for ranking documents by relevance in full-text search.
  • LSM Tree - Log-Structured Merge-Tree — write-optimized storage structure that batches writes in memory before merging to disk, used in RocksDB, Cassandra, and LevelDB.
  • Data skipping - ClickHouse's secondary skipping indexes for filtering data granules without full column scans.
  • High cardinality - Columns with many distinct values and how high cardinality impacts indexing and storage in time-series databases.
  • Bloom filters - Space-efficient probabilistic data structure for fast set-membership testing in query engines and storage layers.
  • Minmax - Parquet page-level min/max statistics used for predicate pushdown to skip irrelevant row groups.
  • Z-ordering - Multi-dimensional data clustering that co-locates related values to speed up multi-column filter queries.

More

Papers

  • Apache Flink state management - Carbone et al. (2017) on Flink's state backend, incremental checkpointing, and exactly-once fault tolerance.
  • Dremel paper - Google's 2010 paper introducing columnar storage for nested data and interactive ad-hoc queries at petabyte scale.
  • RDD - Zaharia et al. (2012) introducing Resilient Distributed Datasets, the fault-tolerant in-memory abstraction that became the foundation of Apache Spark.
  • Spanner paper - Corbett et al. (2012) on Google Spanner, the globally distributed ACID database using TrueTime for external consistency.
  • RocksDB - Facebook's paper on how RocksDB's design priorities evolved to serve large-scale production workloads.
  • Apache Parquet format - Formal specification of the Parquet columnar storage format, including encoding, compression, and nested schema representation.

Querying

  • Cost Based Optimization - How Spark 2.2's cost-based optimizer uses column statistics to choose better join strategies and query plans.
  • Sampling - Statistical sampling techniques used in approximate query processing to return fast estimates over large datasets.
  • GraphX - Apache Spark's graph processing framework and Pregel-based API for iterative graph algorithms at scale.

Transactions

  • ACID properties - Atomicity, Consistency, Isolation, Durability — the four guarantees that define correct database transaction behavior.
  • Serializable transaction - CockroachDB's interactive demo illustrating how serializable isolation prevents anomalies like write skew and phantom reads.

Vectorized query processing