Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

awesome-olap

A curated list of awesome Online Analytical Processing databases, frameworks, ressources and other awesomeness.
https://github.com/samber/awesome-olap

Last synced: 4 days ago
JSON representation

OLAP Databases
- Real-time analytics
  - shared-nothing architecture - second response time. DDL, DML and DCL are operated via SQL. These databases also support tiering for long-term cold storage.
  - Apache Doris
  - Apache Druid
  - Apache HBase
  - Apache Pinot
  - StarRocks
  - Clickhouse
- Search engines
  - Elasticsearch - Search and analytics engine based on Apache Lucene.
  - Meilisearch - Open source search engine, that aims to be a ready-to-go solution.
  - OpenSearch - Apache 2.0 fork of Elasticsearch.
  - Quickwit - Search engine on top of object storage, using shared-everything architecture.
  - Typesense - Оpen-source, typo-tolerant search engine optimized for instant search-as-you-type experiences and developer productivity.
- NewSQL
  - Citus - PostgreSQL compatible distributed table.
  - TiDB - MySQL compatible SQL database that supports hybrid transactional and analytical processing workloads.
- Timeseries
  - Grafana Mimir - Prometheus compatible TSDB on top of object storage.
  - TimeScaleDB - PostgreSQL compatible TSDB.
- Managed cloud services
Data lake
- File formats and serialization
  - Apache Arrow Columnar Format - Columnar format for in-memory Apache Arrow processing.
  - Apache Avro - Row-oriented serialization for data streaming purpose.
  - Apache ORC - Column-oriented serialization for data storage purpose. Part of Hadoop platform.
  - Apache Parquet - Column-oriented serialization for data storage purpose.
  - Apache Thrift - Row-oriented serialization for RPC purpose.
  - Google Protobuf - Row-oriented serialization for RPC purpose.
  - Schema Registry - Centralized repository for validating row-oriented events. Part of Kafka and Confluent platform.
- Open table formats
- Metastore
  - AWS Glue
  - Databricks unity catalog
  - Hive Metastore - Component of Hadoop HiveServer2, that can be used standalone.
  - Nessie
- Object Storage
  - Apache HDFS - Hadoop distributed file system.
  - AWS S3
  - Azure Blob Storage
  - GCP Cloud Storage
  - Minio - S3 compatible and self-hosted object storage.
- Codecs, encoding and compression
  - Bit packing
  - Brotli
  - Deflate
  - Delta
  - LZ4
  - RLE
  - Snappy
  - zstd
  - Gorilla
Brokers and distributed messaging
- Codecs, encoding and compression
Ingestion and querying
- Stream processing
  - Apache Beam - Unified SDK for cross language stream processing. Available in Go, Python, Java, Scala and Typescript.
  - Apache Flink - Stateful stream processing.
  - Apache Kafka stream - Stream processing.
  - Apache Spark streaming - Stream processing on top of Spark.
  - Akka stream - Stream processing.
  - Benthos - Go stream processing.
- Batch processing
  - Apache Spark
  - MapReduce
- In-memory processing
  - Apache Arrow - Low-level in-memory data processing. Zero-copy data manipulation for any language, via gRPC/IPC interfaces.
  - Apache Arrow Datafusion - High level SQL interface for Apache Arrow.
  - Delta Standalone - Standalone DeltaLake driver for Java and Scala. Do not depend on Spark.
  - DuckDB - In-process SQL query engine for processing Parquet files. Built on top of Apache Arrow.
  - Pandas - Python data analysis and manipulation tool.
  - clickhouse-local - Lightweight CLI version of Clickhouse for running SQL queries against CSV, JSON, Parquet, etc files.
  - delta-rs - Standalone DeltaLake driver for Python and Rust. Do not depend on Spark.
  - Apache Arrow Datafusion - High level SQL interface for Apache Arrow.
- Distributed SQL processing
  - Apache Spark SQL - Distributed SQL query engine that sit on top of Spark.
  - ksql - SQL interface for Kafka.
  - PrestoDB - Distributed SQL query engine.
  - Trino - Distributed SQL query engine. Fork of PrestoDB.
Scheduler
- Distributed SQL processing
  - Apache Airflow
  - Dagster
ETL, ELT and reverse ETL
- Distributed SQL processing
  - Airbyte - ELT.
  - Census - Reverse ETL.
  - RudderStack - Customer Data Platform. Pipeline between a tracking plan, event transformation, and destination tools (datawarehouse or SaaS).
Datasets
- Distributed SQL processing
  - awesome-public-datasets
  - CommonCrawl
  - Criteo
  - Entso-e
  - GitHub Archives
  - Kaggle - Community sourced dataset.
  - NYCTaxy
Benchmark
- Distributed SQL processing
  - Jepsen - Distributed databases, queues and consensus protocols testing.
  - TPC family benchmarks - For big data based database.
Readings
- Papers
- Architecture
- Data modeling
  - Schema evolution
  - CDC
- Index
- Vector similarity search
  - ANN (approximate nearest neighbor)
  - kNN (k nearest neighbor)
  - Faiss
  - HNSW
  - HNSW
  - HNSW
  - HNSW
  - HNSW
  - HNSW
  - HNSW
  - HNSW
  - HNSW
  - HNSW
  - HNSW
  - HNSW
  - HNSW
  - HNSW
  - HNSW
- Vectorized query processing
- Querying
- Transactions
  - ACID properties
  - Serializable transaction
- Consensus
  - Paxos
  - Raft
- Challenging platforms
  - Datadog event store
  - Cloudflare logging
- Blogs to follow
👤 Contributors
- More
  - Contributors
💫 Show your support
- More
  - ![GitHub Sponsors
📝 License
- More
  - Samuel Berthe

Programming Languages

Rust 1 Go 1

Ecosyste.ms: Awesome

awesome-olap

OLAP Databases

Real-time analytics

Search engines

NewSQL

Timeseries

Managed cloud services

Data lake

File formats and serialization

Open table formats

Metastore

Object Storage

Codecs, encoding and compression

Brokers and distributed messaging

Codecs, encoding and compression

Ingestion and querying

Stream processing

Batch processing

In-memory processing

Distributed SQL processing

Scheduler

Distributed SQL processing

ETL, ELT and reverse ETL

Distributed SQL processing

Datasets

Distributed SQL processing

Benchmark

Distributed SQL processing

Readings

Papers

Architecture

Data modeling

Index

Vector similarity search

Vectorized query processing

Querying

Transactions

Consensus

Challenging platforms

Blogs to follow

👤 Contributors

More

💫 Show your support

More

📝 License

More