awesome-data-engineering

A curated list of data engineering tools for software developers
https://github.com/igorbarinov/awesome-data-engineering

Last synced: 4 days ago
JSON representation

Batch Processing
- Hadoop MapReduce - A software framework for easily writing applications which process vast amounts of data (multi-terabyte data-sets) - in-parallel on large clusters (thousands of nodes) - of commodity hardware in a reliable, fault-tolerant manner.
- Spark - A multi-language engine for executing data engineering, data science, and machine learning on single-node machines or clusters.
- Spark RDD API Examples - Examples by Zhen He.
- AWS EMR - A web service that makes it easy to quickly and cost-effectively process vast amounts of data.
- Tez - An application framework which allows for a complex directed-acyclic-graph of tasks for processing data.
- Mahout - An environment for quickly creating scalable performant machine learning applications.
- Spark MLlib - Spark's scalable machine learning library consisting of common learning algorithms and utilities, including classification, regression, clustering, collaborative filtering, dimensionality reduction, as well as underlying optimization primitives.
- Giraph - An iterative graph processing system built for high scalability.
- Spark GraphX - Apache Spark's API for graphs and graph-parallel computation.
- Presto - A distributed SQL query engine designed to query large data sets distributed over one or more heterogeneous data sources.
- Drill - Schema-free SQL Query Engine for Hadoop, NoSQL and Cloud Storage.
- Deep Spark - Connecting Apache Spark with different data stores. Deprecated.
- Delight - A free & cross platform monitoring tool (Spark UI / Spark History Server alternative).
- Bistro - A light-weight engine for general-purpose data processing including both batch and stream analytics. It is based on a novel unique data model, which represents data via _functions_ and processes data via _columns operations_ as opposed to having only set operations in conventional approaches like MapReduce or SQL.
- Hivemall - Scalable machine learning library for Hive/Hadoop.
- PyHive - Python interface to Hive and Presto.
- Substation - A cloud native data pipeline and transformation toolkit written in Go.
- GraphLab Create - A machine learning platform that enables data scientists and app developers to easily create intelligent apps at scale.
- dna-claude-analysis - Personal genome analysis toolkit with Python scripts analyzing raw DNA data across 17 categories (health risks, ancestry, pharmacogenomics, nutrition, psychology, etc.) and generating a terminal-style single-page HTML visualization.
- Datatrax - Pure-Go classic machine learning toolkit and data engineering utilities. Eight algorithms with zero external dependencies.
- Zingg - Open source Master Data Management platform using machine learning for entity resolution at scale. Native to Databricks, Microsoft Fabric, Snowflake, AWS, and GCP. Golden records are maintained through a persistent Zingg ID across all systems and sources.
- Spark Packages - A community index of packages for Apache Spark.
- Livy - The REST Spark Server.
- Hive - Data warehouse software facilitates querying and managing large datasets residing in distributed storage.
- Data Mechanics - A cloud-based platform deployed on Kubernetes making Apache Spark more developer-friendly and cost-effective.
Charts and Dashboards
- Highcharts - A charting library written in pure JavaScript, offering an easy way of adding interactive charts to your web site or web application.
- ZingChart - Fast JavaScript charts for any data set.
- D3.js - A JavaScript library for manipulating documents based on data.
- SmoothieCharts - A JavaScript Charting Library for Streaming Data.
- Redash - Make Your Company Data Driven. Connect to any data source, easily visualize and share your data.
- PyQtGraph - A pure-python graphics and GUI library built on PyQt4 / PySide and numpy. It is intended for use in mathematics / scientific / engineering applications.
- PyXley - Python helpers for building dashboards using Flask and React.
- Plotly - Flask, JS, and CSS boilerplate for interactive, web-based visualization apps in Python.
- Metabase - The easy, open source way for everyone in your company to ask questions and learn from data.
- QueryGPT - Natural language database query interface with automatic chart generation, supporting Chinese and English queries.
- Seaborn - A Python visualization library based on matplotlib. It provides a high-level interface for drawing attractive statistical graphics.
- Apache Superset - A modern, enterprise-ready business intelligence web application.
- Dekart - Open-source SQL to map platform for BigQuery, Snowflake, and PostGIS.
- LunaPad - Open-source analytics notebook for reusable SQL workflows, interactive reports, and AI-assisted data exploration.
- C3.js - D3-based reusable chart library.
- D3Plus - D3's simpler, easier to use cousin. Mostly predefined templates that you can just plug data in.
- FlexViz - Open-source Python library for interactive cross-filter dashboards on large datasets. Zoom, pan, and selection are answered by lazy Polars aggregations on a stateless server instead of sending rows to the browser.
- stratif.io - Open-source, self-hosted, warehouse-native product analytics. Runs funnels, retention, and paths directly on DuckDB, Postgres, Snowflake, or ClickHouse.
Community
- Books
  - Snowflake Data Engineering - A practical introduction to data engineering on the Snowflake cloud data platform.
  - Best Data Science Books - This blog offers a curated list of top data science books, categorized by topics and learning stages, to aid readers in building foundational knowledge and staying updated with industry trends.
  - Architecting an Apache Iceberg Lakehouse - A guide to designing an Apache Iceberg lakehouse from scratch.
  - Learn AI Data Engineering in a Month of Lunches - A fast, friendly guide to integrating large language models into your data workflows.
- Conferences
  - Data Council - The first technical conference that bridges the gap between data scientists, data engineers and data analysts.
- Forums
  - /r/dataengineering - News, tips, and background on Data Engineering.
  - /r/etl - Subreddit focused on ETL.
  - AI Dev Jobs - Job board focused on AI, ML, and data engineering roles with 7,400+ listings, salary data, and a free REST API.
- Podcasts
  - Data Engineering Podcast - The show about modern data infrastructure.
  - The Data Stack Show - A show where they talk to data engineers, analysts, and data scientists about their experience around building and maintaining data infrastructure, delivering data and data products, and driving better outcomes across their businesses with data.
  - Latent Space - Technical deep dives on AI engineering, from model training to deployment.
  - Practical AI - Making AI practical, productive, and accessible to everyone.
  - Software Engineering Daily - Daily interviews about technical software topics, including data infrastructure.
  - The Analytics Engineering Podcast - How analytics engineers build and maintain data pipelines at scale.
  - Chain of Thought - Interviews with AI and data infrastructure leaders on building production systems.
Databases
- MySQL - The world's most popular open source database.
- MariaDB - An enhanced, drop-in replacement for MySQL.
- PostgreSQL - The world's most advanced open source database.
- Amazon RDS - Makes it easy to set up, operate, and scale a relational database in the cloud.
- Redis - An open source, BSD licensed, advanced key-value cache and store.
- Riak - A distributed database designed to deliver maximum data availability by distributing data across multiple servers.
- AWS DynamoDB - A fast and flexible NoSQL database service for all applications that need consistent, single-digit millisecond latency at any scale.
- Cassandra Calculator - This simple form allows you to try out different values for your Apache Cassandra cluster and see what the impact is for your application.
- HBase - The Hadoop database, a distributed, scalable, big data store.
- AWS Redshift - A fast, fully managed, petabyte-scale data warehouse that makes it simple and cost-effective to analyze all your data using your existing business intelligence tools.
- Elasticsearch - Search & Analyze Data in Real Time.
- Couchbase - The highest performing NoSQL distributed database.
- RethinkDB - The open-source database for the realtime web.
- RavenDB - Fully Transactional NoSQL Document Database.
- Neo4j - The world's leading graph database.
- Titan - A scalable graph database optimized for storing and querying graphs containing hundreds of billions of vertices and edges distributed across a multi-machine cluster.
- Apache Geode - An open source, distributed, in-memory database for scale-out applications.
- Riak-TS - Riak TS is the only enterprise-grade NoSQL time series database optimized specifically for IoT and Time Series data.
- Rhombus - A time-series object store for Cassandra that handles all the complexity of building wide row indexes.
- RQLite - Replicated SQLite using the Raft consensus protocol.
- TiDB - A distributed NewSQL database compatible with MySQL protocol.
- mysql_utils - Pinterest MySQL Management Tools.
- HyperDex - A scalable, searchable key-value store. Deprecated.
- IonDB - A key-value store for microcontroller and IoT applications.
- FiloDB - Distributed. Columnar. Versioned. Streaming. SQL.
- MemDB - Distributed Transactional In-Memory Database (based on MongoDB).
- FlockDB - A distributed, fault-tolerant graph database by Twitter. Deprecated.
- Gaffer - A large-scale graph database.
- InfluxDB - Scalable datastore for metrics, events, and real-time analytics.
- OpenTSDB - A scalable, distributed Time Series Database.
- kairosdb - Fast scalable time series database.
- Heroic - A scalable time series database based on Cassandra and Elasticsearch, by Spotify.
- Akumuli - A numeric time-series database. It can be used to capture, store and process time-series data in real-time. The word "akumuli" can be translated from esperanto as "accumulate".
- Dalmatiner DB - Fast distributed metrics database.
- Timely - A time series database application that provides secure access to time series data based on Accumulo and Grafana.
- cayley - An open-source graph database. Google.
- DuckDB - A fast in-process analytical database that has zero external dependencies, runs on Linux/macOS/Windows, offers a rich SQL dialect, and is free and extensible.
- Vertica - Distributed, MPP columnar database with extensive analytics SQL.
- OrientDB - 2nd Generation Distributed Graph Database with the flexibility of Documents in one product with an Open Source commercial friendly license.
- Blueflood - A distributed system designed to ingest and process time series data.
- Tarantool - An in-memory database and application server.
- Snappydata - OLTP + OLAP Database built on Apache Spark.
- SSDB - A high performance NoSQL database supporting many data structures, an alternative to Redis.
- Percona XtraBackup - A free, open source, complete online backup solution for all versions of Percona Server, MySQL® and MariaDB®.
- ScyllaDB - NoSQL data store using the seastar framework, compatible with Apache Cassandra.
- TimescaleDB - Built as an extension on top of PostgreSQL, TimescaleDB is a time-series SQL database providing fast analytics, scalability, with automated data management on a proven storage engine.
- ClickHouse - Distributed columnar DBMS for OLAP. SQL.
- Kyoto Tycoon - A lightweight network server on top of the Kyoto Cabinet key-value database, built for high-performance and concurrency.
- CCM - A script to easily create and destroy an Apache Cassandra cluster on localhost.
- Druid - Column oriented distributed data store ideal for powering interactive applications.
- GreenPlum - The Greenplum Database (GPDB) - An advanced, fully featured, open source data warehouse. It provides powerful and rapid analytics on petabyte scale data volumes.
- Actionbase - A database for user interactions (likes, views, follows) represented as graphs, with precomputed reads served in real-time.
- Percona Server for MongoDB - Percona Server for MongoDB® is a free, enhanced, fully compatible, open source, drop-in replacement for the MongoDB® Community Edition that includes enterprise-grade features and functionality.
- Rivestack - Managed PostgreSQL with pgvector for AI workloads. HNSW indexing, sub-4ms latency, and built-in SQL editor with automatic embedding generation.
- ArcadeDB - Open-source multi-model database with native graph, document, key-value, and vector support. SQL, Cypher, and Gremlin query languages. Apache 2.0 license.
- Omnigraph - Typed graph database where agents branch and merge like Git. S3-native, Rust, traversal + vector + BM25 in one runtime.
- SlothDB - In-process analytical SQL database written in C++20. Reads Parquet, CSV, JSON, Avro, Arrow, SQLite, and Excel directly. Single binary, Python package, and 1.3 MB WASM build for the browser.
- MongoDB - An open-source, document database designed for ease of development and scaling.
- zvec - An embedded vector database for on-device RAG and edge AI, the SQLite of vector databases.
- ReductStore - High-performance blob and time-series storage, with edge deployment, selective replication, and efficient querying of multimodal data.
- Manticore Search - An open-source search database for full-text, vector, and hybrid search with real-time indexing and SQL.
- DAtomic - The fully transactional, cloud-ready, distributed database.
- chDB - Embedded ClickHouse — full ClickHouse SQL dialect, ~80 data formats, and 12+ source connectors (S3, Postgres, MongoDB, Kafka, Iceberg) in core. Python, Go, Rust, Node, Bun, Zig, and Ruby bindings.
Data Comparison
- datacompy - A Python library that facilitates the comparison of two DataFrames in Pandas, Polars, Spark and more. The library goes beyond basic equality checks by providing detailed insights into discrepancies at both row and column levels.
- dvt - Data Validation Tool compares data from source and target tables to ensure that they match. It provides column validation, row validation, schema validation, custom query validation, and ad hoc SQL exploration.
- koala-diff - A high-performance Python library for comparing large datasets (CSV, Parquet) locally using Rust and Polars. It features zero-copy streaming to prevent OOM errors and generates interactive HTML data quality reports.
- FutureSearch SDK - Python SDK that dispatches parallel web-research agents across
Data Ingestion
- Kafka - Publish-subscribe messaging rethought as a distributed commit log.
- AWS Kinesis - A fully managed, cloud-based service for real-time data processing over large, distributed data streams.
- RabbitMQ - Robust messaging for applications.
- Apache Pulsar - An open-source distributed pub-sub messaging system.
- Sling - CLI data integration tool specialized in moving data between databases, as well as storage systems.
- Meltano - CLI & code-first ELT.
- BottledWater - Change data capture from PostgreSQL into Kafka. Deprecated.
- kafkat - Simplified command-line administration for Kafka brokers.
- pg-kafka - A PostgreSQL extension to produce messages to Apache Kafka.
- kafka-docker - Kafka in Docker.
- kafka-node - Node.js client for Apache Kafka 0.8.
- Secor - Pinterest's Kafka to S3 distributed consumer.
- Kafka-logger - Kafka-winston logger for Node.js from Uber.
- Heka - Data Acquisition and Processing Made Easy. Deprecated.
- Google Sheets ETL - Live import all your Google Sheets to your data warehouse.
- Artie - Real-time data ingestion tool leveraging change data capture.
- CsvPath Framework - A delimited data preboarding framework that fills the gap between MFT and the data lake.
- kafka-manager - A tool for managing Apache Kafka.
- Pravega - Provides a new storage abstraction - a stream - for continuous and unbounded data.
- dlt - A fast&simple pipeline building library for Python data devs, runs in notebooks, cloud functions, airflow, etc.
- Kroxylicious - A Kafka Proxy, solving problems like encrypting your Kafka data at rest.
- db2lake - Lightweight Node.js ETL framework for databases → data lakes/warehouses.
- AWS Data Wrangler - Utility belt to handle data on AWS.
- Gobblin - Universal data ingestion framework for Hadoop from LinkedIn.
- ingestr - CLI tool to copy data between databases with a single command. Supports 50+ sources including PostgreSQL, MySQL, MongoDB, Salesforce, Shopify to any data warehouse.
- Kreuzberg - Polyglot document intelligence library with a Rust core and bindings for Python, TypeScript, Go, and more. Extracts text, tables, and metadata from 62+ document formats for data pipeline ingestion.
- DataRaven - Managed cloud object storage transfers for ingestion workflows.
- crdt-merge - Conflict-free merge for DataFrames, JSON, ML models & distributed agents — powered by CRDTs.
- librdkafka - The Apache Kafka C/C++ library.
- DataSpoc Pipe - Data ingestion engine that connects 400+ Singer taps to Parquet files in cloud buckets (S3, GCS, Azure). Streaming, incremental, with auto-catalog.
- drt - OSS Reverse ETL CLI. Sync data from warehouses to business tools via YAML.
- data-genie - High-performance, streaming-first ETL engine for Node.js and TypeScript with constant memory footprint.
- pdfmux - Python PDF-to-Markdown orchestrator. Classifies each page and routes to the optimal backend (PyMuPDF, Docling, RapidOCR, Gemini Flash), emitting Markdown plus a per-page confidence score so ingestion pipelines can quarantine low-trust pages before feeding LLMs or retrieval.
- LinkedIn Jobs Scraper - Crawlee-based actor extracting structured LinkedIn job listings at scale without API keys.
- CARQ - Context-Aware RAG Processing Queue for high availability and adaptive rate-limiting.
- Duckle - Local-first, open-source desktop ETL/ELT studio: drag a pipeline onto a canvas (or describe it to a built-in on-device AI assistant) and run it at native speed through DuckDB. 290+ connectors, a scheduler, and an MCP server for driving pipelines from an LLM. No cloud, no servers.
- Rawbbit - Open-source self-hosted game analytics pipeline. HTTP event collector with NATS JetStream buffering, raw Parquet in object storage you own, and ClickHouse for queries via Metabase, SQL, or a read-only MCP server for AI agents. Designed for teams that want to own their raw game data.
- Enrich.sh - Managed event ingestion service that converts JSON sent to a REST API into Hive-partitioned Parquet on Cloudflare R2, queryable from DuckDB, ClickHouse, BigQuery, Snowflake, and Python.
- enrich-companies - CLI tool to enrich CSV files with company data (financials, contacts, metadata) from 250M+ company records. Available on [npm](https://www.npmjs.com/package/enrich-companies).
- faucet-stream - Config-driven data-movement platform for Rust with pluggable source and sink connectors, running ETL, CDC, and streaming pipelines declaratively from YAML or embedded as a library.
- Embulk - An open source bulk data loader that helps data transfer between various databases, storages, file formats, and cloud services.
- Apache Sqoop - A tool designed for efficiently transferring bulk data between Apache Hadoop and structured datastores such as relational databases.
- Nakadi - An open source event messaging platform that provides a REST API on top of Kafka-like queues.
- DBConvert Streams - Self-hosted database migration and change data capture (CDC) tool with built-in SQL IDE.
- Singer SDK - The fastest way to build custom data extractors and loaders compliant with the Singer Spec.
- Jitsu - An open-source Customer Data Platform. Captures event data from websites, apps, and servers and streams it into ClickHouse, Snowflake, BigQuery, Redshift, Postgres, and MySQL in real time.
- Scriptella ETL - Open-source, Java-based ETL and script execution tool for transferring and transforming data between databases, files, and other sources.
- FluentD - An open source data collector for unified logging layer.
- Airbyte - Open-source data integration for modern data teams.
- Estuary Flow - No/low-code data pipeline platform that handles both batch and real-time data ingestion.
- Crustdata - A real-time B2B data API for company and people intelligence, providing firmographics, headcount signals, job listings, web traffic, and funding events via REST API and webhooks for data enrichment pipelines.
Data Lake Management
- lakeFS - An open source platform that delivers resilience and manageability to object-storage based data lakes.
- Project Nessie - A Transactional Catalog for Data Lakes with Git-like semantics. Works with Apache Iceberg tables.
- Ilum - A modular Data Lakehouse platform that simplifies the management and monitoring of Apache Spark clusters across Kubernetes and Hadoop environments.
- Gravitino - An open-source, unified metadata management for data lakes, data warehouses, and external catalogs.
- FlightPath Data - FlightPath is a gateway to a data lake's bronze layer, protecting it from invalid external data file feeds as a trusted publisher.
- rawquery - Managed lakehouse platform on Apache Iceberg with DuckDB query compute, S3 storage, Postgres wire protocol, and SQL transforms.
Datasets
- Data Dumps
  - GitHub Archive - GitHub's public timeline since 2011, updated every hour.
  - Common Crawl - Open source repository of web crawl data.
  - Wikipedia - Wikipedia's complete copy of all wikis, in the form of Wikitext source and metadata embedded in XML. A number of raw database tables in SQL form are also available.
  - FirstData - The world's most comprehensive authoritative data source knowledge base. 160+ curated sources from governments, international organizations, and research institutions with MCP integration.
  - The Quiet-Broke Index - A 30-metro composite of US household cost burdens (housing, taxes, childcare, healthcare, transport) aggregated from Census ACS, BLS Consumer Expenditure Survey, and HUD Fair Market Rents. Open methodology, free, no email gate.
  - Mindweave Synthetic Business Data - 42-table synthetic SME dataset with double-entry accounting, tax compliance (AU/US/UK), and temporal realism. CSV, SQL, Parquet, SQLite. Ideal for ETL pipeline testing.
  - LatAm Synth - Synthetic financial savings behavior generator for Latin America: users, savings goals, and transactions calibrated on 506K real records (2015–2024). Reproducible by seed, 100% synthetic.
- Realtime
  - Reddit - Real-time data is available including comments, submissions and links posted to reddit.
  - Eventsim - Event data simulator. Generates a stream of pseudo-random events from a set of users, designed to simulate web traffic.
  - Twitter Realtime - The Streaming APIs give developers low latency access to Twitter's global stream of Tweet data.
  - Helium MCP - Remote MCP server for real-time financial data, 3.2M+ news articles, ML options pricing, and news bias analysis. Free, no API key. [MCP](https://heliumtrades.com/mcp)
  - Sorsa API - Real-time X (Twitter) data API providing tweets, profiles, search, communities and engagement metrics. Up to 50x cheaper than the official X API with 20 req/sec rate limit, JSON output.
  - Eventum - Data generation platform for producing synthetic event streams with complex correlations.
Docker
- Rancher - RancherOS is a 20mb Linux distro that runs the entire OS as Docker containers.
- ImageLayers - Visualize Docker images and the layers that compose them.
- Gockerize - Package golang service into minimal Docker containers.
- Flocker - Easily manage Docker containers & their data.
- Weave - Weaving Docker containers into applications.

Programming Languages

Python 24 Go 18 Java 17 Rust 8 JavaScript 7 Scala 6 C++ 6 C 4 TypeScript 4 Shell 3

ecosyste.ms

Data

Tools

Indexes

Applications

Experiments

Awesome

awesome-data-engineering

Batch Processing

Charts and Dashboards

Community

Books

Conferences

Forums

Podcasts

Databases

Data Comparison

Data Ingestion

Data Lake Management

Datasets

Data Dumps

Realtime

Docker