{"id":128151,"url":"https://github.com/bricefotzo/awesome-modern-data-stack","name":"awesome-modern-data-stack","description":"A curated list of awesome tools, frameworks, and resources for the Modern Data Stack (MDS).","projects_count":205,"last_synced_at":"2026-06-02T06:00:23.552Z","repository":{"id":336067383,"uuid":"1148171322","full_name":"bricefotzo/awesome-modern-data-stack","owner":"bricefotzo","description":"A curated list of awesome tools, frameworks, and resources for the Modern Data Stack (MDS).","archived":false,"fork":false,"pushed_at":"2026-05-07T06:49:05.000Z","size":73,"stargazers_count":34,"open_issues_count":1,"forks_count":9,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-05-16T17:28:29.750Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bricefotzo.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-02-02T16:56:35.000Z","updated_at":"2026-05-07T06:49:10.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/bricefotzo/awesome-modern-data-stack","commit_stats":null,"previous_names":["bricefotzo/awesome-modern-data-stack"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/bricefotzo/awesome-modern-data-stack","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bricefotzo%2Fawesome-modern-data-stack","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bricefotzo%2Fawesome-modern-data-stack/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bricefotzo%2Fawesome-modern-data-stack/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bricefotzo%2Fawesome-modern-data-stack/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bricefotzo","download_url":"https://codeload.github.com/bricefotzo/awesome-modern-data-stack/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bricefotzo%2Fawesome-modern-data-stack/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33808702,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-02T02:00:07.132Z","response_time":109,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"created_at":"2026-04-14T06:00:57.710Z","updated_at":"2026-06-02T06:00:23.553Z","primary_language":null,"list_of_lists":false,"displayable":true,"categories":["Community","License","Data Lakes \u0026 Storage","Data Contracts","Streaming \u0026 Real-Time","Data Warehouses \u0026 Lakehouses","Data Integration \u0026 Ingestion","Data Notebooks \u0026 Exploration","Data Orchestration","Data Transformation","Query Engines","Vector Databases","ML Platforms \u0026 MLOps","Business Intelligence \u0026 Analytics","Learning Resources","Metrics Layer \u0026 Semantic Layer","Reverse ETL","DataOps \u0026 Version Control","Data Observability","Data Quality \u0026 Testing","Data Catalog \u0026 Discovery","Feature Stores","Data Sharing","Uncategorized"],"sub_categories":["Community Resources","Data Lake Formats","Commercial / Managed","Message Brokers \u0026 Streaming Platforms","Cloud Data Warehouses","Object Storage","Open Source","Python-Based Transformation","Change Data Capture (CDC)","Books","Distributed Computing","Lakehouse Platforms","Stream Processing","Courses \u0026 Certifications","Podcasts","SQL-Based Transformation","Slack Communities","Blogs \u0026 Newsletters","Conferences","Uncategorized"],"readme":"# Awesome Modern Data Stack [![Awesome](https://awesome.re/badge.svg)](https://awesome.re)\n\n\u003e A curated list of awesome tools, frameworks, and resources for the Modern Data Stack (MDS).\n\nThe **Modern Data Stack** refers to a collection of cloud-native tools and technologies that work together to help organizations collect, store, transform, and analyze data. This list aims to help data professionals navigate the ever-evolving data ecosystem.\n\n## Contents\n\n- [Data Integration \u0026 Ingestion](#data-integration--ingestion)\n- [Data Warehouses \u0026 Lakehouses](#data-warehouses--lakehouses)\n- [Data Lakes \u0026 Storage](#data-lakes--storage)\n- [Data Transformation](#data-transformation)\n- [Data Orchestration](#data-orchestration)\n- [Data Quality \u0026 Testing](#data-quality--testing)\n- [Data Observability](#data-observability)\n- [Data Catalog \u0026 Discovery](#data-catalog--discovery)\n- [Data Governance](#data-governance)\n- [Business Intelligence \u0026 Analytics](#business-intelligence--analytics)\n- [Metrics Layer \u0026 Semantic Layer](#metrics-layer--semantic-layer)\n- [Reverse ETL](#reverse-etl)\n- [Data Contracts](#data-contracts)\n- [Streaming \u0026 Real-Time](#streaming--real-time)\n- [Query Engines](#query-engines)\n- [Data Notebooks \u0026 Exploration](#data-notebooks--exploration)\n- [Feature Stores](#feature-stores)\n- [ML Platforms \u0026 MLOps](#ml-platforms--mlops)\n- [Vector Databases](#vector-databases)\n- [Data Privacy \u0026 Security](#data-privacy--security)\n- [Data Sharing](#data-sharing)\n- [DataOps \u0026 Version Control](#dataops--version-control)\n- [Learning Resources](#learning-resources)\n- [Community](#community)\n\n---\n- [querybear.com](https://querybear.com) - Ask your data anything – with persistent memory, schema learning, and more.\n\n## Data Integration \u0026 Ingestion\n\n*Tools for extracting data from various sources and loading it into data warehouses or lakes.*\n\n### Open Source\n\n- [Airbyte](https://airbyte.com/) - Open-source data integration platform with 300+ connectors. ELT-first approach.\n- [dlt (data load tool)](https://dlthub.com/) - Python library for data loading with automatic schema inference.\n- [Singer](https://www.singer.io/) - Open-source standard for writing scripts that move data (taps and targets).\n- [Meltano](https://meltano.com/) - Open-source DataOps platform built on Singer. CLI-first, version-controlled pipelines.\n- [Ingestr](https://github.com/bruin-data/ingestr) - CLI tool to copy data between databases with a single command.\n\n### Commercial / Managed\n\n- [Boomi](https://boomi.com/) - Modern data integration and agent management platform.\n- [Fivetran](https://www.fivetran.com/) - Automated data integration with 500+ connectors. Industry leader in managed ELT.\n- [Stitch](https://www.stitchdata.com/) - Simple, extensible ETL built for developers. Part of Talend.\n- [Hevo Data](https://hevodata.com/) - No-code data pipeline platform.\n- [Informatica](https://www.informatica.com/) - Enterprise data integration and management platform.\n- [Talend](https://www.talend.com/) - Enterprise data integration suite.\n\n---\n\n## Data Warehouses \u0026 Lakehouses\n\n*Cloud data warehouses and lakehouse platforms for storing and querying analytical data.*\n\n### Cloud Data Warehouses\n\n- [Snowflake](https://www.snowflake.com/) - Cloud-native data warehouse with separation of storage and compute.\n- [Google BigQuery](https://cloud.google.com/bigquery) - Serverless, highly scalable data warehouse by Google.\n- [Amazon Redshift](https://aws.amazon.com/redshift/) - Fast, scalable data warehouse by AWS.\n- [Azure Synapse Analytics](https://azure.microsoft.com/en-us/products/synapse-analytics/) - Limitless analytics service by Microsoft.\n- [Databricks SQL](https://www.databricks.com/product/databricks-sql) - Serverless SQL analytics on the Lakehouse.\n- [Firebolt](https://www.firebolt.io/) - Cloud data warehouse built for high-performance analytics.\n- [ClickHouse Cloud](https://clickhouse.com/cloud) - Managed ClickHouse for real-time analytics.\n- [StarRocks](https://www.starrocks.io/) - High-performance analytical database.\n- [MotherDuck](https://motherduck.com/) - Serverless analytics powered by DuckDB.\n\n### Lakehouse Platforms\n\n- [Databricks](https://www.databricks.com/) - Unified analytics platform combining data lake and warehouse.\n- [Apache Iceberg](https://iceberg.apache.org/) - Open table format for huge analytic datasets.\n- [Delta Lake](https://delta.io/) - Open-source storage layer with ACID transactions on data lakes.\n- [Apache Hudi](https://hudi.apache.org/) - Data lake platform for incremental data processing.\n- [Dremio](https://www.dremio.com/) - Lakehouse platform with Apache Iceberg support.\n- [Onehouse](https://www.onehouse.ai/) - Managed lakehouse platform built on Apache Hudi.\n- [Tabular](https://tabular.io/) - Managed Apache Iceberg service by its creators.\n\n---\n\n## Data Lakes \u0026 Storage\n\n*Object storage and data lake solutions for storing raw and processed data.*\n\n### Object Storage\n\n- [Amazon S3](https://aws.amazon.com/s3/) - Industry-standard object storage by AWS.\n- [Google Cloud Storage](https://cloud.google.com/storage) - Object storage by Google Cloud.\n- [Azure Blob Storage](https://azure.microsoft.com/en-us/products/storage/blobs/) - Object storage by Microsoft Azure.\n- [MinIO](https://min.io/) - High-performance, S3-compatible object storage.\n- [Cloudflare R2](https://www.cloudflare.com/products/r2/) - S3-compatible storage with zero egress fees.\n\n### Data Lake Formats\n\n- [Apache Parquet](https://parquet.apache.org/) - Columnar storage file format.\n- [Apache ORC](https://orc.apache.org/) - Columnar storage format for Hadoop.\n- [Apache Avro](https://avro.apache.org/) - Row-based data serialization format.\n- [Lance](https://github.com/lancedb/lance) - Modern columnar format for ML datasets.\n\n---\n\n## Data Transformation\n\n*Tools for transforming, modeling, and preparing data for analysis.*\n\n### SQL-Based Transformation\n\n- [dbt (data build tool)](https://www.getdbt.com/) - Industry-standard SQL-first transformation tool. Version-controlled, tested, documented.\n- [SQLMesh](https://sqlmesh.com/) - Data transformation framework with built-in data quality and CI/CD.\n- [SDF (Semantic Data Fabric)](https://www.sdf.com/) - SQL compiler for data transformation with static analysis.\n- [Coalesce](https://coalesce.io/) - Data transformation platform purpose-built for Snowflake.\n- [Dataform](https://dataform.co/) - SQL-based data transformation (acquired by Google).\n\n### Python-Based Transformation\n\n- [Pandas](https://pandas.pydata.org/) - Data manipulation and analysis library for Python.\n- [Polars](https://www.pola.rs/) - Lightning-fast DataFrame library written in Rust.\n- [DuckDB](https://duckdb.org/) - In-process SQL OLAP database. Perfect for local data transformation.\n- [PySpark](https://spark.apache.org/docs/latest/api/python/) - Python API for Apache Spark.\n- [Vaex](https://vaex.io/) - Out-of-core DataFrames for large datasets.\n- [Modin](https://modin.readthedocs.io/) - Parallel pandas using Ray or Dask.\n- [Ibis](https://ibis-project.org/) - Python DataFrame API that compiles to SQL.\n- [Fugue](https://fugue-tutorials.readthedocs.io/) - Unified interface for distributed computing.\n- [Hamilton](https://github.com/dagworks-inc/hamilton) - Micro-framework for dataframe generation.\n\n### Distributed Computing\n\n- [Apache Spark](https://spark.apache.org/) - Unified analytics engine for large-scale data processing.\n- [Dask](https://www.dask.org/) - Flexible parallel computing library for analytics.\n- [Ray](https://www.ray.io/) - Unified framework for scaling AI and Python applications.\n- [Apache Flink](https://flink.apache.org/) - Stream and batch processing framework.\n- [Apache Beam](https://beam.apache.org/) - Unified programming model for batch and streaming.\n- [Asgarde](https://github.com/tosun-si/asgarde) - Java library for simplified error handling in Beam pipelines.\n\n---\n\n## Data Orchestration\n\n*Tools for scheduling, monitoring, and managing data pipelines and workflows.*\n\n### Open Source\n\n- [Apache Airflow](https://airflow.apache.org/) - Platform to programmatically author, schedule, and monitor workflows.\n- [Dagster](https://dagster.io/) - Cloud-native orchestration platform with software-defined assets.\n- [Prefect](https://www.prefect.io/) - Modern workflow orchestration with Python-native approach.\n- [Mage](https://www.mage.ai/) - Open-source data pipeline tool with notebook-style interface.\n- [Kestra](https://kestra.io/) - Event-driven orchestration platform with declarative YAML.\n- [Luigi](https://github.com/spotify/luigi) - Python module for building complex pipelines (by Spotify).\n- [Argo Workflows](https://argoproj.github.io/argo-workflows/) - Kubernetes-native workflow engine.\n\n### Commercial / Managed\n\n- [Astronomer](https://www.astronomer.io/) - Managed Airflow platform.\n- [Dagster Cloud](https://dagster.io/cloud) - Managed Dagster platform.\n- [Prefect Cloud](https://www.prefect.io/cloud) - Managed Prefect platform.\n- [Google Cloud Composer](https://cloud.google.com/composer) - Managed Apache Airflow by Google.\n- [Amazon MWAA](https://aws.amazon.com/managed-workflows-for-apache-airflow/) - Managed Airflow by AWS.\n- [Azure Data Factory](https://azure.microsoft.com/en-us/products/data-factory/) - Cloud-based data integration service.\n- [Orchestra](https://www.getorchestra.io/) - Unified data orchestration platform.\n\n---\n\n## Data Quality \u0026 Testing\n\n*Tools for ensuring data accuracy, completeness, and reliability.*\n\n### Open Source\n\n- [Great Expectations](https://greatexpectations.io/) - Python library for data validation and documentation.\n- [dbt Tests](https://docs.getdbt.com/docs/build/tests) - Built-in testing framework in dbt.\n- [Elementary](https://www.elementary-data.com/) - Open-source data observability for dbt.\n- [Provero](https://github.com/provero-org/provero) - Vendor-neutral, declarative data quality engine with YAML-based checks.\n\n\n### Commercial / Managed\n\n- [Soda](https://www.soda.io/) - Data quality platform with SodaCL language.\n- [Monte Carlo](https://www.montecarlodata.com/) - Data observability platform with ML-powered anomaly detection.\n\n---\n\n## Data Observability\n\n*Tools for monitoring, alerting, and understanding data pipeline health.*\n\n- [Datadog](https://www.datadoghq.com/) - Monitoring and security platform for developers.\n- [Elementary](https://www.elementary-data.com/) - Open-source dbt-native data observability.\n- [Datafold](https://www.datafold.com/) - Data reliability platform with data diff and regression testing.\n- [Sifflet](https://www.siffletdata.com/) - Full-stack data observability platform.\n\n---\n\n## Data Catalog \u0026 Discovery\n\n*Tools for discovering, understanding, and documenting data assets.*\n\n### Open Source\n\n- [DataHub](https://datahubproject.io/) - Open-source metadata platform by LinkedIn.\n- [Amundsen](https://www.amundsen.io/) - Open-source data discovery platform by Lyft.\n- [OpenMetadata](https://open-metadata.org/) - Open-source metadata platform with discovery and governance.\n\n\n### Commercial / Managed\n\n- [Atlan](https://atlan.com/) - Active metadata platform with collaboration features.\n- [Alation](https://www.alation.com/) - Enterprise data intelligence platform.\n- [Collibra](https://www.collibra.com/) - Data intelligence platform with governance and catalog.\n\n---\n\n## Data Governance\n\n*Tools for managing data policies, access, compliance, and security.*\n\n- [Collibra](https://www.collibra.com/) - Comprehensive data governance platform.\n- [Alation](https://www.alation.com/) - Data catalog with governance capabilities.\n\n---\n\n## Business Intelligence \u0026 Analytics\n\n*Tools for data visualization, reporting, and self-service analytics.*\n\n### Open Source\n\n- [Apache Superset](https://superset.apache.org/) - Modern data exploration and visualization platform.\n- [Metabase](https://www.metabase.com/) - Open-source business intelligence with SQL and visual query builder.\n- [Redash](https://redash.io/) - Connect and query data sources, build dashboards.\n- [Lightdash](https://www.lightdash.com/) - Open-source BI for dbt users.\n- [Evidence](https://evidence.dev/) - Code-based BI with Markdown and SQL.\n- [Grafana](https://grafana.com/) - Open-source analytics and monitoring platform.\n- [stratif.io](https://stratif.io) - Open-source warehouse-native product analytics for DuckDB, Postgres, Snowflake, and ClickHouse.\n\n### Commercial / Managed\n\n- [Looker](https://looker.com/) - Enterprise BI with semantic modeling (Google Cloud).\n- [Tableau](https://www.tableau.com/) - Visual analytics platform (Salesforce).\n- [Power BI](https://powerbi.microsoft.com/) - Business analytics by Microsoft.\n- [Qlik](https://www.qlik.com/) - Data analytics and business intelligence platform.\n- [Preset](https://preset.io/) - Managed Apache Superset.\n- [Omni](https://omni.co/) - Shared-model BI platform.\n- [Holistics](https://www.holistics.io/) - Self-service BI platform with data modeling.\n\n\n---\n\n## Metrics Layer \u0026 Semantic Layer\n\n*Tools for defining consistent business metrics and semantic models.*\n\n- [dbt Semantic Layer](https://www.getdbt.com/product/semantic-layer) - Define metrics in dbt, query from any tool.\n- [Cube](https://cube.dev/) - Headless BI and semantic layer with caching.\n- [MetricFlow](https://github.com/dbt-labs/metricflow) - Semantic layer engine (now part of dbt).\n- [Looker](https://looker.com/) - LookML semantic modeling language.\n- [Transform (Acquired by dbt Labs)](https://transform.co/) - Metrics store platform.\n- [Minerva (Airbnb)](https://medium.com/airbnb-engineering/how-airbnb-achieved-metric-consistency-at-scale-f23cc53dea70) - Airbnb's internal metrics platform.\n\n\n---\n\n## Reverse ETL\n\n*Tools for syncing data from warehouses back to operational tools.*\n\n- [Census](https://www.getcensus.com/) - Operational analytics platform for syncing data to business tools.\n- [Hightouch](https://hightouch.com/) - Data activation platform for reverse ETL.\n- [Polytomic](https://www.polytomic.com/) - Sync data bidirectionally between databases and SaaS.\n\n---\n\n## Data Contracts\n\n*Tools and frameworks for defining and enforcing data contracts between producers and consumers.*\n\n- [Soda Data Contracts](https://www.soda.io/data-contracts) - Define and verify data contracts.\n- [Bitol](https://bitol.io/) - Open-source data contract specification.\n- [DataContract CLI](https://github.com/datacontract/datacontract-cli) - CLI for managing data contracts.\n- [Great Expectations](https://greatexpectations.io/) - Can be used for contract-like validations.\n- [JSON Schema](https://json-schema.org/) - Schema specification for JSON data.\n- [Protobuf](https://protobuf.dev/) - Protocol Buffers for schema definition.\n- [Apache Avro](https://avro.apache.org/) - Data serialization with schema evolution.\n\n---\n\n## Streaming \u0026 Real-Time\n\n*Tools for real-time data processing and event streaming.*\n\n### Message Brokers \u0026 Streaming Platforms\n\n- [Apache Kafka](https://kafka.apache.org/) - Distributed event streaming platform.\n- [Confluent](https://www.confluent.io/) - Enterprise Kafka platform with managed cloud.\n- [Amazon Kinesis](https://aws.amazon.com/kinesis/) - Real-time streaming data service by AWS.\n- [Google Pub/Sub](https://cloud.google.com/pubsub) - Messaging service by Google Cloud.\n- [Azure Event Hubs](https://azure.microsoft.com/en-us/products/event-hubs/) - Big data streaming platform by Azure.\n- [RabbitMQ](https://www.rabbitmq.com/) - Open-source message broker.\n\n\n### Stream Processing\n\n- [Apache Flink](https://flink.apache.org/) - Stateful stream processing framework.\n- [Apache Kafka Streams](https://kafka.apache.org/documentation/streams/) - Client library for stream processing.\n- [ksqlDB](https://ksqldb.io/) - Database for stream processing on Kafka.\n- [Apache Spark Streaming](https://spark.apache.org/streaming/) - Spark module for stream processing.\n\n\n### Change Data Capture (CDC)\n\n- [Boomi Data Integration](https://boomi.com/platform/boomi-data-integration/) - Automated CDC for operational databases and apps.\n- [Debezium](https://debezium.io/) - Open-source distributed CDC platform.\n- [Airbyte CDC](https://airbyte.com/connectors?category=cdc) - CDC connectors in Airbyte.\n- [Fivetran CDC](https://www.fivetran.com/solutions/change-data-capture) - Managed CDC by Fivetran.\n\n---\n\n## Query Engines\n\n*Distributed SQL query engines for data lakes and federated queries.*\n\n- [PrestoDB](https://prestodb.io/) - Distributed SQL query engine by Meta.\n- [Apache Drill](https://drill.apache.org/) - Schema-free SQL query engine.\n- [Apache Impala](https://impala.apache.org/) - Massively parallel SQL query engine.\n- [DuckDB](https://duckdb.org/) - In-process analytical database.\n- [ClickHouse](https://clickhouse.com/) - Column-oriented DBMS for OLAP.\n- [Apache Druid](https://druid.apache.org/) - Real-time analytics database.\n\n---\n\n## Data Notebooks \u0026 Exploration\n\n*Interactive environments for data exploration and analysis.*\n\n### Open Source\n\n- [Jupyter](https://jupyter.org/) - Web-based interactive computing platform.\n- [JupyterLab](https://jupyterlab.readthedocs.io/) - Next-generation Jupyter interface.\n- [Zeppelin](https://zeppelin.apache.org/) - Web-based notebook for data analytics.\n- [Marimo](https://marimo.io/) - Reactive Python notebook with reproducibility.\n- [Streamlit](https://streamlit.io/) - Python framework for data apps.\n\n### Commercial / Managed\n\n- [Google Colab](https://colab.research.google.com/) - Free Jupyter notebooks by Google.\n- [Amazon SageMaker Studio](https://aws.amazon.com/sagemaker/studio/) - ML IDE by AWS.\n- [Databricks Notebooks](https://www.databricks.com/) - Collaborative notebooks on Databricks.\n- [Hex](https://hex.tech/) - Collaborative data workspace with notebooks and apps.\n- [Deepnote](https://deepnote.com/) - Collaborative data notebook for teams.\n\n---\n\n## Feature Stores\n\n*Platforms for managing, storing, and serving ML features.*\n\n- [Vertex AI Feature Store](https://cloud.google.com/vertex-ai/docs/featurestore) - Google Cloud feature store.\n- [Amazon SageMaker Feature Store](https://aws.amazon.com/sagemaker/feature-store/) - AWS managed feature store.\n- [Databricks Feature Store](https://www.databricks.com/product/feature-store) - Feature store in Databricks.\n- [Feast](https://feast.dev/) - Open-source feature store for ML.\n- [Hopsworks](https://www.hopsworks.ai/) - Platform with feature store and MLOps.\n\n\n\n\n\n---\n\n## ML Platforms \u0026 MLOps\n\n*End-to-end platforms for building, deploying, and managing ML models.*\n\n### Open Source\n\n- [MLflow](https://mlflow.org/) - Open-source platform for ML lifecycle management.\n- [Kubeflow](https://www.kubeflow.org/) - ML toolkit for Kubernetes.\n- [Metaflow](https://metaflow.org/) - Framework for real-life data science (by Netflix).\n- [ZenML](https://zenml.io/) - MLOps framework for reproducible pipelines.\n\n\n### Commercial / Managed\n\n- [Google Vertex AI](https://cloud.google.com/vertex-ai) - Unified ML platform by Google Cloud.\n- [Databricks MLflow](https://www.databricks.com/product/managed-mlflow) - Managed MLflow on Databricks.\n- [Amazon SageMaker](https://aws.amazon.com/sagemaker/) - Fully managed ML service by AWS.\n- [Azure Machine Learning](https://azure.microsoft.com/en-us/products/machine-learning/) - ML platform by Microsoft.\n- [Weights \u0026 Biases](https://wandb.ai/) - ML experiment tracking and model management.\n\n---\n\n## Vector Databases\n\n*Databases optimized for storing and querying vector embeddings for AI/ML applications.*\n\n- [MongoDB](https://www.mongodb.com/docs/atlas/atlas-vector-search/vector-search-overview/) - Managed vector store alognside your operational data in MongoDB Atlas.\n- [Pinecone](https://www.pinecone.io/) - Managed vector database for similarity search.\n- [Weaviate](https://weaviate.io/) - Open-source vector search engine.\n- [Qdrant](https://qdrant.tech/) - Open-source vector similarity search engine.\n- [Chroma](https://www.trychroma.com/) - Open-source embedding database.\n- [pgvector](https://github.com/pgvector/pgvector) - Vector similarity search for PostgreSQL.\n- [Elasticsearch](https://www.elastic.co/) - Search engine with vector search capabilities.\n\n\n\n---\n\n## Data Sharing\n\n*Platforms for securely sharing data across organizations.*\n\n- [Google Analytics Hub](https://cloud.google.com/analytics-hub) - Data exchange by Google Cloud.\n- [Snowflake Data Marketplace](https://www.snowflake.com/data-marketplace/) - Data sharing and marketplace by Snowflake.\n- [Databricks Delta Sharing](https://www.databricks.com/product/delta-sharing) - Open protocol for secure data sharing.\n- [AWS Data Exchange](https://aws.amazon.com/data-exchange/) - Data marketplace by AWS.\n\n\n---\n\n## DataOps \u0026 Version Control\n\n*Tools for DataOps practices, version control, and CI/CD for data.*\n\n- [dbt](https://www.getdbt.com/) - Version-controlled data transformation.\n- [DVC](https://dvc.org/) - Version control for data and ML models.\n\n---\n\n## Learning Resources\n\n### Books\n\n- [Fundamentals of Data Engineering](https://www.oreilly.com/library/view/fundamentals-of-data/9781098108298/) - by Joe Reis \u0026 Matt Housley\n- [The Data Warehouse Toolkit](https://www.kimballgroup.com/data-warehouse-business-intelligence-resources/books/) - by Ralph Kimball\n- [Designing Data-Intensive Applications](https://dataintensive.net/) - by Martin Kleppmann\n- [Data Mesh](https://www.oreilly.com/library/view/data-mesh/9781492092384/) - by Zhamak Dehghani\n- [Building a Scalable Data Warehouse with Data Vault 2.0](https://www.elsevier.com/books/building-a-scalable-data-warehouse-with-data-vault-20/linstedt/978-0-12-802510-9) - by Dan Linstedt\n\n### Courses \u0026 Certifications\n\n- [dbt Learn](https://courses.getdbt.com/) - Free dbt fundamentals course.\n- [DataCamp](https://www.datacamp.com/) - Data science and engineering courses.\n- [DataTalks.Club](https://datatalks.club/) - Free data engineering zoomcamp.\n- [Coursera Data Engineering](https://www.coursera.org/professional-certificates/google-cloud-data-engineering) - Google Cloud Data Engineering certificate.\n- [Snowflake Training](https://www.snowflake.com/education-and-training/) - Snowflake certifications.\n- [Databricks Training](https://www.databricks.com/learn/training) - Databricks certifications.\n\n### Blogs \u0026 Newsletters\n\n\n- [Blef.fr](https://www.blef.fr/) - The hub to explore Data News links.\n- [Joe Reis's Substack](https://joereis.substack.com/) - Insights on data engineering.\n- [dbt Blog](https://www.getdbt.com/blog/) - Articles on analytics engineering.\n- [Airbyte Blog](https://airbyte.com/blog) - Data integration and engineering content.\n- [DataStackGuide](https://datastackguide.com) - Independent reviews of B2B data stack tools (CRMs, enrichment, BI, sales engagement) backed by analysis of 23,000+ job postings.\n\n### Podcasts\n\n- [DataGen](https://open.spotify.com/show/27XP61URSuKu9oeWR793D6) - Interviews with French data practitioners \u0026 leaders.\n- [The Data Engineering Podcast](https://www.dataengineeringpodcast.com/) - Interviews with data engineering practitioners.\n- [Data Engineering Show](https://www.dataengineeringshow.com/) - Discussions on data engineering topics.\n\n\n---\n\n## Community\n\n### Slack Communities\n\n- [dbt Community Slack](https://www.getdbt.com/community/) - 50k+ members discussing analytics engineering.\n- [Airbyte Slack](https://airbyte.com/community) - Data integration community.\n- [Apache Airflow Slack](https://apache-airflow.slack.com/) - Workflow orchestration.\n\n### Conferences\n\n\n- [Coalesce](https://coalesce.getdbt.com/) - dbt's annual conference.\n- [Snowflake Summit](https://www.snowflake.com/summit/) - Snowflake user conference.\n\n\u003e Comming soon...\n### Community Resources\n\n- [r/dataengineering](https://www.reddit.com/r/dataengineering/) - Reddit community.\n- [Data Engineering Wiki](https://dataengineering.wiki/) - Community-maintained wiki.\n- [Awesome Data Engineering](https://github.com/igorbarinov/awesome-data-engineering) - Another curated list.\n- [Modern Data Stack Glossary](https://www.secoda.co/glossary) - Data terminology.\n\n---\n\n## Contributing\n\nContributions are welcome! Please read the [contribution guidelines](CONTRIBUTING.md) first.\n\n## License\n\n[![CC0](https://mirrors.creativecommons.org/presskit/buttons/88x31/svg/cc-zero.svg)](https://creativecommons.org/publicdomain/zero/1.0/)\n\nTo the extent possible under law, the contributors have waived all copyright and related or neighboring rights to this work.\n\n---\n\n## Acknowledgments\n\nThis list is maintained by the community. Special thanks to all contributors who help keep it up-to-date.\n\nIf you find this resource helpful, please give it a star and share it with others!\n","projects_url":"https://awesome.ecosyste.ms/api/v1/lists/bricefotzo%2Fawesome-modern-data-stack/projects"}