Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

data-engineering-collection

A collection of awesome software, libraries, Learning Tutorials, documents, books, resources and interesting stuff about Big Data Science & Engineering
https://github.com/exajobs/data-engineering-collection

Last synced: about 12 hours ago
JSON representation

  • `RDBMS `

  • `Distributed Programming `

    • Apache APEX - a unified, enterprise platform for big data stream and batch processing.
    • Apache Beam - an unified model and set of language-specific SDKs for defining and executing data processing workflows.
    • Apache Gearpump - real-time big data streaming engine based on Akka.
    • Apache MapReduce - programming model for processing large data sets with a parallel, distributed algorithm on a cluster.
    • Apache Pig - high level language to express data analysis programs for Hadoop.
    • Apache Spark Streaming - framework for stream processing, part of Spark.
    • Apache Twill - abstraction over YARN that reduces the complexity of developing distributed applications.
    • Baidu Bigflow - an interface that allows for writing distributed computing programs providing lots of simple, flexible, powerful APIs to easily handle data of any scale.
    • Cascalog - data processing and querying library.
    • Cheetah - High Performance, Custom Data Warehouse on Top of MapReduce.
    • DataTorrent StrAM - real-time engine is designed to enable distributed, asynchronous, real time in-memory big-data computations in as unblocked a way as possible, with minimal overhead and impact on performance.
    • Google MapReduce - map reduce framework.
    • Google MillWheel - fault tolerant stream processing framework.
    • Kite - is a set of libraries, tools, examples, and documentation focused on making it easier to build systems on top of the Hadoop ecosystem.
    • Nokia Disco - MapReduce framework developed by Nokia.
    • Pinterest Pinlater - asynchronous job execution system.
    • Rackerlabs Blueflood - multi-tenant distributed metric processing system
    • Stratosphere - general purpose cluster computing framework.
    • Streamdrill - useful for counting activities of event streams over different time windows and finding the most active one.
    • Wallaroo - The ultrafast and elastic data processing engine. Big or fast data - no fuss, no Java needed.
    • Google Dataflow - create data pipelines to help themæingest, transform and analyze data.
    • Twitter TSAR - TimeSeries AggregatoR by Twitter.
    • Facebook Peregrine - Map Reduce framework.

  • `Distributed Filesystem `

  • `Document Data Model `

    • MongoDB - Document-oriented database system.
    • MarkLogic - Schema-agnostic Enterprise NoSQL database technology.

  • `Key Map Data Model `

  • Key-value Data Model

    • Badger - a fast, simple, efficient, and persistent key-value store written natively in Go.
    • Ignite - is an in-memory key-value data store providing full SQL-compliant data access that can optionally be backed by disk storage.
    • LinkedIn Krati - is a simple persistent data store with very low latency and high throughput.
    • Linkedin Voldemort - distributed key/value storage system.

  • Graph Data Model

    • GraphX - resilient Distributed Graph System on Spark.
    • MapGraph - Massively Parallel Graph processing on GPUs.
    • NodeXL - A free, open-source template for Microsoft® Excel® 2007, 2010, 2013 and 2016 that makes it easy to explore network graphs.
    • Titan - distributed graph database, built over Cassandra.
    • Google Pregel - graph processing framework.
    • AgensGraph - a new generation multi-model graph database for the modern complex data environment.

  • Databases

  • `Columnar Databases`

    • MonetDB - column store database.
    • Vertica - is designed to manage large, fast-growing volumes of data and provide very fast query performance when used for data warehouses.
    • EventQL - a distributed, column-oriented database built for large-scale event collection and analytics.
    • Amazon Redshift - Amazon's cloud offering, also based on a columnar datastore backend.

  • `NewSQL Databases`

    • CitusDB - scales out PostgreSQL through sharding and replication.
    • Google F1 - distributed SQL database built on Spanner.
    • Google Spanner - globally distributed semi-relational database.
    • InfiniSQL - infinity scalable RDBMS.
    • Map-D - GPU in-memory database, big data analysis and visualization platform.
    • Pivotal GemFire XD - Low-latency, in-memory, distributed SQL data store. Provides SQL interface to in-memory table data, persistable in HDFS.
    • SAP HANA - is an in-memory, column-oriented, relational database management system.
    • Sky - database used for flexible, high performance analysis of behavioral data.
    • Sky - database used for flexible, high performance analysis of behavioral data.
    • FoundationDB - distributed database, inspired by F1.

  • Time-Series Databases

    • Axibase Time Series Database - Integrated time series database on top of HBase with built-in visualization, rule-engine and SQL support.
    • InfluxDB - a time series database with optimised IO and queries, supports pgsql and influx wire protocols.
    • M3DB - a distributed time series database that can be used for storing realtime metrics at long retention.
    • Prometheus - a time series database and service monitoring system.
    • Rhombus - series object store for Cassandra that handles all the complexity of building wide row indexes.
    • QuestDB - high-performance, open-source SQL database for applications in financial services, IoT, machine learning, DevOps and observability.

  • SQL-like processing

    • Actian SQL for Hadoop - high performance interactive SQL access to all Hadoop data.
    • Apache HCatalog - table and storage management layer for Hadoop.
    • Aster Database - SQL-like analytic processing for MapReduce.
    • Dremio - an open-source, SQL-like Data-as-a-Service Platform based on Apache Arrow.
    • Facebook PrestoDB - distributed SQL query engine.
    • Google BigQuery - framework for interactive analysis, implementation of Dremel.
    • Spark Catalyst - is a Query Optimization Framework for Spark and Shark.
    • Splice Machine - a full-featured SQL-on-Hadoop RDBMS with ACID transactions.
    • Trafodion - enterprise-class SQL-on-HBase solution targeting big data transactional or operational workloads.
    • Cloudera Impala - framework for interactive analysis, Inspired by Dremel.
    • Datasalt Splout SQL - full SQL query engine for big datasets.

  • Data Ingestion

    • Amazon Kinesis - real-time processing of streaming data at massive scale.
    • Amazon Web Services Glue - serverless fully managed extract, transform, and load (ETL) service
    • Apache NiFi - Apache NiFi is an integrated data logistics platform for automating the movement of data between disparate systems.
    • Google Photon - geographically distributed system for joining multiple continuously flowing streams of data in real-time with high scalability and low latency.
    • Kestrel - distributed message queue system.
    • StreamSets Data Collector - continuous big data ingest infrastructure with a simple to use IDE.
    • Alooma - data pipeline as a service enabling moving data sources such as MySQL into data warehouses.
    • redpanda - A Kafka® replacement for mission critical systems; 10x faster. Written in C++.

  • Service Programming

    • Google Chubby - a lock service for loosely-coupled distributed systems.
    • OpenMPI - message passing framework.
    • Serf - decentralized solution for service discovery and orchestration.

  • Scheduling

  • Machine Learning

    • Azure ML Studio - Cloud-based AzureML, R, Python Machine Learning platform
    • DataVec - A vectorization and data preprocessing library for deep learning in Java and Scala. Part of the Deeplearning4j ecosystem.
    • Deeplearning4j - Fast, open deep learning for the JVM (Java, Scala, Clojure). A neural network configuration layer powered by a C++ library. Uses Spark and Hadoop to train nets on multiple GPUs and CPUs.
    • etcML - text classification with machine learning.
    • GraphLab Create - A machine learning platform in Python with a broad collection of ML toolkits, data engineering, and deployment tools.
    • MLbase - distributed machine learning libraries for the BDAS stack.
    • MonkeyLearn - Text mining made easy. Extract and classify data from text.
    • ND4J - A matrix library for the JVM. Numpy for Java.
    • RL4J - Reinforcement learning for Java and Scala. Includes Deep-Q learning and A3C algorithms, and integrates with Open AI's Gym. Runs in the Deeplearning4j ecosystem.
    • Sibyl - System for Large Scale Machine Learning at Google.
    • Theano - A Python-focused machine learning library supported by the University of Montreal.
    • Torch - A deep learning library with a Lua API, supported by NYU and Facebook.
    • WEKA - suite of machine learning software.
    • WEKA - suite of machine learning software.

  • Benchmarking

  • System Deployment

    • Brooklyn - library that simplifies application deployment and management.
    • Buildoop - Similar to Apache BigTop based on Groovy language.
    • Google Borg - job scheduling and monitoring system.
    • Google Omega - job scheduling and monitoring system.
    • Kubernetes - a system for automating deployment, scaling, and management of containerized applications.
    • Apache YARN - Cluster manager.

  • Applications

    • Apache Tika - content analysis toolkit.
    • Qubole - auto-scaling Hadoop cluster, built-in data connectors.
    • Hunk - Splunk analytics for Hadoop.
    • Imhotep - Large scale analytics platform by indeed.
    • Indicative - Web & mobile analytics tool, with data warehouse (AWS, BigQuery) integration.
    • Jupyter - Notebook and project application for interactive data science and scientific computing across all programming languages.
    • MADlib - data-processing library of an RDBMS to analyze data.
    • Splunk - analyzer for machine-generated data.
    • Sumo Logic - cloud based analyzer for machine-generated data.
    • Talend - unified open source environment for YARN, Hadoop, HBASE, Hive, HCatalog & Pig.
    • PivotalR - R on Pivotal HD / HAWQ and PostgreSQL.

  • MySQL forks and evolutions

    • Drizzle - evolution of MySQL 6.0.
    • Google Cloud SQL - MySQL databases in Google's cloud.
    • MySQL Cluster - MySQL implementation using NDB Cluster storage engine.
    • TokuDB - TokuDB is a storage engine for MySQL and MariaDB.
    • WebScaleSQL - is a collaboration among engineers from several companies that face similar challenges in running MySQL at scale.
    • Amazon RDS - MySQL databases in Amazon's cloud.
    • MariaDB - enhanced, drop-in replacement for MySQL.

  • PostgreSQL forks and evolutions

    • RecDB - Open Source Recommendation Engine Built Entirely Inside PostgreSQL.
    • Stado - open source MPP database system solely targeted at data warehousing and data mart applications.
    • Yahoo Everest - multi-peta-byte database / MPP derived by PostgreSQL.
    • Postgres-XL - Scalable Open Source PostgreSQL-based Database Cluster.

  • Embedded Databases

    • LMDB - ultra-fast, ultra-compact key-value embedded data store developed by Symas.
    • Actian PSQL - ACID-compliant DBMS developed by Pervasive Software, optimized for embedding in applications.

  • Business Intelligence

    • BIME Analytics - business intelligence platform in the cloud.
    • GoodData - platform for data products and embedded analytics.
    • Jaspersoft - powerful business intelligence suite.
    • Jedox Palo - customisable Business Intelligence platform.
    • Jethrodata - Interactive Big Data Analytics.
    • Microstrategy - software platforms for business intelligence, mobile intelligence, and network applications.
    • Qlik - business intelligence and analytics platform.
    • Redash - Open source business intelligence platform, supporting multiple data sources and planned queries.
    • Saiku Analytics - Open source analytics platform.
    • Knowage - open source business intelligence platform. (former [SpagoBi](http://www.spagobi.org/))
    • SparklineData SNAP - modern B.I platform powered by Apache Spark.
    • Tableau - business intelligence platform.
    • Dekart - Large scale geospatial analytics for Google BigQuery based on Kepler.gl.

  • Data Visualization

    • chartd - responsive, retina-compatible charts with just an img tag.
    • D3 - javaScript library for manipulating documents.
    • DevExtreme React Chart - High-performance plugin-based React chart for Bootstrap and Material Design.
    • FnordMetric - write SQL queries that return SVG charts rather than tables
    • Grafana - graphite dashboard frontend, editor and graph composer.
    • Graphite - scalable Realtime Graphing.
    • Highcharts - simple and flexible charting API.
    • Metricsgraphic.js - a library built on top of D3 that is optimized for time-series data
    • Zing Charts - JavaScript charting library for big data.
    • Lumify - open source big data analysis and visualization platform

  • Internet of things and sensor data

    • Apache Edgent (Incubating) - a programming model and micro-kernel style runtime that can be embedded in gateways and small footprint edge devices enabling local, real-time, analytics on the edge devices.
    • TempoIQ - Cloud-based sensor analytics.
    • Pubnub - Data stream network
    • IFTTT - If this then that
    • Evrything - Making products smart
    • Ably - Pub/sub messaging platform for IoT

  • Interesting Readings

  • Interesting Papers

    • 2015 - 2016

      • 2015 - **Facebook** - One Trillion Edges: Graph Processing at Facebook-Scale.

    • 2013 - 2014

      • 2014 - **Stanford** - Mining of Massive Datasets.
      • 2013 - **AMPLab** - Presto: Distributed Machine Learning and Graph Processing with Sparse Matrices.
      • 2013 - **AMPLab** - MLbase: A Distributed Machine-learning System.
      • 2013 - **AMPLab** - Shark: SQL and Rich Analytics at Scale.
      • 2013 - **AMPLab** - GraphX: A Resilient Distributed Graph System on Spark.
      • 2013 - **Google** - HyperLogLog in Practice: Algorithmic Engineering of a State of The Art Cardinality Estimation Algorithm.
      • 2013 - **Metamarkets** - Druid: A Real-time Analytical Data Store.
      • 2013 - **Google** - F1: A Distributed SQL Database That Scales.
      • 2013 - **Facebook** - Scaling Memcache at Facebook.

    • 2011 - 2012

      • 2012 - **Twitter** - The Unified Logging Infrastructure
      • 2012 - **AMPLab** - Fast and Interactive Analytics over Hadoop Data with Spark.
      • 2012 - **AMPLab** - Shark: Fast Data Analysis Using Coarse-grained Distributed Memory.
      • 2012 - **Microsoft** - Paxos Replicated State Machines as the Basis of a High-Performance Data Store.
      • 2012 - **AMPLab** - BlinkDB: Queries with Bounded Errors and Bounded Response Times on Very Large Data.
      • 2012 - **Google** - Processing a trillion cells per mouse click.
      • 2011 - **AMPLab** - Scarlett: Coping with Skewed Popularity Content in MapReduce Clusters.
      • 2011 - **AMPLab** - Mesos: A Platform for Fine-Grained Resource Sharing in the Data Center.
      • 2011 - **Google** - Megastore: Providing Scalable, Highly Available Storage for Interactive Services.
      • 2012 - **AMPLab** - Blink and It’s Done: Interactive Queries on Very Large Data.
      • 2012 - **Google** - Spanner: Google’s Globally-Distributed Database.
      • 2012 - **AMPLab** - BlinkDB: Queries with Bounded Errors and Bounded Response Times on Very Large Data.

    • 2001 - 2010

      • 2010 - **AMPLab** - Spark: Cluster Computing with Working Sets.
      • 2010 - **Google** - Dremel: Interactive Analysis of Web-Scale Datasets.
      • 2010 - **Yahoo** - S4: Distributed Stream Computing Platform.
      • 2009 - HadoopDB: An Architectural Hybrid of MapReduce and DBMS Technologies for Analytical Workloads.
      • 2006 - **Google** - The Chubby lock service for loosely-coupled distributed systems.
      • 2004 - **Google** - MapReduce: Simplied Data Processing on Large Clusters.
      • 2010 - **Facebook** - Finding a needle in Haystack: Facebook’s photo storage.
      • 2010 - **Google** - Large-scale Incremental Processing Using Distributed Transactions and Notifications base of Percolator and Caffeine.

  • Videos

  • Books

  • `Frameworks `

    • Bistro - general-purpose data processing engine for both batch and stream analytics. It is based on a novel data model, which represents data via *functions* and processes data via *column operations* as opposed to having only set operations in conventional approaches like MapReduce or SQL.
Programming Languages
Java 1 Ruby 1
Categories
Interesting Papers 30 `Distributed Programming ` 23 Books 21 Machine Learning 14 Business Intelligence 13 Databases 13 SQL-like processing 11 Applications 11 Data Visualization 10 `NewSQL Databases` 10 Data Ingestion 8 `Distributed Filesystem ` 7 MySQL forks and evolutions 7 Graph Data Model 6 System Deployment 6 Time-Series Databases 6 Internet of things and sensor data 6 Search engine and framework 6 Benchmarking 5 `Key Map Data Model ` 4 PostgreSQL forks and evolutions 4 `Columnar Databases` 4 Key-value Data Model 4 Videos 4 `RDBMS ` 4 Service Programming 3 `Document Data Model ` 2 License 2 Embedded Databases 2 Interesting Readings 2 `Frameworks ` 1 Scheduling 1
Sub Categories
2001 - 2010 26 2011 - 2012 12 2013 - 2014 9 Data Visualization 6 Contributing 3 2015 - 2016 1
Keywords
opendata 1 datasets 1 awesome-public-datasets 1 aaron-swartz 1