Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

data-engineering-collection

A collection of awesome software, libraries, Learning Tutorials, documents, books, resources and interesting stuff about Big Data Science & Engineering
https://github.com/exajobs/data-engineering-collection

Last synced: about 4 hours ago
JSON representation

  • Key-value Data Model

    • Badger - a fast, simple, efficient, and persistent key-value store written natively in Go.
    • Ignite - is an in-memory key-value data store providing full SQL-compliant data access that can optionally be backed by disk storage.
    • LinkedIn Krati - is a simple persistent data store with very low latency and high throughput.
    • Linkedin Voldemort - distributed key/value storage system.
    • Bolt - an embedded key-value database for Go.
    • BTDB - Key Value Database in .Net with Object DB Layer, RPC, dynamic IL and much more
    • BuntDB - a fast, embeddable, in-memory key/value database for Go with custom indexing and geospatial support.
    • Edis - is a protocol-compatible Server replacement for Redis.
    • ElephantDB - Distributed database specialized in exporting data from Hadoop.
    • GhostDB - a distributed, in-memory, general purpose key-value data store that delivers microsecond performance at any scale.
    • Graviton - a simple, fast, versioned, authenticated, embeddable key-value store database in pure Go(lang).
    • HyperDex - a scalable, next generation key-value and document store with a wide array of features, including consistency, fault tolerance and high performance.
    • Linkedin Voldemort - distributed key/value storage system.
    • Oracle NoSQL Database - distributed key-value database by Oracle Corporation.
    • Riak - a decentralized datastore.
    • Storehaus - library to work with asynchronous key value stores, by Twitter.
    • SummitDB - an in-memory, NoSQL key/value database, with disk persistance and using the Raft consensus algorithm.
    • Tarantool - an efficient NoSQL database and a Lua application server.
    • TiKV - a distributed key-value database powered by Rust and inspired by Google Spanner and HBase.
    • Tile38 - a geolocation data store, spatial index, and realtime geofence, supporting a variety of object types including latitude/longitude points, bounding boxes, XYZ tiles, Geohashes, and GeoJSON
    • TreodeDB - key-value store that's replicated and sharded and provides atomic multirow writes.
    • Badger - a fast, simple, efficient, and persistent key-value store written natively in Go.
    • EventStore - distributed time series database.
    • GridDB - suitable for sensor data stored in a timeseries.

  • Machine Learning

    • Azure ML Studio - Cloud-based AzureML, R, Python Machine Learning platform
    • DataVec - A vectorization and data preprocessing library for deep learning in Java and Scala. Part of the Deeplearning4j ecosystem.
    • Deeplearning4j - Fast, open deep learning for the JVM (Java, Scala, Clojure). A neural network configuration layer powered by a C++ library. Uses Spark and Hadoop to train nets on multiple GPUs and CPUs.
    • etcML - text classification with machine learning.
    • GraphLab Create - A machine learning platform in Python with a broad collection of ML toolkits, data engineering, and deployment tools.
    • MLbase - distributed machine learning libraries for the BDAS stack.
    • MonkeyLearn - Text mining made easy. Extract and classify data from text.
    • ND4J - A matrix library for the JVM. Numpy for Java.
    • RL4J - Reinforcement learning for Java and Scala. Includes Deep-Q learning and A3C algorithms, and integrates with Open AI's Gym. Runs in the Deeplearning4j ecosystem.
    • Sibyl - System for Large Scale Machine Learning at Google.
    • Theano - A Python-focused machine learning library supported by the University of Montreal.
    • Torch - A deep learning library with a Lua API, supported by NYU and Facebook.
    • WEKA - suite of machine learning software.
    • brain - Neural networks in JavaScript.
    • Oryx - Lambda architecture on Apache Spark, Apache Kafka for real-time large scale machine learning.
    • Concurrent Pattern - machine learning library for Cascading.
    • convnetjs - Deep Learning in Javascript. Train Convolutional Neural Networks (or ordinary ones) in your browser.
    • Decider - Flexible and Extensible Machine Learning in Ruby.
    • Etsy Conjecture - scalable Machine Learning in Scalding.
    • Karate Club - An unsupervised machine learning library for graph structured data. Python
    • Lambdo - Lambdo is a workflow engine which significantly simplifies the analysis process by unifying feature engineering and machine learning operations.
    • Little Ball of Fur - A subsampling library for graph structured data. Python
    • MLPNeuralNet - Fast multilayer perceptron neural network library for iOS and Mac OS X.
    • ML Workspace - All-in-one web-based IDE specialized for machine learning and data science.
    • PyTorch Geometric Temporal - a temporal extension library for PyTorch Geometric .
    • scikit-learn - scikit-learn: machine learning in Python.
    • Shapley - A data-driven framework to quantify the value of classifiers in a machine learning ensemble.
    • TensorFlow - Library from Google for machine learning using data flow graphs.
    • Velox - System for serving machine learning predictions.
    • BidMach - CPU and GPU-accelerated Machine Learning Library.
    • H2O - statistical, machine learning and math runtime with Hadoop. R and Python.
    • WEKA - suite of machine learning software.
    • Vowpal Wabbit - learning system sponsored by Microsoft and Yahoo!.
    • Keras - An intuitive neural net API inspired by Torch that runs atop Theano and Tensorflow.
    • nupic - Numenta Platform for Intelligent Computing: a brain-inspired machine intelligence platform, and biologically accurate neural network based on cortical learning algorithms.
    • Sibyl - System for Large Scale Machine Learning at Google.
    • MonkeyLearn - Text mining made easy. Extract and classify data from text.
    • ENCOG - machine learning framework that supports a variety of advanced algorithms, as well as support classes to normalize and process data.
    • Feast - A feature store for the management, discovery, and access of machine learning features. Feast provides a consistent view of feature data for both model training and model serving.
    • MOA - MOA performs big data stream mining in real time, and large scale machine learning.
    • PredictionIO - machine learning server buit on Hadoop, Mahout and Cascading.
    • SAMOA - distributed streaming machine learning framework.

  • Memcached forks and evolutions

  • MySQL forks and evolutions

    • MariaDB - enhanced, drop-in replacement for MySQL.
    • Amazon RDS - MySQL databases in Amazon's cloud.
    • Drizzle - evolution of MySQL 6.0.
    • Google Cloud SQL - MySQL databases in Google's cloud.
    • MySQL Cluster - MySQL implementation using NDB Cluster storage engine.
    • TokuDB - TokuDB is a storage engine for MySQL and MariaDB.
    • WebScaleSQL - is a collaboration among engineers from several companies that face similar challenges in running MySQL at scale.
    • Drizzle - evolution of MySQL 6.0.
    • Percona Server - enhanced, drop-in replacement for MySQL.
    • ProxySQL - High Performance Proxy for MySQL.
    • WebScaleSQL - is a collaboration among engineers from several companies that face similar challenges in running MySQL at scale.
    • Google Cloud SQL - MySQL databases in Google's cloud.
    • Percona Server - enhanced, drop-in replacement for MySQL.

  • `NewSQL Databases`

    • CitusDB - scales out PostgreSQL through sharding and replication.
    • FoundationDB - distributed database, inspired by F1.
    • Google F1 - distributed SQL database built on Spanner.
    • Google Spanner - globally distributed semi-relational database.
    • InfiniSQL - infinity scalable RDBMS.
    • Map-D - GPU in-memory database, big data analysis and visualization platform.
    • Pivotal GemFire XD - Low-latency, in-memory, distributed SQL data store. Provides SQL interface to in-memory table data, persistable in HDFS.
    • SAP HANA - is an in-memory, column-oriented, relational database management system.
    • Sky - database used for flexible, high performance analysis of behavioral data.
    • Sky - database used for flexible, high performance analysis of behavioral data.
    • ActorDB - a distributed SQL database with the scalability of a KV store, while keeping the query capabilities of a relational database.
    • BayesDB - statistic oriented SQL database.
    • Cockroach - Scalable, Geo-Replicated, Transactional Datastore.
    • Comdb2 - a clustered RDBMS built on optimistic concurrency control techniques.
    • Haeinsa - linearly scalable multi-row, multi-table transaction library for HBase based on Percolator.
    • InfiniSQL - infinity scalable RDBMS.
    • KarelDB - a relational database backed by Apache Kafka.
    • SenseiDB - distributed, realtime, semi-structured database.
    • yugabyteDB - open source, high-performance, distributed SQL database compatible with PostgreSQL.
    • SymmetricDS - open source software for both file and database synchronization.
    • Actian Ingres - commercially supported, open-source SQL relational database management system.
    • NuoDB - SQL/ACID compliant distributed database.
    • H-Store - is an experimental main-memory, parallel database management system that is optimized for on-line transaction processing (OLTP) applications.
    • Oracle TimesTen in-Memory Database - in-memory, relational database management system with persistence and recoverability.
    • Bedrock - a simple, modular, networked and distributed transaction layer built atop SQLite.
    • Datomic - distributed database designed to enable scalable, flexible and intelligent applications.
    • Google F1 - distributed SQL database built on Spanner.
    • HandlerSocket - NoSQL plugin for MySQL/MariaDB.

  • PostgreSQL forks and evolutions

    • RecDB - Open Source Recommendation Engine Built Entirely Inside PostgreSQL.
    • Stado - open source MPP database system solely targeted at data warehousing and data mart applications.
    • Yahoo Everest - multi-peta-byte database / MPP derived by PostgreSQL.
    • Postgres-XL - Scalable Open Source PostgreSQL-based Database Cluster.
    • HadoopDB - hybrid of MapReduce and DBMS.
    • IBM Netezza - high-performance data warehouse appliances.
    • TimescaleDB - An open-source time-series database optimized for fast ingest and complex queries
    • Stado - open source MPP database system solely targeted at data warehousing and data mart applications.

  • `RDBMS `

  • Scheduling

    • Linkedin Azkaban - batch workflow job scheduler.
    • Apache Airflow - a platform to programmatically author, schedule and monitor workflows.
    • Cronicle - Distributed, easy to install, NodeJS based, task scheduler
    • Dagster - a data orchestrator for machine learning, analytics, and ETL.
    • Schedoscope - Scala DSL for agile scheduling of Hadoop jobs.
    • Sparrow - scheduling platform.
    • Azure Data Factory - cloud-based pipeline orchestration for on-prem, cloud and HDInsight
    • Apache Aurora - is a service scheduler that runs on top of Apache Mesos.
    • Apache Falcon - data management framework.
    • Chronos - distributed and fault-tolerant scheduler.
    • ElasticSearch - Search and analytics engine based on Apache Lucene.
    • Enigma.io
    • Lily HBase Indexer - quickly and easily search for any content stored in HBase.
    • LinkedIn Galene - search architecture at LinkedIn.
    • Sphinx Search Server - fulltext search engine.
    • Elassandra - is a fork of Elasticsearch modified to run on top of Apache Cassandra in a scalable and resilient peer-to-peer architecture.
    • LinkedIn Bobo - is a Faceted Search implementation written purely in Java, an extension to Apache Lucene.
    • LinkedIn Zoie - is a realtime search/indexing system written in Java.
    • MG4J - MG4J (Managing Gigabytes for Java) is a full-text search engine for large document collections written in Java. It is highly customisable, high-performance and provides state-of-the-art features and new research algorithms.
    • Sphinx Search Server - fulltext search engine.
    • Facebook Faiss - is a library for efficient similarity search and clustering of dense vectors. It contains algorithms that search in sets of vectors of any size, up to ones that possibly do not fit in RAM. It also contains supporting code for evaluation and parameter tuning. Faiss is written in C++ with complete wrappers for Python/numpy.
    • Annoy - is a C++ library with Python bindings to search for points in space that are close to a given query point. It also creates large read-only file-based data structures that are mmapped into memory so that many processes may share the same data.
    • Weaviate - Weaviate is a GraphQL-based semantic search engine with build-in (word) embeddings.
    • LinkedIn Cleo - is a flexible software library for enabling rapid development of partial, out-of-order and real-time typeahead search.
    • Vespa - is an engine for low-latency computation over large data sets. It stores and indexes your data such that queries, selection and processing over the data can be performed at serving time.
    • Google Caffeine - continuous indexing system.
    • HBase Coprocessor - implementation of Percolator, part of HBase.

  • Security

    • BDA - The vulnerability detector for Hadoop and Spark
    • Apache Ranger - Central security admin & fine-grained authorization for Hadoop
    • Apache Eagle - real time monitoring solution

  • Service Programming

    • Google Chubby - a lock service for loosely-coupled distributed systems.
    • OpenMPI - message passing framework.
    • Hydrosphere Mist - a service for exposing Apache Spark analytics jobs and machine learning models as realtime, batch or reactive web services.
    • Spotify Luigi - a Python package for building complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization, handling failures, command line integration, and much more.
    • Twitter Elephant Bird - libraries for working with LZOP-compressed data.
    • Twitter Finagle - asynchronous network stack for the JVM.
    • Serf - decentralized solution for service discovery and orchestration.
    • Spring XD - distributed and extensible system for data ingestion, real time analytics, batch processing, and data export.
    • Mara - A lightweight opinionated ETL framework, halfway between plain scripts and Apache Airflow
    • Akka Toolkit - runtime for distributed, and fault tolerant event-driven applications on the JVM.
    • Apache Curator - Java libaries for Apache ZooKeeper.

  • SQL-like processing

    • Actian SQL for Hadoop - high performance interactive SQL access to all Hadoop data.
    • Apache HCatalog - table and storage management layer for Hadoop.
    • Aster Database - SQL-like analytic processing for MapReduce.
    • Dremio - an open-source, SQL-like Data-as-a-Service Platform based on Apache Arrow.
    • Facebook PrestoDB - distributed SQL query engine.
    • Spark Catalyst - is a Query Optimization Framework for Spark and Shark.
    • Splice Machine - a full-featured SQL-on-Hadoop RDBMS with ACID transactions.
    • Trafodion - enterprise-class SQL-on-HBase solution targeting big data transactional or operational workloads.
    • Cloudera Impala - framework for interactive analysis, Inspired by Dremel.
    • Concurrent Lingual - SQL-like query language for Cascading.
    • Materialize - is a streaming database for real-time applications using SQL for queries and supporting a large fraction of PostgreSQL.
    • Invantive SQL - SQL engine for online and on-premise use with integrated local data replication and 70+ connectors.
    • Pivotal HDB - SQL-like data warehouse system for Hadoop.
    • Actian SQL for Hadoop - high performance interactive SQL access to all Hadoop data.
    • PipelineDB - an open-source relational database that runs SQL queries continuously on streams, incrementally storing results in tables.
    • Actian SQL for Hadoop - high performance interactive SQL access to all Hadoop data.
    • Apache Phoenix - SQL skin over HBase.
    • RainstorDB - database for storing petabyte-scale volumes of structured and semi-structured data.
    • Apache Drill - framework for interactive analysis, inspired by Dremel.
    • Apache Calcite - framework that allows efficient translation of queries involving heterogeneous and federated data.
    • Google BigQuery - framework for interactive analysis, implementation of Dremel.
    • SparkSQL - Manipulating Structured Data Using Spark.
    • Stinger - interactive query for Hive.
    • Tajo - distributed data warehouse system on Hadoop.

  • System Deployment

    • Brooklyn - library that simplifies application deployment and management.
    • Buildoop - Similar to Apache BigTop based on Groovy language.
    • Google Borg - job scheduling and monitoring system.
    • Google Omega - job scheduling and monitoring system.
    • Kubernetes - a system for automating deployment, scaling, and management of containerized applications.
    • Brooklyn - library that simplifies application deployment and management.
    • Buildoop - Similar to Apache BigTop based on Groovy language.
    • Marathon - Mesos framework for long-running services.
    • Apache Slider - is a YARN application to deploy existing distributed applications on YARN.
Programming Languages
Java 30 Python 19 Go 18 JavaScript 15 Scala 11 C++ 9 C 6 Erlang 5 Ruby 3 Rust 3
Categories
`Distributed Programming ` 53 Interesting Papers 46 Data Visualization 44 Databases 44 Machine Learning 42 Books 35 Applications 31 `NewSQL Databases` 28 Data Ingestion 26 SQL-like processing 24 Graph Data Model 24 Key-value Data Model 24 Business Intelligence 20 Time-Series Databases 18 Search engine and framework 17 `Distributed Filesystem ` 17 System Deployment 14 MySQL forks and evolutions 13 `Columnar Databases` 13 Service Programming 11 `Key Map Data Model ` 11 Scheduling 10 Internet of things and sensor data 10 PostgreSQL forks and evolutions 8 `Frameworks ` 6 Benchmarking 6 `Document Data Model ` 5 Memcached forks and evolutions 5 Videos 4 Embedded Databases 4 Interesting Readings 4 `RDBMS ` 4 Security 3 License 2 `Distributed Index ` 1
Sub Categories
2001 - 2010 33 Data Visualization 19 2013 - 2014 15 2011 - 2012 15 Contributing 3 2015 - 2016 2
Keywords
database 21 machine-learning 13 deep-learning 11 python 10 data-science 9 go 8 sql 7 analytics 7 kafka 6 java 6 time-series 6 data-visualization 6 graph 6 mysql 5 network-embedding 5 visualization 5 awesome-list 5 spark 5 awesome 5 metrics 5 distributed-database 5 golang 5 kubernetes 4 tensorflow 4 network-science 4 classifier 4 graph-embedding 4 random-forest 4 pytorch 4 monitoring 4 geospatial 4 distributed 4 node-embedding 4 tsdb 3 hadoop 3 workflow 3 c-plus-plus 3 graph-database 3 in-memory 3 data-analysis 3 big-data 3 postgresql 3 jupyter 3 gradient-boosting 3 rust 3 stream-processing 3 nosql 3 distributed-systems 3 etl 3 node2vec 3