Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

data-engineering-collection

A collection of awesome software, libraries, Learning Tutorials, documents, books, resources and interesting stuff about Big Data Science & Engineering
https://github.com/exajobs/data-engineering-collection

Last synced: about 3 hours ago
JSON representation

  • Data Visualization

    • Gephi - An award-winning open-source platform for visualizing and manipulating large graphs and network connections. It's like Photoshop, but for graphs. Available for Windows and Mac OS X.
    • Google Charts - simple charting API.
    • Kibana - visualize logs and time-stamped data
    • Matplotlib - plotting with Python.
    • Peity - Progressive SVG bar, line and pie charts.
    • Plotly.js
    • Recline - simple but powerful library for building data applications in pure Javascript and HTML.
    • Redash - open-source platform to query and visualize data.
    • Sigma.js - JavaScript library dedicated to graph drawing.
    • Vega - a visualization grammar.
    • Zeppelin - a notebook-style collaborative data analysis.
    • DataSphere Studio - one-stop data application development management portal.
    • Crossfilter - JavaScript library for exploring large multivariate datasets in the browser. Works well with dc.js and d3.js.
    • Echarts - Baidus enterprise charts.
    • Superset - a data exploration platform designed to be visual, intuitive and interactive, making it easy to slice, dice and visualize data and perform analytics at the speed of thought.
    • NVD3 - chart components for d3.js.
    • Cytoscape - JavaScript library for visualizing complex networks.
    • C3 - D3-based reusable chart library
    • Chart.js - open source HTML5 Charts visualizations.

  • `Distributed Filesystem `

  • `Distributed Index `

  • `Distributed Programming `

    • Apache APEX - a unified, enterprise platform for big data stream and batch processing.
    • Apache Beam - an unified model and set of language-specific SDKs for defining and executing data processing workflows.
    • Apache Gearpump - real-time big data streaming engine based on Akka.
    • Apache MapReduce - programming model for processing large data sets with a parallel, distributed algorithm on a cluster.
    • Apache Pig - high level language to express data analysis programs for Hadoop.
    • Apache Spark Streaming - framework for stream processing, part of Spark.
    • Apache Twill - abstraction over YARN that reduces the complexity of developing distributed applications.
    • Baidu Bigflow - an interface that allows for writing distributed computing programs providing lots of simple, flexible, powerful APIs to easily handle data of any scale.
    • Cascalog - data processing and querying library.
    • Cheetah - High Performance, Custom Data Warehouse on Top of MapReduce.
    • DataTorrent StrAM - real-time engine is designed to enable distributed, asynchronous, real time in-memory big-data computations in as unblocked a way as possible, with minimal overhead and impact on performance.
    • Google MapReduce - map reduce framework.
    • Kite - is a set of libraries, tools, examples, and documentation focused on making it easier to build systems on top of the Hadoop ecosystem.
    • Nokia Disco - MapReduce framework developed by Nokia.
    • Pinterest Pinlater - asynchronous job execution system.
    • Rackerlabs Blueflood - multi-tenant distributed metric processing system
    • Stratosphere - general purpose cluster computing framework.
    • Streamdrill - useful for counting activities of event streams over different time windows and finding the most active one.
    • Wallaroo - The ultrafast and elastic data processing engine. Big or fast data - no fuss, no Java needed.
    • Google Dataflow - create data pipelines to help themæingest, transform and analyze data.
    • Twitter TSAR - TimeSeries AggregatoR by Twitter.
    • Facebook Peregrine - Map Reduce framework.
    • AddThis Hydra - distributed data processing and storage system originally developed at AddThis.
    • AMPLab SIMR - run Spark on Hadoop MapReduce v1.
    • Concurrent Cascading - framework for data management/analytics on Hadoop.
    • Damballa Parkour - MapReduce library for Clojure.
    • Datasalt Pangool - alternative MapReduce paradigm.
    • Kite - is a set of libraries, tools, examples, and documentation focused on making it easier to build systems on top of the Hadoop ecosystem.
    • Netflix PigPen - map-reduce for Clojure which compiles to Apache Pig.
    • Nokia Disco - MapReduce framework developed by Nokia.
    • Pydoop - Python MapReduce and HDFS API for Hadoop.
    • Ray - A fast and simple framework for building and running distributed applications.
    • Rackerlabs Blueflood - multi-tenant distributed metric processing system
    • Skale - High performance distributed data processing in NodeJS.
    • Stratosphere - general purpose cluster computing framework.
    • streamsx.topology - Libraries to enable building IBM Streams application in Java, Python or Scala.
    • Tuktu - Easy-to-use platform for batch and streaming computation, built using Scala, Akka and Play!
    • Twitter Heron - Heron is a realtime, distributed, fault-tolerant stream processing engine from Twitter replacing Storm.
    • Twitter Scalding - Scala library for Map Reduce jobs, built on Cascading.
    • Twitter Summingbird - Streaming MapReduce with Scalding and Storm, by Twitter.
    • Onyx - Distributed computation for the cloud.
    • JAQL - declarative programming language for working with structured, semi-structured and unstructured data.
    • Cheetah - High Performance, Custom Data Warehouse on Top of MapReduce.
    • Cascalog - data processing and querying library.
    • Streamdrill - useful for counting activities of event streams over different time windows and finding the most active one.
    • Apache Gearpump - real-time big data streaming engine based on Akka.
    • Apache REEF - retainable evaluator execution framework to simplify and unify the lower layers of big data systems.
    • Apache S4 - framework for stream processing, implementation of S4.
    • Apache Spark - framework for in-memory cluster computing.
    • Apache Storm - framework for stream processing by Twitter also on YARN.
    • Apache Samza - stream processing framework, based on Kafka and YARN.
    • Apache Tez - application framework for executing a complex DAG (directed acyclic graph) of tasks, built on YARN.
    • Google MillWheel - fault tolerant stream processing framework.
    • Metamarkets Druid - framework for real-time analysis of large datasets.

  • `Document Data Model `

  • Embedded Databases

    • Actian PSQL - ACID-compliant DBMS developed by Pervasive Software, optimized for embedding in applications.
    • HanoiDB - Erlang LSM BTree Storage.
    • LevelDB - a fast key-value storage library written at Google that provides an ordered mapping from string keys to string values.
    • BerkeleyDB - a software library that provides a high-performance embedded database for key/value data.

  • `Frameworks `

    • IBM Streams - platform for distributed processing and real-time analytics. Integrates with many of the popular technologies in the Big Data ecosystem (Kafka, HDFS, Spark, etc.)
    • Bistro - general-purpose data processing engine for both batch and stream analytics. It is based on a novel data model, which represents data via *functions* and processes data via *column operations* as opposed to having only set operations in conventional approaches like MapReduce or SQL.
    • Tigon - High Throughput Real-time Stream Processing Framework.
    • Polyaxon - A platform for reproducible and scalable machine learning and deep learning.
    • Smooks - An extensible Java framework for building XML and non-XML (CSV, EDI, Java, etc...) streaming applications.
    • Pachyderm - Pachyderm is a data storage platform built on Docker and Kubernetes to provide reproducible data processing and analysis.

  • Graph Data Model

    • Google Pregel - graph processing framework.
    • GraphX - resilient Distributed Graph System on Spark.
    • MapGraph - Massively Parallel Graph processing on GPUs.
    • NodeXL - A free, open-source template for Microsoft® Excel® 2007, 2010, 2013 and 2016 that makes it easy to explore network graphs.
    • Titan - distributed graph database, built over Cassandra.
    • AgensGraph - a new generation multi-model graph database for the modern complex data environment.
    • AgensGraph - a new generation multi-model graph database for the modern complex data environment.
    • DGraph - A scalable, distributed, low latency, high throughput graph database aimed at providing Google production level scale and throughput, with low enough latency to be serving real time user queries, over terabytes of structured data.
    • EliasDB - a lightweight graph based database that does not require any third-party libraries.
    • GCHQ Gaffer - Gaffer by GCHQ is a framework that makes it easy to store large-scale graphs in which the nodes and edges have statistics.
    • Google Cayley - open-source graph database.
    • Gremlin - graph traversal Language.
    • Infovore - RDF-centric Map/Reduce framework.
    • Microsoft Graph Engine - a distributed in-memory data processing engine, underpinned by a strongly-typed in-memory key-value store and a general distributed computation engine.
    • Phoebus - framework for large scale graph processing.
    • Titan - distributed graph database, built over Cassandra.
    • Twitter FlockDB - distributed graph database.
    • Facebook TAO - TAO is the distributed data store that is widely used at facebook to store and serve the social graph.
    • OrientDB - document and graph database.
    • GraphLab PowerGraph - a core C++ GraphLab API and a collection of high-performance machine learning and data mining toolkits built on top of the GraphLab API.
    • Intel GraphBuilder - tools to construct large-scale graphs on top of Hadoop.
    • GraphX - resilient Distributed Graph System on Spark.
    • Apache Spark Bagel - implementation of Pregel, part of Spark.
    • JanusGraph - open-source, distributed graph database

  • Interesting Papers

    • 2001 - 2010

      • 2010 - **Facebook** - Finding a needle in Haystack: Facebook’s photo storage.
      • 2010 - **AMPLab** - Spark: Cluster Computing with Working Sets.
      • 2010 - **Google** - Large-scale Incremental Processing Using Distributed Transactions and Notifications base of Percolator and Caffeine.
      • 2010 - **Google** - Dremel: Interactive Analysis of Web-Scale Datasets.
      • 2010 - **Yahoo** - S4: Distributed Stream Computing Platform.
      • 2009 - HadoopDB: An Architectural Hybrid of MapReduce and DBMS Technologies for Analytical Workloads.
      • 2006 - **Google** - The Chubby lock service for loosely-coupled distributed systems.
      • 2004 - **Google** - MapReduce: Simplied Data Processing on Large Clusters.
      • 2010 - **Google** - Large-scale Incremental Processing Using Distributed Transactions and Notifications base of Percolator and Caffeine.
      • 2010 - **Yahoo** - S4: Distributed Stream Computing Platform.
      • 2009 - HadoopDB: An Architectural Hybrid of MapReduce and DBMS Technologies for Analytical Workloads.
      • 2008 - **AMPLab** - Chukwa: A large-scale monitoring system.
      • 2007 - **Amazon** - Dynamo: Amazon’s Highly Available Key-value Store.
      • 2006 - **Google** - Bigtable: A Distributed Storage System for Structured Data.

    • 2011 - 2012

      • 2012 - **Twitter** - The Unified Logging Infrastructure
      • 2012 - **AMPLab** - Blink and It’s Done: Interactive Queries on Very Large Data.
      • 2012 - **AMPLab** - Fast and Interactive Analytics over Hadoop Data with Spark.
      • 2012 - **AMPLab** - Shark: Fast Data Analysis Using Coarse-grained Distributed Memory.
      • 2012 - **Microsoft** - Paxos Replicated State Machines as the Basis of a High-Performance Data Store.
      • 2012 - **AMPLab** - BlinkDB: Queries with Bounded Errors and Bounded Response Times on Very Large Data.
      • 2012 - **Google** - Processing a trillion cells per mouse click.
      • 2012 - **Google** - Spanner: Google’s Globally-Distributed Database.
      • 2011 - **AMPLab** - Scarlett: Coping with Skewed Popularity Content in MapReduce Clusters.
      • 2011 - **AMPLab** - Mesos: A Platform for Fine-Grained Resource Sharing in the Data Center.
      • 2011 - **Google** - Megastore: Providing Scalable, Highly Available Storage for Interactive Services.
      • 2012 - **Twitter** - The Unified Logging Infrastructure
      • 2012 - **AMPLab** - BlinkDB: Queries with Bounded Errors and Bounded Response Times on Very Large Data.
      • 2012 - **Google** - Processing a trillion cells per mouse click.
      • 2012 - **Microsoft** - Paxos Made Parallel.

    • 2013 - 2014

      • 2014 - **Stanford** - Mining of Massive Datasets.
      • 2013 - **AMPLab** - Presto: Distributed Machine Learning and Graph Processing with Sparse Matrices.
      • 2013 - **AMPLab** - MLbase: A Distributed Machine-learning System.
      • 2013 - **AMPLab** - Shark: SQL and Rich Analytics at Scale.
      • 2013 - **AMPLab** - GraphX: A Resilient Distributed Graph System on Spark.
      • 2013 - **Google** - HyperLogLog in Practice: Algorithmic Engineering of a State of The Art Cardinality Estimation Algorithm.
      • 2013 - **Metamarkets** - Druid: A Real-time Analytical Data Store.
      • 2013 - **Google** - F1: A Distributed SQL Database That Scales.
      • 2013 - **Facebook** - Scaling Memcache at Facebook.
      • 2014 - **Stanford** - Mining of Massive Datasets.
      • 2013 - **Google** - HyperLogLog in Practice: Algorithmic Engineering of a State of The Art Cardinality Estimation Algorithm.
      • 2013 - **Metamarkets** - Druid: A Real-time Analytical Data Store.
      • 2013 - **Google** - F1: A Distributed SQL Database That Scales.
      • 2013 - **Facebook** - Scuba: Diving into Data at Facebook.
      • 2013 - **Google** - Online, Asynchronous Schema Change in F1.

    • 2015 - 2016

      • 2015 - **Facebook** - One Trillion Edges: Graph Processing at Facebook-Scale.
      • 2015 - **Facebook** - One Trillion Edges: Graph Processing at Facebook-Scale.

  • Interesting Readings

  • Internet of things and sensor data

    • Apache Edgent (Incubating) - a programming model and micro-kernel style runtime that can be embedded in gateways and small footprint edge devices enabling local, real-time, analytics on the edge devices.
    • TempoIQ - Cloud-based sensor analytics.
    • Pubnub - Data stream network
    • IFTTT - If this then that
    • Evrything - Making products smart
    • Ably - Pub/sub messaging platform for IoT
    • Azure IoT Hub - Cloud-based bi-directional monitoring and messaging hub
    • ThingWorx - Rapid development and connection of intelligent systems
    • NetLytics - Analytics platform to process network data on Spark.
    • NetLytics - Analytics platform to process network data on Spark.

  • `Key Map Data Model `

Programming Languages
Java 30 Python 19 Go 18 JavaScript 15 Scala 11 C++ 9 C 6 Erlang 5 Ruby 3 Rust 3
Categories
`Distributed Programming ` 54 Interesting Papers 46 Data Visualization 44 Databases 44 Machine Learning 42 Books 35 Applications 31 `NewSQL Databases` 28 Data Ingestion 26 SQL-like processing 24 Graph Data Model 24 Key-value Data Model 24 Business Intelligence 20 Time-Series Databases 18 Search engine and framework 17 `Distributed Filesystem ` 17 System Deployment 14 MySQL forks and evolutions 13 `Columnar Databases` 13 Service Programming 11 `Key Map Data Model ` 11 Scheduling 10 Internet of things and sensor data 10 PostgreSQL forks and evolutions 8 `Frameworks ` 6 Benchmarking 6 `Document Data Model ` 5 Memcached forks and evolutions 5 Videos 4 Embedded Databases 4 Interesting Readings 4 `RDBMS ` 4 Security 3 License 2 `Distributed Index ` 1
Sub Categories
2001 - 2010 33 Data Visualization 19 2013 - 2014 15 2011 - 2012 15 Contributing 3 2015 - 2016 2
Keywords
database 21 machine-learning 13 deep-learning 11 python 10 data-science 9 go 8 sql 7 analytics 7 kafka 6 java 6 time-series 6 data-visualization 6 graph 6 mysql 5 network-embedding 5 visualization 5 awesome-list 5 spark 5 awesome 5 metrics 5 distributed-database 5 golang 5 kubernetes 4 tensorflow 4 network-science 4 classifier 4 graph-embedding 4 random-forest 4 pytorch 4 monitoring 4 geospatial 4 distributed 4 node-embedding 4 tsdb 3 hadoop 3 workflow 3 c-plus-plus 3 graph-database 3 in-memory 3 data-analysis 3 big-data 3 postgresql 3 jupyter 3 gradient-boosting 3 rust 3 stream-processing 3 nosql 3 distributed-systems 3 etl 3 node2vec 3