Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

awesome-bigdata

A curated list of awesome big data frameworks, ressources and other awesomeness.
https://github.com/oxnr/awesome-bigdata

Last synced: 4 days ago
JSON representation

  • Books

  • System Deployment

    • Kubernetes - a system for automating deployment, scaling, and management of containerized applications.
    • Brooklyn - library that simplifies application deployment and management.
    • Buildoop - Similar to Apache BigTop based on Groovy language.
    • Google Borg - job scheduling and monitoring system.
    • Google Omega - job scheduling and monitoring system.
    • Marathon - Mesos framework for long-running services.
    • Brooklyn - library that simplifies application deployment and management.
    • Buildoop - Similar to Apache BigTop based on Groovy language.
    • Apache Slider - is a YARN application to deploy existing distributed applications on YARN.
    • Linkis - Linkis helps easily connect to various back-end computation/storage engines.

  • Document Data Model

    • MongoDB - Document-oriented database system.
    • RavenDB - A transactional, open-source Document Database.
    • RethinkDB - document database that supports queries like table joins and group by.
    • Microsoft Azure DocumentDB - NoSQL cloud database service with protocol support for MongoDB

  • Machine Learning

    • Deeplearning4j - Fast, open deep learning for the JVM (Java, Scala, Clojure). A neural network configuration layer powered by a C++ library. Uses Spark and Hadoop to train nets on multiple GPUs and CPUs.
    • Azure ML Studio - Cloud-based AzureML, R, Python Machine Learning platform
    • DataVec - A vectorization and data preprocessing library for deep learning in Java and Scala. Part of the Deeplearning4j ecosystem.
    • etcML - text classification with machine learning.
    • GraphLab Create - A machine learning platform in Python with a broad collection of ML toolkits, data engineering, and deployment tools.
    • MLbase - distributed machine learning libraries for the BDAS stack.
    • MonkeyLearn - Text mining made easy. Extract and classify data from text.
    • ND4J - A matrix library for the JVM. Numpy for Java.
    • RL4J - Reinforcement learning for Java and Scala. Includes Deep-Q learning and A3C algorithms, and integrates with Open AI's Gym. Runs in the Deeplearning4j ecosystem.
    • Sibyl - System for Large Scale Machine Learning at Google.
    • Theano - A Python-focused machine learning library supported by the University of Montreal.
    • Torch - A deep learning library with a Lua API, supported by NYU and Facebook.
    • TensorFlow - Library from Google for machine learning using data flow graphs.
    • scikit-learn - scikit-learn: machine learning in Python.
    • convnetjs - Deep Learning in Javascript. Train Convolutional Neural Networks (or ordinary ones) in your browser.
    • ML Workspace - All-in-one web-based IDE specialized for machine learning and data science.
    • brain - Neural networks in JavaScript.
    • Karate Club - An unsupervised machine learning library for graph structured data. Python
    • Oryx - Lambda architecture on Apache Spark, Apache Kafka for real-time large scale machine learning.
    • PyTorch Geometric Temporal - a temporal extension library for PyTorch Geometric .
    • Little Ball of Fur - A subsampling library for graph structured data. Python
    • Shapley - A data-driven framework to quantify the value of classifiers in a machine learning ensemble.
    • BidMach - CPU and GPU-accelerated Machine Learning Library.
    • Decider - Flexible and Extensible Machine Learning in Ruby.
    • nupic - Numenta Platform for Intelligent Computing: a brain-inspired machine intelligence platform, and biologically accurate neural network based on cortical learning algorithms.
    • Keras - An intuitive neural net API inspired by Torch that runs atop Theano and Tensorflow.
    • MLPNeuralNet - Fast multilayer perceptron neural network library for iOS and Mac OS X.
    • Etsy Conjecture - scalable Machine Learning in Scalding.
    • Concurrent Pattern - machine learning library for Cascading.
    • Lambdo - Lambdo is a workflow engine which significantly simplifies the analysis process by unifying feature engineering and machine learning operations.
    • Velox - System for serving machine learning predictions.
    • H2O - statistical, machine learning and math runtime with Hadoop. R and Python.
    • WEKA - suite of machine learning software.

  • Data Ingestion

    • Amazon Web Services Glue - serverless fully managed extract, transform, and load (ETL) service
    • Amazon Kinesis - real-time processing of streaming data at massive scale.
    • Apache NiFi - Apache NiFi is an integrated data logistics platform for automating the movement of data between disparate systems.
    • Google Photon - geographically distributed system for joining multiple continuously flowing streams of data in real-time with high scalability and low latency.
    • Kestrel - distributed message queue system.
    • Alooma - data pipeline as a service enabling moving data sources such as MySQL into data warehouses.
    • Heka - open source stream processing software system.
    • RudderStack - an open source customer data infrastructure (segment, mParticle alternative) written in go.
    • Apache Pulsar - a distributed pub-sub messaging platform with a very flexible messaging model and an intuitive client API.
    • Facebook Scribe - streamed log data aggregator.
    • Netflix Suro - log agregattor like Storm and Samza based on Chukwa.
    • Zilla - An API gateway built for event-driven architectures and streaming that supports standard protocols such as HTTP, SSE, gRPC, MQTT and the native Kafka protocol.
    • Gazette - Distributed streaming infrastructure built on cloud storage which makes it easy to mix and match batch and streaming paradigms.
    • Pinterest Secor - is a service implementing Kafka log persistance.
    • HIHO - framework for connecting disparate data sources with Hadoop.
    • Logstash - a tool for managing events and logs.
    • StreamSets Data Collector - continuous big data ingest infrastructure with a simple to use IDE.
    • Skizze - sketch data store to deal with all problems around counting and sketching using probabilistic data-structures.
    • redpanda - A Kafka® replacement for mission critical systems; 10x faster. Written in C++.
    • Census - A reverse ETL product that let you sync data from your data warehouse to SaaS Applications. No engineering favors required—just SQL.
    • Estuary - SaaS platform based on Gazette with plug-and-play connectors.

  • Columnar Databases

    • Amazon Redshift - Amazon's cloud offering, also based on a columnar datastore backend.
    • MonetDB - column store database.
    • EventQL - a distributed, column-oriented database built for large-scale event collection and analytics.
    • IndexR - an open-source columnar storage format for fast & realtime analytic with big data.
    • LocustDB - an experimental analytics database aiming to set a new standard for query performance on commodity hardware.
    • EventQL - a distributed, column-oriented database built for large-scale event collection and analytics.
    • Pivotal Greenplum - purpose-built, dedicated analytic data warehouse that offers a columnar engine as well as a traditional row-based one.
    • Vertica - is designed to manage large, fast-growing volumes of data and provide very fast query performance when used for data warehouses.
    • ClickHouse - an open-source column-oriented database management system that allows generating analytical data reports in real time.

  • Key-value Data Model

    • Amazon DynamoDB - distributed key/value store, implementation of Dynamo paper.
    • Redis - in memory key value datastore.
    • Ignite - is an in-memory key-value data store providing full SQL-compliant data access that can optionally be backed by disk storage.
    • LinkedIn Krati - is a simple persistent data store with very low latency and high throughput.
    • Linkedin Voldemort - distributed key/value storage system.
    • Tile38 - a geolocation data store, spatial index, and realtime geofence, supporting a variety of object types including latitude/longitude points, bounding boxes, XYZ tiles, Geohashes, and GeoJSON
    • GhostDB - a distributed, in-memory, general purpose key-value data store that delivers microsecond performance at any scale.
    • BuntDB - a fast, embeddable, in-memory key/value database for Go with custom indexing and geospatial support.
    • Bolt - an embedded key-value database for Go.
    • HyperDex - a scalable, next generation key-value and document store with a wide array of features, including consistency, fault tolerance and high performance.
    • SummitDB - an in-memory, NoSQL key/value database, with disk persistence and using the Raft consensus algorithm.
    • Graviton - a simple, fast, versioned, authenticated, embeddable key-value store database in pure Go(lang).
    • Tarantool - an efficient NoSQL database and a Lua application server.
    • Edis - is a protocol-compatible Server replacement for Redis.
    • Riak - a decentralized datastore.
    • ElephantDB - Distributed database specialized in exporting data from Hadoop.
    • Storehaus - library to work with asynchronous key value stores, by Twitter.
    • TiKV - a distributed key-value database powered by Rust and inspired by Google Spanner and HBase.
    • BTDB - Key Value Database in .Net with Object DB Layer, RPC, dynamic IL and much more
    • Linkedin Voldemort - distributed key/value storage system.
    • Oracle NoSQL Database - distributed key-value database by Oracle Corporation.
    • TreodeDB - key-value store that's replicated and sharded and provides atomic multirow writes.
    • Badger - a fast, simple, efficient, and persistent key-value store written natively in Go.
    • EventStore - distributed time series database.

  • MySQL forks and evolutions

    • Amazon RDS - MySQL databases in Amazon's cloud.
    • Drizzle - evolution of MySQL 6.0.
    • MariaDB - enhanced, drop-in replacement for MySQL.
    • MySQL Cluster - MySQL implementation using NDB Cluster storage engine.
    • TokuDB - TokuDB is a storage engine for MySQL and MariaDB.
    • WebScaleSQL - is a collaboration among engineers from several companies that face similar challenges in running MySQL at scale.
    • Drizzle - evolution of MySQL 6.0.
    • ProxySQL - High Performance Proxy for MySQL.
    • WebScaleSQL - is a collaboration among engineers from several companies that face similar challenges in running MySQL at scale.
    • Percona Server - enhanced, drop-in replacement for MySQL.

  • Internet of things and sensor data

    • IFTTT - If this then that
    • Apache Edgent (Incubating) - a programming model and micro-kernel style runtime that can be embedded in gateways and small footprint edge devices enabling local, real-time, analytics on the edge devices.
    • TempoIQ - Cloud-based sensor analytics.
    • Pubnub - Data stream network
    • Evrything - Making products smart
    • Ably - Pub/sub messaging platform for IoT
    • Azure IoT Hub - Cloud-based bi-directional monitoring and messaging hub
    • NetLytics - Analytics platform to process network data on Spark.
    • ThingWorx - Rapid development and connection of intelligent systems

  • Applications

    • Jupyter - Notebook and project application for interactive data science and scientific computing across all programming languages.
    • Apache Tika - content analysis toolkit.
    • Hunk - Splunk analytics for Hadoop.
    • Imhotep - Large scale analytics platform by indeed.
    • Indicative - Web & mobile analytics tool, with data warehouse (AWS, BigQuery) integration.
    • Qubole - auto-scaling Hadoop cluster, built-in data connectors.
    • Splunk - analyzer for machine-generated data.
    • Sumo Logic - cloud based analyzer for machine-generated data.
    • Comet - Comet provides an end-to-end model evaluation platform for AI developers, with best in class LLM evaluations, experiment tracking, and production monitoring.
    • Eventhub - open source event analytics platform.
    • Snowplow - enterprise-strength web and event analytics, powered by Hadoop, Kinesis, Redshift and Postgres.
    • ElastAert - ElastAlert is a simple framework for alerting on anomalies, spikes, or other patterns of interest from data in ElasticSearch.
    • Atlas - a backend for managing dimensional time series data.
    • Kapacitor - an open source framework for processing, monitoring, and alerting on time series data.
    • 411 - an web application for alert management resulting from scheduled searches into Elasticsearch.
    • Hermes - asynchronous message broker built on top of Kafka.
    • Substation - Substation is a cloud native data pipeline and transformation toolkit written in Go.
    • Argus - Time series monitoring and alerting platform.
    • Adobe spindle - Next-generation web analytics processing with Scala, Spark, and Parquet.
    • PivotalR - R on Pivotal HD / HAWQ and PostgreSQL.
    • SnappyData - a distributed in-memory data store for real-time operational analytics, delivering stream analytics, OLTP (online transaction processing) and OLAP (online analytical processing) built on Spark in a single integrated cluster.
    • Opik - Debug, evaluate, and monitor your LLM applications, RAG systems, and agentic workflows with comprehensive tracing, automated evaluations, and production-ready dashboards.
    • AthenaX - a streaming analytics platform that enables users to run production-quality, large scale streaming analytics using Structured Query Language (SQL).
    • Hunk - Splunk analytics for Hadoop.
    • SparkR - R frontend for Spark.
    • Rakam - open-source real-time custom analytics platform powered by Postgresql, Kinesis and PrestoDB.

  • Time-Series Databases

    • InfluxDB - a time series database with optimised IO and queries, supports pgsql and influx wire protocols.
    • Prometheus - a time series database and service monitoring system.
    • Axibase Time Series Database - Integrated time series database on top of HBase with built-in visualization, rule-engine and SQL support.
    • QuestDB - high-performance, open-source SQL database for applications in financial services, IoT, machine learning, DevOps and observability.
    • M3DB - a distributed time series database that can be used for storing realtime metrics at long retention.
    • Rhombus - series object store for Cassandra that handles all the complexity of building wide row indexes.
    • VictoriaMetrics - fast, scalable and resource-effective open-source TSDB compatible with Prometheus. Single-node and cluster versions included
    • Akumuli - series database. It can be used to capture, store and process time-series data in real-time. The word "akumuli" can be translated from esperanto as "accumulate".
    • Kairosdb - similar to OpenTSDB but allows for Cassandra.
    • Timely
    • Dalmatiner DB
    • Blueflood
    • Beringei - Facebook's in-memory time-series database.
    • Cube - uses MongoDB to store time series data.
    • Heroic - is a scalable time series database based on Cassandra and Elasticsearch.
    • Newts - a time series database based on Apache Cassandra.
    • Chronix - a time series storage built to store time series highly compressed and for fast access times.
    • TDengine - a time series database in C utilizing unique features of IoT to improve read/write throughput and reduce space needed to store data
    • Druid
    • IronDB - scalable, general-purpose time series database.

  • Distributed Programming

    • Cascalog - data processing and querying library.
    • Apache Beam - an unified model and set of language-specific SDKs for defining and executing data processing workflows.
    • Apache APEX - a unified, enterprise platform for big data stream and batch processing.
    • Apache Gearpump - real-time big data streaming engine based on Akka.
    • Apache MapReduce - programming model for processing large data sets with a parallel, distributed algorithm on a cluster.
    • Apache Pig - high level language to express data analysis programs for Hadoop.
    • Apache Spark Streaming - framework for stream processing, part of Spark.
    • Apache Twill - abstraction over YARN that reduces the complexity of developing distributed applications.
    • Baidu Bigflow - an interface that allows for writing distributed computing programs providing lots of simple, flexible, powerful APIs to easily handle data of any scale.
    • Cheetah - High Performance, Custom Data Warehouse on Top of MapReduce.
    • DataTorrent StrAM - real-time engine is designed to enable distributed, asynchronous, real time in-memory big-data computations in as unblocked a way as possible, with minimal overhead and impact on performance.
    • Google MapReduce - map reduce framework.
    • Kite - is a set of libraries, tools, examples, and documentation focused on making it easier to build systems on top of the Hadoop ecosystem.
    • Nokia Disco - MapReduce framework developed by Nokia.
    • Pinterest Pinlater - asynchronous job execution system.
    • Rackerlabs Blueflood - multi-tenant distributed metric processing system
    • Stratosphere - general purpose cluster computing framework.
    • Streamdrill - useful for counting activities of event streams over different time windows and finding the most active one.
    • Wallaroo - The ultrafast and elastic data processing engine. Big or fast data - no fuss, no Java needed.
    • Google Dataflow - create data pipelines to help themæingest, transform and analyze data.
    • Twitter TSAR - TimeSeries AggregatoR by Twitter.
    • Facebook Peregrine - Map Reduce framework.
    • Ray - A fast and simple framework for building and running distributed applications.
    • Twitter Scalding - Scala library for Map Reduce jobs, built on Cascading.
    • Netflix PigPen - map-reduce for Clojure which compiles to Apache Pig.
    • Twitter Summingbird - Streaming MapReduce with Scalding and Storm, by Twitter.
    • AddThis Hydra - distributed data processing and storage system originally developed at AddThis.
    • Twitter Heron - Heron is a realtime, distributed, fault-tolerant stream processing engine from Twitter replacing Storm.
    • Damballa Parkour - MapReduce library for Clojure.
    • Skale - High performance distributed data processing in NodeJS.
    • Concurrent Cascading - framework for data management/analytics on Hadoop.
    • Datasalt Pangool - alternative MapReduce paradigm.
    • Kite - is a set of libraries, tools, examples, and documentation focused on making it easier to build systems on top of the Hadoop ecosystem.
    • Nokia Disco - MapReduce framework developed by Nokia.
    • Rackerlabs Blueflood - multi-tenant distributed metric processing system
    • Stratosphere - general purpose cluster computing framework.
    • streamsx.topology - Libraries to enable building IBM Streams application in Java, Python or Scala.
    • Tuktu - Easy-to-use platform for batch and streaming computation, built using Scala, Akka and Play!
    • AMPLab SIMR - run Spark on Hadoop MapReduce v1.
    • Pydoop - Python MapReduce and HDFS API for Hadoop.
    • Onyx - Distributed computation for the cloud.
    • JAQL - declarative programming language for working with structured, semi-structured and unstructured data.
    • Cheetah - High Performance, Custom Data Warehouse on Top of MapReduce.

  • RDBMS

    • ElasticSearch - Search and analytics engine based on Apache Lucene.
    • Enigma.io
    • Lily HBase Indexer - quickly and easily search for any content stored in HBase.
    • LinkedIn Galene - search architecture at LinkedIn.
    • Sphinx Search Server - fulltext search engine.
    • Facebook Faiss - is a library for efficient similarity search and clustering of dense vectors. It contains algorithms that search in sets of vectors of any size, up to ones that possibly do not fit in RAM. It also contains supporting code for evaluation and parameter tuning. Faiss is written in C++ with complete wrappers for Python/numpy.
    • Annoy - is a C++ library with Python bindings to search for points in space that are close to a given query point. It also creates large read-only file-based data structures that are mmapped into memory so that many processes may share the same data.
    • Elassandra - is a fork of Elasticsearch modified to run on top of Apache Cassandra in a scalable and resilient peer-to-peer architecture.
    • Weaviate - Weaviate is a GraphQL-based semantic search engine with build-in (word) embeddings.
    • LinkedIn Zoie - is a realtime search/indexing system written in Java.
    • MG4J - MG4J (Managing Gigabytes for Java) is a full-text search engine for large document collections written in Java. It is highly customisable, high-performance and provides state-of-the-art features and new research algorithms.
    • Sphinx Search Server - fulltext search engine.
    • LinkedIn Bobo - is a Faceted Search implementation written purely in Java, an extension to Apache Lucene.
    • LinkedIn Cleo - is a flexible software library for enabling rapid development of partial, out-of-order and real-time typeahead search.

  • Data Visualization

    • Grafana - graphite dashboard frontend, editor and graph composer.
    • Metricsgraphic.js - a library built on top of D3 that is optimized for time-series data
    • DevExtreme React Chart - High-performance plugin-based React chart for Bootstrap and Material Design.
    • D3 - javaScript library for manipulating documents.
    • Highcharts - simple and flexible charting API.
    • Dekart - Large scale geospatial analytics for Google BigQuery based on Kepler.gl.
    • chartd - responsive, retina-compatible charts with just an img tag.
    • FnordMetric - write SQL queries that return SVG charts rather than tables
    • Graphite - scalable Realtime Graphing.
    • Zing Charts - JavaScript charting library for big data.
    • Lumify - open source big data analysis and visualization platform
    • Matplotlib - plotting with Python.
    • Chartist.js - another open source HTML5 Charts visualization.
    • Plotly.js
    • Zeppelin - a notebook-style collaborative data analysis.
    • Vega - a visualization grammar.
    • Dash - Analytical Web Apps for Python, R, Julia, and Jupyter. Built on top of plotly, no JS required
    • Sigma.js - JavaScript library dedicated to graph drawing.
    • DataSphere Studio - one-stop data application development management portal.
    • Redash - open-source platform to query and visualize data.
    • Peity - Progressive SVG bar, line and pie charts.
    • CartoDB - open-source or freemium hosting for geospatial databases with powerful front-end editing capabilities and a robust API.
    • Cubism - JavaScript library for time series visualization.
    • Airpal - Web UI for PrestoDB.
    • Bloomery - Web UI for Impala.
    • Freeboard - pen source real-time dashboard builder for IOT and other web mashups.
    • Gephi - An award-winning open-source platform for visualizing and manipulating large graphs and network connections. It's like Photoshop, but for graphs. Available for Windows and Mac OS X.
    • D3.compose - Compose complex, data-driven visualizations from reusable charts and components.
    • Envisionjs - dynamic HTML5 visualization.
    • Arbor - graph visualization library using web workers and jQuery.
    • Kibana - visualize logs and time-stamped data
    • Superset - a data exploration platform designed to be visual, intuitive and interactive, making it easy to slice, dice and visualize data and perform analytics at the speed of thought.
    • Frappe Charts - GitHub-inspired simple and modern SVG charts for the web with zero dependencies.
    • Echarts - Baidus enterprise charts.
    • Google Charts - simple charting API.
    • Banana - visualize logs and time-stamped data stored in Solr. Port of Kibana.
    • chartd - responsive, retina-compatible charts with just an img tag.
    • Crossfilter - JavaScript library for exploring large multivariate datasets in the browser. Works well with dc.js and d3.js.
    • DC.js - Dimensional charting built to work natively with crossfilter rendered using d3.js. Excellent for connecting charts/additional metadata to hover events in D3.
    • Recline - simple but powerful library for building data applications in pure Javascript and HTML.
    • Crossfilter - JavaScript library for exploring large multivariate datasets in the browser. Works well with dc.js and d3.js.

  • Service Programming

    • OpenMPI - message passing framework.
    • Google Chubby - a lock service for loosely-coupled distributed systems.
    • Spotify Luigi - a Python package for building complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization, handling failures, command line integration, and much more.
    • Hydrosphere Mist - a service for exposing Apache Spark analytics jobs and machine learning models as realtime, batch or reactive web services.
    • Twitter Elephant Bird - libraries for working with LZOP-compressed data.
    • Twitter Finagle - asynchronous network stack for the JVM.
    • Serf - decentralized solution for service discovery and orchestration.
    • Spring XD - distributed and extensible system for data ingestion, real time analytics, batch processing, and data export.

  • Business Intelligence

    • Tableau - business intelligence platform.
    • BIME Analytics - business intelligence platform in the cloud.
    • GoodData - platform for data products and embedded analytics.
    • Jaspersoft - powerful business intelligence suite.
    • Jedox Palo - customisable Business Intelligence platform.
    • Jethrodata - Interactive Big Data Analytics.
    • Redash - Open source business intelligence platform, supporting multiple data sources and planned queries.
    • Saiku Analytics - Open source analytics platform.
    • Knowage - open source business intelligence platform. (former [SpagoBi](http://www.spagobi.org/))
    • SparklineData SNAP - modern B.I platform powered by Apache Spark.
    • Microstrategy - software platforms for business intelligence, mobile intelligence, and network applications.
    • Metabase - The simplest, fastest way to get business intelligence and analytics to everyone in your company.
    • Blazer - business intelligence made simple.
    • Lightdash - The open source Looker alternative built on dbt
    • Numeracy - Fast, clean SQL client and business intelligence.
    • Pentaho - business intelligence platform.
    • Zoomdata - Big Data Analytics.

  • Interesting Papers

    • 2001 - 2010

      • 2003 - **Google** - The Google File System.
      • 2010 - **Google** - Pregel: A System for Large-Scale Graph Processing.
      • 2010 - **Facebook** - Finding a needle in Haystack: Facebook’s photo storage.
      • 2010 - **AMPLab** - Spark: Cluster Computing with Working Sets.
      • 2010 - **Google** - Large-scale Incremental Processing Using Distributed Transactions and Notifications base of Percolator and Caffeine.
      • 2010 - **Google** - Dremel: Interactive Analysis of Web-Scale Datasets.
      • 2010 - **Yahoo** - S4: Distributed Stream Computing Platform.
      • 2009 - HadoopDB: An Architectural Hybrid of MapReduce and DBMS Technologies for Analytical Workloads.
      • 2006 - **Google** - The Chubby lock service for loosely-coupled distributed systems.
      • 2004 - **Google** - MapReduce: Simplied Data Processing on Large Clusters.
      • 2010 - **Google** - Large-scale Incremental Processing Using Distributed Transactions and notifications base of Percolator and Caffeine.
      • 2010 - **Yahoo** - S4: Distributed Stream Computing Platform.
      • 2009 - HadoopDB: An Architectural Hybrid of MapReduce and DBMS Technologies for Analytical Workloads.
      • 2008 - **AMPLab** - Chukwa: A large-scale monitoring system.

    • 2015 - 2016

      • 2015 - **Facebook** - One Trillion Edges: Graph Processing at Facebook-Scale.
      • 2015 - **Facebook** - One Trillion Edges: Graph Processing at Facebook-Scale.

    • 2013 - 2014

      • 2014 - **Stanford** - Mining of Massive Datasets.
      • 2013 - **AMPLab** - Presto: Distributed Machine Learning and Graph Processing with Sparse Matrices.
      • 2013 - **AMPLab** - MLbase: A Distributed Machine-learning System.
      • 2013 - **AMPLab** - Shark: SQL and Rich Analytics at Scale.
      • 2013 - **AMPLab** - GraphX: A Resilient Distributed Graph System on Spark.
      • 2013 - **Google** - HyperLogLog in Practice: Algorithmic Engineering of a State of The Art Cardinality Estimation Algorithm.
      • 2013 - **Metamarkets** - Druid: A Real-time Analytical Data Store.
      • 2013 - **Google** - F1: A Distributed SQL Database That Scales.
      • 2013 - **Facebook** - Scaling Memcache at Facebook.
      • 2014 - **Stanford** - Mining of Massive Datasets.
      • 2013 - **Google** - HyperLogLog in Practice: Algorithmic Engineering of a State of The Art Cardinality Estimation Algorithm.
      • 2013 - **Metamarkets** - Druid: A Real-time Analytical Data Store.
      • 2013 - **Google** - F1: A Distributed SQL Database That Scales.

    • 2011 - 2012

      • 2012 - **Twitter** - The Unified Logging Infrastructure
      • 2012 - **AMPLab** - Blink and It’s Done: Interactive Queries on Very Large Data.
      • 2012 - **AMPLab** - Fast and Interactive Analytics over Hadoop Data with Spark.
      • 2012 - **AMPLab** - Shark: Fast Data Analysis Using Coarse-grained Distributed Memory.
      • 2012 - **Microsoft** - Paxos Replicated State Machines as the Basis of a High-Performance Data Store.
      • 2012 - **AMPLab** - BlinkDB: Queries with Bounded Errors and Bounded Response Times on Very Large Data.
      • 2012 - **Google** - Processing a trillion cells per mouse click.
      • 2012 - **Google** - Spanner: Google’s Globally-Distributed Database.
      • 2011 - **AMPLab** - Scarlett: Coping with Skewed Popularity Content in MapReduce Clusters.
      • 2011 - **AMPLab** - Mesos: A Platform for Fine-Grained Resource Sharing in the Data Center.
      • 2011 - **Google** - Megastore: Providing Scalable, Highly Available Storage for Interactive Services.
      • 2012 - **Twitter** - The Unified Logging Infrastructure
      • 2012 - **Google** - Processing a trillion cells per mouse click.
      • 2012 - **AMPLab** - BlinkDB: Queries with Bounded Errors and Bounded Response Times on Very Large Data.
      • 2012 - **Microsoft** - Paxos Made Parallel.

  • Distributed Filesystem

  • Key Map Data Model

  • Graph Data Model

    • MapGraph - Massively Parallel Graph processing on GPUs.
    • Neo4j - graph database written entirely in Java.
    • NodeXL - A free, open-source template for Microsoft® Excel® 2007, 2010, 2013 and 2016 that makes it easy to explore network graphs.
    • Titan - distributed graph database, built over Cassandra.
    • AgensGraph - a new generation multi-model graph database for the modern complex data environment.
    • EliasDB - a lightweight graph based database that does not require any third-party libraries.
    • Google Cayley - open-source graph database.
    • Twitter FlockDB - distributed graph database.
    • Gremlin - graph traversal Language.
    • GCHQ Gaffer - Gaffer by GCHQ is a framework that makes it easy to store large-scale graphs in which the nodes and edges have statistics.
    • Phoebus - framework for large scale graph processing.
    • DGraph - A scalable, distributed, low latency, high throughput graph database aimed at providing Google production level scale and throughput, with low enough latency to be serving real time user queries, over terabytes of structured data.
    • Infovore - RDF-centric Map/Reduce framework.
    • AgensGraph - a new generation multi-model graph database for the modern complex data environment.
    • Microsoft Graph Engine - a distributed in-memory data processing engine, underpinned by a strongly-typed in-memory key-value store and a general distributed computation engine.
    • Titan - distributed graph database, built over Cassandra.
    • Facebook TAO - TAO is the distributed data store that is widely used at facebook to store and serve the social graph.
    • OrientDB - document and graph database.

  • Embedded Databases

    • Actian PSQL - ACID-compliant DBMS developed by Pervasive Software, optimized for embedding in applications.
    • LevelDB - a fast key-value storage library written at Google that provides an ordered mapping from string keys to string values.
    • HanoiDB - Erlang LSM BTree Storage.

  • NewSQL Databases

    • CitusDB - scales out PostgreSQL through sharding and replication.
    • FoundationDB - distributed database, inspired by F1.
    • Google F1 - distributed SQL database built on Spanner.
    • Google Spanner - globally distributed semi-relational database.
    • InfiniSQL - infinity scalable RDBMS.
    • Map-D - GPU in-memory database, big data analysis and visualization platform.
    • Pivotal GemFire XD - Low-latency, in-memory, distributed SQL data store. Provides SQL interface to in-memory table data, persistable in HDFS.
    • SAP HANA - is an in-memory, column-oriented, relational database management system.
    • Sky - database used for flexible, high performance analysis of behavioral data.
    • yugabyteDB - open source, high-performance, distributed SQL database compatible with PostgreSQL.
    • TiDB - TiDB is a distributed SQL database. Inspired by the design of Google F1.
    • Cockroach - Scalable, Geo-Replicated, Transactional Datastore.
    • ActorDB - a distributed SQL database with the scalability of a KV store, while keeping the query capabilities of a relational database.
    • BayesDB - statistic oriented SQL database.
    • Comdb2 - a clustered RDBMS built on optimistic concurrency control techniques.
    • KarelDB - a relational database backed by Apache Kafka.
    • Haeinsa - linearly scalable multi-row, multi-table transaction library for HBase based on Percolator.
    • InfiniSQL - infinity scalable RDBMS.
    • SenseiDB - distributed, realtime, semi-structured database.
    • SymmetricDS - open source software for both file and database synchronization.
    • Actian Ingres - commercially supported, open-source SQL relational database management system.

  • SQL-like processing

    • Actian SQL for Hadoop - high performance interactive SQL access to all Hadoop data.
    • Apache HCatalog - table and storage management layer for Hadoop.
    • Aster Database - SQL-like analytic processing for MapReduce.
    • Dremio - an open-source, SQL-like Data-as-a-Service Platform based on Apache Arrow.
    • Facebook PrestoDB - distributed SQL query engine.
    • Spark Catalyst - is a Query Optimization Framework for Spark and Shark.
    • Splice Machine - a full-featured SQL-on-Hadoop RDBMS with ACID transactions.
    • Trafodion - enterprise-class SQL-on-HBase solution targeting big data transactional or operational workloads.
    • Iceberg - an open table format for huge analytic datasets. Iceberg adds tables to Trino and Spark that use a high-performance format that works just like a SQL table.
    • Cloudera Impala - framework for interactive analysis, Inspired by Dremel.
    • Materialize - is a streaming database for real-time applications using SQL for queries and supporting a large fraction of PostgreSQL.
    • Concurrent Lingual - SQL-like query language for Cascading.
    • Pivotal HDB - SQL-like data warehouse system for Hadoop.
    • Invantive SQL - SQL engine for online and on-premise use with integrated local data replication and 70+ connectors.
    • Actian SQL for Hadoop - high performance interactive SQL access to all Hadoop data.

  • Scheduling

    • Linkedin Azkaban - batch workflow job scheduler.
    • Cronicle - Distributed, easy to install, NodeJS based, task scheduler
    • Dagster - a data orchestrator for machine learning, analytics, and ETL.
    • Apache Airflow - a platform to programmatically author, schedule and monitor workflows.
    • Schedoscope - Scala DSL for agile scheduling of Hadoop jobs.
    • Sparrow - scheduling platform.

  • Benchmarking

  • PostgreSQL forks and evolutions

    • RecDB - Open Source Recommendation Engine Built Entirely Inside PostgreSQL.
    • Stado - open source MPP database system solely targeted at data warehousing and data mart applications.
    • Yahoo Everest - multi-peta-byte database / MPP derived by PostgreSQL.
    • Postgres-XL - Scalable Open Source PostgreSQL-based Database Cluster.
    • HadoopDB - hybrid of MapReduce and DBMS.
    • IBM Netezza - high-performance data warehouse appliances.

  • Interesting Readings

  • Videos

  • Memcached forks and evolutions

  • Frameworks

    • Polyaxon - A platform for reproducible and scalable machine learning and deep learning.
    • Smooks - An extensible Java framework for building XML and non-XML (CSV, EDI, Java, etc...) streaming applications.
    • IBM Streams - platform for distributed processing and real-time analytics. Integrates with many of the popular technologies in the Big Data ecosystem (Kafka, HDFS, Spark, etc.)
    • Tigon - High Throughput Real-time Stream Processing Framework.
    • Bistro - general-purpose data processing engine for both batch and stream analytics. It is based on a novel data model, which represents data via *functions* and processes data via *column operations* as opposed to having only set operations in conventional approaches like MapReduce or SQL.

  • Distributed Index

  • Security

    • BDA - The vulnerability detector for Hadoop and Spark
Programming Languages
Java 29 Python 19 Go 18 JavaScript 14 Scala 10 C++ 10 C 5 Erlang 5 Ruby 3 Clojure 3
Categories
Interesting Papers 44 Distributed Programming 43 Data Visualization 41 Books 33 Machine Learning 33 Applications 26 Key-value Data Model 24 NewSQL Databases 21 Data Ingestion 21 Time-Series Databases 20 Graph Data Model 18 Business Intelligence 17 SQL-like processing 15 Search engine and framework 14 Distributed Filesystem 11 MySQL forks and evolutions 10 System Deployment 10 Columnar Databases 9 Internet of things and sensor data 9 Service Programming 8 Key Map Data Model 7 Benchmarking 7 PostgreSQL forks and evolutions 6 Scheduling 6 Frameworks 5 Interesting Readings 4 RDBMS 4 Videos 4 Document Data Model 4 Embedded Databases 3 Memcached forks and evolutions 3 Security 1 Distributed Index 1
Sub Categories
2001 - 2010 32 Data Visualization 19 2011 - 2012 15 2013 - 2014 13 2015 - 2016 2
Keywords
database 19 machine-learning 13 deep-learning 11 python 10 data-science 9 data-visualization 7 analytics 7 go 6 graph 6 kafka 6 sql 6 golang 5 awesome-list 5 awesome 5 visualization 5 java 5 network-embedding 5 spark 5 mysql 4 node-embedding 4 geospatial 4 distributed-database 4 graph-embedding 4 classifier 4 business-intelligence 4 tensorflow 4 kubernetes 4 pytorch 4 network-science 4 random-forest 4 time-series 3 data-analysis 3 distributed 3 jupyter 3 big-data 3 nosql 3 metrics 3 graph-database 3 event-streaming 3 gradient-boosting 3 etl 3 node2vec 3 workflow 3 networkx 3 c-plus-plus 3 postgresql 3 deepwalk 3 hadoop 3 stream-processing 3 in-memory 3