Applications

• 411 - an web application for alert management resulting from scheduled searches into Elasticsearch.
• Adobe spindle - Next-generation web analytics processing with Scala, Spark, and Parquet.
• Argus - Time series monitoring and alerting platform.
• AthenaX - a streaming analytics platform that enables users to run production-quality, large scale streaming analytics using Structured Query Language (SQL).
• Atlas - a backend for managing dimensional time series data.
• Eclipse BIRT - Eclipse-based reporting system.
• ElastAert - ElastAlert is a simple framework for alerting on anomalies, spikes, or other patterns of interest from data in ElasticSearch.
• Eventhub - open source event analytics platform.
• Hermes - asynchronous message broker built on top of Kafka.
• Imhotep - Large scale analytics platform by indeed.
• Kapacitor - an open source framework for processing, monitoring, and alerting on time series data.
• PivotalR - R on Pivotal HD / HAWQ and PostgreSQL.
• SnappyData - a distributed in-memory data store for real-time operational analytics, delivering stream analytics, OLTP (online transaction processing) and OLAP (online analytical processing) built on Spark in a single integrated cluster.
• Snowplow - enterprise-strength web and event analytics, powered by Hadoop, Kinesis, Redshift and Postgres.
• SparkR - R frontend for Spark.
• Substation - Substation is a cloud native data pipeline and transformation toolkit written in Go.
• Rakam - open-source real-time custom analytics platform powered by Postgresql, Kinesis and PrestoDB.
• SparkR - R frontend for Spark.

Benchmarking

• Berkeley SWIM Benchmark - real-world big data workload benchmark.
• Yahoo Gridmix3 - Hadoop cluster benchmarking from Yahoo engineer team.
• Deeplearning4j Benchmarks
• UCSB - extended Yahoo Cloud Serving Benchmark for NoSQL databases.
• Estuary Benchmark Report - reproducible, vendor-neutral data warehouse benchmark.
• Intel HiBench - a Hadoop benchmark suite.

Books

2001 - 2010

• Fundamentals of Stream Processing: Application Design, Systems, and Analytics - This comprehensive, hands-on guide combining the fundamental building blocks and emerging research in stream processing is ideal for application designers, system builders, analytic developers, as well as students and researchers in the field.
• Stream Data Processing: A Quality of Service Perspective - Presents a new paradigm suitable for stream and complex event processing.
• Distributed Systems for fun and profit

Data Visualization

• awesome-awesomeness
• awesome
• list
• awesome-awesome-awesome
• awesome-analytics
• awesome-public-datasets
• awesome-graph-classification
• awesome-network-embedding
• awesome-community-detection
• awesome-decision-tree-papers
• awesome-fraud-detection-papers
• awesome-gradient-boosting-papers
• awesome-monte-carlo-tree-search-papers
• awesome-kafka
• Google Bigtable
• awesome-open-data-annotation

Business Intelligence

• Blazer - business intelligence made simple.
• Lightdash - The open source Looker alternative built on dbt
• Metabase - The simplest, fastest way to get business intelligence and analytics to everyone in your company.
• Pentaho - business intelligence platform.
• SparklineData SNAP - modern B.I platform powered by Apache Spark.
• Microsoft - business intelligence software and platform.

Columnar Databases

• EventQL - a distributed, column-oriented database built for large-scale event collection and analytics.
• IndexR - an open-source columnar storage format for fast & realtime analytic with big data.
• LocustDB - an experimental analytics database aiming to set a new standard for query performance on commodity hardware.
• Actian Vector - column-oriented analytic database.
• SQream DB - A GPU powered big data database, designed for analytics and data warehousing, with ANSI-92 compliant SQL, suitable for data sets from 10TB to 1PB.

Data Ingestion

• Apache Pulsar - a distributed pub-sub messaging platform with a very flexible messaging model and an intuitive client API.
• Facebook Scribe - streamed log data aggregator.
• Gazette - Distributed streaming infrastructure built on cloud storage which makes it easy to mix and match batch and streaming paradigms.
• Heka - open source stream processing software system.
• HIHO - framework for connecting disparate data sources with Hadoop.
• Kestrel - distributed message queue system.
• LinkedIn Kamikaze - utility package for compressing sorted integer arrays.
• Netflix Suro - log agregattor like Storm and Samza based on Chukwa.
• Pinterest Secor - is a service implementing Kafka log persistance.
• Skizze - sketch data store to deal with all problems around counting and sketching using probabilistic data-structures.
• StreamSets Data Collector - continuous big data ingest infrastructure with a simple to use IDE.
• RudderStack - an open source customer data infrastructure (segment, mParticle alternative) written in go.
• Zilla - An API gateway built for event-driven architectures and streaming that supports standard protocols such as HTTP, SSE, gRPC, MQTT and the native Kafka protocol.
• LinkedIn White Elephant - log aggregator and dashboard.
• Linkedin Gobblin - linkedin's universal data ingestion framework.
• Bruin - end-to-end data pipeline tool combining ingestion, transformations, and data quality checks.
• ingestr - CLI tool for copying data between sources and destinations.
• Duckle - open-source visual ETL/ELT platform built on DuckDB with connectors, data quality checks, and lineage.

Data Visualization

• Airpal - Web UI for PrestoDB.
• Arbor - graph visualization library using web workers and jQuery.
• Banana - visualize logs and time-stamped data stored in Solr. Port of Kibana.
• Bloomery - Web UI for Impala.
• CartoDB - open-source or freemium hosting for geospatial databases with powerful front-end editing capabilities and a robust API.
• chartd - responsive, retina-compatible charts with just an img tag.
• Chartist.js - another open source HTML5 Charts visualization.
• Crossfilter - JavaScript library for exploring large multivariate datasets in the browser. Works well with dc.js and d3.js.
• Cubism - JavaScript library for time series visualization.
• DC.js - Dimensional charting built to work natively with crossfilter rendered using d3.js. Excellent for connecting charts/additional metadata to hover events in D3.
• D3.compose - Compose complex, data-driven visualizations from reusable charts and components.
• Dash - Analytical Web Apps for Python, R, Julia, and Jupyter. Built on top of plotly, no JS required
• Envisionjs - dynamic HTML5 visualization.
• Freeboard - pen source real-time dashboard builder for IOT and other web mashups.
• Gephi - An award-winning open-source platform for visualizing and manipulating large graphs and network connections. It's like Photoshop, but for graphs. Available for Windows and Mac OS X.
• Graphite - scalable Realtime Graphing.
• Lumify - open source big data analysis and visualization platform
• Matplotlib - plotting with Python.
• Peity - Progressive SVG bar, line and pie charts.
• Plotly.js
• Redash - open-source platform to query and visualize data.
• Sigma.js - JavaScript library dedicated to graph drawing.
• Vega - a visualization grammar.
• Zeppelin - a notebook-style collaborative data analysis.
• DataSphere Studio - one-stop data application development management portal.
• Crossfilter - JavaScript library for exploring large multivariate datasets in the browser. Works well with dc.js and d3.js.
• Recline - simple but powerful library for building data applications in pure Javascript and HTML.
• Superset - a data exploration platform designed to be visual, intuitive and interactive, making it easy to slice, dice and visualize data and perform analytics at the speed of thought.
• Echarts - Baidus enterprise charts.
• NVD3 - chart components for d3.js.
• Cytoscape - JavaScript library for visualizing complex networks.
• C3 - D3-based reusable chart library

Distributed Filesystem

• Ambry - a distributed object store that supports storage of trillion of small immutable objects as well as billions of large objects.
• Google GFS - distributed filesystem.
• Seaweed-FS - simple and highly scalable distributed file system.
• Baidu File System - distributed filesystem.
• Disco DDFS - distributed filesystem.
• Alluxio - reliable file sharing at memory speed across cluster frameworks.
• Lustre file system - high-performance distributed filesystem.
• JuiceFS - distributed POSIX file system built on object storage.
• Red Hat GlusterFS - scale-out network-attached storage file system.

Distributed Index

• Pilosa

Distributed Programming

• AddThis Hydra - distributed data processing and storage system originally developed at AddThis.
• AMPLab SIMR - run Spark on Hadoop MapReduce v1.
• Baidu Bigflow - an interface that allows for writing distributed computing programs providing lots of simple, flexible, powerful APIs to easily handle data of any scale.
• Concurrent Cascading - framework for data management/analytics on Hadoop.
• Damballa Parkour - MapReduce library for Clojure.
• Datasalt Pangool - alternative MapReduce paradigm.
• Facebook Peregrine - Map Reduce framework.
• Kite - is a set of libraries, tools, examples, and documentation focused on making it easier to build systems on top of the Hadoop ecosystem.
• Netflix PigPen - map-reduce for Clojure which compiles to Apache Pig.
• Nokia Disco - MapReduce framework developed by Nokia.
• Pydoop - Python MapReduce and HDFS API for Hadoop.
• Ray - A fast and simple framework for building and running distributed applications.
• Rackerlabs Blueflood - multi-tenant distributed metric processing system
• Skale - High performance distributed data processing in NodeJS.
• Stratosphere - general purpose cluster computing framework.
• streamsx.topology - Libraries to enable building IBM Streams application in Java, Python or Scala.
• Tuktu - Easy-to-use platform for batch and streaming computation, built using Scala, Akka and Play!
• Twitter Heron - Heron is a realtime, distributed, fault-tolerant stream processing engine from Twitter replacing Storm.
• Twitter Scalding - Scala library for Map Reduce jobs, built on Cascading.
• Twitter Summingbird - Streaming MapReduce with Scalding and Storm, by Twitter.
• Wallaroo - The ultrafast and elastic data processing engine. Big or fast data - no fuss, no Java needed.
• AMPLab SIMR - run Spark on Hadoop MapReduce v1.
• Onyx - Distributed computation for the cloud.
• Pydoop - Python MapReduce and HDFS API for Hadoop.
• Cheetah - High Performance, Custom Data Warehouse on Top of MapReduce.
• Cascalog - data processing and querying library.

Document Data Model

• jumboDB - document oriented datastore over Hadoop.

Embedded Databases

• HanoiDB - Erlang LSM BTree Storage.
• LevelDB - a fast key-value storage library written at Google that provides an ordered mapping from string keys to string values.

Frameworks

• Bistro - general-purpose data processing engine for both batch and stream analytics. It is based on a novel data model, which represents data via functions and processes data via column operations as opposed to having only set operations in conventional approaches like MapReduce or SQL.
• Tigon - High Throughput Real-time Stream Processing Framework.
• Polyaxon - A platform for reproducible and scalable machine learning and deep learning.
• Smooks - An extensible Java framework for building XML and non-XML (CSV, EDI, Java, etc...) streaming applications.
• Numaflow - Kubernetes-native stream processing platform.
• Pachyderm - Pachyderm is a data storage platform built on Docker and Kubernetes to provide reproducible data processing and analysis.

Graph Data Model

• AgensGraph - a new generation multi-model graph database for the modern complex data environment.
• DGraph - A scalable, distributed, low latency, high throughput graph database aimed at providing Google production level scale and throughput, with low enough latency to be serving real time user queries, over terabytes of structured data.
• EliasDB - a lightweight graph based database that does not require any third-party libraries.
• GCHQ Gaffer - Gaffer by GCHQ is a framework that makes it easy to store large-scale graphs in which the nodes and edges have statistics.
• Google Cayley - open-source graph database.
• Google Pregel - graph processing framework.
• Gremlin - graph traversal Language.
• Infovore - RDF-centric Map/Reduce framework.
• Microsoft Graph Engine - a distributed in-memory data processing engine, underpinned by a strongly-typed in-memory key-value store and a general distributed computation engine.
• Phoebus - framework for large scale graph processing.
• Titan - distributed graph database, built over Cassandra.
• Twitter FlockDB - distributed graph database.
• OrientDB - document and graph database.
• Actionbase - a database for user interactions (likes, views, follows) with precomputed reads, supports HBase.
• AgensGraph - transactional graph database based on PostgreSQL.

Interesting Papers

2001 - 2010

• 2010 - Google - Large-scale Incremental Processing Using Distributed Transactions and notifications base of Percolator and Caffeine.
• 2010 - Google - Dremel: Interactive Analysis of Web-Scale Datasets.
• 2010 - Yahoo - S4: Distributed Stream Computing Platform.
• 2009 - HadoopDB: An Architectural Hybrid of MapReduce and DBMS Technologies for Analytical Workloads.
• 2006 - Google - The Chubby lock service for loosely-coupled distributed systems.
• 2004 - Google - MapReduce: Simplied Data Processing on Large Clusters.
• 2007 - Amazon - Dynamo: Amazon’s Highly Available Key-value Store.

2011 - 2012

• 2012 - Twitter - The Unified Logging Infrastructure
• 2012 - Google - Processing a trillion cells per mouse click.
• 2012 - Google - Spanner: Google’s Globally-Distributed Database.
• 2011 - Google - Megastore: Providing Scalable, Highly Available Storage for Interactive Services.
• 2012 - Microsoft - Paxos Made Parallel.

2013 - 2014

• 2014 - Stanford - Mining of Massive Datasets.
• 2013 - Google - HyperLogLog in Practice: Algorithmic Engineering of a State of The Art Cardinality Estimation Algorithm.
• 2013 - Metamarkets - Druid: A Real-time Analytical Data Store.
• 2013 - Google - F1: A Distributed SQL Database That Scales.
• 2013 - Facebook - Scuba: Diving into Data at Facebook.
• 2013 - Google - Online, Asynchronous Schema Change in F1.

2015 - 2016

• 2015 - Facebook - One Trillion Edges: Graph Processing at Facebook-Scale.

Internet of things and sensor data

• Apache Edgent (Incubating) - a programming model and micro-kernel style runtime that can be embedded in gateways and small footprint edge devices enabling local, real-time, analytics on the edge devices.
• NetLytics - Analytics platform to process network data on Spark.
• NetLytics - Analytics platform to process network data on Spark.

Key Map Data Model

• Distinguishing two major types of Column Stores
• Baidu Tera - an Internet-scale database, inspired by BigTable.
• Google BigTable - column-oriented distributed datastore.
• InfiniDB - is accessed through a MySQL interface and use massive parallel processing to parallelize queries.
• ScyllaDB - column-oriented distributed datastore written in C++, totally compatible with Apache Cassandra.
• Hypertable - column-oriented distributed datastore, inspired by BigTable.

Key-value Data Model

• Bolt - an embedded key-value database for Go.
• BTDB - Key Value Database in .Net with Object DB Layer, RPC, dynamic IL and much more
• BuntDB - a fast, embeddable, in-memory key/value database for Go with custom indexing and geospatial support.
• Edis - is a protocol-compatible Server replacement for Redis.
• ElephantDB - Distributed database specialized in exporting data from Hadoop.
• GhostDB - a distributed, in-memory, general purpose key-value data store that delivers microsecond performance at any scale.
• Graviton - a simple, fast, versioned, authenticated, embeddable key-value store database in pure Go(lang).
• HyperDex - a scalable, next generation key-value and document store with a wide array of features, including consistency, fault tolerance and high performance.