Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

awesome-bigdata

A curated list of awesome big data frameworks, resources and other awesomeness.
https://github.com/eric-erki/awesome-bigdata

Last synced: 1 day ago
JSON representation

  • NewSQL Databases

    • Pivotal GemFire XD - Low-latency, in-memory, distributed SQL data store. Provides SQL interface to in-memory table data, persistable in HDFS.
    • SAP HANA - is an in-memory, column-oriented, relational database management system.
    • Sky - database used for flexible, high performance analysis of behavioral data.
    • Sky - database used for flexible, high performance analysis of behavioral data.
    • ActorDB - a distributed SQL database with the scalability of a KV store, while keeping the query capabilities of a relational database.
    • Cockroach - Scalable, Geo-Replicated, Transactional Datastore.
    • Comdb2 - a clustered RDBMS built on optimistic concurrency control techniques.
    • Haeinsa - linearly scalable multi-row, multi-table transaction library for HBase based on Percolator.
    • InfiniSQL - infinity scalable RDBMS.
    • NuoDB - SQL/ACID compliant distributed database.
    • SenseiDB - distributed, realtime, semi-structured database.
    • TiDB - TiDB is a distributed SQL database. Inspired by the design of Google F1.
    • SenseiDB - distributed, realtime, semi-structured database.
    • SymmetricDS - open source software for both file and database synchronization.
    • Actian Ingres - commercially supported, open-source SQL relational database management system.
    • H-Store - is an experimental main-memory, parallel database management system that is optimized for on-line transaction processing (OLTP) applications.
    • NuoDB - SQL/ACID compliant distributed database.
    • Oracle TimesTen in-Memory Database - in-memory, relational database management system with persistence and recoverability.

  • PostgreSQL forks and evolutions

    • RecDB - Open Source Recommendation Engine Built Entirely Inside PostgreSQL.
    • Stado - open source MPP database system solely targeted at data warehousing and data mart applications.
    • Yahoo Everest - multi-peta-byte database / MPP derived by PostgreSQL.
    • Postgres-XL - Scalable Open Source PostgreSQL-based Database Cluster.
    • HadoopDB - hybrid of MapReduce and DBMS.
    • IBM Netezza - high-performance data warehouse appliances.
    • TimescaleDB - An open-source time-series database optimized for fast ingest and complex queries

  • RDBMS

  • Scheduling

    • Linkedin Azkaban - batch workflow job scheduler.
    • Apache Airflow - a platform to programmatically author, schedule and monitor workflows.
    • Chronos - distributed and fault-tolerant scheduler.
    • Schedoscope - Scala DSL for agile scheduling of Hadoop jobs.
    • Sparrow - scheduling platform.
    • Azure Data Factory - cloud-based pipeline orchestration for on-prem, cloud and HDInsight
    • ElasticSearch - Search and analytics engine based on Apache Lucene.
    • Enigma.io
    • Facebook Unicorn - social graph search platform.
    • Lily HBase Indexer - quickly and easily search for any content stored in HBase.
    • LinkedIn Galene - search architecture at LinkedIn.
    • Sphinx Search Server - fulltext search engine.
    • Elassandra - is a fork of Elasticsearch modified to run on top of Apache Cassandra in a scalable and resilient peer-to-peer architecture.
    • LinkedIn Bobo - is a Faceted Search implementation written purely in Java, an extension to Apache Lucene.
    • LinkedIn Cleo - is a flexible software library for enabling rapid development of partial, out-of-order and real-time typeahead search.
    • LinkedIn Zoie - is a realtime search/indexing system written in Java.
    • MG4J - MG4J (Managing Gigabytes for Java) is a full-text search engine for large document collections written in Java. It is highly customisable, high-performance and provides state-of-the-art features and new research algorithms.
    • Sphinx Search Server - fulltext search engine.
    • Facebook Faiss - is a library for efficient similarity search and clustering of dense vectors. It contains algorithms that search in sets of vectors of any size, up to ones that possibly do not fit in RAM. It also contains supporting code for evaluation and parameter tuning. Faiss is written in C++ with complete wrappers for Python/numpy.
    • Annoy - is a C++ library with Python bindings to search for points in space that are close to a given query point. It also creates large read-only file-based data structures that are mmapped into memory so that many processes may share the same data.
    • LinkedIn Bobo - is a Faceted Search implementation written purely in Java, an extension to Apache Lucene.
    • LinkedIn Cleo - is a flexible software library for enabling rapid development of partial, out-of-order and real-time typeahead search.
    • Vespa - is an engine for low-latency computation over large data sets. It stores and indexes your data such that queries, selection and processing over the data can be performed at serving time.

  • Security

    • BDA - The vulnerability detector for Hadoop and Spark
    • BDA - The vulnerability detector for Hadoop and Spark

  • Service Programming

    • Google Chubby - a lock service for loosely-coupled distributed systems.
    • OpenMPI - message passing framework.
    • Hydrosphere Mist - a service for exposing Apache Spark analytics jobs and machine learning models as realtime, batch or reactive web services.
    • Spotify Luigi - a Python package for building complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization, handling failures, command line integration, and much more.
    • Spring XD - distributed and extensible system for data ingestion, real time analytics, batch processing, and data export.
    • Twitter Elephant Bird - libraries for working with LZOP-compressed data.
    • Twitter Finagle - asynchronous network stack for the JVM.
    • Serf - decentralized solution for service discovery and orchestration.
    • Spring XD - distributed and extensible system for data ingestion, real time analytics, batch processing, and data export.
    • Mara - A lightweight opinionated ETL framework, halfway between plain scripts and Apache Airflow

  • SQL-like processing

    • Actian SQL for Hadoop - high performance interactive SQL access to all Hadoop data.
    • Apache HCatalog - table and storage management layer for Hadoop.
    • Aster Database - SQL-like analytic processing for MapReduce.
    • Dremio - an open-source, SQL-like Data-as-a-Service Platform based on Apache Arrow.
    • Facebook PrestoDB - distributed SQL query engine.
    • Spark Catalyst - is a Query Optimization Framework for Spark and Shark.
    • Splice Machine - a full-featured SQL-on-Hadoop RDBMS with ACID transactions.
    • Trafodion - enterprise-class SQL-on-HBase solution targeting big data transactional or operational workloads.
    • Cloudera Impala - framework for interactive analysis, Inspired by Dremel.
    • Concurrent Lingual - SQL-like query language for Cascading.
    • PipelineDB - an open-source relational database that runs SQL queries continuously on streams, incrementally storing results in tables.
    • Pivotal HDB - SQL-like data warehouse system for Hadoop.
    • RainstorDB - database for storing petabyte-scale volumes of structured and semi-structured data.
    • Actian SQL for Hadoop - high performance interactive SQL access to all Hadoop data.
    • PipelineDB - an open-source relational database that runs SQL queries continuously on streams, incrementally storing results in tables.
    • Actian SQL for Hadoop - high performance interactive SQL access to all Hadoop data.
    • Apache Phoenix - SQL skin over HBase.
    • RainstorDB - database for storing petabyte-scale volumes of structured and semi-structured data.

  • System Deployment

    • Brooklyn - library that simplifies application deployment and management.
    • Buildoop - Similar to Apache BigTop based on Groovy language.
    • Google Borg - job scheduling and monitoring system.
    • Google Omega - job scheduling and monitoring system.
    • Kubernetes - a system for automating deployment, scaling, and management of containerized applications.
    • Apache Slider - is a YARN application to deploy existing distributed applications on YARN.
    • Brooklyn - library that simplifies application deployment and management.
    • Buildoop - Similar to Apache BigTop based on Groovy language.
    • Marathon - Mesos framework for long-running services.
    • Apache Slider - is a YARN application to deploy existing distributed applications on YARN.
    • Cloudera HUE - web application for interacting with Hadoop.

  • Time-Series Databases

    • Axibase Time Series Database - Integrated time series database on top of HBase with built-in visualization, rule-engine and SQL support.
    • InfluxDB - distributed time series database.
    • M3DB - a distributed time series database that can be used for storing realtime metrics at long retention.
    • Prometheus - a time series database and service monitoring system.
    • Rhombus - series object store for Cassandra that handles all the complexity of building wide row indexes.
    • Chronix - a time series storage built to store time series highly compressed and for fast access times.
    • Cube - uses MongoDB to store time series data.
    • Heroic - is a scalable time series database based on Cassandra and Elasticsearch.
    • IronDB - scalable, general-purpose time series database.
    • Kairosdb - similar to OpenTSDB but allows for Cassandra.
    • Newts - a time series database based on Apache Cassandra.
    • Beringei - Facebook's in-memory time-series database.
    • Akumuli - series database. It can be used to capture, store and process time-series data in real-time. The word "akumuli" can be translated from esperanto as "accumulate".
    • Dalmatiner DB
    • Blueflood
    • Timely
    • VictoriaMetrics - fast, scalable and resource-effective open-source TSDB compatible with Prometheus. Single-node and cluster versions included
    • Chronix - a time series storage built to store time series highly compressed and for fast access times.
    • TDengine - a time series database in C utilizing unique features of IoT to improve read/write throughput and reduce space needed to store data
    • Druid
    • IronDB - scalable, general-purpose time series database.
    • Thanos - Thanos is a set of components to create a highly available metric system with unlimited storage capacity using multiple (existing) Prometheus deployments.

  • Videos

Programming Languages
Java 24 Go 13 Python 13 JavaScript 13 Scala 10 C++ 9 Erlang 5 C 4 Ruby 3 Clojure 3
Categories
Interesting Papers 47 Distributed Programming 46 Data Visualization 40 Machine Learning 34 Books 32 Applications 29 NewSQL Databases 25 Key-value Data Model 24 Time-Series Databases 22 Graph Data Model 21 Data Ingestion 19 SQL-like processing 18 Business Intelligence 17 Search engine and framework 17 Distributed Filesystem 14 Columnar Databases 11 MySQL forks and evolutions 11 System Deployment 11 Service Programming 10 Key Map Data Model 9 Internet of things and sensor data 9 PostgreSQL forks and evolutions 7 Document Data Model 7 Scheduling 6 Benchmarking 6 Frameworks 5 Interesting Readings 4 RDBMS 4 Embedded Databases 4 Memcached forks and evolutions 3 Security 2 Videos 2 Distributed Index 1
Sub Categories
2001 - 2010 31 Data Visualization 18 2013 - 2014 15 2011 - 2012 15 2015 - 2016 2
Keywords
database 15 machine-learning 10 deep-learning 8 python 8 data-science 8 go 6 analytics 6 data-visualization 5 visualization 5 graph 5 sql 4 geospatial 4 time-series 4 mysql 4 tensorflow 4 spark 4 data-analysis 4 awesome 4 awesome-list 4 nosql 3 big-data 3 distributed 3 golang 3 metrics 3 in-memory 3 kubernetes 3 java 3 c-plus-plus 3 random-forest 3 kafka 3 graph-database 3 classifier 3 reinforcement-learning 3 pytorch 3 gradient-boosting 3 data 3 business-intelligence 3 decision-tree 2 ml 2 clustering 2 classification-trees 2 xgboost 2 catboost 2 data-mining 2 d3 2 webgl 2 hadoop 2 timeseries-database 2 timeseries 2 javascript 2