data-engineering-collection

A collection of awesome software, libraries, Learning Tutorials, documents, books, resources and interesting stuff about Big Data Science & Engineering
https://github.com/exajobs/data-engineering-collection

Last synced: 3 days ago
JSON representation

Applications
- Apache Tika - content analysis toolkit.
- Hunk - Splunk analytics for Hadoop.
- Imhotep - Large scale analytics platform by indeed.
- Indicative - Web & mobile analytics tool, with data warehouse (AWS, BigQuery) integration.
- Jupyter - Notebook and project application for interactive data science and scientific computing across all programming languages.
- MADlib - data-processing library of an RDBMS to analyze data.
- Qubole - auto-scaling Hadoop cluster, built-in data connectors.
- Splunk - analyzer for machine-generated data.
- Sumo Logic - cloud based analyzer for machine-generated data.
- PivotalR - R on Pivotal HD / HAWQ and PostgreSQL.
- 411 - an web application for alert management resulting from scheduled searches into Elasticsearch.
- Adobe spindle - Next-generation web analytics processing with Scala, Spark, and Parquet.
- Argus - Time series monitoring and alerting platform.
- AthenaX - a streaming analytics platform that enables users to run production-quality, large scale streaming analytics using Structured Query Language (SQL).
- Atlas - a backend for managing dimensional time series data.
- ElastAert - ElastAlert is a simple framework for alerting on anomalies, spikes, or other patterns of interest from data in ElasticSearch.
- Eventhub - open source event analytics platform.
- Hermes - asynchronous message broker built on top of Kafka.
- Hunk - Splunk analytics for Hadoop.
- Kapacitor - an open source framework for processing, monitoring, and alerting on time series data.
- PivotalR - R on Pivotal HD / HAWQ and PostgreSQL.
- Snowplow - enterprise-strength web and event analytics, powered by Hadoop, Kinesis, Redshift and Postgres.
- SparkR - R frontend for Spark.
- Rakam - open-source real-time custom analytics platform powered by Postgresql, Kinesis and PrestoDB.
- Apache Metron - a platform that integrates a variety of open source big data technologies in order to offer a centralized tool for security monitoring and analysis.
- Apache OODT - capturing, processing and sharing of data for NASA's scientific archives.
- Eclipse BIRT - Eclipse-based reporting system.
- HASH - open source simulation and visualization platform.
- MADlib - data-processing library of an RDBMS to analyze data.
- Kylin - open source Distributed Analytics Engine from eBay.
- Talend - unified open source environment for YARN, Hadoop, HBASE, Hive, HCatalog & Pig.
Benchmarking
- Apache Hadoop Benchmarking - micro-benchmarks for testing Hadoop performances.
- Berkeley SWIM Benchmark - real-world big data workload benchmark.
- PUMA Benchmarking - benchmark suite for MapReduce applications.
- Yahoo Gridmix3 - Hadoop cluster benchmarking from Yahoo engineer team.
- Deeplearning4j Benchmarks
- Intel HiBench - a Hadoop benchmark suite.
Books
- 2001 - 2010
  - Streaming Data - Streaming Data introduces the concepts and requirements of streaming and real-time data systems.
  - Storm Applied - Storm Applied is a practical guide to using Apache Storm for the real-world tasks associated with processing and analyzing real-time data streams.
  - Fundamentals of Stream Processing: Application Design, Systems, and Analytics - This comprehensive, hands-on guide combining the fundamental building blocks and emerging research in stream processing is ideal for application designers, system builders, analytic developers, as well as students and researchers in the field.
  - Stream Data Processing: A Quality of Service Perspective - Presents a new paradigm suitable for stream and complex event processing.
  - Unified Log Processing - Unified Log Processing is a practical guide to implementing a unified log of event streams (Kafka or Kinesis) in your business
  - Kafka Streams in Action - Kafka Streams in Action teaches you everything you need to know to implement stream processing on data flowing into your Kafka platform, allowing you to focus on getting more from your data without sacrificing time or effort.
  - Big Data - Big Data teaches you to build big data systems using an architecture that takes advantage of clustered hardware along with new tools designed specifically to capture and analyze web-scale data.
  - Spark in Action - in-action-second-edition) - Spark in Action teaches you the theory and skills you need to effectively handle batch and streaming data using Spark. Fully updated for Spark 2.0.
  - Kafka in Action - Kafka in Action is a fast-paced introduction to every aspect of working with Kafka you need to really reap its benefits.
  - Fusion in Action - Fusion in Action teaches you to build a full-featured data analytics pipeline, including document and data search and distributed data clustering.
  - Reactive Data Handling - Reactive Data Handling is a collection of five hand-picked chapters, selected by Manuel Bernhardt, that introduce you to building reactive applications capable of handling real-time processing with large data loads--free eBook!
  - Grokking Streaming Systems - Grokking Streaming Systems helps you unravel what streaming systems are, how they work, and whether they’re right for your business. Written to be tool-agnostic, you’ll be able to apply what you learn no matter which framework you choose.
  - Distributed Systems for fun and profit
  - Graph-Powered Machine Learning - Alessandro Negro. Combine graph theory and models to improve machine learning projects
  - Data Science at Scale with Python and Dask - Data Science at Scale with Python and Dask teaches you how to build distributed data projects that can handle huge amounts of data.
- Contributing
  - pull request
- Data Visualization
Business Intelligence
- BIME Analytics - business intelligence platform in the cloud.
- Dekart - Large scale geospatial analytics for Google BigQuery based on Kepler.gl.
- GoodData - platform for data products and embedded analytics.
- Jaspersoft - powerful business intelligence suite.
- Jedox Palo - customisable Business Intelligence platform.
- Jethrodata - Interactive Big Data Analytics.
- Microstrategy - software platforms for business intelligence, mobile intelligence, and network applications.
- Qlik - business intelligence and analytics platform.
- Redash - Open source business intelligence platform, supporting multiple data sources and planned queries.
- Saiku Analytics - Open source analytics platform.
- Knowage - open source business intelligence platform. (former [SpagoBi](http://www.spagobi.org/))
- SparklineData SNAP - modern B.I platform powered by Apache Spark.
- Tableau - business intelligence platform.
- Blazer - business intelligence made simple.
- Metabase - The simplest, fastest way to get business intelligence and analytics to everyone in your company.
- Numeracy - Fast, clean SQL client and business intelligence.
- Pentaho - business intelligence platform.
- Zoomdata - Big Data Analytics.
- Microsoft - business intelligence software and platform.
- datapine - self-service business intelligence tool in the cloud.
- intermix.io - Performance Monitoring for Amazon Redshift
- Qlik - business intelligence and analytics platform.
`Columnar Databases`
- Amazon Redshift - Amazon's cloud offering, also based on a columnar datastore backend.
- MonetDB - column store database.
- EventQL - a distributed, column-oriented database built for large-scale event collection and analytics.
- EventQL - a distributed, column-oriented database built for large-scale event collection and analytics.
- Pivotal Greenplum - purpose-built, dedicated analytic data warehouse that offers a columnar engine as well as a traditional row-based one.
- Vertica - is designed to manage large, fast-growing volumes of data and provide very fast query performance when used for data warehouses.
- IndexR - an open-source columnar storage format for fast & realtime analytic with big data.
- LocustDB - an experimental analytics database aiming to set a new standard for query performance on commodity hardware.
- ClickHouse - an open-source column-oriented database management system that allows generating analytical data reports in real time.
- Columnar Storage - an explanation of what columnar storage is and when you might want it.
- Actian Vector - column-oriented analytic database.
- SQream DB - A GPU powered big data database, designed for analytics and data warehousing, with ANSI-92 compliant SQL, suitable for data sets from 10TB to 1PB.
- Google BigQuery - Google's cloud offering backed by their pioneering work on Dremel.
Databases
- RavenDB
- RethinkDB - source database for the realtime web.
- AWS DynamoDB - digit millisecond latency at any scale.
- Redis - value cache and store.
- Neo4j
- Riak
- Cassandra Calculator
- HBase
- Couchbase
- Titan - machine cluster.
- Apache Geode - memory database for scale-out applications.
- Percona XtraBackup
- RQLite
- TiDB
- Percona XtraBackup
- mysql_utils
- SSDB
- IonDB - value store for microcontroller and IoT applications
- ScyllaDB
- FiloDB
- Vertica
- MemDB - Memory Database (based on MongoDB)
- OrientDB
- InfluxDB - time analytics.
- OpenTSDB
- kairosdb
- Heroic
- Akumuli - series database. It can be used to capture, store and process time-series data in real-time. The word "akumuli" can be translated from esperanto as "accumulate".
- Dalmatiner DB
- Blueflood
- Timely
- Tarantool - memory database and application server.
- Snappydata
- TimescaleDB - series SQL database providing fast analytics, scalability, with automated data management on a proven storage engine.
- ClickHouse
- Kyoto Tycoon - value database, built for high-performance and concurrency
- Druid
- CCM
- GreenPlum
- Percona Server for MongoDB - in replacement for the MongoDB® Community Edition that includes enterprise-grade features and functionality.
- Riak
- Riak-TS - grade NoSQL time series database optimized specifically for IoT and Time Series data
Data Ingestion
- Amazon Kinesis - real-time processing of streaming data at massive scale.
- Amazon Web Services Glue - serverless fully managed extract, transform, and load (ETL) service
- Apache NiFi - Apache NiFi is an integrated data logistics platform for automating the movement of data between disparate systems.
- Google Photon - geographically distributed system for joining multiple continuously flowing streams of data in real-time with high scalability and low latency.
- Kestrel - distributed message queue system.
- Alooma - data pipeline as a service enabling moving data sources such as MySQL into data warehouses.
- Apache Pulsar - a distributed pub-sub messaging platform with a very flexible messaging model and an intuitive client API.
- Facebook Scribe - streamed log data aggregator.
- Gazette - Distributed streaming infrastructure built on cloud storage which makes it easy to mix and match batch and streaming paradigms.
- Heka - open source stream processing software system.
- HIHO - framework for connecting disparate data sources with Hadoop.
- Logstash - a tool for managing events and logs.
- Netflix Suro - log agregattor like Storm and Samza based on Chukwa.
- Pinterest Secor - is a service implementing Kafka log persistance.
- Skizze - sketch data store to deal with all problems around counting and sketching using probabilistic data-structures.
- StreamSets Data Collector - continuous big data ingest infrastructure with a simple to use IDE.
- RudderStack - an open source customer data infrastructure (segment, mParticle alternative) written in go.
- redpanda - A Kafka® replacement for mission critical systems; 10x faster. Written in C++.
- Census - A reverse ETL product that let you sync data from your data warehouse to SaaS Applications. No engineering favors required—just SQL.
- LinkedIn White Elephant - log aggregator and dashboard.
- Linkedin Gobblin - linkedin's universal data ingestion framework.
- Apache Chukwa - data collection system.
- Apache Flume - service to manage large amount of log data.
- Apache Sqoop - tool to transfer data between Hadoop and a structured datastore.
- Embulk - open-source bulk data loader that helps data transfer between various databases, storages, file formats, and cloud services.
- Fluentd - tool to collect events and logs.
- Google Photon - geographically distributed system for joining multiple continuously flowing streams of data in real-time with high scalability and low latency.
- LinkedIn Kamikaze - utility package for compressing sorted integer arrays.
Data Visualization
- chartd - responsive, retina-compatible charts with just an img tag.
- D3 - javaScript library for manipulating documents.
- DevExtreme React Chart - High-performance plugin-based React chart for Bootstrap and Material Design.
- FnordMetric - write SQL queries that return SVG charts rather than tables
- Grafana - graphite dashboard frontend, editor and graph composer.
- Graphite - scalable Realtime Graphing.
- Highcharts - simple and flexible charting API.
- Metricsgraphic.js - a library built on top of D3 that is optimized for time-series data
- Zing Charts - JavaScript charting library for big data.
- Lumify - open source big data analysis and visualization platform
- Airpal - Web UI for PrestoDB.
- Arbor - graph visualization library using web workers and jQuery.
- Banana - visualize logs and time-stamped data stored in Solr. Port of Kibana.
- Bloomery - Web UI for Impala.
- CartoDB - open-source or freemium hosting for geospatial databases with powerful front-end editing capabilities and a robust API.
- chartd - responsive, retina-compatible charts with just an img tag.
- Chartist.js - another open source HTML5 Charts visualization.
- Crossfilter - JavaScript library for exploring large multivariate datasets in the browser. Works well with dc.js and d3.js.
- Cubism - JavaScript library for time series visualization.
- DC.js - Dimensional charting built to work natively with crossfilter rendered using d3.js. Excellent for connecting charts/additional metadata to hover events in D3.
- D3.compose - Compose complex, data-driven visualizations from reusable charts and components.
- Dash - Analytical Web Apps for Python, R, Julia, and Jupyter. Built on top of plotly, no JS required
- Envisionjs - dynamic HTML5 visualization.

Programming Languages

Java 30 Python 19 Go 18 JavaScript 15 Scala 11 C++ 9 C 6 Erlang 5 Ruby 3 Rust 3

ecosyste.ms

Data

Tools

Indexes

Applications

Experiments

Awesome

data-engineering-collection

Applications

Benchmarking

Books

2001 - 2010

Contributing

Data Visualization

Business Intelligence

`Columnar Databases`

Databases

Data Ingestion

Data Visualization