Packages

Bioinformatics

• ADAM - commit/bigdatagenomics/adam.svg"> - Set of tools designed to analyse genomics data.
• Hail - commit/hail-is/hail.svg"> - Genetic analysis framework.

Data quality

• deequ - commit/awslabs/deequ.svg"> - Deequ is a library built on top of Apache Spark for defining "unit tests for data", which measure data quality in large datasets.
• python-deequ - commit/awslabs/python-deequ.svg"> - Python API for Deequ.

General Purpose Libraries

• itachi - commit/yaooqinn/itachi.svg"> - A library that brings useful functions from modern database management systems to Apache Spark.
• spark-daria - commit/mrpowers-io/spark-daria.svg"> - A Scala library with essential Spark functions and extensions to make you more productive.
• Joblib Apache Spark Backend - commit/joblib/joblib-spark.svg"> - joblib backend for running tasks on Spark clusters.
• quinn - commit/mrpowers-io/quinn.svg"> - A native PySpark implementation of spark-daria.
• Apache DataFu - commit/apache/datafu.svg"> - A library of general purpose functions and UDF's.

GIS

• Apache Sedona - commit/apache/incubator-sedona.svg"> - Cluster computing system for processing large-scale spatial data.

Graph Processing

• GraphFrames - commit/graphframes/graphframes.svg"> - Data frame based graph API.

Interfaces

• Apache Beam - commit/apache/beam.svg"> - Unified data processing engine supporting both batch and streaming applications. Apache Spark is one of the supported execution environments.
• Koalas - commit/databricks/koalas.svg"> - Pandas DataFrame API on top of Apache Spark.

Language Bindings

• sparkle - commit/tweag/sparkle.svg"> - Haskell on Apache Spark.
• Kotlin for Apache Spark - commit/Kotlin/kotlin-spark-api.svg"> - Kotlin API bindings and extensions.
• .NET for Apache Spark - commit/dotnet/spark.svg"> - .NET bindings.
• spark-connect-rs - commit/sjrusso8/spark-connect-rs.svg"> - Rust bindings.
• spark-connect-go - commit/apache/spark-connect-go.svg"> - Golang bindings.
• spark-connect-csharp - commit/mdrakiburrahman/spark-connect-csharp.svg"> - C# bindings.

Machine Learning Extension

• Mahout Spark Bindings - linear algebra DSL and optimizer with R-like syntax.
• KeystoneML - Type safe machine learning pipelines with RDDs.
• ModelDB - commit/mitdbg/modeldb.svg"> - A system to manage machine learning models for spark.ml and scikit-learn .
• Sparkling Water - commit/h2oai/sparkling-water.svg"> - H2O interoperability layer.
• BigDL - commit/intel-analytics/BigDL.svg"> - Distributed Deep Learning library.
• MLeap - commit/combust/mleap.svg"> - Execution engine and serialization format which supports deployment of o.a.s.ml models without dependency on SparkSession.
• Microsoft ML for Apache Spark - commit/Azure/mmlspark.svg"> - A distributed ml library with support for LightGBM, Vowpal Wabbit, OpenCV, Deep Learning, Cognitive Services, and Model Deployment.

Middleware

• Livy - commit/apache/incubator-livy.svg"> - REST server with extensive language support (Python, R, Scala), ability to maintain interactive sessions and object sharing.
• spark-jobserver - commit/spark-jobserver/spark-jobserver.svg"> - Simple Spark as a Service which supports objects sharing using so called named objects. JVM only.
• Apache Toree - commit/apache/incubator-toree.svg"> - IPython protocol based middleware for interactive applications.
• Apache Kyuubi - commit/apache/kyuubi.svg"> - A distributed multi-tenant JDBC server for large-scale data processing and analytics, built on top of Apache Spark.

Monitoring

• Data Mechanics Delight - commit/datamechanics/delight.svg"> - Cross-platform monitoring tool (Spark UI / Spark History Server replacement).

Natural Language Processing

• spark-nlp - commit/JohnSnowLabs/spark-nlp.svg"> - Natural language processing library built on top of Apache Spark ML.

Notebooks and IDEs

• almond - commit/almond-sh/almond.svg"> - A scala kernel for Jupyter.
• Apache Zeppelin - commit/apache/zeppelin.svg"> - Web-based notebook that enables interactive data analytics with plugable backends, integrated plotting, and extensive Spark support out-of-the-box.
• Polynote - commit/polynote/polynote.svg"> - Polynote: an IDE-inspired polyglot notebook. It supports mixing multiple languages in one notebook, and sharing data between them seamlessly. It encourages reproducible notebooks with its immutable data model. Originating from Netflix.
• sparkmagic - commit/jupyter-incubator/sparkmagic.svg"> - Jupyter magics and kernels for working with remote Spark clusters, for interactively working with remote Spark clusters through Livy, in Jupyter notebooks.

SQL Data Sources

• serveral built-in Data Sources
• Spark XML - commit/databricks/spark-xml.svg"> - XML parser and writer.
• Mongo-Spark - commit/mongodb/mongo-spark.svg"> - Official MongoDB connector.
• Spark Cassandra Connector - commit/datastax/spark-cassandra-connector.svg"> - Cassandra support including data source and API and support for arbitrary queries.

Storage

• lakeFS - commit/treeverse/lakefs.svg"> - Integration with the lakeFS atomic versioned storage layer.
• Delta Lake - commit/delta-io/delta.svg"> - Storage layer with ACID transactions.
• Apache Hudi - commit/apache/hudi.svg"> - Upserts, Deletes And Incremental Processing on Big Data..
• Apache Iceberg - commit/apache/iceberg.svg"> - Upserts, Deletes And Incremental Processing on Big Data..

Streaming

• Apache Bahir - commit/apache/bahir.svg"> - Collection of the streaming connectors excluded from Spark 2.0 (Akka, MQTT, Twitter. ZeroMQ).

Testing

• spark-testing-base - commit/holdenk/spark-testing-base.svg"> - Collection of base test classes.
• spark-fast-tests - commit/mrpowers-io/spark-fast-tests.svg"> - A lightweight and fast testing framework.
• chispa - commit/MrPowers/chispa.svg"> - PySpark test helpers with beautiful error messages.

Utilities

• sparkly - commit/Tubular/sparkly.svg"> - Helpers & syntactic sugar for PySpark.
• Flintrock - commit/nchammas/flintrock.svg"> - A command-line tool for launching Spark clusters on EC2.
• Optimus - commit/ironmussa/Optimus.svg"> - Data Cleansing and Exploration utilities with the goal of simplifying data cleaning.

Web Archives

• Archives Unleashed Toolkit - commit/archivesunleashed/aut.svg"> - Open-source toolkit for analyzing web archives.

Workflow Management

• Cromwell - commit/broadinstitute/cromwell.svg"> - Workflow management system with Spark backend.

Resources

Books

• Learning Spark, 2nd Edition - Introduction to Spark API with Spark 3.0 covered. Good source of knowledge about basic concepts.
• Spark in Action - New book in the Manning's "in action" family with +400 pages. Starts gently, step-by-step and covers large number of topics. Free excerpt on how to setup Eclipse for Spark application development and how to bootstrap a new application using the provided Maven Archetype. You can find the accompanying GitHub repo here.
• Advanced Analytics with Spark - Useful collection of Spark processing patterns. Accompanying GitHub repository: sryza/aas.
• Mastering Apache Spark - Interesting compilation of notes by Jacek Laskowski. Focused on different aspects of Spark internals.

Docker Images

• apache/spark - Apache Spark Official Docker images.
• jupyter/docker-stacks/pyspark-notebook - PySpark with Jupyter Notebook and Mesos client.
• datamechanics/spark - An easy to setup Docker image for Apache Spark from Data Mechanics.
• sequenceiq/docker-spark - Yarn images from SequenceIQ.

Miscellaneous

• Apache Spark User List - spark-developers-list.1001551.n3.nabble.com/) - Mailing lists dedicated to usage questions and development topics respectively.
• Spark with Scala Gitter channel - "A place to discuss and ask questions about using Scala for Spark programming" started by @deanwampler.

MOOCS

• Data Science and Engineering with Apache Spark (edX XSeries) - Series of five courses (Introduction to Apache Spark, Distributed Machine Learning with Apache Spark, Big Data Analysis with Apache Spark, Advanced Apache Spark for Data Science and Data Engineering, Advanced Distributed Machine Learning with Apache Spark) covering different aspects of software engineering and data science. Python oriented.

Papers

• Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing - Paper introducing a core distributed memory abstraction.
• Spark SQL: Relational Data Processing in Spark - Paper introducing relational underpinnings, code generation and Catalyst optimizer.
• Large-Scale Intelligent Microservices - Microsoft paper that presents an Apache Spark-based micro-service orchestration framework that extends database operations to include web service primitives.
• Structured Streaming: A Declarative API for Real-Time Applications in Apache Spark - Structured Streaming is a new high-level streaming API, it is a declarative API based on automatically incrementalizing a static relational query.

Projects Using Spark

• PredictionIO - Machine Learning server for developers and data scientists to build and deploy predictive applications in a fraction of the time.
• Oryx 2 - Lambda architecture platform built on Apache Spark and Apache Kafka with specialization for real-time large scale machine learning.
• Photon ML - A machine learning library supporting classical Generalized Mixed Model and Generalized Additive Mixed Effect Model.
• Crossdata - Data integration platform with extended DataSource API and multi-user environment.

Workshops

• AMP Camp - Periodical training event organized by the UC Berkeley AMPLab. A source of useful exercise and recorded workshops covering different tools from the Berkeley Data Analytics Stack.