Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/okdp/spark-images
Collection of Apache Spark docker images for OKDP
https://github.com/okdp/spark-images
apache-spark big-data docker k8s-spark kubernetes spark-kubernetes spark-python spark-r spark-sql
Last synced: 3 months ago
JSON representation
Collection of Apache Spark docker images for OKDP
- Host: GitHub
- URL: https://github.com/okdp/spark-images
- Owner: OKDP
- License: apache-2.0
- Created: 2024-03-26T07:55:47.000Z (9 months ago)
- Default Branch: main
- Last Pushed: 2024-06-10T22:35:43.000Z (7 months ago)
- Last Synced: 2024-10-12T00:03:15.608Z (3 months ago)
- Topics: apache-spark, big-data, docker, k8s-spark, kubernetes, spark-kubernetes, spark-python, spark-r, spark-sql
- Language: Dockerfile
- Homepage: https://okdp.io
- Size: 84 KB
- Stars: 0
- Watchers: 5
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- License: LICENSE
Awesome Lists containing this project
README
[](https://github.com/okdp/spark-images/actions/workflows/ci.yml)
[](https://github.com/okdp/spark-images/releases/latest)
[](http://www.apache.org/licenses/LICENSE-2.0)
Collection of [Apache Spark](https://spark.apache.org/) docker images for [OKDP Platform](https://okdp.io/).
Currently, the images are built from the [Apache Spark project distribution](https://archive.apache.org/dist/spark) and the requirement may evolve to produce them from the [source code](https://github.com/apache/spark).
The image relashionship is described by the following diagram:
| Image | Description |
|:---------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `JRE` | The JRE LTS base image supported by Apache Spark depending on the version. This includes Java 11/17/21. Please, check the [reference versions](.build/reference-versions.yml) or [Apache Spark website](https://spark.apache.org/docs/latest/) for more information. |
| `spark-base` | The Apache Spark base image with official spark binaries (scala/java) and without OKDP extensions. |
| `spark` | The Apache Spark image with official spark binaries (scala/java) and OKDP extensions. |
| `spark-py` | The Apache Spark image with official spark binaries (scala/java), OKDP extensions and python support. |
| `spark-r` | The Apache Spark image with official spark binaries (scala/java), OKDP extensions and R support. |
# Tagging
The project builds the images with a long format tags. Each tag combines multiple compatible versions combinations.
There are multiple tags levels and the format to use depends on your convenience in term of stability and reproducibility.
The images are pushed to [quay.io/okdp](https://quay.io/organization/okdp) repository with the following [tags](.build/images.yml):
| Images | Tags |
|:--------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| spark-base, spark | spark--scala--java-spark--scala--java--spark--scala--java--spark--scala--java--- |
| spark-py | spark--python--scala--java-spark--python--scala--java--spark--python--scala--java--spark--python--scala--java--- |
| spark-r | spark--r--scala--java- spark--r--scala--java--spark--r--scala--java--spark--r--scala--java--- |
> [!NOTE]
> 1. `` corresponds to the Github [release version](https://github.com/okdp/spark-images/releases) or [git tag](https://github.com/okdp/spark-images/tags) without the leading `v`.
> Ex.: 1.0.0
>
> 2. `` corresponds to the images build date with the `YYYY-MM-DD` format. The latest release tag is rebuilt every week to ensure the OS image is up to date against the latest security updates.
>
> You may need to switch to the latest release version if your are using the long form tag image with a ``. Please, check the [changelog](https://github.com/okdp/spark-images/releases) to see the notable impacts.
>
> An example of `py-spark` image with a long form tag including `spark/java/scala/python` compatible versions and a `` with a `` is:
>
> `quay.io/okdp/spark-py:spark-3.5.1-python-3.11-scala-2.13-java-17-2024-04-04-1.0.0`.
>
> The corresponding changelog is [releases/tag/v1.0.0](https://github.com/okdp/spark-images/releases/tag/v1.0.0).
>
> 3. You can also use the latest tag without `` and `` which is always up to date with the latest security updates.
>
> An example of `py-spark` image with the latest tag is: `quay.io/okdp/spark-py:spark-3.5.1-python-3.11-scala-2.13-java-17`
>
# Alternatives
- [Official images](https://github.com/apache/spark-docker)