https://github.com/duyet/spark-docker
Spark image for running on Kubernetes
https://github.com/duyet/spark-docker
docker docker-image hacktoberfest spark
Last synced: 6 months ago
JSON representation
Spark image for running on Kubernetes
- Host: GitHub
- URL: https://github.com/duyet/spark-docker
- Owner: duyet
- Created: 2020-06-10T08:53:01.000Z (about 6 years ago)
- Default Branch: master
- Last Pushed: 2025-03-05T00:40:29.000Z (over 1 year ago)
- Last Synced: 2025-03-05T01:29:14.228Z (over 1 year ago)
- Topics: docker, docker-image, hacktoberfest, spark
- Language: Shell
- Homepage:
- Size: 65.4 KB
- Stars: 0
- Watchers: 1
- Forks: 1
- Open Issues: 11
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Spark Image for Kubernetes
# Docker hub
- https://hub.docker.com/r/duyetdev/spark
- https://hub.docker.com/r/duyetdev/spark-py
- https://hub.docker.com/r/duyetdev/spark-operator
# Image information
## spark-py
Base image from `gcr.io/spark-operator/spark-py`, adding some jar files to support AWS.
- hadoop-aws-3.3.1.jar
- aws-java-sdk-bundle-1.11.271.jar
## spark-operator
Base image from `gcr.io/spark-operator/spark-operator`, adding some jar files to support AWS.
- hadoop-aws-3.3.1.jar
- aws-java-sdk-bundle-1.11.271.jar
# Usage
```bash
docker pull duyetdev/spark-py:v3.1.1-hadoop3
docker pull duyetdev/spark-operator:v1beta2-1.2.3-3.1.1
```
# CI/CD
Github Workflows will detect changes, build and publish images to `duyetdev/*` on Docker Hub.
Need to generate the `.github/workflows/ci.yaml` whenever adding new image name or image tags folders.
For example, adding
```
spark-operator//Dockerfile
```
Run the command below to generate
```bash
pip3 install jinja2
python3 gen.py
```