https://github.com/erikerlandson/pyspark-ubi

Minimalist install of pyspark on top of Red Hat UBI
https://github.com/erikerlandson/pyspark-ubi

container-image pyspark spark ubi

Last synced: about 1 month ago
JSON representation

Minimalist install of pyspark on top of Red Hat UBI

• Host: GitHub
• URL: https://github.com/erikerlandson/pyspark-ubi
• Owner: erikerlandson
• License: apache-2.0
• Created: 2021-01-26T21:20:27.000Z (about 4 years ago)
• Default Branch: main
• Last Pushed: 2021-01-30T14:15:08.000Z (about 4 years ago)
• Last Synced: 2024-11-09T23:37:05.149Z (3 months ago)
• Topics: container-image, pyspark, spark, ubi
• Language: Dockerfile
• Homepage:
• Size: 6.84 KB
• Stars: 2
• Watchers: 3
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# pyspark-ubi

Minimalist install of pyspark on top of Red Hat UBI

This image is available at: `quay.io/erikerlandson/pyspark-ubi`

Various steps have been taken to keep this image minimal:

- built on Red Hat `ubi-minimal` base image

- headless java install

- removal of intermediate caching

- only `pyspark` has been installed to the pipenv environment.

The pyspark environment was created using pipenv,

to make it straightforward to include additional python package dependencies.

For example, starting from this base image and adding packages with docker RUN:

```

RUN cd /opt/pyspark \

 && pipenv install ...

```

An example of a script that could run this image against a file `spark-app.py`

```sh

cd /opt/pyspark

pipenv run python3 /your/path/to/spark-app.py

```

In the example above, the script or `spark-app.py` might be staged using S2I or made available via a volume mounted to a pod, etc.