https://github.com/sraashis/spark
Dockerfile for spark-ubunt-scala-python3
https://github.com/sraashis/spark
docker dockerfile python scala sparksql
Last synced: 2 months ago
JSON representation
Dockerfile for spark-ubunt-scala-python3
- Host: GitHub
- URL: https://github.com/sraashis/spark
- Owner: sraashis
- Created: 2017-12-14T01:25:40.000Z (over 7 years ago)
- Default Branch: master
- Last Pushed: 2017-12-17T23:50:44.000Z (over 7 years ago)
- Last Synced: 2025-01-24T05:29:35.502Z (4 months ago)
- Topics: docker, dockerfile, python, scala, sparksql
- Language: Shell
- Size: 35.2 KB
- Stars: 0
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# spark
Dockerfile for spark-ubunt-scala-python3
This image is intended to use as a spark worker, easy to deploy. It consists of Java 8, python 3, pip 3 and spark-2.2.1-bin-hadoop2.7.tgz.
Also, the container will have a new user name "ak" setup with PASSWORD ##>"redhat"<<##.
This is to authenticate the worker to master for password less login that initiates worker-master negotiation for spark.
For password less authentication just add:
~# ssh-copy-id ak@worker and use "redhat" as password for first and the last time.
To run service named worker:
~# ./run.sh worker
In run.sh
-----------------
SERVICE_NAME=$1
docker-compose run --service-ports $SERVICE_NAME
Environment variables lies in a separate file name .env which will be captured by docker-compose.yml file.
Also for bootstraping we have a bootsrap.sh file which has following content for now:
-----------------
#!/bin/bash
/usr/sbin/sshd
/bin/bash -c "$*"