https://github.com/tobilg/docker-mesos-spark-shell
A docker image for creating a Spark shell on a Mesos cluster
https://github.com/tobilg/docker-mesos-spark-shell
Last synced: 8 months ago
JSON representation
A docker image for creating a Spark shell on a Mesos cluster
- Host: GitHub
- URL: https://github.com/tobilg/docker-mesos-spark-shell
- Owner: tobilg
- License: mit
- Created: 2015-04-24T08:56:08.000Z (over 10 years ago)
- Default Branch: master
- Last Pushed: 2015-09-15T08:03:13.000Z (about 10 years ago)
- Last Synced: 2023-02-26T19:16:35.001Z (over 2 years ago)
- Language: Shell
- Size: 152 KB
- Stars: 2
- Watchers: 3
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Mesos Spark Shell
A docker image for creating a Spark 1.4.1 shell on a Mesos cluster. Currently, Mesos 0.23.0 is supported via `.deb` installer.
### Running
To see how to run the Docker image, have a look a the `run_spark_shell.sh` file. Basically, you just need to replace the `MESOS_MASTER_IP` with the actual IP the Mesos Master is living on,
and the `SPARK_EXECUTOR_URI` variable needs to point to a "Spark package" where the Spark binaries are reachable for the Mesos Slaves' Executors to download, see [Spark docs](https://spark.apache.org/docs/latest/running-on-mesos.html#uploading-spark-package).
Example:
```
docker run -i -t \
--net=host \
-e MESOS_MASTER=127.0.0.1:5050 \
-e SPARK_LOCAL_IP=127.0.0.1 \
-e SPARK_EXECUTOR_URI=http://d3kbcqa49mib13.cloudfront.net/spark-1.4.1-bin-hadoop2.6.tgz \
-p 4040:4040 \
-p 5000:5000 \
-p 5001:5001 \
-p 5002:5002 \
-p 5003:5003 \
-p 5004:5004 \
-p 5005:5005 \
--name spark-shell \
tobilg/mesos-spark-shell
```
### Networking
Eventually, you'll also need to set the `SPARK_LOCAL_IP` variable to the public IP address of your Docker host if the Mesos cluster is run in non-local mode, as well as `--net=host`.
To automatically determine the IP address, on RedHat/CentOS/Fedora-based hosts', the `SPARK_LOCAL_IP` line needs to replaced with
-e SPARK_LOCAL_IP=$(/sbin/ifconfig eth0 | grep 'inet ' | awk '{print $2}') \
For Debian/Ubuntu-based hosts, use
-e SPARK_LOCAL_IP=$(ifconfig eth0 | grep 'inet addr:' | cut -d: -f2 | awk '{ print $1}')
Be sure to replace `eth0` with the actual interface your host is using for external access.
### Ports
The ports 5000 - 5005 (see `spark-defaults.conf`) are opened to be reachable for the Mesos Master.