https://github.com/tonycox/spark-ignite-docker
Standalone Spark with Ignite In-memory Rdd sharding on Docker
https://github.com/tonycox/spark-ignite-docker
docker ignite spark
Last synced: about 1 month ago
JSON representation
Standalone Spark with Ignite In-memory Rdd sharding on Docker
- Host: GitHub
- URL: https://github.com/tonycox/spark-ignite-docker
- Owner: tonycox
- Created: 2017-02-01T13:48:42.000Z (over 9 years ago)
- Default Branch: master
- Last Pushed: 2017-02-06T00:49:28.000Z (over 9 years ago)
- Last Synced: 2025-06-21T20:13:02.641Z (12 months ago)
- Topics: docker, ignite, spark
- Language: Shell
- Homepage:
- Size: 6.84 KB
- Stars: 2
- Watchers: 1
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
##start spark master, worker and ignite node
```
docker-compose up
```
**scale mode:**
```
docker-compose scale spark-master=1 spark-slave=2
```
**stop all containers:**
```
docker kill $(docker ps -q)
```
**spark submit:**
```
docker exec -v \
spark-master spark-submit \
--class \
--master spark://spark-master:7077 \
\
[application-arguments]
```
**spark shell:**
```
docker exec -it spark-master spark-shell \
--master spark://spark-master:7077 \
--packages org.apache.ignite:ignite-spark_2.10:1.8.0,org.apache.ignite:ignite-spring:1.8.0
```
###Manual start spark
**build**
```
docker build -t spark-config . &&
docker build -t spark-master ./spark-master &&
docker build -t spark-slave ./spark-slave
```
**start master**
```
docker run -it --name spark-master
```
**start slave**
```
docker run -it -e SPARK_MASTER_URL=spark://:7077 \
--name spark-slave
```
get ip:
```
docker exec -it spark-master hostname -i
```