https://github.com/NashTech-Labs/spark-on-mesos

https://github.com/NashTech-Labs/spark-on-mesos

deployment mesos spark word-count

Last synced: 2 months ago
JSON representation

• Host: GitHub
• URL: https://github.com/NashTech-Labs/spark-on-mesos
• Owner: NashTech-Labs
• Created: 2018-02-05T05:12:51.000Z (almost 7 years ago)
• Default Branch: master
• Last Pushed: 2018-02-05T05:13:15.000Z (almost 7 years ago)
• Last Synced: 2024-07-29T17:53:38.914Z (5 months ago)
• Topics: deployment, mesos, spark, word-count
• Language: Scala
• Size: 5.86 KB
• Stars: 1
• Watchers: 9
• Forks: 1
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
# spark-on-mesos wordcount-example

Simple word count example using spark and running it on mesos. For basic configuration to run spark on mesos 

[refer](https://blog.knoldus.com/2017/10/09/spark-on-mesosinstallation/)

# Clone the repo

```

git clone https://github.com/mahesh2492/spark-on-mesos.git

cd spark-on-mesos

```

# Build the code

If this is your first time running SBT, you have to download the used dependencies.

```

cd spark-on-mesos

sbt clean compile

```

# To run application locally, do following things:

First change the spark master in application.conf and run following command

```

sbt run

```

To run it on mesos, first create the jar to deploy. For that run following command

```

sbt assembly

 ```

 It will create the fat jar of wordcount application.

 

 # Run mesos master and slave

 ```

 cd build

 ./bin/mesos-master.sh --ip=127.0.0.1 --work_dir=/tmp/mesos

 ./bin/mesos-slave.sh --master=127.0.0.1:5050 --work_dir=/tmp/mesos --no-systemd_enable_support

 ```

 

 Deploy the jar using spark-submit

 

 ```

 cd spark-2.2.0-bin-hadoop2.7

 ./bin/spark-submit --class "rdd.WordCount" --master mesos://127.0.0.1:5050 file://path/target/scala-2.11/spark-on-mesos-assembly-1.0.jar

```