https://github.com/florentf9/spark-config-example
:sparkles: Configuring a Spark application with Typesafe config (example application)
https://github.com/florentf9/spark-config-example
configuration spark tutorial
Last synced: 4 months ago
JSON representation
:sparkles: Configuring a Spark application with Typesafe config (example application)
- Host: GitHub
- URL: https://github.com/florentf9/spark-config-example
- Owner: FlorentF9
- Created: 2019-01-07T07:30:42.000Z (about 7 years ago)
- Default Branch: master
- Last Pushed: 2019-01-20T18:37:04.000Z (about 7 years ago)
- Last Synced: 2025-04-10T17:08:29.150Z (10 months ago)
- Topics: configuration, spark, tutorial
- Language: Scala
- Homepage:
- Size: 3.91 KB
- Stars: 5
- Watchers: 1
- Forks: 6
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Configuring a Spark application with Typesafe config (example application)
Example code for my blog post ["Configuring Spark application with Typesafe config"](http://florentforest.xyz/2019/01/07/configuring-spark-applications-with-typesafe-config.html).
## Basic instructions
Build assembly JAR:
```shell
sbt assembly
```
Run (local mode):
```shell
spark-submit --master local[*] --class xyz.florentforest.sparkconfigexample.ConfigWordCount --files wordcount.conf --driver-java-options -Dconfig.file=wordcount.conf sparkconfigexample-assembly-1.0.jar
```