https://github.com/aamend/spark-archetype
Maven archetype is a convenient way to create fully fledged SPARK libraries at minimal cost
https://github.com/aamend/spark-archetype
devops maven spark
Last synced: 9 months ago
JSON representation
Maven archetype is a convenient way to create fully fledged SPARK libraries at minimal cost
- Host: GitHub
- URL: https://github.com/aamend/spark-archetype
- Owner: aamend
- License: apache-2.0
- Created: 2020-01-30T16:46:40.000Z (about 6 years ago)
- Default Branch: master
- Last Pushed: 2020-03-04T23:44:16.000Z (about 6 years ago)
- Last Synced: 2025-07-03T18:53:42.019Z (9 months ago)
- Topics: devops, maven, spark
- Language: Scala
- Homepage:
- Size: 25.4 KB
- Stars: 1
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE.txt
Awesome Lists containing this project
README
# Spark archetype
Maven archetype is a convenient way to create fully fledged Spark libraries at minimal cost.
A Spark module called `com.aamend.spark.ml:my-awesome-module:2.0-SNAPSHOT` can be created as follows.
```
mvn archetype:generate \
-DinteractiveMode=false \
-DarchetypeGroupId=com.aamend.spark \
-DarchetypeArtifactId=archetype \
-DarchetypeVersion=1.0 \
-DgroupId=com.aamend.spark.ml \
-DartifactId=my-awesome-module \
-Dversion=2.0-SNAPSHOT
```
Generated project can be compiled as follows
`mvn clean package`
Should you need to include third parties dependencies, packaging shaded jar as follows
`mvn clean package -Pshaded`
Spark module can be validated as follows
`spark-submit --class com.aamend.spark.ml.App target/my-awesome-module-2.0-SNAPSHOT.jar`
## Compile
`mvn clean install`