https://github.com/47degrees/how-to-test-spark
Different techniques to do unit testing in Spark without the need to rely on anything else than your IDE and SBT
https://github.com/47degrees/how-to-test-spark
Last synced: 5 months ago
JSON representation
Different techniques to do unit testing in Spark without the need to rely on anything else than your IDE and SBT
- Host: GitHub
- URL: https://github.com/47degrees/how-to-test-spark
- Owner: 47degrees
- Created: 2022-06-07T14:16:50.000Z (about 4 years ago)
- Default Branch: main
- Last Pushed: 2022-07-27T08:22:53.000Z (almost 4 years ago)
- Last Synced: 2025-09-25T20:58:32.931Z (10 months ago)
- Language: Scala
- Size: 61.5 KB
- Stars: 3
- Watchers: 1
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# how-to-test-spark
Different techniques to do unit testing in Spark without the need to rely on anything else than your IDE and SBT.
The focus of this project is to share ideas on how to test the small.
Testing the small means testing the basic transformations that every spark application has in an isolated manner.
## Run applications
The project also gives you the oportunity to run your applications locally if you wish to - drop local to run it in a cluster.
```bash
sbt "run $inputPath/inputFile.csv $outputPath/outputFile.csv local"
```