https://github.com/newfront/spark-inception
This project is available free of charge as a companion to my Data+AI Summit (2022) talk.
https://github.com/newfront/spark-inception
Last synced: 11 months ago
JSON representation
This project is available free of charge as a companion to my Data+AI Summit (2022) talk.
- Host: GitHub
- URL: https://github.com/newfront/spark-inception
- Owner: newfront
- License: apache-2.0
- Created: 2022-06-14T21:52:03.000Z (about 4 years ago)
- Default Branch: main
- Last Pushed: 2022-06-28T04:21:01.000Z (almost 4 years ago)
- Last Synced: 2025-04-03T08:11:56.086Z (about 1 year ago)
- Language: Scala
- Size: 58.6 KB
- Stars: 5
- Watchers: 1
- Forks: 3
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# spark-inception
This project is available free of charge as a companion to my Data+AI Summit (2022) talk.
## Setup
> Requirements: Java 11 (jdk), Scala 2.12, SBT (1.5.2)
Export your java 11 location to `~/.zshrc`: (example: `export JAVA11_HOME="/Library/Java/JavaVirtualMachines/openjdk@11.jdk/Contents/Home"`)
## Building
From the root project directory, run the following command.
> Note: Because I am kind of lazy - you need to have redis running to build. Use `docker compose -f local/docker-compose.yml` to spin up redis (and minio if you want to get creative with things - I use it for a local S3 clone).
```bash
sbt clean assembly
docker build . -t mde/spark-inception-controller:1.0.0
```
## Running the Inception Controller
See the [How To: Docs](docs/howto.md) to run everything without needing to build / test / all that.
## Shameless Plug:
I wrote a book on Apache Spark published in 2022. You'll like it, https://www.amazon.com/Modern-Engineering-Apache-Spark-Hands/dp/1484274512/