https://github.com/shixi99/spark-multinode

Standalone Spark cluster on Docker
https://github.com/shixi99/spark-multinode

Last synced: 6 days ago
JSON representation

Standalone Spark cluster on Docker

• Host: GitHub
• URL: https://github.com/shixi99/spark-multinode
• Owner: Shixi99
• Created: 2024-09-10T15:27:43.000Z (2 months ago)
• Default Branch: main
• Last Pushed: 2024-09-11T17:13:04.000Z (2 months ago)
• Last Synced: 2024-09-12T02:59:56.929Z (2 months ago)
• Language: Python
• Homepage:
• Size: 240 KB
• Stars: 0
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
## Configure Spark standalone cluster on Docker

[Read detailed artice on medium](https://shixieyyub.medium.com/dockerd%C9%99-spark-standalone-cluster-qurulmas%C4%B1-2bca5410daf7)

### To run this project on your local machine:

Make sure you have **Docker Desktop**, **git** and **Make** installed on your computer

Open 2 terminals

In the first terminal run commands below

```bash

mkdir my_park_project

```

```

cd my_spark_project

```

```

git clone https://github.com/Shixi99/spark-multinode.git

```

```

cd spark-multinode

```

```

make run-scaled

```

In the second terminal

```bash

cd myproject\spark-multinode

```

```

make run submit app=sales.py

```

To down containers 

```bash

make down

```