https://github.com/alanrma/pyspark-simple-docker-cluster

https://github.com/alanrma/pyspark-simple-docker-cluster

Last synced: about 1 year ago
JSON representation

• Host: GitHub
• URL: https://github.com/alanrma/pyspark-simple-docker-cluster
• Owner: AlanRMA
• Created: 2024-06-08T07:52:44.000Z (about 2 years ago)
• Default Branch: main
• Last Pushed: 2024-06-08T07:56:49.000Z (about 2 years ago)
• Last Synced: 2025-03-04T18:46:17.538Z (over 1 year ago)
• Language: Python
• Size: 125 KB
• Stars: 1
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

          
# Project overview

This project is designed to run on containers using Docker that can be expanded for Kubernetes usage in the future.

# Getting Started

All you need to run your cluster is having Docker installed and follow Steps on terminal/shell.

# Step-by-Step

### 0. Open Docker

### 1. Enter on you terminal

### 2. Go to project root folder

### 3. Run `docker-compose up`

### 4. Go to Containers in Docker

![alt text](image.png)

### 5. Select Master node 

### 6. Go to `Exec` tab

![alt text](image-1.png)

### 7. At this point you can edit your `task.py` code to choose what you want to submit to cluster process

### 8. Run `/opt/bitnami/spark/bin/spark-submit /opt/spark-app/task.py`

 Now you should see output on Exec tab from docker.