https://github.com/yash-chauhan-dev/spark_hdfs_airflow_cluster_docker
Set-up apache spark cluster with hadoop(hdfs) and airflow on docker
https://github.com/yash-chauhan-dev/spark_hdfs_airflow_cluster_docker
apache-airflow apache-spark data-engineering data-pipeline docker docker-compose hadoop hdfs pyspark python
Last synced: 3 months ago
JSON representation
Set-up apache spark cluster with hadoop(hdfs) and airflow on docker
- Host: GitHub
- URL: https://github.com/yash-chauhan-dev/spark_hdfs_airflow_cluster_docker
- Owner: yash-chauhan-dev
- Created: 2025-02-09T07:30:55.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2025-02-11T04:59:13.000Z (over 1 year ago)
- Last Synced: 2025-02-11T05:29:54.970Z (over 1 year ago)
- Topics: apache-airflow, apache-spark, data-engineering, data-pipeline, docker, docker-compose, hadoop, hdfs, pyspark, python
- Language: Dockerfile
- Homepage:
- Size: 1.29 MB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: readme.md
Awesome Lists containing this project
README
# Apache Spark Cluster with Hadoop & Airflow - Docker Setup
This project sets up an **Apache Spark cluster with Hadoop, Airflow, and PostgreSQL** using Docker Compose. The cluster includes:
- **Hadoop NameNode & DataNode for HDFS**
- **PostgreSQL as the metadata store for Airflow**
- **Apache Airflow for workflow orchestration**
- **Apache Spark with a scalable number of worker nodes**
## Setup & Running the Cluster
### 1️⃣ **Start all services (detached mode)**
```bash
docker-compose up -d
```
### 2️⃣ **Stop all services**
```bash
docker-compose down
```
### 3️⃣ **Restart all services**
```bash
docker-compose restart
```
### 4️⃣ **Scale Up/Down Workers**
To start **3 worker nodes**:
```bash
docker-compose up -d --scale spark-worker=3
```
To scale dynamically:
```bash
docker-compose up -d --scale spark-worker=5 # Increase to 5 workers
docker-compose up -d --scale spark-worker=2 # Reduce to 2 workers
```
### 5️⃣ **Check logs of a specific container**
```bash
docker logs -f
```
### 6️⃣ **Access a running container's shell**
```bash
docker exec -it /bin/bash
```
## Working with PostgreSQL
### 1️⃣ **Access PostgreSQL database inside the container**
```bash
docker exec -it postgres-db psql -U user -d airbnb
```
### 2️⃣ **Run a SQL command inside PostgreSQL**
```bash
docker exec -it postgres-db psql -U user -d airbnb -c "SELECT * FROM table_name;"
```
## Managing Spark
### 1️⃣ **Submit a Spark job**
```bash
docker exec -it spark-master-node spark-submit --master spark://spark-master:7077 /path/to/job.py
```
### 2️⃣ **Check Spark UI**
- **Spark Master UI:** http://localhost:9090
- **Spark History Server UI:** http://localhost:18080
## Managing Hadoop (HDFS)
### 1️⃣ **Format the NameNode (Only for first-time setup)**
```bash
docker exec -it namenode hdfs namenode -format
```
### 2️⃣ **List HDFS files**
```bash
docker exec -it namenode hdfs dfs -ls /
```
### 3️ **Upload a file to HDFS**
```bash
docker exec -it namenode hdfs dfs -put /local/path/file.txt /hdfs/path/
```
## Managing Airflow
### 1️⃣ **Access Airflow web UI**
- Airflow UI: http://localhost:8081
### 2️⃣ **Manually trigger an Airflow DAG**
```bash
docker exec -it airflow airflow dags trigger
```
### 3️⃣ **Check Airflow DAGs**
```bash
docker exec -it airflow airflow dags list
```
### 4️⃣ **Restart Airflow**
```bash
docker-compose restart airflow
```
## 📝 Note
- Ensure **.env.spark** contains the correct Spark environment configurations.
- Modify **docker-compose.yml** to adjust cluster settings if needed.