{"id":25510306,"url":"https://github.com/yash-chauhan-dev/spark_cluster_docker","last_synced_at":"2026-05-04T01:34:44.055Z","repository":{"id":276571238,"uuid":"929284034","full_name":"yash-chauhan-dev/SPARK_CLUSTER_DOCKER","owner":"yash-chauhan-dev","description":"Set-up local spark cluster, hadoop (hdfs), airflow, postgresql on docker with ease, without any local installations","archived":false,"fork":false,"pushed_at":"2025-02-21T08:05:03.000Z","size":1364,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-11-27T11:19:24.039Z","etag":null,"topics":["apache-spark","data","data-engineering","data-engineering-pipeline","deployment","docker","docker-compose","hadoop","hdfs","local-development","localhost","pyspark","python"],"latest_commit_sha":null,"homepage":"","language":"Dockerfile","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/yash-chauhan-dev.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-02-08T07:17:08.000Z","updated_at":"2025-02-21T08:05:06.000Z","dependencies_parsed_at":null,"dependency_job_id":"01bf5d88-4136-40a5-86f7-493f6bb3aa97","html_url":"https://github.com/yash-chauhan-dev/SPARK_CLUSTER_DOCKER","commit_stats":null,"previous_names":["yash-chauhan-dev/spark_cluster_docker"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/yash-chauhan-dev/SPARK_CLUSTER_DOCKER","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yash-chauhan-dev%2FSPARK_CLUSTER_DOCKER","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yash-chauhan-dev%2FSPARK_CLUSTER_DOCKER/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yash-chauhan-dev%2FSPARK_CLUSTER_DOCKER/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yash-chauhan-dev%2FSPARK_CLUSTER_DOCKER/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/yash-chauhan-dev","download_url":"https://codeload.github.com/yash-chauhan-dev/SPARK_CLUSTER_DOCKER/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yash-chauhan-dev%2FSPARK_CLUSTER_DOCKER/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32591601,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-03T22:12:39.696Z","status":"ssl_error","status_checked_at":"2026-05-03T22:09:10.534Z","response_time":103,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apache-spark","data","data-engineering","data-engineering-pipeline","deployment","docker","docker-compose","hadoop","hdfs","local-development","localhost","pyspark","python"],"created_at":"2025-02-19T09:38:56.905Z","updated_at":"2026-05-04T01:34:44.035Z","avatar_url":"https://github.com/yash-chauhan-dev.png","language":"Dockerfile","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Set-up Apache Pyspark Cluster Locally using Docker\n\nThis repository provides a **Docker-based Apache Spark cluster** setup with:\n- **Spark Master**\n- **Multiple Spark Workers**\n- **Spark History Server**\n- **HDFS**\n- **Airflow**\n- **PostgreSQL**\n\n## 🚀 Setup \u0026 Running the Cluster\n\n### 1️⃣ **Build the Spark Image**\n```bash\ndocker-compose build\n```\n\n### 2️⃣ **Start the Spark Cluster**\n```bash\ndocker-compose up -d\n```\nThis starts:\n- **Spark Master** (`spark://spark-master:7077`)\n- **Spark Worker(s)**\n- **Spark History Server**\n\n### 3️⃣ **Check Running Containers**\n```bash\ndocker ps\n```\n\n### 4️⃣ **Scale Up/Down Workers**\nTo start **3 worker nodes**:\n```bash\ndocker-compose up -d --scale spark-worker=3\n```\nTo scale dynamically:\n```bash\ndocker-compose up -d --scale spark-worker=5  # Increase to 5 workers\ndocker-compose up -d --scale spark-worker=2  # Reduce to 2 workers\n```\n\n### 5️⃣ **Access Spark Web UI**\n- **Spark Master UI** → [http://localhost:9090](http://localhost:9090)\n- **Spark History Server UI** → [http://localhost:18080](http://localhost:18080)\n\n### 6️⃣ **Submit an ETL Job**\nTo run an **ETL pipeline** (e.g., `main.py`):\n```bash\ndocker exec -it spark-master-node spark-submit \\\n  --master spark://spark-master:7077 \\\n  --deploy-mode client \\\n  /path/to/main.py\n```\nTo pass arguments:\n```bash\ndocker exec -it spark-master-node spark-submit \\\n  --master spark://spark-master:7077 \\\n  --deploy-mode client \\\n  /path/to/main.py arg1 arg2\n```\n\n### 7️⃣ **Stop \u0026 Remove the Cluster**\nTo **stop** the cluster (without removing containers):\n```bash\ndocker-compose stop\n```\nTo **stop \u0026 remove** all containers:\n```bash\ndocker-compose down\n```\nTo **remove everything** (including volumes):\n```bash\ndocker-compose down -v\n```\n\n## Working with PostgreSQL\n\n### 1️⃣ **Access PostgreSQL database inside the container**\n```bash\ndocker exec -it postgres-db psql -U user -d airbnb\n```\n### 2️⃣ **Run a SQL command inside PostgreSQL**\n```bash\ndocker exec -it postgres-db psql -U user -d airbnb -c \"SELECT * FROM table_name;\"\n```\n\n## Managing Hadoop (HDFS)\n\n### 1️⃣ **Format the NameNode (Only for first-time setup)**\n```bash\ndocker exec -it namenode hdfs namenode -format\n```\n\n### 2️⃣ **List HDFS files**\n```bash\ndocker exec -it namenode hdfs dfs -ls /\n```\n\n### 3️ **Upload a file to HDFS**\n```bash\ndocker exec -it namenode hdfs dfs -put /local/path/file.txt /hdfs/path/\n```\n\n## Managing Airflow\n\n### 1️⃣ **Access Airflow web UI**\n- Airflow UI: http://localhost:8081\n\n### 2️⃣ **Manually trigger an Airflow DAG**\n```bash\ndocker exec -it airflow airflow dags trigger \u003cdag_id\u003e\n```\n\n### 3️⃣ **Check Airflow DAGs**\n```bash\ndocker exec -it airflow airflow dags list\n```\n\n### 4️⃣ **Restart Airflow**\n```bash\ndocker-compose restart airflow\n```\n\n## 📝 Note\n- Ensure your ETL scripts (`spark_session.py`, `extract.py`, `transform.py`, `load.py`, `main.py`) are inside the container or mounted.\n- Use `spark-submit` to run your jobs.\n- Dynamically **scale workers** based on workload.\n\n🚀 **Now you're ready to run your Apache Spark cluster with Docker!**\n\n## Youtube Video Link\n\n[![Youtube Link](https://img.youtube.com/vi/Mrp1VAa_HL0/0.jpg)](https://www.youtube.com/watch?v=Mrp1VAa_HL0)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyash-chauhan-dev%2Fspark_cluster_docker","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fyash-chauhan-dev%2Fspark_cluster_docker","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyash-chauhan-dev%2Fspark_cluster_docker/lists"}