{"id":25240404,"url":"https://github.com/yash-chauhan-dev/spark_hdfs_airflow_cluster_docker","last_synced_at":"2026-04-13T22:03:50.483Z","repository":{"id":276913167,"uuid":"929707937","full_name":"yash-chauhan-dev/SPARK_HDFS_AIRFLOW_CLUSTER_DOCKER","owner":"yash-chauhan-dev","description":"Set-up apache spark cluster with hadoop(hdfs) and airflow on docker","archived":false,"fork":false,"pushed_at":"2025-02-11T04:59:13.000Z","size":1350,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-02-11T05:29:54.970Z","etag":null,"topics":["apache-airflow","apache-spark","data-engineering","data-pipeline","docker","docker-compose","hadoop","hdfs","pyspark","python"],"latest_commit_sha":null,"homepage":"","language":"Dockerfile","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/yash-chauhan-dev.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-02-09T07:30:55.000Z","updated_at":"2025-02-11T05:01:29.000Z","dependencies_parsed_at":"2025-02-11T05:40:03.561Z","dependency_job_id":null,"html_url":"https://github.com/yash-chauhan-dev/SPARK_HDFS_AIRFLOW_CLUSTER_DOCKER","commit_stats":null,"previous_names":["yash-chauhan-dev/spark_hdfs_airflow_cluster_docker"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yash-chauhan-dev%2FSPARK_HDFS_AIRFLOW_CLUSTER_DOCKER","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yash-chauhan-dev%2FSPARK_HDFS_AIRFLOW_CLUSTER_DOCKER/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yash-chauhan-dev%2FSPARK_HDFS_AIRFLOW_CLUSTER_DOCKER/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yash-chauhan-dev%2FSPARK_HDFS_AIRFLOW_CLUSTER_DOCKER/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/yash-chauhan-dev","download_url":"https://codeload.github.com/yash-chauhan-dev/SPARK_HDFS_AIRFLOW_CLUSTER_DOCKER/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247393574,"owners_count":20931813,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apache-airflow","apache-spark","data-engineering","data-pipeline","docker","docker-compose","hadoop","hdfs","pyspark","python"],"created_at":"2025-02-11T19:13:56.175Z","updated_at":"2026-04-13T22:03:50.478Z","avatar_url":"https://github.com/yash-chauhan-dev.png","language":"Dockerfile","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Apache Spark Cluster with Hadoop \u0026 Airflow - Docker Setup\n\nThis project sets up an **Apache Spark cluster with Hadoop, Airflow, and PostgreSQL** using Docker Compose. The cluster includes:\n\n- **Hadoop NameNode \u0026 DataNode for HDFS**\n\n- **PostgreSQL as the metadata store for Airflow**\n\n- **Apache Airflow for workflow orchestration**\n\n- **Apache Spark with a scalable number of worker nodes**\n\n## Setup \u0026 Running the Cluster\n\n### 1️⃣ **Start all services (detached mode)**\n```bash\ndocker-compose up -d\n```\n\n### 2️⃣ **Stop all services**\n```bash\ndocker-compose down\n```\n\n### 3️⃣ **Restart all services**\n```bash\ndocker-compose restart\n```\n\n### 4️⃣ **Scale Up/Down Workers**\nTo start **3 worker nodes**:\n```bash\ndocker-compose up -d --scale spark-worker=3\n```\nTo scale dynamically:\n```bash\ndocker-compose up -d --scale spark-worker=5  # Increase to 5 workers\ndocker-compose up -d --scale spark-worker=2  # Reduce to 2 workers\n```\n\n### 5️⃣ **Check logs of a specific container**\n```bash\ndocker logs -f \u003ccontainer_name\u003e\n```\n\n### 6️⃣ **Access a running container's shell**\n```bash\ndocker exec -it \u003ccontainer_name\u003e /bin/bash\n```\n## Working with PostgreSQL\n\n### 1️⃣ **Access PostgreSQL database inside the container**\n```bash\ndocker exec -it postgres-db psql -U user -d airbnb\n```\n### 2️⃣ **Run a SQL command inside PostgreSQL**\n```bash\ndocker exec -it postgres-db psql -U user -d airbnb -c \"SELECT * FROM table_name;\"\n```\n## Managing Spark\n\n### 1️⃣ **Submit a Spark job**\n```bash\ndocker exec -it spark-master-node spark-submit --master spark://spark-master:7077 /path/to/job.py\n```\n\n### 2️⃣ **Check Spark UI**\n- **Spark Master UI:** http://localhost:9090\n- **Spark History Server UI:** http://localhost:18080\n\n## Managing Hadoop (HDFS)\n\n### 1️⃣ **Format the NameNode (Only for first-time setup)**\n```bash\ndocker exec -it namenode hdfs namenode -format\n```\n\n### 2️⃣ **List HDFS files**\n```bash\ndocker exec -it namenode hdfs dfs -ls /\n```\n\n### 3️ **Upload a file to HDFS**\n```bash\ndocker exec -it namenode hdfs dfs -put /local/path/file.txt /hdfs/path/\n```\n\n## Managing Airflow\n\n### 1️⃣ **Access Airflow web UI**\n- Airflow UI: http://localhost:8081\n\n### 2️⃣ **Manually trigger an Airflow DAG**\n```bash\ndocker exec -it airflow airflow dags trigger \u003cdag_id\u003e\n```\n\n### 3️⃣ **Check Airflow DAGs**\n```bash\ndocker exec -it airflow airflow dags list\n```\n\n### 4️⃣ **Restart Airflow**\n```bash\ndocker-compose restart airflow\n```\n\n## 📝 Note\n- Ensure **.env.spark** contains the correct Spark environment configurations.\n\n- Modify **docker-compose.yml** to adjust cluster settings if needed.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyash-chauhan-dev%2Fspark_hdfs_airflow_cluster_docker","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fyash-chauhan-dev%2Fspark_hdfs_airflow_cluster_docker","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyash-chauhan-dev%2Fspark_hdfs_airflow_cluster_docker/lists"}