{"id":21358103,"url":"https://github.com/raycad/spark","last_synced_at":"2025-03-16T06:16:22.479Z","repository":{"id":141975080,"uuid":"169009128","full_name":"raycad/spark","owner":"raycad","description":"Apache Spark cluster docker","archived":false,"fork":false,"pushed_at":"2019-02-06T07:13:08.000Z","size":8,"stargazers_count":3,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-01-22T18:32:33.046Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/raycad.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-02-04T00:24:43.000Z","updated_at":"2021-08-04T04:21:26.000Z","dependencies_parsed_at":null,"dependency_job_id":"b7df6138-ea6d-45d8-958e-651a9c13d9d1","html_url":"https://github.com/raycad/spark","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/raycad%2Fspark","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/raycad%2Fspark/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/raycad%2Fspark/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/raycad%2Fspark/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/raycad","download_url":"https://codeload.github.com/raycad/spark/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243830956,"owners_count":20354856,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-22T05:14:21.466Z","updated_at":"2025-03-16T06:16:22.462Z","avatar_url":"https://github.com/raycad.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Apache Spark Cluster\n\n### 1. Pull the image\n```\n$ sudo docker pull seedotech/spark:2.4.0\n```\n\n### 2. Create a spark network\n```\n$ sudo docker network create --driver=bridge spark\n```\n\n### 3. Create and run docker containers\n\n```sh\n# The 1st argument (CREATE_SPARK_MASTER) is set to YES if you want to create a spark master, the default is YES\n# The 2nd argument (SPARK_MASTER_IP) is the spark master ip (the ip of the host machine of the spark master docker container), the default is spark-master\n# The 3rd argument (SPARK_WORKER_NUMBER) is the number of workers you want to create, the default is 2\n\n# The default spark cluster has 3 nodes includes 1 master and 2 workers\n$ start_containers.sh\n\n# If you want to create 1 spark worker in other machine (assume that the ip of the spark master is 192.168.1.10), you can do like that:\n$ start_containers.sh NO 192.168.1.10 1\n```\n\n### 4. Get into the spark master container\n```\n$ sudo docker exec -it spark-master bash\n```\n\n### 5. Monitor spark cluster\n\nAccess to the http://spark-master:8080 to monitor the spark cluster.\n\n### 6. Run spark application with hadoop\n#### 6.1. Create a shared network for hadoop cluster and spark cluster be able to connect together\n```\n$ sudo docker network create --driver=bridge spark-hadoop\n``` \n\n#### 6.2. Create a hadoop cluster and use the spark-hadoop network\n```\ne.g\n$ sudo docker run -itd \\\n\t\t\t\t\t--net=spark-hadoop \\\n\t\t\t\t\t-p 50070:50070 \\\n\t\t\t\t\t-p 8088:8088 \\\n\t\t\t\t\t-e HADOOP_SLAVE_NUMBER=$HADOOP_SLAVE_NUMBER \\\n\t\t\t\t\t--name hadoop-master \\\n\t\t\t\t\t--hostname hadoop-master \\\n\t\t\t\t\tseedotech/hadoop:2.9.2 \u0026\u003e /dev/null\n```\n\n#### 6.3. Create a spark cluster and use the spark-hadoop network\n```\ne.g\n$ sudo docker run -itd \\\n\t\t\t\t\t--net=spark-hadoop \\\n\t\t\t\t\t-p 6066:6066 -p 7077:7077 -p 8080:8080 \\\n\t\t\t\t\t-e IS_SPARK_MASTER=YES \\\n\t\t\t\t\t--name spark-master \\\n\t\t\t\t\t--hostname spark-master \\\n\t\t\t\t\tseedotech/spark:2.4.0 \u0026\u003e /dev/null\n```\n\n#### 6.4. Execute spark application in other machine\n```\n$ spark-submit --class com.seedotech.spark.SparkJavaApp --master spark://spark-master:7077 --deploy-mode cluster hdfs://hadoop-master:9000/apps/spark/spark-java-1.0.jar hdfs://hadoop-master:9000/apps/spark/demo.txt hdfs://hadoop-master:9000/apps/spark/out\n```\n\n#### 6.5. Debug spark application\n\nUse logs to trace bugs/issues\n```\n$ sudo docker logs -f spark-master\n$ sudo docker logs -f spark-worker1\n```\n\n**NOTE**: you might run failed in the second time, it's because of the folder **hdfs://hadoop-master:9000/apps/spark/out** has been created. To run successfully please remove it before running (use http://localhost:50070/explorer.html to upload/delete folders/files).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fraycad%2Fspark","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fraycad%2Fspark","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fraycad%2Fspark/lists"}