{"id":20745260,"url":"https://github.com/savvydatainsights/spark","last_synced_at":"2026-05-05T13:41:49.455Z","repository":{"id":92519037,"uuid":"188009024","full_name":"savvydatainsights/spark","owner":"savvydatainsights","description":"Apache Spark cluster lab.","archived":false,"fork":false,"pushed_at":"2023-04-27T13:33:13.000Z","size":7441,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-01-18T02:41:19.220Z","etag":null,"topics":["ansible","apache-spark","apache-spark-cluster","vagrant"],"latest_commit_sha":null,"homepage":null,"language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/savvydatainsights.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-05-22T09:43:11.000Z","updated_at":"2023-04-27T13:33:16.000Z","dependencies_parsed_at":null,"dependency_job_id":"6ce34d3b-788f-4908-b8cf-5cf09853cdfd","html_url":"https://github.com/savvydatainsights/spark","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/savvydatainsights%2Fspark","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/savvydatainsights%2Fspark/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/savvydatainsights%2Fspark/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/savvydatainsights%2Fspark/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/savvydatainsights","download_url":"https://codeload.github.com/savvydatainsights/spark/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243039609,"owners_count":20226131,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ansible","apache-spark","apache-spark-cluster","vagrant"],"created_at":"2024-11-17T07:19:24.185Z","updated_at":"2026-05-05T13:41:44.397Z","avatar_url":"https://github.com/savvydatainsights.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Spark\n\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)\n\n[Apache Spark](https://spark.apache.org) cluster lab.\n\n## Setup\n\n`vagrant up`\n\n## WebUIs\n\n| Node | URL |\n| ---- | --- |\n| master | \u003chttp://192.168.33.10:8080\u003e |\n| slave1 | \u003chttp://192.168.33.20:8081\u003e |\n| slave2 | \u003chttp://192.168.33.30:8081\u003e |\n\n## Submitting the sample application\n\nThe sample application counts how many times each word appears in a [lorem ipsum](https://www.lipsum.com) text.\n\nIn order to submit it, execute:\n\n`ansible-playbook submit-spark-application.yml`\n\nThe output then can be viewed under the *output* folder.\n\n### Manually from the master\n\nThe application can also be submitted manually from the master host of the cluster.\n\nFirst, SSH into the master: `vagrant ssh master`.\n\nOnce inside the master, become root: `sudo su -`.\n\nAfter that, build the application: `mvn install -f /vagrant`.\n\nFinally, submit the application:\n\n```bash\n/opt/spark/bin/spark-submit --master spark://192.168.33.10:7077 \\\n    --conf spark.driver.host=192.168.33.10 \\\n    --class uk.co.savvydatainsights.WordCount \\\n    /vagrant/target/spark-examples-1.0-SNAPSHOT.jar \\\n    /vagrant/input/lorem-ipsum.txt\n```\n\n## Submitting your own Java application\n\nYou can also submit to the Spark cluster your own Java application, by setting the parameters *repo* and *class*, like in the example:\n\n```bash\nansible-playbook submit-spark-application.yml \\\n    -e \"repo=https://github.com/project/repo.git\" \\\n    -e \"class=com.domain.spark.JavaApplication\"\n```\n\nYou will be prompted to inform your repository credentials. Then, the application will be cloned, built and submitted to the Spark master instance.\n\nRequirements:\n\n- The project must be a [Maven](https://maven.apache.org) project;\n- It is expected one input file under the *src/main/resources* folder.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsavvydatainsights%2Fspark","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsavvydatainsights%2Fspark","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsavvydatainsights%2Fspark/lists"}