{"id":19163210,"url":"https://github.com/programmer-ke/data-playground","last_synced_at":"2026-03-01T17:03:45.179Z","repository":{"id":49947539,"uuid":"511428305","full_name":"programmer-ke/data-playground","owner":"programmer-ke","description":"Set up a cluster of debian servers with hadoop and spark","archived":false,"fork":false,"pushed_at":"2024-11-27T11:28:11.000Z","size":252,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-02-22T22:42:33.842Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Jinja","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/programmer-ke.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-07-07T07:32:23.000Z","updated_at":"2024-11-27T11:28:14.000Z","dependencies_parsed_at":"2024-11-27T14:49:01.820Z","dependency_job_id":null,"html_url":"https://github.com/programmer-ke/data-playground","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/programmer-ke/data-playground","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/programmer-ke%2Fdata-playground","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/programmer-ke%2Fdata-playground/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/programmer-ke%2Fdata-playground/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/programmer-ke%2Fdata-playground/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/programmer-ke","download_url":"https://codeload.github.com/programmer-ke/data-playground/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/programmer-ke%2Fdata-playground/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29976272,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-01T16:35:47.903Z","status":"ssl_error","status_checked_at":"2026-03-01T16:35:44.899Z","response_time":124,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-09T09:14:29.202Z","updated_at":"2026-03-01T17:03:40.171Z","avatar_url":"https://github.com/programmer-ke.png","language":"Jinja","funding_links":[],"categories":[],"sub_categories":[],"readme":"# data-playground\n\n![data-playground logo](img/data-playground.jpeg)\n\nSet up a cluster of debian servers with hadoop and spark.\n\n# Local cluster setup\n\nTo set up a cluster in your dev machine, install [vagrant][1] and\n[ansible][2]\n\n[1]: https://www.vagrantup.com/docs/installation\n[2]: https://docs.ansible.com/ansible/latest/installation_guide/intro_installation.html\n\nThen in the project root, run `vagrant up`.\n\nThis should download and install everything in a local 3-node cluster\nif this is the first time.\n\nAfterwards, you can run `vagrant up` to start up the cluster. If you\nwant to run through the setting up process you will need to include the\nprovisioning flag i.e.  `vagrant up --provision`, or run\n`vagrant provision` after the cluster is up.\n\nAfter provisioning, the HDFS web UI will be available at\nhttp://192.168.56.10:9870\n\nYou can then SSH into the master node with `vagrant ssh master` and\nstart the various services as follows.\n\n - Yarn: `./hadoop/sbin/start-yarn.sh`. The web UI will be at\n   http://192.168.56.10:8088\n   \n - Hadoop Job History Server: `./hadoop/bin/mapred --daemon start historyserver`\n   Web UI: http://192.168.56.10:19888\n\n - Spark in standalone mode: `./spark/sbin/start-all.sh`. Web UI: http://192.168.56.10:8080\n\n - Spark History Server: `./spark/sbin/start-history-server.sh`.\n   Web UI: http://192.168.56.10:18080\n\nNote that Spark is configured in standalone mode by default so it is\nnot dependent on Yarn. To make Spark run with Yarn as the resource\nmanager, change the value of `spark_master` in [vars.yml](vars.yml) to\n`yarn` and shut down spark standalone if it is already running with\n`./spark/sbin/stop-all.sh`.\n\nTo shut down the local cluster, run `vagrant halt`. To delete all the\nresources associated with the cluster, run `vagrant destroy`.\n\n## Syncing files to local master node\n\nBy default, vagrant will sync the the project root (where the\nVagrantfile is located) to `/vagrant` within each cluster VM.\n\nAdditionally, a directory named `projects` will be\ncreated in the project's root directory and its contents will synced to\nthe directory `/projects` in the master node. You can place any code\nand data that you want to access from within the master node in it,\nand after you ssh into master via `vagrant ssh master`, you'll find\nthem in `/projects`.\n\nAny files in the `projects` folder will be ignored by git.\n\nThis can be disabled by removing the relevant section in the\n[Vagrantfile](Vagrantfile). Search for 'disable syncing'.\n\n## Access URLs via hostnames\n\nThe local cluster will use hostnames configured internally for\ncommunication amongst the nodes, and the jobs URLs may contain these\nhostname. To access these URLs, one would have to replace the hostnames\nwith the related IP as configured in the [Vagrantfile](Vagrantfile).\nFor example replacing `master` in the URL with `192.168.56.10`.\n\nHowever, you can configured your host machine to recognize these\nURLs by updating the `/etc/hosts` file by adding the following lines\nat the end.\n\n```\n# For use with data-playground cluster\n192.168.56.10 master\n192.168.56.11 worker1\n192.168.56.12 worker2\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fprogrammer-ke%2Fdata-playground","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fprogrammer-ke%2Fdata-playground","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fprogrammer-ke%2Fdata-playground/lists"}