{"id":14984530,"url":"https://github.com/lresende/ansible-spark-cluster","last_synced_at":"2025-10-09T03:07:31.684Z","repository":{"id":102476277,"uuid":"59455499","full_name":"lresende/ansible-spark-cluster","owner":"lresende","description":"Ansible roles to install an Spark Standalone cluster (HDFS/Spark/Jupyter Notebook) or Ambari based Spark cluster","archived":false,"fork":false,"pushed_at":"2024-01-30T13:54:26.000Z","size":207,"stargazers_count":61,"open_issues_count":1,"forks_count":35,"subscribers_count":7,"default_branch":"master","last_synced_at":"2025-10-09T03:03:16.711Z","etag":null,"topics":["anaconda","ansible","ansible-roles","apache-ambari","apache-spark","jupyter-enterprise-gateway","jupyter-notebook"],"latest_commit_sha":null,"homepage":"","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lresende.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-05-23T05:52:30.000Z","updated_at":"2024-12-30T22:22:37.000Z","dependencies_parsed_at":null,"dependency_job_id":"9cc63ac9-a3fd-4cb2-9e09-22c7c4e378af","html_url":"https://github.com/lresende/ansible-spark-cluster","commit_stats":{"total_commits":225,"total_committers":10,"mean_commits":22.5,"dds":0.12,"last_synced_commit":"cb9418161ec8ce9b31924cf1968de8f2de6c44bf"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/lresende/ansible-spark-cluster","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lresende%2Fansible-spark-cluster","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lresende%2Fansible-spark-cluster/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lresende%2Fansible-spark-cluster/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lresende%2Fansible-spark-cluster/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lresende","download_url":"https://codeload.github.com/lresende/ansible-spark-cluster/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lresende%2Fansible-spark-cluster/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279000775,"owners_count":26082911,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-09T02:00:07.460Z","response_time":59,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["anaconda","ansible","ansible-roles","apache-ambari","apache-spark","jupyter-enterprise-gateway","jupyter-notebook"],"created_at":"2024-09-24T14:09:13.353Z","updated_at":"2025-10-09T03:07:31.639Z","avatar_url":"https://github.com/lresende.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"\nThis repository defines multiple ansible roles to help deploying different modes of a Spark cluster and\nData Science Platform based on Anaconda and Jupyter Notebook stack\n\n# Requirements\n\nYou will need a driver machine with ansible installed and a clone of the current repository:\n\n* If you are running on cloud (public/private network)\n  * Install ansible on the edge node (with public ip)\n* if you are running on private cloud (public network access to all nodes)\n  * Install ansible on your laptop and drive the deployment from it\n\n### Installing Ansible on RHEL\n\n```\ncurl -O https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm\nsudo rpm -i epel-release-latest-7.noarch.rpm\nsudo yum update -y\nsudo yum install -y  ansible\n```\n\n### Installing Ansible on Mac\n\n* Install Annaconda\n* Use pip install ansible\n\n```\npip install --upgrade ansible\n```\n\n### Updating Ansible configuration\n\nIn order to have variable overriding from host inventory, please add the following configuration into your ~/.ansible.cfg file\n\n```\n[defaults]\nhost_key_checking = False\nhash_behaviour = merge\n```\n\n### Supported/Tested Platform\n\n* RHEL 7.x\n* Ansible 2.6.3\n\n\n# Defining your cluster deployment metadata (host inventory)\n\nAnsible uses 'host inventory' files to define the cluster configuration, nodes, and groups of nodes\nthat serves a given purpose (e.g. master node).\n\nBelow is a host inventory sample definition:\n\n```\n[all:vars]\nansible_connection=ssh\n#ansible_user=root\n#ansible_ssh_private_key_file=~/.ssh/ibm_rsa\ngather_facts=True\ngathering=smart\nhost_key_checking=False\ninstall_java=True\ninstall_temp_dir=/tmp/ansible-install\ninstall_dir=/opt\npython_version=2\n\n[master]\nlresende-elyra-node-1   ansible_host=IP   ansible_host_private=IP  ansible_host_id=1\n\n[nodes]\nlresende-elyra-node-2   ansible_host=IP   ansible_host_private=IP  ansible_host_id=2\nlresende-elyra-node-3   ansible_host=IP   ansible_host_private=IP  ansible_host_id=3\nlresende-elyra-node-4   ansible_host=IP   ansible_host_private=IP  ansible_host_id=4\nlresende-elyra-node-5   ansible_host=IP   ansible_host_private=IP  ansible_host_id=5\n\n```\n\nSome specific configurations are:\n\n* install_java=True : install/update java 8\n* install_temp_dir=/tmp/ansible-install : temporary folder used for install files\n* install_dir=/opt : where packages are installed (e.g. Spark)\n* python_version=2 : python version to use, influence which version of Anaconda to download\n\n**Note:** ansible_host_id is only used when deploying a \"Spark Standalone\" cluster.\n**Note:** Ambari is currently only supporting Python 2.x\n\n# Deploying Spark using Ambari and HDP distribution\n\nIn this scenario, a minimal blueprint is used to deploy the required components\nto run YARN and Spark.\n\n### Related ansible roles\n\n* **Common**  Deploys Java and common dependencies\n* **Ambari** Deploys Ambari cluster with HDP Stack\n\n### Deployment playbook\n\nThe sample playbook below can be used to deploy an Spark using an HDP distribution\n\n```\n- name: ambari setup\n  hosts: all\n  remote_user: root\n  roles:\n    - role: common\n    - role: ambari\n```\n\n### Deploying\n\n```\nansible-playbook --verbose \u003cdeployment playbook.yml\u003e -i \u003chosts inventory\u003e\n```\n\nExample:\n\n```\nansible-playbook --verbose setup-ambari.yml -c paramiko -i hosts-fyre-ambari\n```\n\n# Deploying Spark standalone\n\nIn this scenario, a Standalone Spark cluster will be deployed with a few optional components.\n\n### Related ansible roles\n\n* **Common**  Deploys Java and common dependencies\n* **HDFS** Deploys HDFS filesystem using slave nodes as data nodes\n* **Spark** Deploys Spark in Standalone mode using slave nodes as workers\n* **Spark-CLuster-Admin** Utility scripts for managing Spark cluster\n* **ElasticSearch** Deploy ElasticSearch nodes on all slave nodes\n* **Zookeeper** Depoys Zookeeper on all nodes (required by Kafka)\n* **Kafka** Deploy Kafka nodes on all slave nodes\n\n### Deployment playbook\n\n```\n- name: spark setup\n  hosts: all\n  remote_user: root\n  roles:\n    - role: common\n    - role: hdfs\n    - role: spark\n    - role: spark-cluster-admin\n\n```\n\n**Note:** When deploying Kafka, the Zookeeper role is required\n\n### Deploying\n\n\n```\nansible-playbook --verbose \u003cdeployment playbook.yml\u003e -i \u003chosts inventory\u003e\n```\n\nExample:\n\n```\nansible-playbook --verbose setup-spark-standalone.yml -c paramiko -i hosts-fyre-spark\n```\n\n# Deploying Data Science Platform components\n\nIn this scenario, an existing Spark cluster is updated with necessary components to build a data science platform\nbased on Anaconda and Jupyter Notebook stack.\n\n### Related ansible roles\n\n* **Anaconda** Deploys Anaconda Python distribution on all nodes\n* **Notebook** Deploys Notebook Platform\n\n### Deployment playbook\n\n```\n- name: anaconda\n  hosts: all\n  vars:\n    anaconda:\n      update_path: true\n  remote_user: root\n  roles:\n   - role: anaconda\n\n- name: notebook platform dependencies\n  hosts: all\n  vars:\n    notebook:\n      use_anaconda: true\n      deploy_kernelspecs_to_workers: false\n  remote_user: root\n  roles:\n    - role: notebook\n```\n\n**Playbook Configuration**\n\n* **use_anaconda**: Flag to identify if anaconda is available and should be used as python package manager\n* **deploy_kernelspecs_to_workers**: optionally deploy kernelspecs for Python, R, and Scala to all nodes\n\n\n# Legal Disclaimers\n\n* The **Ambari** role will install [MySQL community edition](https://www.mysql.com/products/community/)\nwhich is available under GPL license.\n\n* The **Notebook** role will install [R](https://www.r-project.org/) which is available under [GPL2 | GPL 3](https://www.r-project.org/Licenses/)\n\nBy deploying these packages via the ansible utility scripts in this project you are accepting the\nlicense terms for these components.\n \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flresende%2Fansible-spark-cluster","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flresende%2Fansible-spark-cluster","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flresende%2Fansible-spark-cluster/lists"}