{"id":19402855,"url":"https://github.com/cscfi/pouta-virtualcluster","last_synced_at":"2025-04-24T07:32:31.923Z","repository":{"id":20107938,"uuid":"23377603","full_name":"CSCfi/pouta-virtualcluster","owner":"CSCfi","description":"Scripts and Ansible playbooks to assist in running a virtual cluster in pouta.csc.fi","archived":false,"fork":false,"pushed_at":"2021-03-08T15:09:44.000Z","size":240,"stargazers_count":3,"open_issues_count":2,"forks_count":2,"subscribers_count":42,"default_branch":"master","last_synced_at":"2023-08-02T06:25:42.961Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":"amitnv/OSCAR","license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/CSCfi.png","metadata":{"files":{"readme":"README.rst","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2014-08-27T05:12:36.000Z","updated_at":"2021-03-08T15:09:47.000Z","dependencies_parsed_at":"2022-09-02T13:51:03.080Z","dependency_job_id":null,"html_url":"https://github.com/CSCfi/pouta-virtualcluster","commit_stats":null,"previous_names":[],"tags_count":3,"template":null,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CSCfi%2Fpouta-virtualcluster","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CSCfi%2Fpouta-virtualcluster/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CSCfi%2Fpouta-virtualcluster/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CSCfi%2Fpouta-virtualcluster/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/CSCfi","download_url":"https://codeload.github.com/CSCfi/pouta-virtualcluster/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":223945498,"owners_count":17229646,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-10T11:26:03.963Z","updated_at":"2024-11-10T11:26:04.574Z","avatar_url":"https://github.com/CSCfi.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"============\nPoutacluster\n============\n\n**NOTE: This project is not maintained anymore**\n\nPouta-virtualcluster is a helper script and a set of Ansible playbooks to quickly setup a cluster in **pouta.csc.fi**\nIaaS service. It draws heavily from the *ElastiCluster* by *Grid Computing Competence Center, University of Zurich*,\nespecially most of the Ansible playbooks are based on ElastiCluster. Provisioning code, however, is written from scratch\nto support volumes (persistent storage), server affinity and security group relations. It runs directly against\nOpenStack Python API.\n\nCurrently poutacluster can provision:\n\n* Basic cluster infra on top of CentOS 6.6 or Ubuntu 14.04 with one frontend and N compute nodes\n\n  - frontend and compute nodes can have different images, flavors and keys\n  - frontend has a public IP\n  - there is a volume for local persistent data\n  - frontend has a separate shared volume, exported with NFS to the worker nodes from */mnt/shared_data*\n  - frontend also exports */home*\n\n* Ganglia for monitoring\n* GridEngine for batch processing\n* Apache Hadoop 1.2.1\n* Apache Spark 1.3.1\n\nHow it works\n============\n\nThere are two separate parts:\n\n- provisioning the VMs and related resources\n- configuring the VMs with Ansible\n\nProvisioning\n------------\n\nProvisioning VMs for cluster goes roughly like this:\n\n* The cluster configuration is read from YAML file provided by the user.\n* The current state of provisioned resources is loaded using OpenStack APIs from pouta.csc.fi\n* Missing VMs are provisioned\n\n  - first frontend, then appropriate number of nodes\n  - naming: *[cluster-name]-fe* and *[cluster-name]-node[number]*. If your cluster name was be *my-cluster*,\n    you would get\n\n      + my-cluster-fe\n      + my-cluster-node01\n      + my-cluster-node02\n      + ...\n\n  - VMs are launched from the specified image with specified flavor. They are placed in an OpenStack server group with\n    anti-affinity policy to distribute them on separate hosts for better fault tolerance.\n  - volumes are created or reused and attached\n  - template security groups are created if these don't exist already\n\n\n\n* Ansible host inventory file is created, mapping VMs to assigned roles\n\nShutdown is done in reverse order, starting with the last nodes and finally shutting down frontend. Only after the cluster\nhas been shut down, you can wipe the persistent storage for the cluster.\n\nSee the example below.\n\nConfiguration\n-------------\n\nThere is a collection of Ansible playbooks in *ansible/playbooks* directory, all collected to *ansible/playbooks/site.yml*.\nBased on the role/group assignment in Ansible inventory (which in turn is generated from provisioning state and cluster\nconfiguration in cluster.yml by poutacluster script) a set of tasks is launched on each VM.\n\nThe playbooks are designed to be idempotent, so that you should be able to run them at any time, and they will only make\nchanges in the configuration if necessary. Also, if you change the number of nodes in the cluster, playbooks can be\nre-applied to reflect the change.\n\nPrerequisites\n=============\n\nGetting started with pouta.csc.fi\n---------------------------------\n\nTo use CSC Pouta cloud environment you will need\n\n* credentials for Pouta\n* basic knowledge of Pouta and OpenStack\n\nSee https://research.csc.fi/pouta-user-guide for details\n\nSetting up bastion host\n-----------------------\n\nNote: Here we assume that you are already past the basic steps mentioned above.\n\nCreate a small management VM to act as a \"bastion\" host (http://en.wikipedia.org/wiki/Bastion_host)\n\n* Log into https://pouta.csc.fi\n* If you are member of multiple projects, select the desired one from the drop down list on top left\n* Create a new security group called, for example, 'bastion'\n\n  - go to *Access and Security -\u003e Security groups -\u003e Create Security Group*\n  - add rules to allow ssh for yourself and other admins\n  - normal users do not need to access this hosts\n  - keep the access list as small as possible to minimize exposure\n\n* Create an access key if you don't already have one\n\n  - go to *Access and Security -\u003e Keypairs -\u003e Create/Import Keypair*\n\n* Boot a new VM from the latest CentOS 6 image that is provided by CSC\n\n  - go to *Instances -\u003e Launch Instance*\n  - Image: Latest public Centos image (CentOS 6.6 at the time of writing)\n  - Flavor: tiny\n  - Keypair: select your key\n  - Security Groups: select only *bastion*\n  - Network: select the desired network (you probably only have one, which is the default and ok)\n  - Launch\n\n* Associate a floating IP (allocate one for the project if you don't already have a spare)\n\n* Log in to the bastion host with ssh as *cloud-user* user, depending on the image::\n\n    ssh cloud-user@86.50.16X.XXX:\n    \n* update the system and reboot to bring the host up to date::\n\n    sudo yum update -y \u0026\u0026 sudo reboot\n\n* add OpenStack repository for installing the client packages::\n\n    sudo yum install -y yum-plugin-priorities\n    sudo yum install -y http://repos.fedorapeople.org/repos/openstack/openstack-icehouse/rdo-release-icehouse-3.noarch.rpm\n\n* install openssh-clients, bash-completion, git, Python yaml-support, Ansible and OpenStack clients::\n\n    sudo yum install -y bash-completion openssh-clients python-openstackclient git python-yaml ansible\n\n* import your OpenStack command line access configuration\n\n  - see https://research.csc.fi/pouta-credentials how to export the openrc\n  - use scp to copy the file to bastion from your workstation::\n\n    [me@workstation]$ scp openrc.sh cloud-user@86.50.168.XXX:\n\n* test the clients (enter your Pouta password when asked for)::\n\n    source openrc.sh\n\n    nova image-list\n\n* create a new key for cluster access (keeping bastion access and cluster access separate is a good practice)::\n\n    ssh-keygen\n\n* import the key::\n\n    nova keypair-add  --pub-key .ssh/id_rsa.pub cluster-key\n\n* make a backup copy of the keypair, so you don't lose it if something bad happens to your bastion host\n\n    [me@workstation]$ scp -r cloud-user@86.50.168.XXX:.ssh dot_ssh_from_bastion\n\n\nInstallation\n============\n\nNext we install *poutacluster* on the bastion host::\n\n    cd\n    git clone https://github.com/CSC-IT-Center-for-Science/pouta-virtualcluster\n    mkdir ~/bin\n    ln -s ~/pouta-virtualcluster/python/poutacluster.py ~/bin/poutacluster\n    ln -s ~/pouta-virtualcluster/ansible ~/ansible\n    cp ~/pouta-virtualcluster/ansible/cfg/ansible-centos6.cfg ~/.ansible.cfg\n\nNow *poutacluster -h* should give you basic usage. See examples below for more details.\n\nExamples\n========\n\nCluster life-cycle walk-through\n-------------------------------\n\nLog in to the bastion host, source the openrc.sh and start deploying the cluster:\n\n* create a new subdirectory for the cluster configuration in your home directory::\n\n    mkdir ~/my-cluster\n    cd ~/my-cluster\n\n* copy *cluster.yml.template* to *~/my-cluster/cluster.yml* and open it for editing::\n\n    cp ~/pouta-virtualcluster/cluster.yml.template cluster.yml\n    vi cluster.yml\n\n* you can also edit the definition on your workstation and then copy it over to the bastion. The template can\n  be found at https://github.com/CSC-IT-Center-for-Science/pouta-virtualcluster\n\n* check, edit or fill in:\n\n  - cluster name (only characters a-z, 0-9 and a hyphen '-' are allowed)\n  - ssh-key name\n  - public IP (use 'auto' for any unused floating IP available for project)\n  - image\n  - flavors\n  - volume sizes (NOTE: when testing, keep the volume size small, otherwise deleting the cluster storage will take a long time). Keep the volume names and order as they are.\n  - groups (you can comment out software groups that you don't need)\n\n* bring the cluster up with a frontend and two nodes::\n\n    poutacluster up 2\n\n* check what *info* shows about the state::\n\n    poutacluster info\n\n* ssh in to the the frontend and test the cluster\n\n* check the web interfaces for Ganglia, Hadoop and Spark. Urls are printed out at the end of the run\n\n* try resetting the nodes::\n\n    poutacluster reset_nodes\n\n* bring the cluster down to save credits (permanent data on volumes is still preserved)::\n\n    poutacluster down\n\n* bring the cluster up again, this time with 4 nodes::\n\n    poutacluster up 4\n\n* destroy the cluster by first bringing it down and then getting rid of the volumes::\n\n    poutacluster down\n    poutacluster destroy_volumes\n\nGeneral cluster\n---------------\nCheck uptime on all the hosts on cluster frontend::\n\n    pdsh -w mycluster-node[01-04] uptime\n\nReboot the nodes::\n\n    sudo pdsh -w mycluster-node[01-04] reboot\n\nAdd a user and test NFS::\n\n    sudo useradd --create-home --shell=/bin/bash -u 1010 bill\n    sudo passwd bill\n    sudo pdsh -w mycluster-node[01-04] useradd --shell=/bin/bash -u 1010 --no-create-home bill\n    sudo su - bill\n    ssh mycluster-node01 touch hello-from-node01\n    ls\n    exit\n\n\nGridEngine\n----------\n\nAs a normal user (or cloud-user), test job submission::\n\n    cd\n    for i in {001..016}; do qsub -b y -N uname-$i uname -a; done\n    cat uname-0*.o*\n\nThe jobs are probably executed on different nodes.\n\nCreate a few empty 1G files on the NFS share and calculate sha256 sums over zero data::\n\n    sudo mkdir /mnt/shared_data/tmp\n    sudo chmod 1777 /mnt/shared_data/tmp\n    for i in {001..050}; do truncate --size 1G /mnt/shared_data/tmp/zeroes.1G.$i; done\n    for i in {001..050}; do qsub -b y -N shasum-$i sha256sum /mnt/shared_data/tmp/zeroes.1G.$i; done\n    cat shasum-*.o*\n\nDuring the test, you should see quite a lot of network traffic from frontend out to the nodes, as the sparse files are\nread and NFS is feeding a lot of zeroes to the sha256sum -processes on the nodes. You can open another terminal (or use\na multiplexer like *tmux* or *screen*) and run *dstat -taf 10* for some real time monitoring on the frontend.\n\nHadoop\n------\n\nRunning terasort with 100GB dataset. Make sure you have big enough *shared_data* and *local_data* -volumes provisioned.::\n\n    # generate data (with 8 'small' nodes, this should take around 6 minutes)\n    # map tasks tuned to match the size of the cluster (8 small nodes, 4 cores each)\n\n    hadoop jar /usr/share/hadoop/hadoop-examples-1.2.1.jar teragen -Dmapred.map.tasks=32 1000000000 /user/hduser/terasort-input\n\n    # sort (with 8 'small' nodes, this should take around 15 minutes)\n    # reduce tasks tuned to match the size of the cluster (8 small nodes, 4 cores each)\n\n    hadoop jar /usr/share/hadoop/hadoop-examples-1.2.1.jar terasort -Dmapred.reduce.tasks=32 /user/hduser/terasort-input /user/hduser/terasort-output\n\n\nSome useful admin commands::\n\n    # get status report for hdfs (HADOOP_USER_NAME is needed for admin access)\n\n    HADOOP_USER_NAME=hdfs hadoop dfsadmin -report\n\n    # balancing the HDFS data across nodes: set the balancer bandwidth to 100MB/sec and run balancer\n    HADOOP_USER_NAME=hdfs hadoop dfsadmin -setBalancerBandwidth 100000000\n    HADOOP_USER_NAME=hdfs hadoop balancer -threshold 1\n\n    # check HDFS\n    HADOOP_USER_NAME=hdfs hadoop fsck /\n\n    # list running jobs\n    hadoop job -list\n\nSpark\n-----\nWord count example with English language DBpedia dump. The file is replicated 20 times,\nresulting 50GB of text data. Make sure you have big enough *shared_data* and *local_data* -volumes provisioned.\n\nFirst download the input data and replicate it to get more data::\n\n    sudo mkdir /mnt/shared_data/tmp\n    sudo chmod 1777 /mnt/shared_data/tmp\n    cd /mnt/shared_data/tmp\n    mkdir dbpedia\n    cd dbpedia\n    curl -sS http://data.dws.informatik.uni-mannheim.de/dbpedia/2014/en/long_abstracts_en.ttl.bz2 \\\n    | bunzip2 -c | tee dbpedia_long_abstracts_en.ttl.{01..20} \u003e /dev/null\n\nThen upload it to HDFS::\n\n    hadoop distcp file:///mnt/shared_data/tmp/dbpedia hdfs://$HOSTNAME:9000/sparktest/dbpedia\n\nMake sure Spark is running::\n\n    sudo /opt/spark/sbin/start-all.sh\n\nStart a Spark shell with 8GB worker nodes in the cluster::\n\n    /opt/spark/bin/spark-shell --master spark://$HOSTNAME:7077 --executor-memory 8G\n\nNote that logs will be printed to the shell and it might look like the prompt is not ready. Hit *Enter* a few times to\nget the *scala\u003e* -prompt.\n\nFirst we can test reading the input from NFS and writing the results to HDFS::\n\n    import java.net._\n    val hostname = InetAddress.getLocalHost.getHostName\n    val dbpediaText = sc.textFile(\"file:///mnt/shared_data/tmp/dbpedia\")\n    val counts = dbpediaText.flatMap(line =\u003e line.split(\" \")).map(word =\u003e (word, 1)).reduceByKey(_ + _)\n    counts.saveAsTextFile(\"hdfs://\"+hostname+\":9000/sparktest/output-1\")\n\nNote: Spark is lazy in evaluating the expressions, so no processing will be done before the last line.\n\nThen test HDFS to HDFS (should be faster)::\n\n    import java.net._\n    val hostname = InetAddress.getLocalHost.getHostName\n    val dbpediaText = sc.textFile(\"hdfs://\"+hostname+\":9000/sparktest/dbpedia\")\n    val counts = dbpediaText.flatMap(line =\u003e line.split(\" \")).map(word =\u003e (word, 1)).reduceByKey(_ + _)\n    counts.saveAsTextFile(\"hdfs://\"+hostname+\":9000/sparktest/output-2\")\n\nPrinting the Top 50 words longer than 3 characters in the dbpedia dump::\n\n    import java.net._\n    val hostname = InetAddress.getLocalHost.getHostName\n    val dbpediaText = sc.textFile(\"hdfs://\"+hostname+\":9000/sparktest/dbpedia\")\n    val filtered = dbpediaText.flatMap(line =\u003e line.toLowerCase().split(\" \")).filter(word =\u003e word.matches(\"[a-z]*\")).filter(word =\u003e word.length()\u003e3)\n    val counts=filtered.map(word =\u003e (word, 1)).reduceByKey(_ + _)\n    val top=counts.map(x =\u003e (x._2, x._1)).sortByKey(false).take(50)\n    for (i \u003c- top){ println (i._2+\"\\t\"+i._1) }\n\nProbably these hadoop dfs -commands will be handy, too::\n\n    hadoop dfs -ls /sparktest\n    hadoop dfs -du /sparktest/*\n    hadoop dfs -rmr /sparktest/output-1-gone-wrong\n\nMissing bits\n============\n\n* online resize\n\n* persistent home directory\n\n* HDFS resize has to be done manually when scaling down\n\n* Spark does not start automatically after a reboot. To start it run::\n\n    # sudo /opt/spark/sbin/start-all.sh\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcscfi%2Fpouta-virtualcluster","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcscfi%2Fpouta-virtualcluster","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcscfi%2Fpouta-virtualcluster/lists"}