{"id":21721477,"url":"https://github.com/informaticsmatters/ansible-galaxy-cloud","last_synced_at":"2026-05-16T18:33:22.755Z","repository":{"id":46815108,"uuid":"189268761","full_name":"InformaticsMatters/ansible-galaxy-cloud","owner":"InformaticsMatters","description":"Ansible play and roles to orchestrate the graph-processing cloud instance","archived":false,"fork":false,"pushed_at":"2023-01-24T23:28:13.000Z","size":173,"stargazers_count":2,"open_issues_count":2,"forks_count":1,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-08-02T00:14:50.554Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/InformaticsMatters.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-05-29T17:20:30.000Z","updated_at":"2022-02-07T19:52:38.000Z","dependencies_parsed_at":"2023-02-14T02:31:13.165Z","dependency_job_id":null,"html_url":"https://github.com/InformaticsMatters/ansible-galaxy-cloud","commit_stats":null,"previous_names":[],"tags_count":7,"template":false,"template_full_name":null,"purl":"pkg:github/InformaticsMatters/ansible-galaxy-cloud","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/InformaticsMatters%2Fansible-galaxy-cloud","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/InformaticsMatters%2Fansible-galaxy-cloud/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/InformaticsMatters%2Fansible-galaxy-cloud/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/InformaticsMatters%2Fansible-galaxy-cloud/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/InformaticsMatters","download_url":"https://codeload.github.com/InformaticsMatters/ansible-galaxy-cloud/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/InformaticsMatters%2Fansible-galaxy-cloud/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33114237,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-16T04:41:52.686Z","status":"ssl_error","status_checked_at":"2026-05-16T04:41:52.009Z","response_time":115,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-26T02:17:04.684Z","updated_at":"2026-05-16T18:33:22.739Z","avatar_url":"https://github.com/InformaticsMatters.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Ansible playbooks to form a Galaxy/Slurm cloud\n\n![lint](https://github.com/InformaticsMatters/ansible-galaxy-cloud/workflows/lint/badge.svg)\n\n[![GPLv3 license](https://img.shields.io/badge/License-GPLv3-blue.svg)](http://perso.crans.org/besson/LICENSE.html)\n\nThis repo contains [Ansible] playbooks for deploying a compute cluster. The \nprimary purpose is to act as a HPC resource for executing [Galaxy] and\n[Nextflow] workflows. The playbooks can be used to spin up nodes on an\nOpenStack cluster and deploy a [Slurm] environment to this. You can also\ndeploy [Pulsar] for executing Galaxy tools and Nextflow for running Nextflow\nworkflows. Both Pulsar and Nextflow use the Slurm cluster for execution.\nThe use of Pulsar allows Galaxy workflows to be executed without the need\nfor a shared filesystem between the Galaxy and Slurm environments.\n\n\u003e   **This project should not be confused with Ansible Galaxy**, Ansible's\n    community-driven repository of Ansible Roles.\n     \nCreating a complete cluster takes approx 15 mins depending on the number of\nnodes.\n\nSingularity is deployed to the Slurm cluster allowing jobs to be run using\nthat format of container. Currently Pulsar is not configured to use Singularity\ncontainers and instead deploys tool dependencies using `conda` but we aim to\nswitch to all execution using Singularity in the near future.\n\n\u003e   Deployment and configuration of Galaxy will be described elsewhere\n    in the near future.\n\nThis `ansible-galaxy-cloud` project contains a [site.yaml](site.yaml) file and\n_roles_ for the formation (and removal) of a [Nextflow]/[Slurm]/[MUNGE]/[Pulsar]\ncluster, a cluster suitable for [Galaxy] that consists of a **head** node and\none or more **worker** nodes that share an NFS-mounted volume attached to\nthe head node (mounted as `/data`). \n\n\u003e   As the cloud instances are unlikely to be accessibly from outside the\n    provider network this playbook is expected to be executed from a \n    suitably equipped _bastion_ node on the cloud provider (OpenStack).\n\n## Built-in applications\nThe following applications can be deployed to the cluster, each managed in\ntheir own `app_` roles: -\n\n-   [Nextflow]\n-   [Slurm] (and [MUNGE])\n-   [Pulsar]\n\nApplication components (nextflow, slurm, Pulsar) are deployed when their\ncorresponding `install` variable is set (see [group_vars/all/main.yaml](group_vars/all/main.yaml)).\nThe default in this repository is to enable and install all of them.\nIf you don't want to install a particular component (say nextflow), set its\n`install` variable to `no`, i.e. `install_nextflow: no`.\n\n## Deploying\nYou need to satisfy a few prerequisites before you can deploy the cluster\nand applications, as detailed below.\n\n### Prerequisite - a bastion node\nYou can use the [bastion] project to instantiate a suitable\nBastion instance from a workstation outside your cluster. From that\ninstance you should be able to create the cluster using the pre-configured\nPython virtual environment that it creates there.\n\nCreate your bastion and ssh to it. You'll find a clone of this\nrepository in its `~/git` directory and you can continue reading this\ndocument from there.\n\n\u003e   The bastion playbook will have also added a `parameters` file in the\n    project root containing your OpenStack credentials. Use this file to add\n    additional parameter values. \n\n### Prerequisite - provider environment\nYou will need to set provider-specific environment variables before you\ncan run this playbook. If you're using OpenStack you should `source` the\nkeystone file provided by your stack provider. This sets up the essential\ncredentials to create and access cloud resources.\n\n\u003e   If you've used the [ansible-bastion] playbook it will have written a\n    suitable set of authentication parameters for you in the root\n    of the initial clone of this project so you need not source anything.\n\n### Prerequisite - template environment\nInspect the [setenv-template.sh](setenv-template.sh) file in the root of the\nproject to see if there are any variables you need to define. Instructions for\nproviding these variables can be found in the template file.\n\nThe easiest way to over-ride the built-in values is to provide your\nown YAML-based [parameters] file (called `parameters`). The project\n`parameters` file is excluded from the repository using `.gitignore`.\nTo define your own shared volume size you could provide the following in\na `parameters` file: -\n\n    volume_size_g: 3000\n\n...and add the file to your Ansible command-line using `-e \"@parameters\"`\n\nThe playbooks rely on a number of _roles_ in the project. Where appropriate,\neach role exposes its key variables in a corresponding `defaults/main.yaml`\nfile but the main (common) variables have been placed in\n[group_vars/all/main.yaml](group_vars/all/main.yaml).\n\n\u003e   You are encouraged to review all the variables so that you can decide\n    whether you need to provide your own values for any of them.  \n\n### Prerequisite - playbook variables (OpenStack)\nAt the very least you should provide your own values for: -\n\n-   `instance_base_name`. A tag prepended to the cloud objects created\n    (instances and volumes)\n-   `instance_network`. A network name to use.\n-   `head_address`. The IP address (from a pool you own) to assign to the\n    head node (optional).\n-   `worker_count`. The number of worker instances that will be put in the\n    cluster.\n\n### Prerequisite - playbook variables (AWS)\nAt the very least you should provide your own values for: -\n\n-   `instance_base_name`. A tag prepended to the cloud objects created\n    (instances and volumes)\n-   `aws_vpc_subnet_id`. the ID of your VPC.\n-   `head_type`\n-   `worker_type`\n-   `volume_device`\n-   `worker_count`. The number of worker instances that will be put in the\n    cluster.\n\n### Installing public SSH keys\nAny `.pub` files found in the project's root directory will be considered\npublic SSH key-files and they will be added to the `centos` account of the\nhead node, allowing those users access to it.\n  \n## Running the playbook\nWith environment variables set and a `parameters` file written,\nrun the following on a suitably equipped bastion on your cloud provider: -\n\n    $ pip install --upgrade pip\n    $ pip install -r requirements.txt\n    $ ansible-galaxy install -r requirements.yml\n    \n    $ ansible-playbook site.yaml -e \"@parameters\"\n\n\u003e   You can avoid formatting the shared volume (for instance if\n    you have an existing non-ext4 volume) by adding `volume_initialise: no`\n    to your parameters file. By default the volume is expected to\n    be an `ext4` volume.\n    \n\u003e   CAUTION: The instance creation process creates instances and volumes whose\n    names begin with the value of the `instance_base_name` variable. Use a\n    base name that is unique to your cluster or, on deletion, you may find\n    you've lost more than you expected!\n\nYou can run separate 'sanity checks' with the `site-check` playbook: -\n\n    $ ansible-playbook site-check.yaml -e \"@parameters\"\n\n\u003e   The check ensures that slurm's `sinfo` command, when run on the head node,\n    does not report any _offline_ nodes and that all the total number of\n    nodes matches your `worker_count` value.\n\nAnd, to destroy the cluster: -\n\n    $ ansible-playbook unsite.yaml -e \"@parameters\"\n\n\u003e   To avoid deleting the shared volume, add `volume_delete: no` to your\n    parameters file. By default it is deleted when the cluster is deleted.\n    \n---\n\n[ansible]: https://www.ansible.com/\n[bastion]: https://github.com/InformaticsMatters/ansible-bastion\n[galaxy]: https://docs.galaxyproject.org/en/latest/index.html\n[munge]: https://dun.github.io/munge/\n[nextflow]: https://www.nextflow.io\n[parameters]: https://docs.ansible.com/ansible/latest/user_guide/playbooks_variables.html#passing-variables-on-the-command-line\n[pulsar]: https://pulsar.readthedocs.io/en/latest/index.html\n[slurm]: https://slurm.schedmd.com/documentation.html\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Finformaticsmatters%2Fansible-galaxy-cloud","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Finformaticsmatters%2Fansible-galaxy-cloud","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Finformaticsmatters%2Fansible-galaxy-cloud/lists"}