{"id":13312305,"url":"https://github.com/Xophmeister/osswarm","last_synced_at":"2025-03-10T20:32:42.758Z","repository":{"id":145629260,"uuid":"303712878","full_name":"Xophmeister/osswarm","owner":"Xophmeister","description":"Build and deploy a Docker Swarm cluster to OpenStack","archived":false,"fork":false,"pushed_at":"2020-10-23T15:43:37.000Z","size":79,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-02-18T02:32:38.690Z","etag":null,"topics":["alpine-linux","ansible","docker-swarm","infrastructure-as-code","openstack","packer","qemu","terraform"],"latest_commit_sha":null,"homepage":"","language":"HCL","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Xophmeister.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-10-13T13:30:00.000Z","updated_at":"2021-11-26T22:08:56.000Z","dependencies_parsed_at":null,"dependency_job_id":"7c7d8ffc-8777-4ca1-b414-24b9c5a2d5d2","html_url":"https://github.com/Xophmeister/osswarm","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Xophmeister%2Fosswarm","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Xophmeister%2Fosswarm/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Xophmeister%2Fosswarm/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Xophmeister%2Fosswarm/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Xophmeister","download_url":"https://codeload.github.com/Xophmeister/osswarm/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":242921825,"owners_count":20207103,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["alpine-linux","ansible","docker-swarm","infrastructure-as-code","openstack","packer","qemu","terraform"],"created_at":"2024-07-29T18:03:46.642Z","updated_at":"2025-03-10T20:32:42.749Z","avatar_url":"https://github.com/Xophmeister.png","language":"HCL","funding_links":[],"categories":[],"sub_categories":[],"readme":"# osswarm\n\nBuild and deploy a Docker Swarm cluster to OpenStack.\n\n## Usage\n\nThe [Infoblox][infoblox] configuration will first need to be defined in\n`infrastructure/dns/infoblox.yaml`. See [the example\nconfiguration](infrastructure/dns/infoblox.yaml.example) for details.\n\nThen, to build and deploy a Docker Swarm to the given OpenStack project,\nper the `~/.config/openstack/clouds.yaml` configuration, setting DNS\nrecords under the given `${DOMAIN}`, run:\n\n    make CLOUD=${OS_CLOUD} DOMAIN=${DOMAIN}\n\nOther Make variables can be set to fine-tune the cluster; please see the\nappropriate [documentation](#build-process) below. The following\nadditional Make targets are available:\n\n* **`cloud-clean`** \\\n  Destroy a deployed Docker Swarm and all its related infrastructure\n* **`clean`** \\\n  Clean the local build state, desynchronising it from any cloud state\n  (i.e., run `cloud-clean` first)\n* **`help`** \\\n  This documentation\n\nThe build is optimised for the OpenStack environment provided at the\n[Wellcome Sanger Institute][sanger]. It can be modified for different\nOpenStack environments by focusing on the following\n[configuration](#sanger-specific-configuration).\n\nNote that while Make is used to build and deploy the cluster, there will\nbe a disconnect between cloud and local build artefacts. No attempt is\nmade to synchronise these. For example, once an image is deployed,\nchanges to its sources will not trigger a full rebuild until `make\ncloud-clean` is manually invoked.\n\n\u003c!-- ## Appliance --\u003e\n\u003c!-- ## Deploying to the Cluster --\u003e\n\u003c!-- ## Monitoring --\u003e\n\n## Dependencies\n\n* [GNU Make][make]\n* [OpenSSH][openssh]\n* [Openstack CLI][openstack-cli] (tested with 5.4.0) and a `clouds.yaml`\n* [Packer][packer] (tested with 1.6.4)\n* [QEMU][qemu] (tested with 5.1.0)\n* [Ansible][ansible] (tested with 2.9.13)\n  * Ansible [`community.general` modules][ansible-modules] (tested with\n    1.2.0)\n* [Terraform][terraform] (tested with 0.13.5)\n  * Terraform [OpenStack provider][terraform-openstack] (tested with\n    1.32.0)\n  * Terraform [local provider][terraform-local] (tested with 2.0.0)\n  * Terraform [Infoblox provider][terraform-infoblox] (tested with\n    1.1.0)\n\n## Build Process\n\n### Image\n\nBuild and deploy an [Alpine Linux][alpine] image to the OpenStack\nproject. The image is built inside QEMU and is provisioned with:\n\n* [`cloud-init`][cloud-init]\n* [Netdata][netdata]\n* [Docker][docker]\n\nThe following Make variables are available to `image/Makefile`:\n\n* **`ARCH`** \\\n  The CPU architecture used by the image and the OpenStack environment\n  (defaults to `x86_64`)\n* **`ALPINE_VERSION`** \\\n  The version of Alpine Linux used by the image (defaults to `3.12.0`)\n* **`ALPINE_FLAVOUR`** \\\n  The \"flavour\" of Alpine Linux used by the image (defaults to `virt`)\n\nThe image will be built locally as `image/build/image.qcow2`. It will\nthen be deployed to the OpenStack project as `osswarm-${ARCH}` (e.g.,\n`osswarm-x86_64`).\n\nSSH access to the booted image is controlled by `cloud-init`, under the\n`alpine` user. Root SSH access is forbidden, but direct root access is\navailable from, for example, the OpenStack console, with the password\n`alpine`.\n\n### Infrastructure\n\nThe following Make variables are available to `infrastructure/Makefile`:\n\n* **`NAME`** \\\n  The name used for the cluster infrastructure (defaults to the current\n  username)\n* **`SECRET_KEY`** \\\n  The path to an SSH secret key (defaults to any match of `~/.ssh/id_*`;\n  falling back to `~/.ssh/osswarm-${NAME}_rsa`, which will be generated\n  if it doesn't exist)\n* **`FLAVOUR`** \\\n  The machine flavour for all nodes in the cluster (defaults to\n  `m2.medium`)\n* **`FAULT_TOLERANCE`** \\\n  The number of faults that can be tolerated by the cluster (defaults to\n  0, maximum of 3)\n* **`WORKERS`** \\\n  The number of worker nodes in the cluster (defaults to 1)\n* **`MANAGEMENT_SUBDOMAIN`** \\\n  The subdomain from which to manage the cluster (defaults to\n  `management.osswarm-${NAME}`)\n* **`SERVICE_SUBDOMAIN`** \\\n  The subdomain from which to access cluster services (defaults to\n  `services.osswarm-${NAME}`)\n\n\u003c!-- ### Orchestration --\u003e\n\n## Sanger-Specific Configuration\n\nThe following *may* need to be changed for a general OpenStack cloud:\n\n* **`image/src/docker.json`** \\\n  Configures Docker's networking to avoid internal conflicts\n* **Fault Tolerance** \\\n  Partitioning is simulated using hypervisor affinity (see the\n  `infrastructure/compute` Terraform module for details); distribution\n  over availability zones is preferable, if supported\n* **DNS** \\\n  DNS is provided by Infoblox and is defined in the following locations:\n  * **`infrastructure/Makefile`** \\\n    Checks for the Infoblox configuration\n  * **`infrastructure/dns`** \\\n    Terraform module utilising the Infoblox provider\n\n## Acknowledgements\n\nThanks, as always, to Sanger's [Informatics Support Group][sanger-isg]\nas custodians of OpenStack. Particularly to James Beal, Dave Holland and\nJon Nicholson, in this instance, for their help regarding the finer\npoints of virtualisation and networking infrastructure.\n\n## To Do\n- [ ] Image\n  - [x] Bootstrapping\n  - [ ] Provisioning\n    - [x] `cloud-init`\n    - [x] Netdata\n    - [ ] Docker\n      - [x] Docker\n      - [ ] Python library\n      - [ ] Docker Compose (?)\n    - [ ] Stretch: Cinder/S3 Docker volume drivers\n  - [x] `cloud-init` configuration\n  - [ ] Netdata configuration\n    - [ ] Docker monitoring\n    - [ ] Cluster monitoring\n  - [x] Docker configuration\n- [ ] Infrastructure\n  - [x] SSH key\n  - [x] Networking\n    - [x] Network\n    - [x] Security groups\n    - [x] Load Balancer\n    - [x] Floating IP\n    - [x] DNS\n  - [x] Compute\n  - [ ] Stretch: Fault tolerance\n- [ ] Orchestration\n  - [ ] Prometheus (Docker/Netdata monitoring)\n  - [ ] Swarm manager\n  - [ ] Swarm workers\n  - [ ] Traefik (?)\n- [ ] Documentation\n- [ ] Stretch: Refactoring\n\n\u003c!-- References --\u003e\n[alpine]:              https://alpinelinux.org/\n[ansible-modules]:     https://galaxy.ansible.com/community/general\n[ansible]:             https://www.ansible.com/\n[cloud-init]:          https://cloud-init.io/\n[docker]:              https://www.docker.com/\n[infoblox]:            https://www.infoblox.com/\n[make]:                https://www.gnu.org/software/make\n[netdata]:             https://www.netdata.cloud/\n[openssh]:             https://www.openssh.com/\n[openstack-cli]:       https://docs.openstack.org/python-openstackclient\n[packer]:              https://www.packer.io/\n[qemu]:                https://www.qemu.org/\n[sanger]:              https://www.sanger.ac.uk/\n[sanger-isg]:          https://www.sanger.ac.uk/group/informatics-support-group\n[terraform-infoblox]:  https://www.terraform.io/docs/providers/infoblox\n[terraform-local]:     https://www.terraform.io/docs/providers/local\n[terraform-openstack]: https://registry.terraform.io/providers/terraform-provider-openstack/openstack\n[terraform]:           https://www.terraform.io/\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FXophmeister%2Fosswarm","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FXophmeister%2Fosswarm","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FXophmeister%2Fosswarm/lists"}