{"id":14553365,"url":"https://github.com/ComputeCanada/magic_castle","last_synced_at":"2025-09-03T22:31:00.688Z","repository":{"id":36158244,"uuid":"204500239","full_name":"ComputeCanada/magic_castle","owner":"ComputeCanada","description":"Terraform modules to replicate the HPC user experience in the cloud","archived":false,"fork":false,"pushed_at":"2024-11-19T12:20:31.000Z","size":3557,"stargazers_count":135,"open_issues_count":33,"forks_count":39,"subscribers_count":10,"default_branch":"main","last_synced_at":"2024-11-19T13:24:19.960Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"HCL","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ComputeCanada.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-08-26T15:01:12.000Z","updated_at":"2024-11-19T12:18:04.000Z","dependencies_parsed_at":"2023-09-27T23:25:41.346Z","dependency_job_id":"594deab8-ab5c-4b83-bf34-5f01c9726489","html_url":"https://github.com/ComputeCanada/magic_castle","commit_stats":null,"previous_names":[],"tags_count":106,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ComputeCanada%2Fmagic_castle","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ComputeCanada%2Fmagic_castle/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ComputeCanada%2Fmagic_castle/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ComputeCanada%2Fmagic_castle/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ComputeCanada","download_url":"https://codeload.github.com/ComputeCanada/magic_castle/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":231921736,"owners_count":18446387,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-09-06T16:00:47.789Z","updated_at":"2025-09-03T22:31:00.675Z","avatar_url":"https://github.com/ComputeCanada.png","language":"HCL","funding_links":[],"categories":["Resources","Management Tool"],"sub_categories":["Other/Wikis"],"readme":"# Magic Castle\n\n\u003c!-- markdown-link-check-disable-next-line --\u003e\n[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.4895357.svg)](https://doi.org/10.5281/zenodo.4895357)\n[![Build Status](https://github.com/ComputeCanada/magic_castle/actions/workflows/test.yaml/badge.svg)](https://github.com/ComputeCanada/magic_castle/actions/workflows/test.yaml)\n\n\u003cimg src=\"https://github.com/computecanada/magic_castle/raw/assets/logo.png\" width=\"150\"\u003e\n\n[The Digital Research Alliance of Canada](https://www.alliancecan.ca/) provides HPC infrastructure and support to every academic research institution in Canada. The Alliance [uses CVMFS](https://docs.alliancecan.ca/wiki/Accessing_CVMFS), a software distribution system [developed at CERN](https://cernvm.cern.ch/fs/), to make its research software stack available on its HPC clusters, and anywhere else with internet access. This enables replication of the user experience outside of The Alliance physical infrastructure.\n\nFrom these new possibilities emerged an open-source software project named Magic Castle, which aims to recreate the HPC user experience in public clouds. Magic Castle uses the open-source software [Terraform](https://www.terraform.io) and HashiCorp Language (HCL) to define the virtual machines, volumes, and networks that are required to replicate a virtual HPC infrastructure. The infrastructure definition is packaged as a Terraform module that users can customize as they require. After deployment, the user is provided with a complete HPC cluster software environment including a Slurm scheduler, a Globus Endpoint, JupyterHub, LDAP, DNS, and over 3000 research software applications compiled by experts with [EasyBuild](https://github.com/easybuilders/easybuild). Magic Castle is compatible with AWS, Microsoft Azure, Google Cloud, OpenStack, and OVH.\n\n## Setup\n\n- Install [Terraform](https://releases.hashicorp.com/terraform/) (\u003e= 1.5.7)\n- Download the [latest release of Magic Castle](https://github.com/ComputeCanada/magic_castle/releases) for the cloud provider you wish to use.\n- Uncompress the release\n- Follow the instructions\n  - [OpenStack Cloud](openstack/README.md)\n  - [Amazon Web Services (AWS)](aws/README.md)\n  - [Microsoft Azure](azure/README.md)\n  - [Google Cloud Platform (GCP)](gcp/README.md)\n  - [OVH Public Cloud (OVH)](ovh/README.md)\n  - [Incus](incus/README.md)\n- For more details, refer to [Magic Castle Documentation](docs)\n\n## How Magic Castle Works\n\nThis software project integrates multiple parts that come into play at\ndifferent steps of spawning the cluster. The following list\nenumerates the steps involved in order for users to better\ngrasp what is happening when they create clusters.\n\nWe will refer to the user of Magic Castle as the operator.\n\n1. After downloading the latest release of the cloud provider of choice\nand adapting the main configuration file, the operator launches\nTerraform and accepts the proposed plan.\n2. Terraform communicates with the cloud provider REST API and requests\nthe creation of the virtual machines.\n3. For each virtual machine creation request, Magic Castle\nprovides a [cloud-init](https://cloudinit.readthedocs.io/en/latest/) file. This\nfile is used to initialize the virtual machine base configuration and installs\npuppet agent. The cloud-init file of the `puppet` tagged virtual machine installs\nand configures a Puppet primary server.\n4. Terraform uploads on the Puppet primary server instance a YAML file containing\ninformation about the roles of each instances specified as tags.\n5. The Puppet agents communicate with the Puppet primary server to retrieve\nand apply their configuration based on the tags defined in the preceding YAML file.\n\n## Talks, slides and videos\n\n- [FOSDEM 2020 - Magic Castle: Terraforming the Cloud for HPC](https://fosdem.org/2020/schedule/event/magic_castle/) (2020-02-02)\n- [EasyBuild User Meeting 2021 - Magic Castle: Terraforming the Cloud for HPC](https://www.youtube.com/watch?v=54ph7f2-AL4) (2021-01-27)\n- [Campus Research Computing Consortium - Clusters in the Sky: How Canada is Building Beyond Iaas Cloud with Magic Castle](https://www.youtube.com/watch?v=jWCyUeGmm-8) (2021-05-20)\n- [EESSI User Meeting 2022 - Magic Castle: Preview of MC 12](https://www.youtube.com/watch?v=XGnxbIHJLmw\u0026list=PL6_PkP_6pUtb7_tovj1V__y4ii_AjhroJ\u0026index=6) (2022-09-14)\n- [HashiTalks Québec 2022 - Magic Castle: Le CIP à l'échelle grâce à Terraform Cloud](https://www.youtube.com/watch?v=3Mg4gMmkktM) (2022-09-29)\n- [SIGHPC Education - Magic Castle: Terraforming the cloud to teach HPC](https://sighpceducation.acm.org/events/magic_castle/) (2024-02-08)\n\n## List of other cloud HPC cluster open-source projects\n\n- [AWS ParallelCluster](https://github.com/aws/aws-parallelcluster) [AWS]\n- [Cluster in the cloud](https://github.com/clusterinthecloud) [AWS, GCP, Oracle]\n- [Elasticluster](https://github.com/elasticluster/elasticluster) [AWS, GCP, OpenStack]\n- [Google Cluster Toolkit](https://github.com/GoogleCloudPlatform/cluster-toolkit) [GCP]\n- [illume-v2](https://github.com/jamierajewski/illume-v2/) [OpenStack]\n- [NVIDIA DeepOps](https://github.com/NVIDIA/deepops) [Ansible playbooks only]\n- [StackHPC Ansible Role OpenHPC](https://github.com/stackhpc/ansible-role-openhpc) [Ansible Role for OpenStack]\n\n\n\u003e When I think about the DevOps landscape, we have so many people just like chefs in a restaurant that are experimenting with different ways of doing things. Once they get it, then they create those recipes. Those recipes in our world is source code. [...] That's why we will always have duplicates and similar projects, because there's going to be one ingredient that's going to be slightly different to make you preferred over something else\n\n[Kelsey Hightower, Sourcegraph Podcast, Episode 16, 2020](https://web.archive.org/web/20240527182945/https://sourcegraph.com/podcast/kelsey-hightower)\n\n## Contributing / Customizing\n\nRefer to the [reference design](docs/design.md) and the [developer documentation](docs/developers.md).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FComputeCanada%2Fmagic_castle","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FComputeCanada%2Fmagic_castle","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FComputeCanada%2Fmagic_castle/lists"}