Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/zenml-io/terraform-gcp-zenml-stack
A Terraform module for provisioning and registering a cloud ZenML stack in GCP.
https://github.com/zenml-io/terraform-gcp-zenml-stack
gcp terraform-module zenml
Last synced: 11 days ago
JSON representation
A Terraform module for provisioning and registering a cloud ZenML stack in GCP.
- Host: GitHub
- URL: https://github.com/zenml-io/terraform-gcp-zenml-stack
- Owner: zenml-io
- Created: 2024-07-18T13:09:00.000Z (7 months ago)
- Default Branch: main
- Last Pushed: 2025-02-04T09:04:18.000Z (16 days ago)
- Last Synced: 2025-02-04T10:19:38.485Z (16 days ago)
- Topics: gcp, terraform-module, zenml
- Language: HCL
- Homepage:
- Size: 40 KB
- Stars: 0
- Watchers: 2
- Forks: 1
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
ZenML Cloud Infrastructure Setup
---
## ⭐️ Show Your Support
If you find this project helpful, please consider giving ZenML a star on GitHub. Your support helps promote the project and lets others know it's worth checking out.
Thank you for your support! 🌟
[](https://github.com/zenml-io/zenml/stargazers)
## 🚀 Overview
This Terraform module sets up the necessary GCP infrastructure for a [ZenML](https://zenml.io) stack. It provisions various GCP services and resources, and registers [a ZenML stack](https://docs.zenml.io/user-guide/production-guide/understand-stacks) using these resources with your ZenML server, allowing you to create an internal MLOps platform for your entire machine learning team.
## 🛠 Prerequisites
- Terraform installed (version >= 1.9")
- GCP account set up
- To authenticate with GCP, you need to have [the `gcloud` CLI](https://cloud.google.com/sdk/gcloud)
installed on your machine and you need to have run `gcloud auth application-default login`
to set up your credentials.
- You'll need a Zenml server (version >= 0.62.0) deployed in a remote setting where it can be accessed from GCP. You have the option to either [self-host a ZenML server](https://docs.zenml.io/getting-started/deploying-zenml) or [register for a free ZenML Pro account](https://cloud.zenml.io/signup). Once you have a ZenML Server set up, you also need to create [a ZenML Service Account API key](https://docs.zenml.io/how-to/connecting-to-zenml/connect-with-a-service-account) for your ZenML Server. You can do this by running the following command in a terminal where you have the ZenML CLI installed:
```bash
zenml service-account create
```
- This Terraform module uses [the ZenML Terraform provider](https://registry.terraform.io/providers/zenml-io/zenml/latest/docs). It is recommended to use environment variables to configure the ZenML Terraform provider with the API key and server URL. You can set the environment variables as follows:
```bash
export ZENML_SERVER_URL="https://your-zenml-server.com"
export ZENML_API_KEY="your-api-key"
```
## 🏗 GCP Resources Created
The Terraform module in this repository creates the following resources in your GCP project:
1. a GCS bucket
2. a Google Artifact Registry
3. a Cloud Composer environment (only if the `orchestrator` variable is set to `airflow`)
4. a Service Account with the minimum necessary permissions to access the GCS bucket, the Google Artifact Registry and the GCP project to build and push container images with Google Cloud Build, store artifacts and run pipelines with Vertex AI, SkyPilot or GCP Cloud Composer.
5. depending on the target ZenML Server capabilities, different authentication methods are used:
* for a self-hosted ZenML server, a Service Account Key is generated and shared with the ZenML server
* for a ZenML Pro account, GCP Workload Identity Federation is used to authenticate with the ZenML server, so that no sensitive credentials are shared with the ZenML server. For this, a GCP Workload Identity Pool and a GCP Workload Identity Provider are created and linked to the GCP Service Account. There's only one exception: when the SkyPilot orchestrator is used, this authentication method is not supported, so the Service Account Key is used instead.
## 🧩 ZenML Stack Components
The Terraform module automatically registers a fully functional GCP [ZenML stack](https://docs.zenml.io/user-guide/production-guide/understand-stacks) directly with your ZenML server. The ZenML stack is based on the provisioned GCP resources and is ready to be used to run machine learning pipelines.
The ZenML stack configuration is the following:
1. an GCP Artifact Store linked to the GCS bucket via an AWS Service Connector configured with IAM role credentials
2. an GCP Container Registry linked to the Google Artifact Registry via an AWS Service Connector configured with IAM role credentials
3. depending on the `orchestrator` input variable:
* if `orchestrator` is set to `local`: a local Orchestrator. This can be used in combination with the Vertex AI Step Operator to selectively run some steps locally and some on Vertex AI.
* if `orchestrator` is set to `vertex` (default): a Vertex AI Orchestrator linked to the GCP project via an AWS Service Connector configured with IAM role credentials
* if `orchestrator` is set to `skypilot`: a SkyPilot Orchestrator linked to the GCP project via an AWS Service Connector configured with IAM role credentials
* if `orchestrator` is set to `airflow`: an Airflow Orchestrator linked to the Cloud Composer environment
4. a Google Cloud Build Image Builder linked to the GCP project via an AWS Service Connector configured with IAM role credentials
5. a Vertex AI Step Operator linked to the GCP project via an AWS Service Connector configured with IAM role credentials
6. a Vertex AI Experiment Tracker linked to the GCP project via an AWS Service Connector configured with IAM role credentials
To use the ZenML stack, you will need to install the required integrations:
* for Vertex AI:
```shell
zenml integration install gcp
```
* for SkyPilot:
```shell
zenml integration install gcp skypilot_gcp
```
* for Airflow:
```shell
zenml integration install gcp airflow
```
## 🚀 Usage
To use this module, aside from the prerequisites mentioned above, you also need to create [a ZenML Service Account API key](https://docs.zenml.io/how-to/connecting-to-zenml/connect-with-a-service-account) for your ZenML Server. You can do this by running the following command in a terminal where you have the ZenML CLI installed:
```bash
zenml service-account create
```
### Basic Configuration
```hcl
terraform {
required_providers {
google = {
source = "hashicorp/google"
}
zenml = {
source = "zenml-io/zenml"
}
}
}
provider "google" {
region = "europe-west3"
project = "my-project"
}
provider "zenml" {
# server_url =
# api_key =
}
module "zenml_stack" {
source = "zenml-io/zenml-stack/gcp"
orchestrator = "vertex" # or "skypilot", "airflow" or "local"
zenml_stack_name = "my-zenml-stack"
}
output "zenml_stack_id" {
value = module.zenml_stack.zenml_stack.id
}
output "zenml_stack_name" {
value = module.zenml_stack.zenml_stack.name
}
```
## 🎓 Learning Resources
[ZenML Documentation](https://docs.zenml.io/)
[ZenML Starter Guide](https://docs.zenml.io/user-guide/starter-guide)
[ZenML Examples](https://github.com/zenml-io/zenml/tree/main/examples)
[ZenML Blog](https://www.zenml.io/blog)
## 🆘 Getting Help
If you need assistance, join our Slack community or open an issue on our GitHub repo.
