{"id":17924428,"url":"https://github.com/anna-geller/prefect-cloud-gcp","last_synced_at":"2025-03-24T02:33:42.971Z","repository":{"id":65553018,"uuid":"578667332","full_name":"anna-geller/prefect-cloud-gcp","owner":"anna-geller","description":"Repository template to get started with Prefect Cloud \u0026 Google Cloud","archived":false,"fork":false,"pushed_at":"2023-02-09T12:22:32.000Z","size":86,"stargazers_count":24,"open_issues_count":1,"forks_count":9,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-19T02:37:52.627Z","etag":null,"topics":["automation","cloud","compute-engine","dataflow","google-cloud-platform","orchestration","prefect","python"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/anna-geller.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-12-15T15:42:39.000Z","updated_at":"2024-06-21T10:51:47.000Z","dependencies_parsed_at":"2024-10-28T20:48:51.809Z","dependency_job_id":"65188dc9-c5a9-4334-8190-217b774d379a","html_url":"https://github.com/anna-geller/prefect-cloud-gcp","commit_stats":null,"previous_names":[],"tags_count":0,"template":true,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/anna-geller%2Fprefect-cloud-gcp","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/anna-geller%2Fprefect-cloud-gcp/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/anna-geller%2Fprefect-cloud-gcp/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/anna-geller%2Fprefect-cloud-gcp/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/anna-geller","download_url":"https://codeload.github.com/anna-geller/prefect-cloud-gcp/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245199100,"owners_count":20576490,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["automation","cloud","compute-engine","dataflow","google-cloud-platform","orchestration","prefect","python"],"created_at":"2024-10-28T20:48:45.554Z","updated_at":"2025-03-24T02:33:42.702Z","avatar_url":"https://github.com/anna-geller.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Getting started with Prefect Cloud \u0026 Google Cloud\n\nThis repository includes example project structure with:\n- your first flows and blocks\n- all-in-one GitHub Action deploying a custom image and container to a VM hosted in the Google Compute Engine - you can easily modify details such as region, instance type or names for all components\n- modular GitHub Actions in case you only want to use this template to deploy a VM, or to only build and push a Docker image to the Google Artifact Registry - the custom modules give you the freedom to choose which component you want to use, and you can deploy those one by one to set up everything more incrementally, and have more control over how you do it.  \n\n\n[Blog post: GCP and Prefect Cloud — from Docker Container to Cloud VM on Google Compute Engine](https://medium.com/the-prefect-blog/gcp-and-prefect-cloud-from-docker-container-to-cloud-vm-on-google-compute-engine-2dffa026d16b)\n\n\n## Prerequisites\n\n### Prefect Cloud\n\nSign up for a [Prefect Cloud](https://app.prefect.cloud/) account.\n\n1. Create a workspace and an API key.\n2. Add both ``PREFECT_API_KEY`` and ``PREFECT_API_URL``as GitHub Actions secrets.\n\n\n### Google Cloud\n\nCreate a GCP project and a service account. Here's how you can create all that from ``gcloud`` CLI, e.g. using Cloud Shell terminal (_feel free to customize any names based on your needs_):\n\n```bash\n# Create GCP account + project =\u003e here we use project named \"prefect-community\" - replace it with your project name\n# This will also set default project and region:\nexport CLOUDSDK_CORE_PROJECT=\"prefect-community\"\nexport CLOUDSDK_COMPUTE_REGION=us-east1\nexport GCP_AR_REPO=prefect\nexport GCP_SA_NAME=prefect\n\n# enable required GCP services:\ngcloud services enable iamcredentials.googleapis.com\ngcloud services enable artifactregistry.googleapis.com\ngcloud services enable run.googleapis.com\ngcloud services enable compute.googleapis.com\n\n# create service account named e.g. prefect:\ngcloud iam service-accounts create $GCP_SA_NAME\nexport MEMBER=serviceAccount:\"$GCP_SA_NAME\"@\"$CLOUDSDK_CORE_PROJECT\".iam.gserviceaccount.com\ngcloud projects add-iam-policy-binding $CLOUDSDK_CORE_PROJECT --member=$MEMBER --role=\"roles/run.admin\"\ngcloud projects add-iam-policy-binding $CLOUDSDK_CORE_PROJECT --member=$MEMBER --role=\"roles/compute.instanceAdmin.v1\"\ngcloud projects add-iam-policy-binding $CLOUDSDK_CORE_PROJECT --member=$MEMBER --role=\"roles/artifactregistry.admin\"\ngcloud projects add-iam-policy-binding $CLOUDSDK_CORE_PROJECT --member=$MEMBER --role=\"roles/iam.serviceAccountUser\"\n\n# create JSON credentials file as follows, then copy-paste its content into your GHA Secret + Prefect GcpCredentials block:\ngcloud iam service-accounts keys create prefect.json --iam-account=\"$GCP_SA_NAME\"@\"$CLOUDSDK_CORE_PROJECT\".iam.gserviceaccount.com\n```\n\nThis will generate the JSON key file, of which contents you can copy and paste into your ``GCP_CREDENTIALS`` secret in:\n\n1. GitHub Action secret named ``GCP_CREDENTIALS``\n2. Prefect ``GcpCredentials`` block named ``default``- if you save this Prefect block with a different name, make sure to adjust it in your GitHub Action inputs, e.g. in [`.github/workflows/getting-started.yml`](.github/workflows/getting-started.yml):\n\n```yaml\n  gcp_creds_block_name:\n    description: 'Name of the GcpCredentials block'\n    required: false\n    default: \"default\"\n    type: string\n```\n\n\n# What does the main Getting Started action do?\n\n1. It creates an Artifact Registry repository if one doesn't exist yet. That's why the permissions on your service account must be set to admin ``\"roles/artifactregistry.admin\"`` rather than just a writer: ``\"roles/artifactregistry.writer\"`` - if you prefer to manually create the repository and limit the service account permissions, you can do that.\n2. It builds a Docker image and pushes it to that Artifact Registry repository, based on the [``Dockerfile``](Dockerfile)\n3. It deploys a VM (if one such VM with the same name already exists, it gets deleted before a new VM gets created) and a Docker container running a Prefect worker process that deploys flow runs. By default, the flows are configured to be deployed as serverless containers using Google Cloud Run jobs. This makes it easy to scale your project as your needs grow - no need to monitor and maintain the underlying infrastructure - serverless containers gets spun up based on the provided Artifact Registry image and the resource allocation can be adjusted any time on the ``CloudRunJob`` block, even from the Prefect UI.\n4. It automatically deploys your first [Prefect blocks](.github/actions/blocks-quickstart/blocks.py)\n5. It automatically deploys your first [Prefect flows](flows)\n\n\n\n# How does this automated GitHub Actions process deploy flows?\n\n## ``deploy-flows`` action\n\nThis action assumes that the name of your `flow_script.py` matches the name of the flow, e.g. the flow script ``parametrized.py`` has a function named ``parametrized()`` decorated with `@flow`. This means that if your script `parametrized.py` has multiple flows within, only the flow `parametrized` gets deployed (and potentially scheduled) as part of your Prefect Cloud deployment:\n\n\n```python\nfrom prefect import get_run_logger, flow\nfrom typing import Any\n\n\n@flow(log_prints=True)\ndef some_subflow():\n    print(\"I'm a subflow\")\n\n    \n@flow\ndef parametrized(\n    user: str = \"Marvin\", question: str = \"Ultimate\", answer: Any = 42\n) -\u003e None:\n    logger = get_run_logger()\n    logger.info(\"Hello from Prefect, %s! 👋\", user)\n    logger.info(\"The answer to the %s question is %s! 🤖\", question, answer)\n\n\nif __name__ == \"__main__\":\n    parametrized(user=\"World\")\n```\n\nYou can still create a deployment for the flow ``some_subflow`` if you want to, but the default GitHub Action here won't do it for you - this is not a Prefect limitation, it's only a choice made in this demo to make the example project here easier to follow (and more standardized to deploy in an automated CI/CD pipeline).\n\n\u003e As an alternative approach to naming the main flow the same way as the script name, you could consider naming each main flow that needs to get schedule as ``main``.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fanna-geller%2Fprefect-cloud-gcp","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fanna-geller%2Fprefect-cloud-gcp","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fanna-geller%2Fprefect-cloud-gcp/lists"}