{"id":19307395,"url":"https://github.com/artefactory/scheduled-pipelines","last_synced_at":"2026-02-27T20:33:15.961Z","repository":{"id":196625115,"uuid":"668712252","full_name":"artefactory/scheduled-pipelines","owner":"artefactory","description":"This repository enables to easily schedule existing Vertex pipelines","archived":false,"fork":false,"pushed_at":"2023-10-03T09:43:14.000Z","size":796,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-02-24T02:49:53.523Z","etag":null,"topics":["engineering"],"latest_commit_sha":null,"homepage":"https://artefact.roadie.so/docs/default/Component/scheduled-pipelines","language":"HCL","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/artefactory.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-07-20T12:30:54.000Z","updated_at":"2023-11-05T16:27:05.000Z","dependencies_parsed_at":"2024-11-10T00:10:57.241Z","dependency_job_id":"7619e2a7-d488-452a-bd43-bfe4fbadce50","html_url":"https://github.com/artefactory/scheduled-pipelines","commit_stats":null,"previous_names":["artefactory/scheduled-pipelines"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/artefactory/scheduled-pipelines","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/artefactory%2Fscheduled-pipelines","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/artefactory%2Fscheduled-pipelines/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/artefactory%2Fscheduled-pipelines/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/artefactory%2Fscheduled-pipelines/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/artefactory","download_url":"https://codeload.github.com/artefactory/scheduled-pipelines/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/artefactory%2Fscheduled-pipelines/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29912222,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-27T19:37:42.220Z","status":"ssl_error","status_checked_at":"2026-02-27T19:37:41.463Z","response_time":57,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["engineering"],"created_at":"2024-11-10T00:10:47.049Z","updated_at":"2026-02-27T20:33:15.910Z","avatar_url":"https://github.com/artefactory.png","language":"HCL","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Vertex Pipelines Scheduler Accelerator\n\nThis repository enables to easily schedule existing Vertex pipelines.\n\nIt uploads Vertex pipelines templates to an Artifact Registry repository and schedules pipelines using Cloud Scheduler and Cloud Functions.\n\n\u003cimg src=\"assets/infra.png\"\u003e\n\nIt does for you the creation of the required service accounts, configures the required permissions and creates the necessary cloud resources.\n\n## Table of Contents\n\n- [Vertex Pipelines Scheduler Accelerator](#vertex-pipelines-scheduler-accelerator)\n  - [Table of Contents](#table-of-contents)\n  - [Prerequisites](#prerequisites)\n  - [Setup](#setup)\n  - [Installation](#installation)\n  - [Usage](#usage)\n  - [Sanity check](#sanity-check)\n  - [Troubleshooting](#troubleshooting)\n  - [More details](#more-details)\n\n## Prerequisites\n\n- Unix-like environment (Linux, macOS, WSL, etc... Tested on MacOS Monterey, M1 chip \u0026 GNU/Linux 10)\n- Google SDK (gcloud) (instructions [here](https://cloud.google.com/sdk/docs/install#installation_instructions))\n- Terraform (tested for version v1.5.6) (instructions [here](https://developer.hashicorp.com/terraform/tutorials/aws-get-started/install-cli#install-terraform))\n- Having `wget` installed (instructions [here](https://www.gnu.org/software/wget/) for Linux and for MacOS: `brew install wget`)\n- Having `jq` installed (instructions [here](https://stedolan.github.io/jq/download/) for Linux and for MacOS: `brew install jq`. Tested for version jq-1.6)\n- Having `yq` installed (instructions [here](https://github.com/mikefarah/yq/#install) for Linux and for MacOS: `brew install yq`. Tested for version 4.35.2)\n- Having a compiled Vertex pipeline (instructions [here](https://cloud.google.com/vertex-ai/docs/pipelines/build-pipeline#compile_your_pipeline_into_a_yaml_file)) or  use the [`hello_world_pipeline.yaml`](pipelines/hello_world_pipeline.yaml) file in the `pipelines` directory to test the scheduling rapidly.\n\n## Setup\n\nFirst, execute these Google Cloud commands:\n\n```bash\nexport GCP_PROJECT_ID=\u003cgcp_project_id\u003e\ngcloud config set project $GCP_PROJECT_ID\ngcloud auth login\ngcloud auth application-default login\n```\n\n## Installation\n\nCreate a working directory and download this repository:\n\n```bash\nwget -O scheduled-pipelines.zip https://github.com/artefactory/scheduled-pipelines/archive/main.zip\n```\n\nThen unzip it:\n\n```bash\nunzip scheduled-pipelines.zip \\\n\u0026\u0026 rm scheduled-pipelines.zip \\\n\u0026\u0026 mv scheduled-pipelines-main scheduled-pipelines \\\n\u0026\u0026 cd scheduled-pipelines\n```\n\n## Usage\n\nTo use this repository, you need to:\n\n1. In the [`pipelines`](./pipelines/) directory, put your compiled pipeline(s) (YAML file) inside or directly use the dummy pipeline \"`hello_world_pipeline.yaml`\" (already in the `pipelines` directory).\n\n2. Replace the values in your configuration file  [`scheduled_pipelines_config.yaml`](scheduled_pipelines_config.yaml) with the values **corresponding to your project**.\n\n3. Enable the required APIs:\n\n```bash\ngcloud services enable \\\n  cloudscheduler.googleapis.com \\\n  cloudfunctions.googleapis.com \\\n  cloudbuild.googleapis.com \\\n  artifactregistry.googleapis.com \\\n  storage-component.googleapis.com \\\n  aiplatform.googleapis.com \\\n  --project=$GCP_PROJECT_ID\n```\n\n4. Deploy the scheduled pipeline(s) and its (their) infrastructure:\n\n```bash\nmake deploy_scheduled_pipelines\n```\n\nThis command will:\n\n- Create the service accounts used to run the scheduled pipelines and schedule them.\n- Create the necessary cloud resources (Cloud Scheduler, Cloud Functions, Artifact Registry repository).\n- Give the appropriate permissions to the service accounts.\n- Upload the pipeline templates to the Artifact Registry repository.\n\n5. (Optional) If you modify the compiled pipeline(s) and/or you modify the configuration file, just run the previous command again.\n\n## Sanity check\n\nGo to the [Cloud Scheduler page](https://console.cloud.google.com/cloudscheduler) in the Google Cloud console and make sure the right schedulers are present.\n\nTrigger a force run of the scheduler:\n\n⚠️ It might take a few minutes for the scheduler to work properly with the cloud function. If the command fails, wait a few minutes and try again. ⚠️\n\n\u003cimg src=\"assets/cloud_schedulers.png\" alt=\"Cloud schedulers\" /\u003e\n\nIf the \"Status of last execution\" is failure, check the [troubleshooting](#troubleshooting) section below.\n\nIf the \"Status of last execution\" is \"Success\", check that the [Vertex pipeline](https://console.cloud.google.com/vertex-ai/pipelines) is running as expected (make sure you selected the right region). If it it not running, check the logs of the cloud function.\n\n\u003cimg src=\"assets/vertex_pipelines.png\" alt=\"Vertex pipelines\" /\u003e\n\n## Troubleshooting\n\n1. Check that the \"Status of last execution\" of the scheduler is \"Success\". If this is not the case:\n\nFirst check the logs of the cloud scheduler to see whether the error is coming from the scheduler (permission denied) or the cloud function (internal error).\n\nIf the error is a permission denied, check that the service account of the cloud scheduler has the right permissions on the cloud function. If the error is an internal error, check the [cloud function](https://console.cloud.google.com/functions) logs (Go to the Cloud functions page, click on the cloud function name and go to the \"LOGS\" tab).\n\n2. If the \"Status of last execution\" is \"Success\" but the pipeline is not running, check the logs of the cloud function to debug.\n\n## More details\n\nThe required permissions required to execute the `make deploy_scheduled_pipelines` command are:\n\n| Resource creation        | Permission(s) required                                    |\n| ------------------------ | --------------------------------------------------------- |\n| Create service account   | iam.serviceAccounts.create                                |\n| Create artifact registry | artifactregistry.repositories.create                      |\n| Creation cloud function  | cloudfunctions.functions.create, cloudbuild.builds.create |\n| Create cloud scheduler   | cloudscheduler.jobs.create                                |\n\n| Resource to give permission on (iam-policy-binding) | Permission required                        |\n| ----------------- | ------------------------------------------ |\n| Project           | resourcemanager.projects.setIamPolicy      |\n| Cloud storage     | storage.buckets.setIamPolicy               |\n| Cloud function    | cloudfunctions.functions.setIamPolicy      |\n| Artifact registry | artifactregistry.repositories.setIamPolicy |\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fartefactory%2Fscheduled-pipelines","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fartefactory%2Fscheduled-pipelines","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fartefactory%2Fscheduled-pipelines/lists"}