{"id":15221590,"url":"https://github.com/googlecloudplatform/automlops","last_synced_at":"2025-05-15T17:03:42.380Z","repository":{"id":65593577,"uuid":"585676969","full_name":"GoogleCloudPlatform/automlops","owner":"GoogleCloudPlatform","description":"Build MLOps Pipelines in Minutes","archived":false,"fork":false,"pushed_at":"2025-04-18T16:08:48.000Z","size":18978,"stargazers_count":239,"open_issues_count":11,"forks_count":42,"subscribers_count":25,"default_branch":"main","last_synced_at":"2025-04-19T22:18:04.210Z","etag":null,"topics":["ci-cd","google-cloud-platform","jupyter-notebook","machine-learning","mlops","pipeline","vertex-ai"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/GoogleCloudPlatform.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-01-05T19:36:05.000Z","updated_at":"2025-03-31T17:32:52.000Z","dependencies_parsed_at":"2023-07-04T10:45:56.088Z","dependency_job_id":"e6f0ee62-4814-4449-a49b-6fa1bd725d48","html_url":"https://github.com/GoogleCloudPlatform/automlops","commit_stats":{"total_commits":57,"total_committers":3,"mean_commits":19.0,"dds":0.03508771929824561,"last_synced_commit":"5c8ca94a05ff4eb0a6d99fd4b27a26b153d271a5"},"previous_names":[],"tags_count":24,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GoogleCloudPlatform%2Fautomlops","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GoogleCloudPlatform%2Fautomlops/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GoogleCloudPlatform%2Fautomlops/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GoogleCloudPlatform%2Fautomlops/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/GoogleCloudPlatform","download_url":"https://codeload.github.com/GoogleCloudPlatform/automlops/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254384982,"owners_count":22062422,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ci-cd","google-cloud-platform","jupyter-notebook","machine-learning","mlops","pipeline","vertex-ai"],"created_at":"2024-09-28T15:06:12.694Z","updated_at":"2025-05-15T17:03:42.340Z","avatar_url":"https://github.com/GoogleCloudPlatform.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# AutoMLOps\n\nAutoMLOps is a service that generates, provisions, deploys, and monitors CI/CD integrated MLOps pipelines, bridging the gap between Data Science and DevOps. AutoMLOps provides a repeatable process that dramatically reduces the time required to build MLOps pipelines. The service generates a containerized MLOps codebase, provides infrastructure-as-code to provision and maintain the underlying MLOps infra, provides deployment functionalities to trigger and run MLOps pipelines, monitoring capabilities to monitor models deployed to live endpoints, and optional automatic retraining of models on a recurring basis or if anomalies are detected during monitoring.\n\nAutoMLOps gives flexibility over the tools and technologies used in the MLOps pipelines, allowing users to choose from a wide range of options for artifact repositories, build tools, provisioning tools, orchestration frameworks, and source code repositories. AutoMLOps can be configured to either use existing infra, or provision new infra, including source code repositories for versioning the generated MLOps codebase, build configs and triggers, artifact repositories for storing docker containers, storage buckets, service accounts, IAM permissions, APIs needed to run pipelines, RESTful services to allow for triggering and running pipelines asynchronous, and Cloud Scheduler jobs for triggering and running pipelines on a recurring basis.\n\nThese automatic integrations empower data scientists to take their experiments to production more quickly, allowing them to focus on what they do best: providing actionable insights through data.\n\n# Install\n\nInstall AutoMLOps from [PyPI](https://pypi.org/project/google-cloud-automlops/): `pip install google-cloud-automlops`\n\nOr Install locally by cloning the repo and running `pip install .`\n\n# Dependencies\n- `docopt==0.6.2`\n- `docstring-parser==0.15`\n- `google-api-python-client==2.97.0`\n- `google-auth==2.22.0`\n- `importlib-resources==6.0.1`\n- `Jinja2==3.1.2`\n- `kfp\u003e=2.0.0`\n- `packaging==23.1`\n- `pipreqs==0.4.13`\n- `pydantic==2.3.0`\n- `PyYAML==6.0.1`\n- `yarg==0.1.9`\n\n# Usage\n\nAutoMLOps provides 2 functions for defining MLOps pipelines:\n\n- `@AutoMLOps.component(...)`: Defines a component, which is a containerized python function.\n- `@AutoMLOps.pipeline(...)`: Defines a pipeline, which is a series of components.\n\nAutoMLOps provides 6 functions for building and maintaining MLOps pipelines:\n\n- `AutoMLOps.generate(...)`: Generates the MLOps codebase. Users can specify the tooling and technologies they would like to use in their MLOps pipeline.\n- `AutoMLOps.provision(...)`: Runs provisioning scripts to create and maintain necessary infra for MLOps.\n- `AutoMLOps.deprovision(...)`: Runs deprovisioning scripts to tear down MLOps infra created using AutoMLOps.\n- `AutoMLOps.deploy(...)`: Builds and pushes component container, then triggers the pipeline job.\n- `AutoMLOps.launchAll(...)`: Runs `generate()`, `provision()`, and `deploy()` all in succession.\n- `AutoMLOps.monitor(...)`: Creates model monitoring jobs on deployed endpoints.\n\nFor a full user-guide, please view these [slides](https://github.com/GoogleCloudPlatform/automlops/blob/main/AutoMLOps_User_Guide.pdf).\n\n# List of Examples\n\nTraining\n- [00_introduction_training_example](https://github.com/GoogleCloudPlatform/automlops/blob/main/examples/training/00_introduction_training_example.ipynb) \u003c- start here\n- [00_introduction_training_example_no_notebook](https://github.com/GoogleCloudPlatform/automlops/blob/main/examples/training/00_introduction_training_example_no_notebook.py)\n- [01_clustering_example](https://github.com/GoogleCloudPlatform/automlops/blob/main/examples/training/01_clustering_example.ipynb)\n- [02_tensorflow_transfer_learning_gpu_example](https://github.com/GoogleCloudPlatform/automlops/blob/main/examples/training/02_tensorflow_transfer_learning_gpu_example.ipynb)\n- [03_bqml_introduction_training_example](https://github.com/GoogleCloudPlatform/automlops/blob/main/examples/training/03_bqml_introduction_training_example.ipynb)\n- [04_bqml_forecasting-retail-demand](https://github.com/GoogleCloudPlatform/automlops/blob/main/examples/training/04_bqml_forecasting-retail-demand.ipynb)\n\nInferencing\n- [00_batch_prediction_example](https://github.com/GoogleCloudPlatform/automlops/blob/main/examples/inferencing/00_batch_prediction_example.ipynb)\n- [01_customer_churn_model_monitoring_example](https://github.com/GoogleCloudPlatform/automlops/blob/main/examples/inferencing/01_customer_churn_model_monitoring_example.ipynb)\n\n# Supported Tools and Technologies\n\n**Artifact Repositories**: Stores component docker containers\n- Artifact Registry\n\n**Deployment Frameworks**: Builds component docker containers, compiles pipelines, and submits Pipeline Jobs\n- Github Actions\n- Cloud Build\n- [coming soon] Gitlab CI\n- [coming soon] Bitbucket Pipelines\n- [coming soon] Jenkins\n\n**Orchestration Frameworks**: Executes and orchestrates pipelines jobs\n- Kubeflow Pipelines (KFP) - Runs on [Vertex AI Pipelines](https://cloud.google.com/vertex-ai/docs/pipelines/introduction)\n- [coming soon] Tensorflow Extended (TFX) - Runs on [Vertex AI Pipelines](https://cloud.google.com/vertex-ai/docs/pipelines/introduction)\n- [coming soon] Argo Workflows - Runs on [GKE](https://cloud.google.com/kubernetes-engine/docs/concepts/kubernetes-engine-overview)\n- [coming soon] Airflow - Runs on [Cloud Composer](https://cloud.google.com/composer/docs)\n- [coming soon] Ray - Runs on [GKE](https://cloud.google.com/kubernetes-engine/docs/concepts/kubernetes-engine-overview)\n\n**Submission Service Compute Environments**: RESTful service for submitting pipeline jobs to the orchestrator (e.g. Vertex AI, Cloud Composer, etc.)\n- Cloud Functions\n- Cloud Run\n\n**Provisioning Frameworks**: Stands up necessary infra to run MLOps pipelines\n- gcloud\n- terraform\n- [coming soon] pulumi\n\n**Source Code Repositories**: Repository for versioning generated MLOps code\n- Github\n- Bitbucket\n- Gitlab\n- [deprecated] Cloud Source Repositories\n\n# Prerequisites\n### Generate\nIn order to use `AutoMLOps.generate(...)`, the following are required:\n- Python 3.7 - 3.10\n\n### Provision\nIn order to use `AutoMLOps.provision(...)` with `provisioning_framework='gcloud'`, the following are recommended:\n- [Google Cloud SDK 407.0.0](https://cloud.google.com/sdk/gcloud/reference)\n- [beta 2022.10.21](https://cloud.google.com/sdk/gcloud/reference/beta)\n\nIn order to use `AutoMLOps.provision(...)` with `provisioning_framework='terraform'`, the following are recommended:\n- [Terraform v1.5.6](https://www.terraform.io/downloads.html)\n\n### Deploy\nIn order to use `AutoMLOps.deploy(...)` with `use_ci=False`, the following are required:\n- Local python environment with these packages installed:\n    - `kfp\u003e=2.0.0`\n    - `google-cloud-aiplatform`\n    - `google-cloud-pipeline-components`\n    - `google-cloud-storage`\n    - `pyyaml`\n\nIn order to use `AutoMLOps.deploy(...)` with `use_ci=True`, the following are required:\n- `git` installed\n- `git` logged-in:\n```\n  git config --global user.email \"you@example.com\"\n  git config --global user.name \"Your Name\"\n```\n- Registered and setup your SSH key if you are using Github, Gitlab, or Bitbucket\n\n### Monitor\nIn order to use `AutoMLOps.monitor(...)`, the following are required:\n- Local python environment with these packages installed:\n    - `google-cloud-aiplatform`\n    - `google-cloud-logging`\n    - `google-cloud-storage`\n    - `pyyaml`\n\n- [Application Default Credentials (ADC)](https://cloud.google.com/docs/authentication/provide-credentials-adc) are set up. This can be done through the following commands:\n```\ngcloud auth application-default login\ngcloud config set account \u003caccount@example.com\u003e\n```\n\n# GCP Services\nAutoMLOps makes use of the following products by default:\n- [Google Cloud Storage](https://cloud.google.com/storage/docs/introduction)\n\nAutoMLOps will makes use of the following products based on user selected options:\n\n1. if `artifact_repo_type='artifact-registry'`, AutoMLOps will use:\n- [Artifact Registry](https://cloud.google.com/artifact-registry/docs/overview)\n\n2. if `use_ci=False` and `orchestration_framework='kfp'` and `deployment_framework='cloud-build'`, AutoMLOps will use:\n- [Vertex AI Pipelines](https://cloud.google.com/vertex-ai/docs/pipelines/introduction)\n- [Cloud Build](https://cloud.google.com/build/docs/overview)\n\n2. if `use_ci=True` and `orchestration_framework='kfp'` and `deployment_framework='cloud-build'`, AutoMLOps will use:\n- [Vertex AI Pipelines](https://cloud.google.com/vertex-ai/docs/pipelines/introduction)\n- [Cloud Pub/Sub](https://cloud.google.com/pubsub/docs/overview)\n- [Cloud Build](https://cloud.google.com/build/docs/overview)\n- [Cloud Build Triggers](https://cloud.google.com/build/docs/triggers)\n\n4. if `use_ci=True` and `pipeline_job_submission_service_type='cloud-functions'`, AutoMLOps will use:\n- [Cloud Functions](https://cloud.google.com/functions/docs/concepts/overview)\n\n5. if `use_ci=True` and `pipeline_job_submission_service_type='cloud-run'`, AutoMLOps will use:\n- [Cloud Run](https://cloud.google.com/run/docs/overview/what-is-cloud-run)\n\n6. if `use_ci=True` and `schedule_pattern` is specified, AutoMLOps will use:\n- [Cloud Scheduler](https://cloud.google.com/scheduler/docs/overview)\n\n7. if `use_ci=True` and `setup_model_monitoring=True`, AutoMLOps will use:\n- [Vertex AI Model Monitoring](https://cloud.google.com/vertex-ai/docs/model-monitoring/overview)\n- [Cloud Logging](https://cloud.google.com/logging/docs/overview)\n\n\n# APIs \u0026 IAM\nBased on the above user selection, AutoMLOps will enable up to the following APIs during the provision step:\n- [aiplatform.googleapis.com](https://cloud.google.com/vertex-ai/docs/reference/rest)\n- [artifactregistry.googleapis.com](https://cloud.google.com/artifact-registry/docs/reference/rest)\n- [cloudbuild.googleapis.com](https://cloud.google.com/build/docs/api/reference/rest)\n- [cloudfunctions.googleapis.com](https://cloud.google.com/functions/docs/reference/rest)\n- [cloudresourcemanager.googleapis.com](https://cloud.google.com/resource-manager/reference/rest)\n- [cloudscheduler.googleapis.com](https://cloud.google.com/scheduler/docs/reference/rest)\n- [compute.googleapis.com](https://cloud.google.com/compute/docs/reference/rest/v1)\n- [iam.googleapis.com](https://cloud.google.com/iam/docs/reference/rest)\n- [iamcredentials.googleapis.com](https://cloud.google.com/iam/docs/reference/credentials/rest)\n- [logging.googleapis.com](https://cloud.google.com/logging/docs/reference/v2/rest)\n- [pubsub.googleapis.com](https://cloud.google.com/pubsub/docs/reference/rest)\n- [run.googleapis.com](https://cloud.google.com/run/docs/reference/rest)\n- [storage.googleapis.com](https://cloud.google.com/storage/docs/apis)\n\n\nAutoMLOps will create the following service account and update [IAM permissions](https://cloud.google.com/iam/docs/understanding-roles) during the provision step:\n1. Pipeline Runner Service Account (defaults to: vertex-pipelines@PROJECT_ID.iam.gserviceaccount.com). Roles added:\n- roles/aiplatform.user\n- roles/artifactregistry.reader\n- roles/bigquery.user\n- roles/bigquery.dataEditor\n- roles/iam.serviceAccountUser\n- roles/storage.admin\n- roles/cloudfunctions.admin\n\n# Prechecks and Warnings\n\nAutoMLOps provides a number of optional prechecks and warnings to provide visibility to the user into what IAM permissions are required to run certain operations. \n\n1. `AutoMLOps.provision(...hide_warnings=False)`: This will check installation versions and account permissions to determine if the current account has the permissions to provision successfully.\n2. `AutoMLOps.deploy(...hide_warnings=False)`: This will check installation versions and account permissions to determine if the current account has the permissions to deploy successfully.\n3. `AutoMLOps.deploy(...precheck=True)`: This will check for the necessary infrastructure to deploy successfully (e.g. does the specified artifact registry exist, does the specified storage bucket exist, etc.)\n4. `AutoMLOps.monitor(...hide_warnings=False)`: This will check installation versions and account permissions to determine if the current account has the permissions to create model monitoring jobs successfully.\n\n# Code Generation Options\n\nAutoMLOps CI/CD options:\n1. `use_ci`: Bool that specifies whether to execute using generated files and scripts locally or use cloud CI/CD workflow. Defaults to False. See [CI/CD Workflow](#deployment)\n\nRequired parameters:\n1. `project_id: str`\n2. `pipeline_params: dict`\n\nOptional parameters (defaults shown):\n1. `artifact_repo_location: str = 'us-central1'`\n2. `artifact_repo_name: str = f'{naming_prefix}-artifact-registry'`\n3. `artifact_repo_type: str = 'artifact-registry'`\n4. `base_image: str = 'python:3.9-slim'`\n5. `build_trigger_location: str = 'us-central1'`\n6. `build_trigger_name: str = f'{naming_prefix}-build-trigger'`\n7. `custom_training_job_specs: list[dict] = None`\n8. `deployment_framework: str = 'github-actions'`\n9. `naming_prefix: str = 'automlops-default-prefix'`\n10. `orchestration_framework: str = 'kfp'`\n11. `pipeline_job_location: str = 'us-central1'`\n12. `pipeline_job_runner_service_account: str = f'vertex-pipelines@{project_id}.iam.gserviceaccount.com'`\n13. `pipeline_job_submission_service_location: str = 'us-central1'`\n14. `pipeline_job_submission_service_name: str = f'{naming_prefix}-job-submission-svc'`\n15. `pipeline_job_submission_service_type: str = 'cloud-functions'`\n16. `project_number: str = None`\n17. `provision_credentials_key: str = None`\n18. `provisioning_framework: str = 'gcloud'`\n19. `pubsub_topic_name: str = f'{naming_prefix}-queueing-svc'`\n20. `schedule_location: str = 'us-central1'`\n21. `schedule_name: str = f'{naming_prefix}-schedule'`\n22. `schedule_pattern: str = 'No Schedule Specified'`\n23. `setup_model_monitoring: Optional[bool] = False`\n24. `source_repo_branch: str = 'automlops'`\n25. `source_repo_name: str = f'{naming_prefix}-repository'`\n26. `source_repo_type: str = 'github'`\n27. `storage_bucket_location: str = 'us-central1'`\n28. `storage_bucket_name: str = f'{project_id}-{naming_prefix}-bucket'`\n29. `use_ci: bool = False`\n30. `vpc_connector: str = 'No VPC Specified'`\n31. `workload_identity_pool: str = None`\n32. `workload_identity_provider: str = None`\n33. `workload_identity_service_account: str = None`\n\nParameter Options:\n- `artifact_repo_type=`:\n    - 'artifact-registry' (default)\n- `deployment_framework=`:\n    - 'github-actions' (default)\n    - 'cloud-build'\n    - [coming soon] 'gitlab-ci'\n    - [coming soon] 'bitbucket-pipelines'\n    - [coming soon] 'jenkins'\n- `orchestration_framework=`:\n    - 'kfp' (default)\n    - [coming soon] 'tfx'\n    - [coming soon] 'argo-workflows'\n    - [coming soon] 'airflow'\n    - [coming soon] 'ray'\n- `pipeline_job_submission_service_type=`:\n    - 'cloud-functions' (default)\n    - 'cloud-run'\n- `provisioning_framework=`:\n    - 'gcloud' (default)\n    - 'terraform'\n    - [coming soon] 'pulumi'\n- `source_repo_type=`:\n    - 'github' (default)\n    - 'gitlab'\n    - 'bitbucket'\n\nA description of the parameters is below:\n- `project_id`: The project ID.\n- `pipeline_params`: Dictionary containing runtime pipeline parameters.\n- `artifact_repo_location`: Region of the artifact repo (default use with Artifact Registry).\n- `artifact_repo_name`: Artifact repo name where components are stored (default use with Artifact Registry).\n- `artifact_repo_type`: The type of artifact repository to use (e.g. Artifact Registry, JFrog, etc.)        \n- `base_image`: The image to use in the component base dockerfile.\n- `build_trigger_location`: The location of the build trigger (for cloud build).\n- `build_trigger_name`: The name of the build trigger (for cloud build).\n- `custom_training_job_specs`: Specifies the specs to run the training job with.\n- `deployment_framework`: The CI tool to use (e.g. cloud build, github actions, etc.)\n- `naming_prefix`: Unique value used to differentiate pipelines and services across AutoMLOps runs.\n- `orchestration_framework`: The orchestration framework to use (e.g. kfp, tfx, etc.)\n- `pipeline_job_location`: The location to run the Pipeline Job in.\n- `pipeline_job_runner_service_account`: Service Account to run PipelineJobs (specify the full string).\n- `pipeline_job_submission_service_location`: The location of the cloud submission service.\n- `pipeline_job_submission_service_name`: The name of the cloud submission service.\n- `pipeline_job_submission_service_type`: The tool to host for the cloud submission service (e.g. cloud run, cloud functions).\n- `precheck`: Boolean used to specify whether to check for provisioned resources before deploying.\n- `project_number`: The project number.\n- `provision_credentials_key`: Either a path to or the contents of a service account key file in JSON format.\n- `provisioning_framework`: The IaC tool to use (e.g. Terraform, Pulumi, etc.)\n- `pubsub_topic_name`: The name of the pubsub topic to publish to.\n- `schedule_location`: The location of the scheduler resource.\n- `schedule_name`: The name of the scheduler resource.\n- `schedule_pattern`: Cron formatted value used to create a Scheduled retrain job.\n- `setup_model_monitoring`: Boolean parameter which specifies whether to set up a Vertex AI Model Monitoring Job.\n- `source_repo_branch`: The branch to use in the source repository.\n- `source_repo_name`: The name of the source repository to use.\n- `source_repo_type`: The type of source repository to use (e.g. gitlab, github, etc.)\n- `storage_bucket_location`: Region of the GS bucket.\n- `storage_bucket_name`: GS bucket name where pipeline run metadata is stored.\n- `hide_warnings`: Boolean used to specify whether to show provision/deploy permission warnings\n- `use_ci`: Flag that determines whether to use Cloud CI/CD.\n- `vpc_connector`: The name of the vpc connector to use.\n- `workload_identity_pool`: Pool for workload identity federation. \n- `workload_identity_provider`: Provider for workload identity federation.\n- `workload_identity_service_account`: Service account for workload identity federation (specify the full string).\n\nAutoMLOps will generate the resources specified by these parameters (e.g. Artifact Registry, GCS bucket, etc.). If use_ci is set to True, AutoMLOps will turn the outputted AutoMLOps/ directory into a Git repo and use it for the source repo. If a cron formatted str is given as an arg for `schedule_pattern` then it will set up a Cloud Schedule to run accordingly. If `setup_model_monitoring` is set to true, a model_monitoring/ directory will be created and a monitoring section will be added to config/defaults.yaml with empty values. These values are then set by running `AutoMLOps.monitor()`.\n\n# Generating Code\n\nAutoMLOps generates code that is compatible with `kfp\u003e=2.0.0`. Upon running `AutoMLOps.generate(project_id='project-id', pipeline_params=pipeline_params, setup_model_monitoring=True, use_ci=True)`, a series of directories will be generated automatically:\n\n```bash\n.\n├── components                                     : Custom vertex pipeline components.\n    ├──component_base                              : Contains all the python files, Dockerfile and requirements.txt\n        ├── Dockerfile                             : Dockerfile containing all the python files for the components.\n        ├── requirements.txt                       : Package requirements for all the python files for the components.\n        ├── src                                    : Python source code directory.\n            ├──component_a.py                      : Python file containing code for the component.\n            ├──...(for each component)\n    ├──component_a                                 : Components specs generated using AutoMLOps\n        ├── component.yaml                         : Component yaml spec, acts as an I/O wrapper around the Docker container.\n    ├──...(for each component)\n├── configs                                        : Configurations for defining vertex ai pipeline and MLOps infra.\n    ├── defaults.yaml                              : Runtime configuration variables.\n├── images                                         : Custom container images for training models (optional).\n├── pipelines                                      : Vertex ai pipeline definitions.\n    ├── pipeline.py                                : Full pipeline definition; compiles pipeline spec and uploads to GCS.\n    ├── pipeline_runner.py                         : Sends a PipelineJob to Vertex AI.\n    ├── requirements.txt                           : Package requirements for running pipeline.py.\n    ├── runtime_parameters                         : Variables to be used in a PipelineJob.\n        ├── pipeline_parameter_values.json         : Json containing pipeline parameters.\n├── provision                                      : Provision configurations and details.\n    ├── provision_resources.sh                     : Provisions the necessary infra to run the MLOps pipeline.\n    ├── provisioning_configs                       : (Optional) Relevant terraform/Pulumi config files for provisioning infa.\n├── scripts                                        : Scripts for manually triggering the cloud run service.\n    ├── build_components.sh                        : Submits a Cloud Build job that builds and pushes the components to the registry.\n    ├── build_pipeline_spec.sh                     : Compiles the pipeline specs.\n    ├── run_pipeline.sh                            : Submit the PipelineJob to Vertex AI.\n    ├── run_all.sh                                 : Builds components, compiles pipeline specs, and submits the PipelineJob.\n    ├── publish_to_topic.sh                        : Publishes a message to a Pub/Sub topic to invoke the pipeline job submission service.\n    ├── create_model_monitoring_job.sh             : Creates or updated a Vertex AI model monitoring job for a given deployed model endpoint.\n├── model_monitoring                               : Code for building and maintaining model monitoring jobs.\n    ├── requirements.txt                           : Package requirements for creating and updating model monitoring jobs.\n    ├── monitor.py                                 : Creates a ModelDeploymentMonitoringJob and optionally creates a Log Sink for automatic retraining.\n├── services                                       : MLOps services related to continuous training.\n    ├── submission_service                         : REST API service used to submit pipeline jobs to Vertex AI.\n        ├── Dockerfile                             : Dockerfile for running the REST API service.\n        ├── requirements.txt                       : Package requirements for the REST API service.\n        ├── main.py                                : Python REST API source code.\n├── README.md                                      : Readme markdown file describing the contents of the generated directories.\n└── Build config yaml                              : Build configuration file for building custom components.\n```\n\n# Provisioning\nAutoMLOps currently provides 2 primary options for provisioning infrastructure: `gcloud` and `terraform`. In the diagram below dashed boxes show areas users can select and customize their tooling. \n\n\u003cp align=\"center\"\u003e\n    \u003cimg src=\"https://raw.githubusercontent.com/GoogleCloudPlatform/automlops/main/assets/provision/provision-default.png\" alt=\"CICD\" width=\"1000\"/\u003e\n\u003c/p\u003e\n\n\n# Deployment\n### Cloud Continuous Integration and Continuous Deployment Workflow\nIf `use_ci=True`, AutoMLOps will generate and use a fully featured CI/CD environment for the pipeline. Otherwise, it will use the local scripts to build and run the pipeline. In the diagrams below dashed boxes show areas users can select and customize their tooling. \n\n**\u003ccenter\u003eGithub Actions option:\u003c/center\u003e**\n\u003cp align=\"center\"\u003e\n    \u003cimg src=\"https://raw.githubusercontent.com/GoogleCloudPlatform/automlops/main/assets/deploy/CICD-github.png\" alt=\"CICD\" width=\"1000\"/\u003e\n\u003c/p\u003e\n\n**\u003ccenter\u003eGitlab CI option:\u003c/center\u003e**\n\u003cp align=\"center\"\u003e\n    \u003cimg src=\"https://raw.githubusercontent.com/GoogleCloudPlatform/automlops/main/assets/deploy/CICD-gitlab.png\" alt=\"CICD\" width=\"1000\"/\u003e\n\u003c/p\u003e\n\n**\u003ccenter\u003eBitbucket Pipelines option:\u003c/center\u003e**\n\u003cp align=\"center\"\u003e\n    \u003cimg src=\"https://raw.githubusercontent.com/GoogleCloudPlatform/automlops/main/assets/deploy/CICD-bitbucket.png\" alt=\"CICD\" width=\"1000\"/\u003e\n\u003c/p\u003e\n\n# Model Monitoring\n\nTo use AutoMLOps for setting up and creating model monitoring jobs, the `setup_model_monitoring` parameter must be set to `True` during the generate step. This will create the necessary files under AutoMLOps/scripts and AutoMLOps/model_monitoring. From there, configurations can be set using `AutoMLOps.monitor`. This operation requires a functioning Vertex AI model endpoint, and the name of the endpoint's prediction column. Monitoring can be set to check for anomalies in 2 ways:\n1. Data Drift: This compares incoming feature distributions to past feature distributions that the model endpoint has seen, within a given window of time.\n2. Data Skew: This compares incoming feature distributions to feature distributions within the training dataset to check for training/serving skew.\n\nAt least one of these monitoring types must be used. To specify drift and skew, provide a dictionary of feature names and their threshold values to the `drift_thresholds` and `skew_thresholds` parameters.\n\nWhen an anomaly is detected, actions can be taken in 2 ways:\n1. Generate email alerts: This will send an email to the specified email(s) informing the user of a detected data drift or skew.\n2. Automatically retrain the model: This will send anomaly alerts to the Pub/Sub Queueing service and trigger a retraining of the model, with the parameters specified, when a data drift or skew is detected.\n\nThese can be configured using the `alert_emails` and `auto_retraining_params` parameters, which are left null by default. If `auto_retraining_params` are specified, `AutoMLOps.monitor` will set up a Log Sink to connect the Vertex AI Model Monitoring Job to the Pub/Sub Queueing service, and filter on only ModelMonitoringJob anomalies.\n\nBefore running `AutoMLOps.monitor()`, be sure to install the package dependencies using the following command: `pip3 install -r AutoMLOps/model_monitoring/requirements.txt --user`\n\n### `AutoMLOps.monitor()` parameter list:\n\nRequired parameters:\n1. `target_field: str`\n2. `model_endpoint: str`\n\nOptional parameters (defaults shown):\n1. `alert_emails: list = None`\n2. `auto_retraining_params: dict = None`\n3. `drift_thresholds: dict = None`\n4. `hide_warnings: bool = True`\n5. `job_display_name: str = f'{naming_prefix}-model-monitoring-job'`\n6. `monitoring_interval: int = 1`\n7. `monitoring_location: str = 'us-central1'`\n8. `sample_rate: float = 0.8`\n9. `skew_thresholds: dict = None`\n10. `training_dataset: str = None`\n\nA description of the parameters is below:\n- `target_field`: Prediction target column name in training dataset.\n- `model_endpoint`: Endpoint resource name of the deployed model to monitoring. Format: projects/{project}/locations/{location}/endpoints/{endpoint}\n- `alert_emails`: Optional list of emails to send monitoring alerts. Email alerts not used if this value is set to None.\n- `auto_retraining_params`: Pipeline parameter values to use when retraining the model. Defaults to None; if left None, the model will not be retrained if an alert is generated.\n- `drift_thresholds`: Compares incoming data to data previously seen to check for drift.\n- `hide_warnings`: Boolean that specifies whether to show permissions warnings before monitoring.\n- `job_display_name`: Display name of the ModelDeploymentMonitoringJob. The name can be up to 128 characters long and can be consist of any UTF-8 characters.\n- `monitoring_interval`: Configures model monitoring job scheduling interval in hours. This defines how often the monitoring jobs are triggered.\n- `monitoring_location`: Location to retrieve ModelDeploymentMonitoringJob from.\n- `sample_rate`: Used for drift detection, specifies what percent of requests to the endpoint are randomly sampled for drift detection analysis. This value most range between (0, 1].\n- `skew_thresholds`: Compares incoming data to the training dataset to check for skew.\n- `training_dataset`: Training dataset used to train the deployed model. This field is required if using skew detection.\n\n\n### Model Monitoring Diagram\n\n\u003cp align=\"center\"\u003e\n    \u003cimg src=\"assets/monitor/monitor-default.png\" alt=\"CICD\" width=\"1000\"/\u003e\n\u003c/p\u003e\n\n# Other Customizations\n\n**Using Github Actions:**\n\nTo use Github Actions integration, you must first have a [Workload Identity Federation](https://cloud.google.com/iam/docs/workload-identity-federation) set up properly. You must use a pre-existing Github repo, and you must also have already set up and registered your [ssh keys](https://docs.github.com/en/authentication/connecting-to-github-with-ssh/adding-a-new-ssh-key-to-your-github-account) with your Github Repo. If you meet all of these prerequisites, you can use github actions as follows:\n```\nAutoMLOps.generate(project_id=PROJECT_ID,\n                   pipeline_params=pipeline_params,\n                   use_ci=True,\n                   deployment_framework='github-actions',\n                   project_number='\u003cproject_number\u003e',\n                   source_repo_type='github',\n                   source_repo_name='source/repo/string',\n                   workload_identity_pool='identity_pool_string',\n                   workload_identity_provider='identity_provider_string',\n                   workload_identity_service_account='workload_identity_sa')\n```\n\nMore specific details for setting up AutoMLOps to use Github and Github Actions can be found in [this doc](docs/Using%20Github%20With%20AMO.md).\n\n**Set scheduled run:**\n\nUse the `schedule_pattern` parameter to specify a cron job schedule to run the pipeline job on a recurring basis.\n```\nschedule_pattern = '0 */12 * * *'\n```\n\n**Use Vertex AI Experiments:**\n\nTo use Vertex AI Experiments, include key-value pair for `vertex_experiment_tracking_name` in your pipeline parameters dictionary. An experiment will be created if one does not already exist with the specified name.\n```\npipeline_params = {\n    'project_id': PROJECT_ID,\n    'region': 'us-central1',\n    'vertex_experiment_tracking_name': 'my-experiment-name'\n}\nAutoMLOps.generate(project_id=PROJECT_ID,\n                   pipeline_params=pipeline_params)\n```\n\n**Set pipeline compute resources:**\n\nUse the `base_image` and `custom_training_job_specs` parameter to specify resources for any custom component in the pipeline.\n```\nbase_image = 'us-docker.pkg.dev/vertex-ai/training/tf-gpu.2-11.py310:latest',\ncustom_training_job_specs = [{\n    'component_spec': 'train_model',\n    'display_name': 'train-model-accelerated',\n    'machine_type': 'a2-highgpu-1g',\n    'accelerator_type': 'NVIDIA_TESLA_A100',\n    'accelerator_count': 1\n}]\n```\n\n**Use a VPC connector:**\n\nUse the `vpc_connector` parameter to specify a vpc connector.\n```\nvpc_connector = 'example-vpc-connector'\n```\n\n**Specify package versions:**\n\nUse the `packages_to_install` parameter of `@AutoMLOps.component` to explicitly specify packages and versions.\n```\n@AutoMLOps.component(\n    packages_to_install=[\n        \"google-cloud-bigquery==2.34.4\",\n        \"pandas\",\n        \"pyarrow\",\n        \"db_dtypes\"\n    ]\n)\ndef create_dataset(\n    bq_table: str,\n    data_path: str,\n    project_id: str\n):\n...\n```\n\n# Contributors\n\n[Sean Rastatter](mailto:srastatter@google.com): Tech Lead\n\n[Tony DiLoreto](mailto:tonydiloreto@google.com): Project Manager\n\n[Allegra Noto](mailto:allegranoto@google.com): Senior Project Engineer\n\n[Ahmad Khan](mailto:ahmadkh@google.com): Engineer\n\n[Jesus Orozco](mailto:jesusfc@google.com): Engineer\n\n[Erin Horning](mailto:ehorning@google.com): Engineer\n\n[Alex Ho](mailto:alexanderho@google.com): Engineer\n\n[Kyle Sorensen](mailto:kylesorensen@google.com): Engineer\n\n[Matt Sokoloff](mailto:msokoloff@google.com): Engineer\n\n[Andrew Chasin](mailto:andrewchasin@google.com): Engineer\n\n[Atulan Zaman](mailto:atulanz@google.com): Use Case Development\n\n[Nelly Wilson](mailto:nellywilson@google.com): Use Case Development\n\n# Disclaimer\n\n**This is not an officially supported Google product.**\n\nCopyright 2024 Google LLC. All Rights Reserved.\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use this file except in compliance with the License.\nYou may obtain a copy of the License at\n\nhttp://www.apache.org/licenses/LICENSE-2.0\n\nUnless required by applicable law or agreed to in writing, software\ndistributed under the License is distributed on an \"AS IS\" BASIS,\nWITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nSee the License for the specific language governing permissions and\nlimitations under the License.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgooglecloudplatform%2Fautomlops","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgooglecloudplatform%2Fautomlops","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgooglecloudplatform%2Fautomlops/lists"}