https://github.com/sayakpaul/ci-cd-for-model-training

This repository holds files and scripts for incorporating simple CI/CD practices for model training in ML.
https://github.com/sayakpaul/ci-cd-for-model-training

ci-cd cloud-build cloud-functions cloud-scheduler gcp kubeflow-pipelines mlops model-training pub tensorflow-extended vertex-ai

Last synced: about 7 hours ago
JSON representation

This repository holds files and scripts for incorporating simple CI/CD practices for model training in ML.

• Host: GitHub
• URL: https://github.com/sayakpaul/ci-cd-for-model-training
• Owner: sayakpaul
• License: apache-2.0
• Created: 2021-08-20T08:40:32.000Z (about 3 years ago)
• Default Branch: main
• Last Pushed: 2021-10-26T02:04:56.000Z (almost 3 years ago)
• Last Synced: 2024-04-21T17:24:39.160Z (6 months ago)
• Topics: ci-cd, cloud-build, cloud-functions, cloud-scheduler, gcp, kubeflow-pipelines, mlops, model-training, pub, tensorflow-extended, vertex-ai
• Language: Jupyter Notebook
• Homepage: https://cloud.google.com/blog/topics/developers-practitioners/model-training-cicd-system-part-ii
• Size: 716 KB
• Stars: 18
• Watchers: 2
• Forks: 6
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# CI/CD for Model Training in ML

This repository holds files and scripts for incorporating simple CI/CD practices for model training in ML. Following

resources have been used to prepare the files under the `build` directory:

* [Vertex AI Training with TFX and Vertex Pipelines](https://www.tensorflow.org/tfx/tutorials/tfx/gcp/vertex_pipelines_vertex_training)

* [MLOps with Vertex AI by GCP](https://github.com/GoogleCloudPlatform/mlops-with-vertex-ai)

Another accompanying repository for doing CI/CD for model training can be found here: [deep-diver/Model-Training-as-a-CI-CD-System](https://github.com/deep-diver/Model-Training-as-a-CI-CD-System). 

This repository acts as a sister repository to [deep-diver/Model-Training-as-a-CI-CD-System](https://github.com/deep-diver/Model-Training-as-a-CI-CD-System). It's recommended that one goes through this repository with the accompanying blog post from Google Cloud: [Model training as a CI/CD system: Part II](https://cloud.google.com/blog/topics/developers-practitioners/model-training-cicd-system-part-ii). The figures below schematically present what this project implements:

We first get a compiled pipeline specification for model re-training capable of accepting hyperparameter values as [`RuntimeParameters`](https://www.tensorflow.org/tfx/api_docs/python/tfx/v1/dsl/experimental/RuntimeParameter):



  



Then we schedule and trigger model training jobs:



  



## Acknowledgements

[ML-GDE program](https://developers.google.com/programs/experts/) for providing GCP credits. Thanks to [Karl Weinmeister](https://twitter.com/kweinmeister?lang=hr) for providing review feedback on this project.