https://github.com/sayakpaul/ci-cd-for-model-training
This repository holds files and scripts for incorporating simple CI/CD practices for model training in ML.
https://github.com/sayakpaul/ci-cd-for-model-training
ci-cd cloud-build cloud-functions cloud-scheduler gcp kubeflow-pipelines mlops model-training pub tensorflow-extended vertex-ai
Last synced: 12 days ago
JSON representation
This repository holds files and scripts for incorporating simple CI/CD practices for model training in ML.
- Host: GitHub
- URL: https://github.com/sayakpaul/ci-cd-for-model-training
- Owner: sayakpaul
- License: apache-2.0
- Created: 2021-08-20T08:40:32.000Z (over 3 years ago)
- Default Branch: main
- Last Pushed: 2021-10-26T02:04:56.000Z (over 3 years ago)
- Last Synced: 2025-03-31T01:13:27.110Z (about 2 months ago)
- Topics: ci-cd, cloud-build, cloud-functions, cloud-scheduler, gcp, kubeflow-pipelines, mlops, model-training, pub, tensorflow-extended, vertex-ai
- Language: Jupyter Notebook
- Homepage: https://cloud.google.com/blog/topics/developers-practitioners/model-training-cicd-system-part-ii
- Size: 716 KB
- Stars: 21
- Watchers: 1
- Forks: 6
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# CI/CD for Model Training in ML
This repository holds files and scripts for incorporating simple CI/CD practices for model training in ML. Following
resources have been used to prepare the files under the `build` directory:
* [Vertex AI Training with TFX and Vertex Pipelines](https://www.tensorflow.org/tfx/tutorials/tfx/gcp/vertex_pipelines_vertex_training)
* [MLOps with Vertex AI by GCP](https://github.com/GoogleCloudPlatform/mlops-with-vertex-ai)
Another accompanying repository for doing CI/CD for model training can be found here: [deep-diver/Model-Training-as-a-CI-CD-System](https://github.com/deep-diver/Model-Training-as-a-CI-CD-System).
This repository acts as a sister repository to [deep-diver/Model-Training-as-a-CI-CD-System](https://github.com/deep-diver/Model-Training-as-a-CI-CD-System). It's recommended that one goes through this repository with the accompanying blog post from Google Cloud: [Model training as a CI/CD system: Part II](https://cloud.google.com/blog/topics/developers-practitioners/model-training-cicd-system-part-ii). The figures below schematically present what this project implements:
We first get a compiled pipeline specification for model re-training capable of accepting hyperparameter values as [`RuntimeParameters`](https://www.tensorflow.org/tfx/api_docs/python/tfx/v1/dsl/experimental/RuntimeParameter):
Then we schedule and trigger model training jobs:
## Acknowledgements
[ML-GDE program](https://developers.google.com/programs/experts/) for providing GCP credits. Thanks to [Karl Weinmeister](https://twitter.com/kweinmeister?lang=hr) for providing review feedback on this project.