Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/dsdatsme/ml-circleci-integration
Building Machine Learning pipeline using CircleCI integration on docker and kubernetes
https://github.com/dsdatsme/ml-circleci-integration
circleci docker kubernetes machine-learning
Last synced: 6 days ago
JSON representation
Building Machine Learning pipeline using CircleCI integration on docker and kubernetes
- Host: GitHub
- URL: https://github.com/dsdatsme/ml-circleci-integration
- Owner: DSdatsme
- Created: 2020-04-05T10:04:58.000Z (over 4 years ago)
- Default Branch: master
- Last Pushed: 2022-06-22T01:36:49.000Z (over 2 years ago)
- Last Synced: 2024-10-17T00:16:44.871Z (21 days ago)
- Topics: circleci, docker, kubernetes, machine-learning
- Language: Python
- Homepage:
- Size: 220 KB
- Stars: 0
- Watchers: 2
- Forks: 2
- Open Issues: 2
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Machine Learning Pipeline with CircleCI
[](https://circleci.com/gh/DSdatsme/ML-circleci-integration)
## Project Overview
In this project, you will apply the skills you have acquired in this course to operationalize a Machine Learning Microservice API.
You are given a pre-trained, `sklearn` model that has been trained to predict housing prices in Boston according to several features, such as average rooms in a home and data about highway access, teacher-to-pupil ratios, and so on. You can read more about the data, which was initially taken from Kaggle, on [the data source site](https://www.kaggle.com/c/boston-housing). This project tests your ability to operationalize a Python flask app—in a provided file, `app.py`—that serves out predictions (inference) about housing prices through API calls. This project could be extended to any pre-trained machine learning model, such as those for image recognition and data labeling.
### Project Tasks
Your project goal is to operationalize this working, machine learning microservice using [kubernetes](https://kubernetes.io/), which is an open-source system for automating the management of containerized applications. In this project you will:
* Test your project code using linting
* Complete a Dockerfile to containerize this application
* Deploy your containerized application using Docker and make a prediction
* Improve the log statements in the source code for this application
* Configure Kubernetes and create a Kubernetes cluster
* Deploy a container using Kubernetes and make a prediction
* Upload a complete Github repo with CircleCI to indicate that your code has been tested
You can find a detailed [project rubric, here](https://review.udacity.com/#!/rubrics/2576/view).
**The final implementation of the project will showcase your abilities to operationalize production microservices.**
---
## Setup the Environment
* Create a virtualenv and activate it
* Run `make install` to install the necessary dependencies
### Running `app.py`
1. Standalone: `python app.py`
2. Run in Docker: `./run_docker.sh`
3. Run in Kubernetes: `./run_kubernetes.sh`
### Make Predictions
After running `app.py` inside container, now you can make predictions using the following command.
```bash
bash make_prediction.sh
```
### Kubernetes Steps
* Setup and Configure Docker locally
* Setup and Configure Kubernetes locally
* Create Flask app in Container
* Run via kubectl
### Push Images To Docker
Before you run the upload command make sure you have username and password of DockerHub set in shell variables as below
```bash
username=[DOCKER-HUB-USERNAME]
password=[DOCKER-HUB-PASSWORD]
```
After setting up variables, you can run the following command to Upload Docker image.
```bash
bash upload_docker.sh
```
### Files
* run_docker.sh : to run app server in docker
* run_kubernetes.sh : run app server on kubernetes
* upload_docker.sh : upload locally built docker image to DockerHub
* model_data : folder containing model files
* config.yaml : Circle CI build file