https://github.com/leoho0722/mnist-openfaas-example
Example for MLOps Pipeline using OpenFaaS Functions and MNIST handwritten CNN model
https://github.com/leoho0722/mnist-openfaas-example
kubernetes mlops mnist openfaas openfaas-function pipeline
Last synced: about 1 year ago
JSON representation
Example for MLOps Pipeline using OpenFaaS Functions and MNIST handwritten CNN model
- Host: GitHub
- URL: https://github.com/leoho0722/mnist-openfaas-example
- Owner: leoho0722
- Created: 2024-03-01T09:41:49.000Z (about 2 years ago)
- Default Branch: main
- Last Pushed: 2024-04-13T12:22:18.000Z (almost 2 years ago)
- Last Synced: 2024-04-14T02:28:27.837Z (almost 2 years ago)
- Topics: kubernetes, mlops, mnist, openfaas, openfaas-function, pipeline
- Language: Python
- Homepage: https://leoho0722.github.io/docs-repo/mnist-openfaas-example.html
- Size: 82 KB
- Stars: 0
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# mnist-openfaas-example
## Environment Install
### Docker
[Docker Official Documentation](https://docs.docker.com/engine/install/ubuntu/)
#### Install Docker Engine on Ubuntu
* Step 1: Set up Docker's apt repository.
```shell
# Add Docker's official GPG key:
sudo apt-get update
sudo apt-get install ca-certificates curl
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc
# Add the repository to Apt sources:
echo \
"deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
$(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update
```
* Step 2: Install the Docker packages.
```shell
sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
```
### OpenFaaS
See [openfaas-example README.md](https://github.com/leoho0722/openfaas-example/blob/main/README.md)
#### OpenFaaS Command Line Tool (faas-cli)
* macOS
* ```brew install faas-cli```
* Ubuntu
* Root user
* ```curl -sSL https://cli.openfaas.com | sudo -E sh```
* Non root user
* ```curl -sSL https://cli.openfaas.com | sh```
#### Kubernetes
* Step 1: Create Kubernetes Namespace
```shell
kubectl apply -f https://raw.githubusercontent.com/openfaas/faas-netes/master/namespaces.yml
```
* Step 2: Helm charts
```shell
helm repo add openfaas https://openfaas.github.io/faas-netes/
helm repo update
helm upgrade openfaas --install openfaas/openfaas --namespace openfaas
```
* (**Optional**) Step 3: Expose OpenFaaS Gateway Service to LoadBalancer
```shell
kubectl patch svc gateway-external -n openfaas -p '{"spec":{"type": "LoadBalancer"}}'
```
* Step 4: Get OpenFaaS admin Password
```shell
PASSWORD=$(kubectl -n openfaas get secret basic-auth -o jsonpath="{.data.basic-auth-password}" | base64 --decode) && \
echo "OpenFaaS admin password: $PASSWORD"
```
* Step 5: Login OpenFaaS WebUI via credential
```shell
# Default gateway endpoint is http://127.0.0.1:8080
faas-cli login -u admin -p $PASSWORD
# Specify gateway endpoint, for example http://10.0.0.156:31112
faas-cli login -u admin -p $PASSWORD -g http://10.0.0.156:31112
```
### MinIO Server
[MinIO Official Documentation](https://min.io/docs/minio/linux/index.html)
#### Install
* Ubuntu
```shell
wget https://dl.min.io/server/minio/release/linux-amd64/archive/minio_20240305044844.0.0_amd64.deb -O minio.deb
sudo dpkg -i minio.deb
```
#### Launch
```shell
mkdir ~/minio
minio server ~/minio --console-address :9001
```
#### Connect browser to MinIO Server
MinIO WebUI default account: minioadmin
MinIO WebUI default password: minioadmin
```text
http://:9001
# Example
# http://10.0.0.156:9001
```
### MinIO Client
[MinIO Official Documentation](https://min.io/docs/minio/linux/index.html)
#### Install
```shell
wget https://dl.min.io/client/mc/release/linux-amd64/mc
chmod +x mc
sudo mv mc /usr/local/bin/mc
```
#### Config
```shell
mc alias set http://:9000
mc admin info
# Example
# mc alias set local http://10.0.0.156:9000 minioadmin minioadmin
# mc admin info local
```
##### Python Client API
[MinIO Python Client API Official Documentation](https://min.io/docs/minio/linux/developers/python/API.html)
```python
from minio import Minio
def connect_minio():
"""Connect to MinIO Server"""
MINIO_API_ENDPOINT = os.environ["minio_api_endpoint"]
MINIO_ACCESS_KEY = os.environ["minio_access_key"]
MINIO_SECRET_KEY = os.environ["minio_secret_key"]
return Minio(
MINIO_API_ENDPOINT,
access_key=MINIO_ACCESS_KEY,
secret_key=MINIO_SECRET_KEY,
secure=False # Disable HTTPS
)
```
## Build, Push, Deploy, Remove OpenFaaS Functions
### One-key to build, push, deploy (**Recommand**)
```shell
# Default architecture is amd64
make faas-up
# Specify architecture, for example arm64
make faas-up ARCH=arm64
```
### Manual
* Build
```shell
# Default architecture is amd64
make faas-build
# Specify architecture, for example arm64
make faas-build ARCH=arm64
```
* Push
```shell
# Default architecture is amd64
make faas-push
# Specify architecture, for example arm64
make faas-push ARCH=arm64
```
* Deploy
```shell
# Default architecture is amd64
make faas-deploy
# Specify architecture, for example arm64
make faas-deploy ARCH=arm64
```
* Remove
```shell
# Default architecture is amd64
make faas-remove
# Specify architecture, for example arm64
make faas-remove ARCH=arm64
```
## References
1.