{"id":19797665,"url":"https://github.com/viveksyngh/openfaas-pipeline","last_synced_at":"2025-05-01T03:31:56.886Z","repository":{"id":38975046,"uuid":"176008086","full_name":"viveksyngh/openfaas-pipeline","owner":"viveksyngh","description":"Event driven pipeline using OpenFaaS, Minio and Tensorflow inception","archived":false,"fork":false,"pushed_at":"2022-06-01T23:32:36.000Z","size":5168,"stargazers_count":15,"open_issues_count":2,"forks_count":5,"subscribers_count":3,"default_branch":"master","last_synced_at":"2024-06-21T18:04:22.299Z","etag":null,"topics":["golang","minio","nodejs","openfaas"],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/viveksyngh.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-03-16T18:16:24.000Z","updated_at":"2022-03-05T04:51:37.000Z","dependencies_parsed_at":"2022-09-19T16:53:27.642Z","dependency_job_id":null,"html_url":"https://github.com/viveksyngh/openfaas-pipeline","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/viveksyngh%2Fopenfaas-pipeline","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/viveksyngh%2Fopenfaas-pipeline/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/viveksyngh%2Fopenfaas-pipeline/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/viveksyngh%2Fopenfaas-pipeline/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/viveksyngh","download_url":"https://codeload.github.com/viveksyngh/openfaas-pipeline/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224234032,"owners_count":17277889,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["golang","minio","nodejs","openfaas"],"created_at":"2024-11-12T07:26:14.322Z","updated_at":"2024-11-12T07:26:14.926Z","avatar_url":"https://github.com/viveksyngh.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# OpenFaas Pipeline\nEvent driven pipeline using OpenFaaS, Minio and Tensorflow inception\n\n\n![OpenFaaS Pipeline](https://github.com/viveksyngh/openfaas-pipeline/blob/master/media/openfaas-pipeline.jpg?raw=true)\n\n\n## Kubernetes \nFor Kubernetes, first install Helm and Tiller\n\n### Install Helm \n\n#### Install Helm CLI Client\n\n* On Linux and Mac/Darwin\n```\ncurl https://raw.githubusercontent.com/kubernetes/helm/master/scripts/get | bash\n```\n\n* On Mac Via Homebrew \n```\nbrew install kubernetes-helm \n```\n\n#### Install Tiller\n\n* Create RBAC permissions for tiller\n```\nkubectl -n kube-system create sa tiller \\\n  \u0026\u0026 kubectl create clusterrolebinding tiller \\\n  --clusterrole cluster-admin \\\n  --serviceaccount=kube-system:tiller\n```\n\n* Install the server-side Tiller component on your cluster\n```\nhelm init --skip-refresh --upgrade --service-account tiller\n```\n\n### Install OpenFaaS on Kubernetes\n\n* Create OpenFaaS namespace \n```sh\nkubectl apply -f https://raw.githubusercontent.com/openfaas/faas-netes/master/namespaces.yml\n```\n\n* Add OpenFaaS helm repository \n```sh\nhelm repo add openfaas https://openfaas.github.io/faas-netes/\n```\n\n* Create basic-auth credentials\n```sh\nPASSWORD=$(head -c 12 /dev/urandom | shasum| cut -d' ' -f1)\n```\n\n```sh\nkubectl -n openfaas create secret generic basic-auth \\\n--from-literal=basic-auth-user=admin \\\n--from-literal=basic-auth-password=\"$PASSWORD\"\n```\n\n* Install OpenFaaS on kubernetes cluster\n```sh\nhelm repo update \\\n \u0026\u0026 helm upgrade openfaas --install openfaas/openfaas \\\n    --namespace openfaas  \\\n    --set basic_auth=true \\\n    --set functionNamespace=openfaas-fn \\\n    --set operator.create=true\n\n```\n\n### Install Minio Client\n\n* MacOS\n\n```sh\nbrew install minio/stable/mc\n```\n\n* Linux\n\n```sh\nwget https://dl.min.io/client/mc/release/linux-amd64/mc\nchmod +x mc\n```\n\n\n### Install and Configure minio\n\n* Create OpenFaaS namespaces\n```\nkubectl apply -f https://raw.githubusercontent.com/openfaas/faas-netes/master/namespaces.yml\n```\n\n* Generate secrets for Minio\n```\nSECRET_KEY=$(head -c 12 /dev/urandom | shasum| cut -d' ' -f1)\nACCESS_KEY=$(head -c 12 /dev/urandom | shasum| cut -d' ' -f1)\n```\n\n* Create Secrets in kubernetes\n```\nkubectl create secret generic -n openfaas-fn \\\n s3-secret-key --from-literal s3-secret-key=\"$SECRET_KEY\"\n```\n\n```\nkubectl create secret generic -n openfaas-fn \\\n s3-access-key --from-literal s3-access-key=\"$ACCESS_KEY\"\n```\n\n* Install minio with helm chart\n```\nhelm install --name cloud-minio \\\n    --namespace openfaas \\\n    --set accessKey=$ACCESS_KEY,secretKey=$SECRET_KEY,replicas=1,persistence.enabled=false,service.port=9000,service.type=NodePort stable/minio\n```\n\n* Get Minio NodePort\n\n```\nMINIO_PORT=$(kubectl get svc/cloud-minio -n openfaas --output=jsonpath='{.spec.ports[?(@.name==\"service\")].nodePort}')\n``` \n\n* Configure minio client\n```sh\nmc config host add minio-kube http://127.0.0.1:$MINIO_PORT $ACCESS_KEY $SECRET_KEY\n```\n\n* Get minio config and edit the JSON to add webhook handler\n```sh\nmc admin config get minio-kube \u003e myconfig.json\n```\nedit webhook section of `myconfig.json` and save it\n```json\n\"webhook\":{\n    \"1\":{\n        \"enable\":true,\n        \"endpoint\":\"http://gateway.openfaas:8080/function/minio-webhook-hanlder\"\n        }\n    }\n}\n```\n\n* Update minio config and restart mino server\nUpdate the mini config and restart minio server\n```sh\nmc admin config set minio-kube \u003c myconfig.json\n```\n\n```sh\nmc admin service restart minio-kube\n```\n\n* Create buckets\n```sh\nmc mb minio-kube/images\n```\n```sh\nmc mb minio-kube/images-thumbnail\n```\n```sh\nmc mb minio-kube/inception\n```\n\n* Add event for the webhook\n```sh\nmc event add minio-kube/images arn:minio:sqs::1:webhook --event put --suffix .jpg\n```\n\n* Change `images` bucket policy to public so that inception function can download the image without secret\n```sh\nmc policy public minio-kube/images\n```\n\n#### Deploy Functions\n\n* Login with faas-cli\n```sh\nfaas-cli login -u admin -p $PASSWORD -g http://127.0.0.1:31112\n```\n\n* Deploy functions\n```sh\nfaas-cli deploy -f stack.kubernetes.yml\n```\n\n## Docker swarm\n\n### Install OpenFaaS\n```sh\ngit clone https://www.github.com/openfaas/faas \u0026\u0026 \\\n        cd faas \u0026\u0026 ./deploy_stack.sh\n```\n\n### Install and Configure minio\n\n* Create minio secret and access key\n```sh\nSECRET_KEY=$(head -c 12 /dev/urandom | shasum| cut -d' ' -f1)\nACCESS_KEY=$(head -c 12 /dev/urandom | shasum| cut -d' ' -f1)\n\necho -n \"$SECRET_KEY\" | docker secret create s3-secret-key -\necho -n \"$ACCESS_KEY\" | docker secret create s3-access-key -\n```\n\n* Deploy minio to cluster\n```sh\ndocker service create --constraint=\"node.role==manager\" \\\n --name minio \\\n --mount type=bind,source=~/minio/data,target=/data \\\n --mount type=bind,source=~/minio/config,target=/root/.minio \\\n --detach=true --network func_functions \\\n --secret s3-access-key \\\n --secret s3-secret-key \\\n --env MINIO_SECRET_KEY_FILE=s3-secret-key \\\n --env MINIO_ACCESS_KEY_FILE=s3-access-key \\\nminio/minio:latest server /data\n```\n\n* Publish port `9000`\n```sh\ndocker service update minio --publish-add 9000:9000\n``` \n\n* Configure minio client\n```sh\nmc config host add minio http://127.0.0.1:9000 $ACCESS_KEY $SECRET_KEY\n```\n\n* Get minio config and edit the JSON to add webhook handler\n```sh\nmc admin config get minio \u003e myconfig.json\n```\nedit webhook section of `myconfig.json` and save it\n```json\n\"webhook\":{\n    \"1\":{\n        \"enable\":true,\n        \"endpoint\":\"http://gateway:8080/function/minio-webhook-hanlder\"\n        }\n    }\n}\n```\n\n* Update minio config and restart mino server\nUpdate the mini config and restart minio server\n```sh\nmc admin config set minio \u003c myconfig.json\n```\n\n```sh\nmc admin service restart minio\n```\n\n* Create required buckets\n```sh\nmc mb minio/images\n```\n```sh\nmc mb minio/images-thumbnail\n```\n```sh\nmc mb minio/inception\n```\n\n* Add event for the webhook\n```sh\nmc event add minio/images arn:minio:sqs::1:webhook --event put --suffix .jpg\n```\n\n* Change `images` bucket policy to public so that inception function can download the image without secret\n```sh\nmc policy public minio/images\n```\n\n#### Deploy Functions\n\n* Login with faas-cli\n```sh\nfaas-cli login -u admin -p $PASSWORD -g http://127.0.0.1:31112\n```\n\n* Deploy functions\n```sh\nfaas-cli deploy -f stack.swarm.yml\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fviveksyngh%2Fopenfaas-pipeline","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fviveksyngh%2Fopenfaas-pipeline","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fviveksyngh%2Fopenfaas-pipeline/lists"}