{"id":21184881,"url":"https://github.com/r-gg/serverless-computing","last_synced_at":"2025-03-14T19:48:39.333Z","repository":{"id":233608309,"uuid":"720728112","full_name":"r-gg/serverless-computing","owner":"r-gg","description":"Implementation of Federated Learning on a Serverless platform Openwhisk","archived":false,"fork":false,"pushed_at":"2024-02-01T15:35:09.000Z","size":21092,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-01-21T12:46:56.430Z","etag":null,"topics":["federated-learning","kubernetes","openwhisk","serverless"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/r-gg.png","metadata":{"files":{"readme":"README-dev.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-11-19T12:20:36.000Z","updated_at":"2024-04-16T17:21:08.000Z","dependencies_parsed_at":"2024-05-01T17:46:32.136Z","dependency_job_id":null,"html_url":"https://github.com/r-gg/serverless-computing","commit_stats":null,"previous_names":["r-gg/serverless-computing"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/r-gg%2Fserverless-computing","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/r-gg%2Fserverless-computing/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/r-gg%2Fserverless-computing/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/r-gg%2Fserverless-computing/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/r-gg","download_url":"https://codeload.github.com/r-gg/serverless-computing/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243639414,"owners_count":20323505,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["federated-learning","kubernetes","openwhisk","serverless"],"created_at":"2024-11-20T18:12:10.626Z","updated_at":"2025-03-14T19:48:39.325Z","avatar_url":"https://github.com/r-gg.png","language":"Jupyter Notebook","readme":"# TODOs\n\n- currently the actions need to be manually warmed up before starting training.\n- EMNIST instead of MNIST\n\n# Order at setup\n\n1. setup-kind-openwhisk\n2. setup-minio (added a little update to the setup script and did not test it, so it might not work) + add minioadmin1 access key in the dashboard \n3. setup-kafka\n4. start-main-app\n5. goto localhost:5000/build-everything ~ creates a bucket, splits and uploads MNIST dataset to the bucket. creates a dedicated kafka topic called 'federated'\n6. prepare the action: `......./python$ ./deploy-python-action.sh learner`\n7. to start learning goto localhost:5000/learn\n\n# To start learning\n\nAfter staring the main-app, it gets served on localhost:5000. After navigating to `/build-everything` the training can start.\n\nTo start training navigate to `/learn`. Pass the arguments with url params: `/learn?nclients=10\u0026nrounds=10\u0026nselected=5`\n\nDefault values are:\n- nclients=3\n- nselected=3\n- nrounds=5\n\nEach round starts nclients and each client gets one split. As soon as the 50 splits get used the splits start getting re-used.\n\n# After changing the action code\n\n- Delete action `wski action delete learner`\n- Redeploy action `......./python$ ./deploy-python-action.sh learner`\n\n# After changing the main-app code\n\n- Delete deployment `kubectl delete deployment main-app -n main-app`\n- Kill tmux session `tmux kill-session -t main-app`\n- rebuild and deploy `...../main-app$ ./build-and-start-main-app.sh`\n\n# Available services to all pods\n\n- kafka at : `kafkica.openwhisk.svc.cluster.local`\n- minio at : `minio-operator9000.minio-dev.svc.cluster.local` (port 9000)\n- openwhisk API (for wsk CLI) : `owdev-nginx.openwhisk.svc.cluster.local`\n\n\n# Developing the main app\n\n- Ports 5000 and 5001 of the main-app pod are open. The default app is served on 5000 and 5001 is left for experimentation.\n- If you want to add custom code to the main flask app, modify the app.py in the pod and start serving the flask app on localhost:5001\n\n# Openwhisk setup (automated with `./setup-kind-openwhisk.sh`)\n\n## Deploy the kubernetes cluster with kind\n\n```\narrow@rg-vm:~/Desktop/openwhisk-test/openwhisk-deploy-kube/deploy/kind$ ./start-kind.sh \n```\n\n## Deploy openwhisk on that k8s cluster\n\n### Label the nodes as invoker and core\n\n```\nkubectl label node kind-worker openwhisk-role=invoker \u0026\u0026 kubectl label node kind-worker2 openwhisk-role=invoker \u0026\u0026 kubectl label node kind-control-plane openwhisk-role=core \n```\n\n### Deploy openwhisk\n```\narrow@rg-vm:~/Desktop/openwhisk-test/openwhisk-deploy-kube/deploy/kind$ helm install owdev ../../helm/openwhisk/ -n openwhisk --create-namespace -f mycluster.yaml\n```\n\n#### Check status with\n\n```\nkubectl get pods -n openwhisk --watch\n```\ninstall-packages should be completed.\n\n\n## Set up kubernetes dashboard and start serving it (automated in `./setup-kind-openwhisk.sh` and `./get-kube-dashboard-token.sh`)\n\n```\nkubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.7.0/aio/deploy/recommended.yaml \u0026\u0026 kubectl proxy\n```\n\nIn order to log in to the dashboard create service account role and assign bindings to get the token\n\n```\nkubectl apply -f dashboard-adminuser.yaml \u0026\u0026 kubectl apply -f dashboard-cluster-role-binding.yaml \u0026\u0026 kubectl apply -f dashboard-admin-secret.yaml \u0026\u0026 kubectl get secret admin-user -n kubernetes-dashboard -o jsonpath={\".data.token\"} | base64 -d\n```\n\nthe AUTH data and API_URL is stored in `~/.wskprop` (also queryable via `wski property get --all`) (`wski` is an alias for `wsk -i`)\n\n\n\n# MinIO\n\nFollowing https://min.io/docs/minio/kubernetes/upstream/\n\nIn default minio-dev.yaml changed (deleted the following lines):\n```\nnodeSelector:\n    kubernetes.io/hostname: kind-control-plane # Specify a node label associated to the Worker Node on which you want to deploy the pod.\n```\n\n(address 9090 was already in use  so `kubectl port-forward pod/minio 9000:9090 -n minio-dev`)\n\n## Expose Minio pod as a service so it is reachable within the cluster (Automated in `./setup-minio.sh`)\n\nhttps://kubernetes.io/docs/tutorials/services/connect-applications-service/\nhttps://kubernetes.io/docs/concepts/services-networking/dns-pod-service/\n\n```\nkubectl expose pod minio --port=9000 --name=minio-operator9000 -n minio-dev\n```\n\nTesting if it is reachable from another pod\n```\nkubectl run curl --image=radial/busyboxplus:curl -i --tty --rm\n[ root@curl:/ ]$ curl -k minio-operator9000.minio-dev.svc.cluster.local:9000\n```\n\n## Create access key in minio dashboard\n\nAutomated in `./setup-minio.sh` (at the end).\n\n\n## New python runtime\n\n \nSince minio needs python version \u003e3.7 and the default python runtime of OpenWhisk actions is 3.6, we use another runtime 3.11 by following the setup steps in https://github.com/apache/openwhisk-runtime-python/blob/master/tutorials/local_build.md and https://github.com/apache/openwhisk-runtime-python/tree/master respectively\n\nTo build the runtime and push it to dockerhub run the script `./rebuild-image-and-push-to-hub.sh` in `new_python_runtime_setup/openwhisk-runtime-python` (from that directory).\n\n\n# Debugging \n\n\n## Debugging ML and Action code with a jupyter notebook pod\n\n```\n./start-jupyter.sh\n```\n\nRuns the jupyter pod as root so that further packages can be installed for debugging. It forwards the localhost:10000 to the pod port where the dashboard is served (8888). Note: No root privileges are available in dashboard. If you want to add packages or generally have sudo access, you have to exec into the pod.\n\nThe script also prints out the token needed for logging into the dashboard.\n\n## Debugging network in kubernetes \n\n```\nkubectl run tmp-shell --rm -i --tty --image nicolaka/netshoot\n```\n\n# Main-App\n\nThe main app image now contains `wsk` and its apihost and auth are already set to `owdev-nginx.openwhisk.svc.cluster.local` and guest credentials (`23bc46b1-71f6-4ed5-8c54-816aa4f8c502:123zO3xZCLrMN6v2BKK1dXYFpXlPkccOFqm12CdAsMgRU4VrNZ9lyGVCGuMDGIwP`) respectively\n\nTo download datasets to minio firstly log into minio dashboard, create access key where access key is `minioadmin1` and secret key is `minioadmin1` as well.\n\nCall `/setup-minio` endpoint of main-app\n\nCall `/download-and-split-dataset` ~ This downloads the entire MNIST dataset into the main-app pod and splits it into 50 pieces.\n\nCall `/upload-splits` ~ This uploads the 50 pieces into MinIO\n\n\n\n# Issues:\n\nTraining data cannot be stored locally but pulled from minio because an activation container cannot have access to the underlying pod's files. (possible with apache.openwhisk.Invoker modification but not sure how to do it with helm)\n\nKafka messages might not be consumed due to unclosed consumers. If it happens, just change the consumer's `group_id`.\n\n\n\n# Misc\n\n## get admin user credentials to access whisk.system namespace\nkubectl -n openwhisk get secret owdev-whisk.auth -o jsonpath='{.data.system}' | base64 --decode\n\nfrom [https://github.com/apache/openwhisk-deploy-kube/issues/253#issuecomment-404533474]\n\n## NOTE: Persistent volume host refers to the node running the pod (not the Ubuntu system hosting docker and kubernetes on it!)\n\nSo in order to have that, bind the volume from ubunto to (each) one of the nodes in the simulated k8s cluster (kind-workers)\n\n\n## Label pod so that it can be exposed as a service\n\n```\nkubectl label pod -n openwhisk owdev-kafka-0 application=kafka\n```\n\nThen use following yaml to create a service:\n```\napiVersion: v1\nkind: Service\nmetadata:\n  name: kafkica\n  labels:\n    application: kafka\nspec:\n  ports:\n  - port: 9092\n    protocol: TCP\n  selector:\n    application: kafka\n```\n\nit will expose the pod port 9092 \n\n\n\n## Prometheus and Grafana\n\nMix from https://dev.to/aws-builders/your-guide-to-prometheus-monitoring-on-kubernetes-with-grafana-gi8\nand https://semaphoreci.com/blog/prometheus-grafana-kubernetes-helm and https://medium.com/@giorgiodevops/kind-install-prometheus-operator-and-fix-missing-targets-b4e57bcbcb1f\n\n\n```\nhelm repo add prometheus-community https://prometheus-community.github.io/helm-charts\n```\n\n```\nkubectl create namespace prometheus\n```\n\n```\nhelm install stable prometheus-community/kube-prometheus-stack -n prometheus\n```\n\nExpose prometheus to localhost (it was only available to the nodes in the cluster)\n```\nkubectl port-forward -n prometheus svc/stable-kube-prometheus-sta-prometheus 9090:9090\n```\n\n- Note: Port-forward usually goes with pods and not services , hence `svc/` was added\n\nGrafana comes in the helm chart as well so we expose grafana on localhost with port-forwarding as well\n```\nkubectl -n prometheus port-forward svc/stable-grafana 3000:80   \n```\n\n### get admin password for grafana service (username is admin)\n```\nkubectl get secret --namespace prometheus stable-grafana -o jsonpath=\"{.data.admin-password}\" | base64 --decode ; echo\n```\n\n- The prometheus is already added as a data source in grafana :) !\n\n\n### Adding a new Openwhisk user (Optional) ~ Currently the guest account is used with predefined credentials.\n\nGuest credentials: \"--auth 23bc46b1-71f6-4ed5-8c54-816aa4f8c502:123zO3xZCLrMN6v2BKK1dXYFpXlPkccOFqm12CdAsMgRU4VrNZ9lyGVCGuMDGIwP\"\n\n```\nkubectl -n openwhisk  -ti exec owdev-wskadmin -- /bin/sh\n\nwskadmin user create fedUser\n```","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fr-gg%2Fserverless-computing","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fr-gg%2Fserverless-computing","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fr-gg%2Fserverless-computing/lists"}