{"id":23058031,"url":"https://github.com/srodi/data-driven-microservices","last_synced_at":"2026-04-14T14:32:23.788Z","repository":{"id":154979113,"uuid":"241990440","full_name":"SRodi/data-driven-microservices","owner":"SRodi","description":"Sample gRPC implementation and Docker containerisation of Python micro-services. Server generates gRPC tweet stream from a 1.6m tweets stored in csv file","archived":false,"fork":false,"pushed_at":"2020-04-29T12:41:12.000Z","size":13874,"stargazers_count":0,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-04-03T06:16:02.588Z","etag":null,"topics":["docker","docker-compose","flask","grpc","python","redis"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/SRodi.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-02-20T21:09:10.000Z","updated_at":"2020-06-24T05:44:00.000Z","dependencies_parsed_at":null,"dependency_job_id":"eac584c0-d6c2-4925-aef5-09a4b2b1ec31","html_url":"https://github.com/SRodi/data-driven-microservices","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/SRodi/data-driven-microservices","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SRodi%2Fdata-driven-microservices","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SRodi%2Fdata-driven-microservices/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SRodi%2Fdata-driven-microservices/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SRodi%2Fdata-driven-microservices/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/SRodi","download_url":"https://codeload.github.com/SRodi/data-driven-microservices/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SRodi%2Fdata-driven-microservices/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31801314,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-14T11:13:53.975Z","status":"ssl_error","status_checked_at":"2026-04-14T11:13:53.299Z","response_time":153,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["docker","docker-compose","flask","grpc","python","redis"],"created_at":"2024-12-16T02:14:14.746Z","updated_at":"2026-04-14T14:32:23.747Z","avatar_url":"https://github.com/SRodi.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# srodi-gRPC streaming\ngRPC example to implement data streaming and simulate real-time analytics\n\n#### Prerequisites\n* Python version 2.7 or higher\n* Docker version 19.03 or higher\n* Add `.csv` file with relative path `server/static/training.1600000.processed.noemoticon.csv`\n\nFile `training.1600000.processed.noemoticon.csv` can be downloaded from this [link](https://www.kaggle.com/kazanova/sentiment140/data)\n\n#### Architecture \nDocker compose will bring up 4 containers following the architecture described in diagram below.\n\nNote: `An additional service has been deployed [server-reddit]. server-reddit has the \nsame functionality and relations as [Server] service pictured below but it serves a different \ndataset.`\n\nI have chosen to implement a server to read the tweets csv and generate the gRPC \nstream which is then consumed by a client service which live-streams the tweets, \nprocesses some analytics and stores tweets/analytics-results in a redis service. \nTo complete the project I have developed a frontend service with python Flask \nwhich reads from redis and displays tweets (http://localhost:8080). The server is \nstreaming tweets with a 2 seconds delay between each tweet, so if you refresh the frontend \nyou can see new tweets coming in. This architecture allows for separation of concerns \nwhere each service is in charge of a limited number of specific tasks. This architecture \npromotes low coupling and high cohesion where the only dependency is client-depend-on-server \nfor the gRPC stream. \n\n![architecture](static/architecture.png)\n\n# Docker - Compose\nThis steps allow to create and run the containerized micro-services.\nFollowing commands must be executed within `srodi-gRPC/test/` directory.\n\n#### Build and run containers:\n```bash\ndocker-compose up\n```\n\n#### Clean up\nStop execution and remove containers:\n```bash\ndocker-compose stop\ndocker-compose rm\n```\nYou will also have to remove docker images by running `docker images` and `docker rmi [IMAGE_ID]`\n\n# Run locally (without Docker)\nThis refers to server and gRPC-stream client only\n\n#### Clone repo and prepare environment\nInitial steps\n```bash\ngit clone https://github.com/srodi/srodi-gRPC.git\ncd srodi-gRPC\npip install -r test/requirements.txt\n```\n\nView and edit `protos/test.proto` as required, `server.py` and `client.py` will have to match changes in `test.proto`\n\nGenerate interfaces (Only if you edited `protos/test.proto`) \nGenerate the gRPC client and server interfaces from .proto service definition\n```bash\ncd server\npython -m grpc_tools.protoc -I../protos --python_out=. --grpc_python_out=. ../protos/test.proto\ncd ../client\npython -m grpc_tools.protoc -I../protos --python_out=. --grpc_python_out=. ../protos/test.proto\n```\nStart Server\n```bash\npython -m server\n```\nStart Client\n```bash\npython -m client\n```\n\n# Docker\nBelow steps allow to containerize the microservices with Docker and run docker images locally.\n\nFollowing commands must be executed from project root directory `srodi-gRPC/`.\n\n#### Build images \nbuild server docker image\n```bash\ndocker build -f server.Dockerfile -t=srodi/stream-server .\n```\nbuild reddit-server docker image\n```bash\ndocker build -f server_reddit.Dockerfile -t=srodi/reddit-stream-server .\n```\nbuild client docker image\n```bash\ndocker build -f client.Dockerfile -t=srodi/stream-client .\n```\nbuild web-server docker image\n```bash\ndocker build -f web_server.Dockerfile -t=srodi/web-server .\n```\n#### Run Containers\nrun server container\n```bash\ndocker run -it --name server -p 9999:9999 srodi/stream-server\n```\nrun reddit-server container\n```bash\ndocker run -it --name reddit-server -p 9998:9998 srodi/reddit-stream-server\n```\nrun client container\n```bash\ndocker run -it --name client --network=\"host\" srodi/stream-client\n``` \nrun web-server container\n```bash\ndocker run -it --name web-server -p 8080:5000 srodi/web-server\n```\n#### Push image to registry\npush to `DockerHub` container registry\n```bash\ndocker push your_hub_username/image_name\n```\n#### Clean up\nremove containers\n```bash\ndocker stop server web-server \u0026\u0026 docker rm server client web-server\n```\nremove images\n```bash\ndocker rmi grcp-stream-server grcp-stream-client grcp-web-server\n```\n\n#### Redis\nRun the latest `bitnami/redis` image\n`docker run --name redis -e ALLOW_EMPTY_PASSWORD=yes bitnami/redis:latest`\nOpen redis-cli\n```bash\ndocker exec -it [CONTAINER_ID] redis-cli\n```\n\n# Kubernetes\nAll services can be deployed in a `Kubernets` cluster, testing has ben conducted on `microk8` instance.\n\n### multipass\ninstall multipass Ubuntu VMs orchestrator:\n```shell script\nsudo snap install multipass --beta --classic\n```\n### microk8s\ninstall a minimal, lightweight Kubernetes \n```shell script\nmultipass launch --name microk8s-vm --mem 3G\nmultipass exec microk8s-vm -- sudo snap install microk8s --classic --channel=1.17/stable\nmultipass exec microk8s-vm -- sudo iptables -P FORWARD ACCEPT\nmultipass list\n```\naccess the VM via multipass interface and execute the following\n```shell script\nsudo usermod -a -G microk8s $USER\nsu - $USER\nalias kubectl='microk8s.kubectl'\n# enable for our micro-services in-cluster communication\nmicrok8s.enable dns\n# add public key of host\necho \"\u003cssh-host-pub-key\u003e\" \u003e\u003e .ssh/authorized_keys\n```\nfrom host machine run the following to copy kubernetes directory to vm via ssh\n```shell script\nrsync -avz -e 'ssh' /path/to/local/kubernetes/dir user@remotehost:/path/to/remote\n# for example\nrsync -avz -e 'ssh' kubernetes ubuntu@192.168.64.2:/home/ubuntu/\n```\n#### Create services\ncreate deployments and services within `microk8`\n```bash\nbash kubernetes/scripts/create.sh\n```\n\n#### Delete services\ndelete deployments and services within `microk8`\n```bash\nbash kubernetes/scripts/delete.sh\n```\n\n#### Scale deployment\nscale client deployment to 3 replica sets\n```bash\nkubectl scale --replicas=3 deployment/tweet-stream-client-deploy\n```\n\nKubernetes demo\n\n![architecture](kubernetes/static/microk8-demo.png)\n\n#### Kubernetes dashboard\nThis will require additional resource allocation (i.e. it will not work with the microk8s just created above with --mem 3G)\n\nOn multipass VM run:\n```shell script\nmicrok8s.enable dashboard\n# get token for dashboard login\ntoken=$(microk8s.kubectl -n kube-system get secret | grep default-token | cut -d \" \" -f1)\nmicrok8s.kubectl -n kube-system describe secret $token\n# run dashboard\nkubectl proxy\n```\nOn host machine run:\n```shell script\n# get ip of running VMs\nmultipass list\n# port-forward to access dashbord from localhost\nssh -L 8001:localhost:8001 ubuntu@\u003cvm-ip\u003e\n```\nOpen dashboard at `http://localhost:8001/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/#/login` \nusing the `token` obtained in previous step.\n\nremove ssh forwarded ports\n```shell script\nps aux | grep ssh\nkill -9 \u003cid\u003e\n```\n\n## Grafana dashboard\nThis will require additional resource allocation (i.e. it will not work with the microk8s just created above with --mem 3G)\n\nenable prometheus on microk8s VM\n```shell script\nmicrok8s.enable dashboard ingress prometheus\n# get path for grafana\nkubectl cluster-info\n# run dashboard\nkubectl proxy\n```\nOn host machine run:\n```shell script\n# get ip of running VMs\nmultipass list\n# port-forward to access dashbord from localhost\nssh -L 8001:localhost:8001 ubuntu@\u003cvm-ip\u003e\n```\nOpen grafana dashboard at `http://localhost:8001\u003cpath-for-grafana\u003e`.\n\n## Serverless functions\n\nTwo functions have been developed to read and write to Redis, please see \n\nDeploy kubeless\n```shell script\nexport RELEASE=$(curl -s https://api.github.com/repos/kubeless/kubeless/releases/latest | grep tag_name | cut -d '\"' -f 4)\nkubectl create ns kubeless\nkubectl create -f https://github.com/kubeless/kubeless/releases/download/$RELEASE/kubeless-$RELEASE.yaml\n```\nInstall kubeless cli\n```shell script\nexport OS=$(uname -s| tr '[:upper:]' '[:lower:]')\ncurl -OL https://github.com/kubeless/kubeless/releases/download/$RELEASE/kubeless_$OS-amd64.zip \u0026\u0026 \\\n  unzip kubeless_$OS-amd64.zip \u0026\u0026 \\\n  sudo mv bundles/kubeless_$OS-amd64/kubeless /usr/local/bin/\n```\nDeploy functions\n```shell script\n# test function\nkubeless function deploy test --runtime python2.7 \\\n    --from-file test.py \\\n    --handler redis-test.get_from_redis --dependencies requirements.txt\n# redis-set function\nkubeless function deploy redis-set --runtime python2.7 \\\n    --from-file redis-write.py \\\n    --handler redis-write.redis_set --dependencies requirements.txt\n```\nExecute functions\n```shell script\n# test function test (no parameters)\nkubeless function call test\n# test function redis-set (with parameter)\nkubeless function call redis-set --data 'test-value1'\n```\nClean up\n```shell script\nkubectl delete -f https://github.com/kubeless/kubeless/releases/download/$RELEASE/kubeless-$RELEASE.yaml\nkubeless function delete test\nkubeless function delete redis-set\n```\n\n# Tests\n\nUnit tests have been developed to test `redis`, `tweet-server` and `reddit-server`. \nTest file is located in `client/grpc_test.py`. This test uses the same stubs as the client.\n\n# Useful commands\ndelete all evicted pods manually after an incident\n```shell script\nkubectl get pods --all-namespaces -o json | jq '.items[] | select(.status.reason!=null) | select(.status.reason | contains(\"Evicted\")) | \"microk8s.kubectl delete pods \\(.metadata.name) -n \\(.metadata.namespace)\"' | xargs -n 1 bash -c\n```\ngcloud context\n```shell script\ngcloud container clusters get-credentials cluster-1 --zone europe-west1-b --project pulumi-259310\n```\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsrodi%2Fdata-driven-microservices","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsrodi%2Fdata-driven-microservices","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsrodi%2Fdata-driven-microservices/lists"}