{"id":13906325,"url":"https://github.com/luqmansen/gosty","last_synced_at":"2025-07-19T05:12:52.985Z","repository":{"id":37676181,"uuid":"333483140","full_name":"luqmansen/gosty","owner":"luqmansen","description":"Scalable Transcoding Service on Kubernetes","archived":false,"fork":false,"pushed_at":"2022-12-30T21:45:05.000Z","size":20378,"stargazers_count":7,"open_issues_count":18,"forks_count":2,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-07-18T06:51:01.338Z","etag":null,"topics":["distributed-computing","golang","golang-microservice","k8s","kubernetes","linkerd","microk8s","microservices","minikube","transcode-video","transcoding"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/luqmansen.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-01-27T16:03:34.000Z","updated_at":"2025-01-22T14:41:22.000Z","dependencies_parsed_at":"2023-01-31T17:01:35.641Z","dependency_job_id":null,"html_url":"https://github.com/luqmansen/gosty","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/luqmansen/gosty","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/luqmansen%2Fgosty","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/luqmansen%2Fgosty/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/luqmansen%2Fgosty/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/luqmansen%2Fgosty/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/luqmansen","download_url":"https://codeload.github.com/luqmansen/gosty/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/luqmansen%2Fgosty/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":265892712,"owners_count":23845064,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["distributed-computing","golang","golang-microservice","k8s","kubernetes","linkerd","microk8s","microservices","minikube","transcode-video","transcoding"],"created_at":"2024-08-06T23:01:33.584Z","updated_at":"2025-07-19T05:12:52.962Z","avatar_url":"https://github.com/luqmansen.png","language":"Go","funding_links":[],"categories":["HarmonyOS"],"sub_categories":["Windows Manager"],"readme":"# Gosty\n\nScalable cloud transcoding service on Kubernetes\n\n## Table of Content\n - [Gosty](#gosty)\n   * [Table of Content](#table-of-content)\n   * [System Overview](#system-overview)\n   * [Development](#development)\n     + [Requirements](#requirements)\n     + [Using Docker compose](#using-docker-compose)\n     + [Using docker container](#using-docker-container)\n     + [Using Kind](#using-kind)\n   * [Deployment](#deployment)\n     + [Deploy on GKE](#deploy-on-gke)\n       - [Using Managed RabbitMQ and MongoDB](#using-managed-rabbitmq-and-mongodb)\n       - [Deploy RabbitMQ and MongoDB inside Cluster](#deploy-rabbitmq-and-mongodb-inside-cluster)\n     + [RabbitMQ](#rabbitmq)\n     + [MongoDB](#mongodb)\n     + [API Server, File Server, Worker](#api-server--file-server--worker)\n     + [Elasticsearch-Fluentd-Kibana](#elasticsearch-fluentd-kibana)\n     + [Linkerd](#linkerd)\n   * [Additional](#additional)\n     + [Local Image Registry](#local-image-registry)\n       - [Using minikube's local registry](#using-minikube-s-local-registry)\n       - [Using microk8s local registry](#using-microk8s-local-registry)\n     + [K3s local registry](#k3s-local-registry)\n       - [Using Docker compose](#using-docker-compose-1)\n       - [K8s manifest adjustment](#k8s-manifest-adjustment)\n     + [Spekt8](#spekt8)\n     + [Dashboard](#dashboard)\n     + [Chaos Mesh](#chaos-mesh)\n     + [Testing](#testing)\n     + [Resize gke cluster to 0 when not used](#resize-gke-cluster-to-0-when-not-used)\n   * [Issues](#issues)\n   * [What can be improved](#what-can-be-improved)\n   * [Todo](#todo)\n   * [Acknowledgements](#acknowledgements)\n \n \u003csmall\u003e\u003ci\u003e\u003ca href='http://ecotrust-canada.github.io/markdown-toc/'\u003eTable of contents generated with markdown-toc\u003c/a\u003e\u003c/i\u003e\u003c/small\u003e\n\n---\n\n## System Overview\n\n\u003cimg width=\"60%\" src=\"https://github.com/luqmansen/gosty/wiki/out/Diagram/sys-design-overview.png\" /\u003e\n\u003cbr\u003e\n\n---\n\n## Development\n### Requirements \n- go 16.7 \n- docker\n- docker-compose\n- npm\n- yarn\n- minikube\n\n### Using Docker compose\n\n1. `docker-compose up`\n2. Change `config.env` to use the docker compose env\n\n### Using docker container\n\nIf you want to run app on container while run the database and message broker on minikube, make sure to attach minikube\nnetwork to the container\n\n```\ndocker run -p 8000:8000 --network minikube -e GOSTY_FILESERVER_SERVICE_HOST=192.168.49.4 localhost:5000/gosty-apiserver\ndocker run -p 8001:8001 --network=minikube localhost:5000/gosty-fileserver\ndocker run --network minikube -e GOSTY_FILESERVER_SERVICE_HOST=192.168.49.4 localhost:5000/gosty-worker\n```\n\n### Using Kind\nInit the kind cluster (mongodb and rabbitmq will run on host machine using docker-compose)\n```\nbash ./hack/create-kind-local-registry.sh\n```\n\nDeploy using kustomize\n```\nkustomize build deployment/kustomize/environments/gke | kubectl apply -f -\n```\n\n### Using Minikube\n```\nkustomize build deployment/kustomize/environments/minikube | kubectl apply -f -\n```\n\n## Deployment\n\n### Deploy on GKE\n\nInit cluster script\n```bash\nbash ./hack/create-cluster.sh\n```\n\n#### Using Managed RabbitMQ and MongoDB\n\nSet your mongodb \u0026 rabbitmq secret on configmap (change to secret if you want, modify the manifest accordingly)\n\nThen apply linkerd \u0026 gosty component\n\n\n#### Deploy RabbitMQ and MongoDB inside Cluster\n\nRabbitMQ and MongoDB deployed using helm, make sure to install helm before\n\n```\ncurl https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 | bash\n```\n\nalso add bitnami repo\n\n```\nhelm repo add bitnami https://charts.bitnami.com/bitnami\n```\n\n### RabbitMQ\n\n```\nkubectl create -fs k8s/rabbitmq/service.yaml # create nodePort service, skip if you don't need\nhelm install rabbit bitnami/rabbitmq -fs k8s/rabbitmq/helm-values.yaml --create-namespace --namespace gosty\n```\n\n### MongoDB\n\n```\nkubectl create -fs deployment/k8s/mongodb/service.yaml # create nodePort service, skip if you don't need\nhelm install mongodb bitnami/mongodb -fs deployment/k8s/mongodb/helm-values.yaml --create-namespace --namespace gosty\n```\n\n### API Server, File Server, Worker\n\nApply the rest of k8s resource manifest\n\n```bash\nkustomize build deployment/kustomize/environments/gke | kubectl apply -f -\n```\n\n### Elasticsearch-Fluentd-Kibana\n\nThis resource will be deployed on `fluentd-monitoring` namespace. This stack currently will only be used for logs\nmonitoring\n\n```shell\nkubectl apply -fs ./deployment/k8s/fluentd\n```\n\n```shell\nkubectl -n fluentd-monitoring port-forward svc/kibana 5601\n```\n\n### Linkerd\n\nInstall linkerd cli\n\n```\ncurl -sL run.linkerd.io/install | sh                                                                                                                                                                       1 ↵\n```\n\ninstall linkerd component\n\n```\nkubectl apply -fs deployment/k8s/linkerd/manifest\n```\n\naccess linkerd dashboard\n\n```bash\nlinkerd viz dashboard\n```\n\nInjecting linkerd to rabbitmq, mongodb \u0026 ingress\n\n```bash\nkubectl get statefulset -n gosty rabbit-rabbitmq -o yaml  | linkerd inject - | kubectl apply -fs -\nkubectl get statefulset -n gosty mongodb -o yaml  | linkerd inject - | kubectl apply -fs -\nkubectl get statefulset -n gosty -o yaml mongodb-arbiter | linkerd inject - | kubectl apply -fs -   \nkubectl get deployment -n kube-system ingress-nginx-controller -o yaml | linkerd inject - | kubectl apply -fs -                                               1 ↵\n```\n\nPromQL for Add additional kubernetes avg cpu usage in grafana dashboard\n```\nsum (rate (container_cpu_usage_seconds_total{image!=\"\",kubernetes_io_hostname=~\"^$Node$\",namespace=\"gosty\"}[1m]))\n```\n\n## Additional\n\n### Local Image Registry\n\nRun local image registry on my machine for faster dev purposes\n\n#### Using minikube's local registry\n\nIn case you want to run it inside kubernetes cluster (minikube), enable minikube local registry\n\n```\nminikube addons enable registry\n```\n\nforward registry service to local port\n\n```\nkubectl port-forward --namespace kube-system svc/registry 5000:80\n```\n\npush the local image to minikube's local registry\n\n```\ndocker build -t localhost:5000/{image-name} -fs docker/Dockerfile-{image-name} .\ndocker push localhost:5000/{image-name}\n```\n\n#### Using microk8s local registry\n\n```shell\nmicrok8s.enable registry\n```\n\nYou can access it via microk8s's ip (NodePort service)\n\n### K3s local registry\n\ncreate this file on ``/etc/rancher/k3s/registries.yaml``\n\n```yaml\nmirrors:\n  registry.local:\n    endpoint:\n      - \"http://192.168.56.1:5000\" # your local container registry\n\n```\n\n#### Using Docker compose\n\n```shell\ndocker-compose -fs docker-compose-registry.yaml up -d registry\n```\n\nThe pod will be exposed on 0.0.0.0:5000\n\n#### K8s manifest adjustment\n\nDon't forget to change the k8s deployment image\n\n```yaml\nspec:\n  containers:\n    - name: { image-name }\n      image: localhost:\u003cregistry's cluster ip/container registry's ip\u003e/{image-name}:latest \n```\n\n### Spekt8\n\nI setup [spekt8](https://github.com/spekt8/spekt8) for cluster visualization\n\n```\nkubectl create -fs ./deployment/k8s/plugins/spekt8/fabric8-rbac.yaml \nkubectl apply -fs ./deployment/k8s/plugins/spekt8/spekt8-deployment.yaml \nkubectl port-forward -n gosty deployment/spekt8 3000:3000\n```\n\n### Dashboard\n\nAccessing k8s dashboard\n\n```shell\nkubectl -n kube-system port-forward svc/kubernetes-dashboard 8443:443       \n```\n\n### Chaos Mesh\n\nI add some testing scenario on [k8s/chaos](deployment/k8s/chaos) using chaos mesh. First install chaos mesh on the\ncluster\n\n```\ncurl -sSL https://mirrors.chaos-mesh.org/v1.1.2/install.sh | bash\n```\n\n### Testing\n\n***Run Pod for Testing Upload File from Inside Container***\n\n```shell\n# omit --rm to keep instance after exit\nkubectl run curl-test --image=marsblockchain/curl-git-jq-wget  -it -- sh             \n```\n\nDownload The File\n\n```bash\n# this is 20MB Test File\nwget --no-check-certificate -r 'https://docs.google.com/uc?export=download\u0026id=102o0T6XeB0znP-r0dkvKhDYTObniockI' -O sony.mp4\n# This is 200MB Test File (Blender's Foundation Big Bucks Bunny)\nwget --no-check-certificate 'https://docs.google.com/uc?export=download\u0026id=1mw1JHv739M46J6Jv5cXkVHb-_n7O0blK' -r -A 'uc*' -e robots=off -nd # will download 2 files\nmv uc?export\\=download\\\u0026confirm\\=uuRp\\\u0026id\\=1mw1JHv739M46J6Jv5cXkVHb-_n7O0blK bunny.mp4 #rename the files\n\n```\n\nPost data via curl\n```\ncurl http://gosty-apiserver.gosty.svc.cluster.local/api/video/upload -F file=@bunny.mp4 -v\ncurl http://34.149.27.149/api/video/upload -F file=@bunny.mp4 -v\ncurl http://gosty-apiserver.gosty.svc.cluster.local/api/video/upload -F file=@sony.mp4 -v\n```\n\nSubmit query to morph\n```\npython cli_submit.py -l bunny.mp4 -s 256x144 426x240 640x360 854x480 1280x720 1920x1080\npython cli_submit.py -l bunny.mp4 -s 854x480\n\n```\n\n***Execute Flaky Endpoint***\n\n```shell\n while true; do  sleep 60 \u0026\u0026 curl \"http://34.134.157.70/api/scheduler/progress/update\"; done # change the ip accordingly                     \n```\n\n***GCP Compute Metrics MQL for get average  load (for morph comparison)***\n- CPU\n```\nfetch gce_instance\n| metric 'compute.googleapis.com/instance/cpu/utilization'\n| group_by 25m , [value_utilization_aggregate: aggregate(value.utilization)/25]\n```\n- Memory\n```\nfetch gce_instance\n| metric 'agent.googleapis.com/memory/percent_used'\n| group_by 25m , [value_percent_used_mean: aggregate(value.percent_used)/25.15]\n```\n### Resize gke cluster to 0 when not used\nTo save some bill\n```\ngcloud container clusters resize ${CLUSTER_NAME} --zone=us-central1-a --num-nodes=0\n```\n\n## Issues\n\n**Minikube Ingress Change ip when Ingress controller restarted**\n\n```shell\n# change this according to your `kubectl get ingress`\nsed -i -e 's/192.168.59.2/'\"192.168.59.3\"'/g' /etc/hosts\n```\n\n**ImagePullBackOff on MicroK8s**\nWhen internet connection is bad, sometimes this is happened (especially for large image)\nSolution: set image pull timeout limmit for kubelet re-run the kubelet (somehow I can't edit ExecStart kubelet's\nservice, so need to manually run it)\n\n```shell\nsudo sudo systemctl stop snap.microk8s.daemon-kubelet.service \nsudo /snap/microk8s/2094/kubelet --kubeconfig=/var/snap/microk8s/2094/credentials/kubelet.config --cert-dir=/var/snap/microk8s/2094/certs --client-ca-file=/var/snap/microk8s/2094/certs/ca.crt --anonymous-auth=false --network-plugin=cni --root-dir=/var/snap/microk8s/common/var/lib/kubelet --fail-swap-on=false --cni-conf-dir=/var/snap/microk8s/2094/args/cni-network/ --cni-bin-dir=/var/snap/microk8s/2094/opt/cni/bin/ --feature-gates=DevicePlugins=true --eviction-hard=\"memory.available\u003c100Mi,nodefs.available\u003c1Gi,imagefs.available\u003c1Gi\" --container-runtime=remote --container-runtime-endpoint=/var/snap/microk8s/common/run/containerd.sock --containerd=/var/snap/microk8s/common/run/containerd.sock --node-labels=microk8s.io/cluster=true --authentication-token-webhook=true --cluster-domain=cluster.local --cluster-dns=10.152.183.10 --image-pull-progress-deadline=30s\n```\n\n**Random pods evicted**\n\nMicrok8s has issue with, even though when the resource is fine, just restart the deployment and delete old resource\n\n```\nexport NS=\u003cNAMESPACE\u003e \nkubectl -n $NS delete rs $(kubectl -n $NS get rs | awk '{if ($2 + $3 + $4 == 0) print $1}' | grep -v 'NAME')\n```\n\n**Pods stuck on terminating state**\n\nSometimes this is happened on k3s cluster after a while\n\n```bash\nexport NS=\u003cnamespace\u003e\nfor p in $(kubectl get pods -n $NS | grep Terminating | awk '{print $1}'); do kubectl delete pod -n $NS $p --grace-period=0 --force;done\n```\n\n**Hostpath provisioner only writable by root**\n\nIf you're running into this [issue](https://github.com/kubernetes/minikube/issues/1990), where the pod won't start\nbecause it can't write to pv, currently my workaround is change the directory modifier. For every node on your cluster,\nrun below command\n\n````\nsudo chmod -R 777 /tmp/hostpath-provisioner/gosty\n````\n\n**Viper need .env file**\n\nIf you notice, the config.env is still added to final docker images since viper\nhas [this issue](https://github.com/spf13/viper/issues/584). The env value later will be replaced by injected env values\nfrom K8s\n\n**Scaling statefulsets rabbitmq problem**\n\nRabbitMQ using `erlang cookie` as shared secret used for authentication between RabbitMQ nodes. It will be stored on the\nvolume. If `erlang cookie` is not defined on secret, it will generated randomly and in case of scaling via kubernetes\nAPI, each node will have different cookie, so they can't work together. Remove previous volume to make sure cookie is\nrenewed.\n\n**Debug K8s DNS**\n\nSome image has problem for dns resolving, example is this alpine 3.11 and 3.13 (used to use this) with\nthis [issue](https://github.com/gliderlabs/docker-alpine/issues/539)\nBelow command is to run one time pod to debug dns\n\n```shell\nkubectl run --restart=Never --rm -i --tty alpine --image=alpine:3.12 -- nslookup kube-dns.kube-system.svc.cluster.local\n```\n\n**Private registry on MikroK8S\nMicrok8s private registry communication need to be https, else it won't work, here is\nthe [reference](https://microk8s.io/docs/registry-private) for the setup\n\n\n## What can be improved\n\n- Use more proper permanent storage system (consider using object storage, eg: minio, gcs)\n- Currently, every worker will always download a copy of the file and process it on its local pod volume, then remove\n  the original file, then send the processed file to file server, this can use a lot of bandwidth. This can be improved\n  by using shared volume on the node, and check if other worker already download the file, then process it.\n  \n## Todo\n- Run experiment using static mode in CPU Manager K8S \n\n## Acknowledgements\n\n- Credit to [gibbok](https://github.com/gibbok) for [video player](https://github.com/gibbok/react-video-list) in web client,\n(Heavily modified for this project use case)   ","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fluqmansen%2Fgosty","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fluqmansen%2Fgosty","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fluqmansen%2Fgosty/lists"}