{"id":19250936,"url":"https://github.com/imsalmanmalik/video-to-audio-microservices-python-application-k8s-helm-aws-eks","last_synced_at":"2026-04-14T05:32:01.249Z","repository":{"id":207158371,"uuid":"718562679","full_name":"imsalmanmalik/Video-to-Audio-Microservices-Python-Application-K8s-Helm-AWS-EKS","owner":"imsalmanmalik","description":" Python-based microservice application on AWS Elastic Kubernetes Service (EKS).","archived":false,"fork":false,"pushed_at":"2023-11-25T11:28:18.000Z","size":13853,"stargazers_count":1,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-02-23T16:48:37.613Z","etag":null,"topics":["api","aws","cloudwatch","containerization","databases","docker","eks-cluster","helm-charts","jwt-authentication","kubernetes","logging","microservices-architecture","mongodb","observability","postgresql","python","rabbitmq","smtp-server"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/imsalmanmalik.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-11-14T10:44:42.000Z","updated_at":"2024-10-14T16:15:36.000Z","dependencies_parsed_at":"2023-11-25T12:36:20.551Z","dependency_job_id":null,"html_url":"https://github.com/imsalmanmalik/Video-to-Audio-Microservices-Python-Application-K8s-Helm-AWS-EKS","commit_stats":null,"previous_names":["imsalmanmalik/video-to-audio-microservices-python-application-k8s-helm-aws-eks"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/imsalmanmalik/Video-to-Audio-Microservices-Python-Application-K8s-Helm-AWS-EKS","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/imsalmanmalik%2FVideo-to-Audio-Microservices-Python-Application-K8s-Helm-AWS-EKS","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/imsalmanmalik%2FVideo-to-Audio-Microservices-Python-Application-K8s-Helm-AWS-EKS/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/imsalmanmalik%2FVideo-to-Audio-Microservices-Python-Application-K8s-Helm-AWS-EKS/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/imsalmanmalik%2FVideo-to-Audio-Microservices-Python-Application-K8s-Helm-AWS-EKS/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/imsalmanmalik","download_url":"https://codeload.github.com/imsalmanmalik/Video-to-Audio-Microservices-Python-Application-K8s-Helm-AWS-EKS/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/imsalmanmalik%2FVideo-to-Audio-Microservices-Python-Application-K8s-Helm-AWS-EKS/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31784251,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-14T02:24:21.117Z","status":"ssl_error","status_checked_at":"2026-04-14T02:24:20.627Z","response_time":153,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["api","aws","cloudwatch","containerization","databases","docker","eks-cluster","helm-charts","jwt-authentication","kubernetes","logging","microservices-architecture","mongodb","observability","postgresql","python","rabbitmq","smtp-server"],"created_at":"2024-11-09T18:19:20.755Z","updated_at":"2026-04-14T05:32:01.233Z","avatar_url":"https://github.com/imsalmanmalik.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"## Video to Audio Microservices Python Application onto AWS EKS \nConverting mp4 videos to mp3 in a microservices architecture.\n\n## Technology Stack\nThis project leverages a variety of tools, platforms, and skills. Below is a comprehensive list:\n\n1) **Programming Languages \u0026 Frameworks**:\n* Python: Primary programming language used for microservices.\n* Flask: Web framework for building microservices in Python.\n\n2) **Containerization \u0026 Orchestration**:\n* Docker: Used for containerizing the microservices.\n* Kubernetes: Orchestration tool for managing containerized applications.\n* Helm: Package manager for Kubernetes, used for managing Kubernetes applications.\n\n3) **Database \u0026 Messaging**:\n* PostgreSQL: Relational database used for data storage.\n* MongoDB: NoSQL database used for flexible data storage.\n* RabbitMQ: Messaging broker for handling communication between services.\n  \n4) **Cloud \u0026 Infrastructure**:\n* AWS (Amazon Web Services): Cloud platform for hosting the application.\n* Amazon EKS (Elastic Kubernetes Service): Managed Kubernetes service on AWS.\n* AWS CloudWatch: Monitoring and observability service used for logging and metrics.\n* AWS EC2: Node Groups and worker instances.\n* AWS IAM: For cluster and node roles with necessary policies attached. \n  \n5) **Security \u0026 Authentication**:\n* JWT (JSON Web Tokens): Used for secure authentication in the application.\n* Secrets Management: Kubernetes secrets used for managing sensitive data.\n\n## Architecture\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"./images/ProjectArchitecture.png\" width=\"600\" title=\"Architecture\" alt=\"Architecture\"\u003e\n  \u003c/p\u003e\n\n### Introduction\n\nThis document provides a step-by-step guide for deploying a Python-based microservice application on AWS Elastic Kubernetes Service (EKS). The application comprises four major microservices: `auth-server`, `converter-module`, `gateway server` and `notification-server`.\n\n### Prerequisites\n\nBefore you begin, ensure that the following prerequisites are met:\n\n1. **Create an AWS Account:** If you do not have an AWS account, create one by following the steps [here](https://docs.aws.amazon.com/streams/latest/dev/setting-up.html).\n\n2. **Install Helm:** Helm is a Kubernetes package manager. Install Helm by following the instructions provided [here](https://helm.sh/docs/intro/install/).\n\n3. **Python:** Ensure that Python is installed on your system. You can download it from the [official Python website](https://www.python.org/downloads/).\n\n4. **AWS CLI:** Install the AWS Command Line Interface (CLI) following the official [installation guide](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html).\n\n5. **Install kubectl:** Install the latest stable version of `kubectl` on your system. You can find installation instructions [here](https://kubernetes.io/docs/tasks/tools/).\n\n6. **Databases:** Set up PostgreSQL and MongoDB for your application.\n\n### Steps to follow\n\n#### Cluster Creation\n\n1. **Log in to AWS Console:**\n   - Access the AWS Management Console with your AWS account credentials.\n\n2. **Create eksCluster IAM Role**\n   - Follow the steps mentioned in [this](https://docs.aws.amazon.com/eks/latest/userguide/service_IAM_role.html) documentation using root user\n   - After creating it will look like this:\n\n \u003cp align=\"center\"\u003e\n  \u003cimg src=\"./images/ekscluster_role.png\" width=\"600\" title=\"ekscluster_role\" alt=\"ekscluster_role\"\u003e\n  \u003c/p\u003e\n\n   - Please attach `AmazonEKS_CNI_Policy` explicitly if it is not attached by default\n\n3. **Create Node Role - AmazonEKSNodeRole**\n   - Follow the steps mentioned in [this](https://docs.aws.amazon.com/eks/latest/userguide/create-node-role.html#create-worker-node-role) documentation using root user\n   - Please note that you do NOT need to configure any VPC CNI policy mentioned after step 5.e under Creating the Amazon EKS node IAM role\n   - Simply attach the following policies to your role once you have created `AmazonEKS_CNI_Policy` , `AmazonEBSCSIDriverPolicy` , `AmazonEC2ContainerRegistryReadOnly` , `CloudWatchAgentServerPolicy`(for observability \u0026 logging at cluster and application level)\n   - Your AmazonEKSNodeRole will look like this: \n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"./images/eksnode_role_policies.png\" width=\"600\" title=\"Node_IAM\" alt=\"Node_IAM\"\u003e\n  \u003c/p\u003e\n\n4. **Open EKS Dashboard:**\n   - Navigate to the Amazon EKS service from the AWS Console dashboard.\n\n5. **Create EKS Cluster:**\n   - Click \"Create cluster.\"\n   - Choose a name for your cluster.\n   - Configure networking settings (VPC, subnets).\n   - Choose the `eksCluster` IAM role that was created above\n   - Review and create the cluster.\n\n6. **Cluster Creation:**\n   - Wait for the cluster to provision, which may take several minutes.\n\n7. **Cluster Ready:**\n   - Once the cluster status shows as \"Active,\" you can now create node groups.\n\n#### Node Group Creation\n\n1. In the \"Compute\" section, click on \"Add node group.\"\n\n2. Choose the AMI (default), instance type (e.g., t3a.xlarge), and the number of nodes(1 node is sufficient for testing out this project).\n\n3. Click \"Create node group.\"\n\n#### Adding inbound rules in Security Group of Nodes\n\n**NOTE:** Ensure that all the necessary ports are open in the node security group.\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"./images/inbound_rules_sg.png\" width=\"600\" title=\"Inbound_rules_sg\" alt=\"Inbound_rules_sg\"\u003e\n  \u003c/p\u003e\n\n#### Enable EBS CSI and CloudWatch Observability Addon\n1. enable addon `ebs csi` this is for enabling pvc's once cluster is created.\n2. enable addon `Amazon CloudWatch Observability`(For installing the CloudWatch agent to enable Container Insights within the cluster).\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"./images/eks-add-ons.png\" width=\"600\" title=\"ebs_addon\" alt=\"ebs_addon\"\u003e\n  \u003c/p\u003e\n\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"./images/CloudWatch_log_groups.png\" width=\"600\" title=\"ebs_addon\" alt=\"ebs_addon\"\u003e\n  \u003c/p\u003e\n\n- This is how the CloudWatch log group would look like after the `Amazon CloudWatch Observability` add on is enabled. We also have other log groups related to the cluster infrastructure level logs that is explicitly enabled when creating the cluster on the EKS AWS console. \n\n#### Deploying your application on EKS Cluster\n\n1. Clone the code from this repository.\n\n2. Set the cluster context:\n   ```\n   aws eks update-kubeconfig --name \u003ccluster_name\u003e --region \u003caws_region\u003e\n   ```\n\n### Commands\n\nHere are some essential Kubernetes commands for managing your deployment:\n\n\n### MongoDB\n\nTo install MongoDB, set the database username and password in `values.yaml`, then navigate to the MongoDB Helm chart folder and run:\n\n```\ncd Helm_charts/MongoDB\nhelm install mongo .\n```\n\nConnect to the MongoDB instance using:\n\n```\nmongosh 'mongodb://\u003cusername\u003e:\u003cpwd\u003e@\u003cnodeip\u003e:30005/mp3s?authSource=admin'\n```\n**Note**: `nodeip` can be obtained by navigating to EKS \u003eClusters \u003eYour cluster \u003eCompute \u003eNode Groups \u003eNode \u003eInstanceId \u003ePublicIpAddress(copy)\n\n### PostgreSQL\n\nSet the database username and password in `values.yaml`. Install PostgreSQL from the PostgreSQL Helm chart folder and initialize it with the queries in `init.sql`.\n\n```\ncd ..\ncd Postgres\nhelm install postgres .\n```\n\nConnect to the Postgres database and copy all the queries from the \"init.sql\" file.\n```\npsql 'postgres://\u003cusername\u003e:\u003cpwd\u003e@\u003cnodeip\u003e:30003/authdb'\n```\n\n### RabbitMQ\n\nDeploy RabbitMQ by running:\n\n```\nhelm install rabbitmq .\n```\n\nEnsure you have created two queues in RabbitMQ named `mp3` and `video`. To create queues, visit `\u003cnodeIp\u003e:30004\u003e` and use default username `guest` and password `guest`\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"./images/rabbitmq.png\" width=\"600\" title=\"ebs_addon\" alt=\"ebs_addon\"\u003e\n  \u003c/p\u003e\n\n**NOTE:** Ensure that all the necessary ports are open in the node security group.\n\n### Building and Pushing Docker Images\nYou can create your own Docker images from the Dockerfiles present for each microservice and push them to Docker Hub, or you can use my existing images. Below are the microservices available and the commands to build and push their Docker images:\n\n```\n# Build the Docker image\ndocker build -t \u003cyour-docker-hub-username\u003e/\u003cmicroservice-name\u003e:latest ./src/\u003cmicroservice-name\u003e\n\n# Push the image to Docker Hub\ndocker push \u003cyour-docker-hub-username\u003e/\u003cmicroservice-name\u003e:latest\n```\n\n### Apply the manifest file for each microservice:\n\n- **Auth Service:**\n  ```\n  cd auth-service/manifest\n  kubectl apply -f .\n  ```\n\n- **Gateway Service:**\n  ```\n  cd gateway-service/manifest\n  kubectl apply -f .\n  ```\n\n- **Converter Service:**\n  ```\n  cd converter-service/manifest\n  kubectl apply -f .\n  ```\n\n- **Notification Service:**\n  ```\n  cd notification-service/manifest\n  kubectl apply -f .\n  ```\n\n### Application Validation\n\nAfter deploying the microservices, verify the status of all components by running:\n\n```\nkubectl get all\n```\n\n### Notification Configuration\n\nFor configuring email notifications and two-factor authentication (2FA), follow these steps:\n\n1. Go to your Gmail account and click on your profile.\n\n2. Click on \"Manage Your Google Account.\"\n\n3. Navigate to the \"Security\" tab on the left side panel.\n\n4. Enable \"2-Step Verification.\"\n\n5. Search for the application-specific passwords. You will find it in the settings.\n\n6. Click on \"Other\" and provide your name.\n\n7. Click on \"Generate\" and copy the generated password.\n\n8. Paste this generated password in `notification-service/manifest/secret.yaml` along with your email.\n\nRun the application through the following API calls by navigating inside the directory where the mp4 file is present:\n\n# API Definition\n\n- **Login Endpoint**\n  ```http request\n  POST http://nodeIP:30002/login\n  ```\n  \n  ```console\n  curl -X POST http://nodeIP:30002/login -u \u003cemail\u003e:\u003cpassword\u003e\n  ``` \n  Expected output: JWT Token\n  \nOnce the login API endpoint is called, a JWT token is produced. Users can decode this token using the online base64 decoder [here](https://www.base64decode.org/)\n\n\n- **Upload Endpoint**\n  ```http request\n  POST http://nodeIP:30002/upload\n  ```\n\n  ```console\n   curl -X POST -F 'file=@./video.mp4' -H 'Authorization: Bearer \u003cJWT Token\u003e' http://nodeIP:30002/upload\n  ``` \n  \n  Check if you received the File ID(fid) on your email and copy the string for the next step.\n  \n  *Note*: The video upload is going to take some time since it is a long 5 min mp4 video which is a video recording of one of my projects 'MLOps-end-to-end-project'. This is the [link](https://github.com/imsalmanmalik/MLOps-end-to-end-project) to the MLOps project if anyone is interested. Feel free to replace it with a shorter video. \n\n- **Download Endpoint**\n  ```http request\n  GET http://nodeIP:30002/download?fid=\u003cGenerated file identifier\u003e\n  ```\n  ```console\n   curl --output video.mp3 -X GET -H 'Authorization: Bearer \u003cJWT Token\u003e' \"http://nodeIP:30002/download?fid=\u003cGenerated fid\u003e\"\n  ``` \n**Note**: Replace with the `fid` recieved on your personal email along with the `JWT Token` recieved earlier after the Login Endpoint was called and the `nodeIP` of your EKS Node Group worker instance.\n\n## Destroying the Infrastructure\n\nTo clean up the infrastructure, follow these steps:\n\n1. **Delete the Node Group:** Delete the node group associated with your EKS cluster.\n\n2. **Delete the EKS Cluster:** Once the nodes are deleted, you can proceed to delete the EKS cluster itself.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fimsalmanmalik%2Fvideo-to-audio-microservices-python-application-k8s-helm-aws-eks","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fimsalmanmalik%2Fvideo-to-audio-microservices-python-application-k8s-helm-aws-eks","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fimsalmanmalik%2Fvideo-to-audio-microservices-python-application-k8s-helm-aws-eks/lists"}