{"id":20037230,"url":"https://github.com/sagarmaheshwary/microservices","last_synced_at":"2026-04-08T11:31:29.003Z","repository":{"id":234555656,"uuid":"784753526","full_name":"SagarMaheshwary/microservices","owner":"SagarMaheshwary","description":"Microservices-based video streaming platform built with gRPC, RabbitMQ, Kubernetes, Redis, PostgreSQL, AWS — with full observability using Grafana, Prometheus, Loki, and Jaeger.","archived":false,"fork":false,"pushed_at":"2025-11-21T15:59:26.000Z","size":1660,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-11-21T17:29:26.191Z","etag":null,"topics":["distributed-systems","docker","golang","grafana","grpc","kind-kubernetes","kubernetes","loki","microservices","nestjs","nodejs","postgresql","prometheus","rabbitmq","redis"],"latest_commit_sha":null,"homepage":"","language":"Makefile","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/SagarMaheshwary.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2024-04-10T13:42:59.000Z","updated_at":"2025-11-21T15:59:30.000Z","dependencies_parsed_at":"2024-04-19T20:28:48.919Z","dependency_job_id":"d076d857-51ae-4307-a5e6-05600902a0ea","html_url":"https://github.com/SagarMaheshwary/microservices","commit_stats":null,"previous_names":["sagarmaheshwary/microservices"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/SagarMaheshwary/microservices","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SagarMaheshwary%2Fmicroservices","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SagarMaheshwary%2Fmicroservices/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SagarMaheshwary%2Fmicroservices/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SagarMaheshwary%2Fmicroservices/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/SagarMaheshwary","download_url":"https://codeload.github.com/SagarMaheshwary/microservices/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SagarMaheshwary%2Fmicroservices/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31553999,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-08T10:21:54.569Z","status":"ssl_error","status_checked_at":"2026-04-08T10:21:38.171Z","response_time":54,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["distributed-systems","docker","golang","grafana","grpc","kind-kubernetes","kubernetes","loki","microservices","nestjs","nodejs","postgresql","prometheus","rabbitmq","redis"],"created_at":"2024-11-13T10:18:43.496Z","updated_at":"2026-04-08T11:31:28.979Z","avatar_url":"https://github.com/SagarMaheshwary.png","language":"Makefile","funding_links":[],"categories":[],"sub_categories":[],"readme":"# STREAMING PLATFORM - MICROSERVICES\n\nA Streaming platform based on Microservices architecture.\n\n### 📌 Note\n\nThis microservices project was originally created as a **learning and experimentation sandbox** for distributed systems, Kubernetes, and observability.\nAs a result, some parts of the codebase (e.g., structure, patterns, and tests) may not fully reflect production-level practices.\n\nHowever, it still serves as a **useful reference** for understanding:\n- Multi-service architecture\n- gRPC + REST communication\n- Docker Compose orchestration\n- Kubernetes deployment with KIND\n- Full observability stack\n  - **Prometheus** (metrics)\n  - **Loki** (logs)\n  - **Jaeger** (tracing)\n  - **Grafana Dashboards**\n\nThe observability setup is fully available in Docker Compose.\nYou can also run the services on Kubernetes (KIND) by applying the manifests provided.\n\n![Microservices Architecture](./assets/microservices-architecture.png)\n\n### Table of Contents\n\n- [Technologies](#technologies)\n- [Microservices Overview](#microservices-overview)\n- [Setup - Docker Compose](#setup---docker-compose)\n- [Setup - Kubernetes](#setup---kubernetes)\n- [APIs](#apis)\n\n### TECHNOLOGIES\n\n- **Languages:** Golang, NodeJS (NestJS)\n- **Databases:** PostgreSQL, Redis\n- **Message Broker:** RabbitMQ\n- **Cloud:** AWS S3, CloudFront\n- **Monitoring \u0026 Observability:** Grafana, Prometheus, Loki, Jaeger\n\n### **MICROSERVICES OVERVIEW**\n\n#### **API Gateway:**\n\n- Acts as the main entry point for external requests.\n- Uses REST API to communicate with clients.\n- Routes internal service-to-service communication via gRPC.\n\n#### **User Service:**\n\n- Manages user-related operations (e.g., profile management, settings).\n- Stores user data in PostgreSQL.\n- Uses gRPC for request/response communication.\n\n#### **Authentication Service:**\n\n- Handles user registration, login, and authentication.\n- Uses JWT for authentication.\n- Maintains a JWT token blacklist in Redis.\n- Communicates with the User Service for user validation.\n\n#### **Upload Service:**\n\n- Generates pre-signed S3 URLs that allow users to upload videos directly from the frontend, avoiding backend bandwidth usage.\n- Handles upload initialization by validating metadata and upload intent.\n- After a successful upload, it sends video metadata and processing instructions to the Encode Service via RabbitMQ.\n\n#### **Encode Service:**\n\n- Listens for RabbitMQ messages from the Upload Service.\n- Processes video files into MPEG-DASH chunks.\n- Generates DASH manifest files.\n- Uploads processed video chunks and manifest files to S3.\n- Sends metadata and video details to the Video Catalog Service via RabbitMQ.\n\n#### **Video Catalog Service:**\n\n- Receives metadata from the Encode Service after video processing.\n- Stores video metadata (e.g., resolutions, duration, S3 paths) in PostgreSQL.\n- Implements gRPC RPCs for:\n  - Listing available videos.\n  - Fetching details of a single video.\n  - Generating CloudFront CDN URLs for streaming.\n\n#### **Storage and Streaming**\n\n- **S3 + CloudFront:**\n  - Raw and processed video content is stored in an S3 bucket.\n  - CloudFront CDN is used to distribute video chunks efficiently.\n  - Signed URLs are generated for secure streaming access.\n\n#### Monitoring, Logging, and Tracing\n\n- **Metrics \u0026 Observability**\n\n  - Prometheus collects default and custom metrics from Microservices, PostgreSQL, Redis, and RabbitMQ.\n  - Metrics are visualized through Grafana dashboards, showing insights like request rates, error rates, latency, and system health.\n\n- **Logging**\n\n  - Loki collects structured logs from each Microservice.\n  - Logs are forwarded using Grafana Alloy, and visualized in Grafana for easier filtering, searching, and debugging.\n\n- **Tracing**\n  - OpenTelemetry is integrated into all services to instrument request flows.\n  - Jaeger provides a visual trace explorer to follow request chains across HTTP, gRPC, and RabbitMQ boundaries.\n  - Helps identify performance bottlenecks and root causes during failures or latency spikes.\n\n---\n\n### SETUP - DOCKER COMPOSE\n\nOnce you've cloned the repository, you can use **Docker** and **Docker Compose** to set up and run all the services locally. Additionally, a Makefile is included with commands to streamline the setup process.\n\nMake sure you have the latest version of Docker installed, as it includes docker compose. You can download it from [Docker's official website](https://docs.docker.com/engine/install/).\n\nIf you don't have **make** installed on your system, you can install it using:\n\n- **Ubuntu/Debian:** `sudo apt install make`\n- **MacOS (Homebrew):** `brew install make`\n- **Windows (via Chocolatey):** `choco install make`\n\nClone all microservices:\n\n```bash\nmake clone\n```\n\nCreate **.env** file from **.env.docker-example** in each service:\n\n```bash\nmake copy-env-files\n```\n\nUpdate the **env** variables for these services:\n\n- **microservices-encode-service:** Add your AWS S3 credentials\n- **microservices-upload-service:** Add your AWS S3 credentials\n- **microservices-video-catalog-service:** Add your AWS S3 credentials and Cloudfront URL\n\nDownload and build all required Docker images:\n\n```bash\nmake docker-compose-build\n```\n\nStart all Docker containers:\n\n```bash\nmake docker-compose-up\n```\n\nAfter all the containers are up and running, you need to run migrations and scripts to create the database tables for the required services. To do this, open a new terminal and execute the following command:\n\n```bash\nmake docker-db-migrate\n```\n\nAccess the microservices API Gateway at **http://localhost:4000**. Each service is mounted with a **volume**. This means any changes to the code are automatically synced to the container, eliminating the need to restart containers manually.\n\nTo run containers in the background:\n\n```bash\nmake docker-compose-up-detached\n```\n\nTo stop and remove the containers:\n\n```bash\nmake docker-compose-down\n```\n\nDelete the container images:\n\n```bash\nmake docker-delete-images\n```\n\n#### GRAFANA DASHBOARDS\n\nAccess Grafana: Open **[http://localhost:3000](http://localhost:3000)** in your browser.\nLogin Credentials:\n\n- **Default username:** `admin`\n- **Default password:** `admin` (you can set a new password on first login)\n\n#### Available Dashboards\n\n##### **Microservices Overview**\n\nDisplays high-level metrics from all microservices collected via Prometheus. Includes:\n\n- Service health status\n- Request count and rates\n- Latency (avg, p95)\n- Error rates (4xx, 5xx)\n\n![Grafana Dashboard Microservices](./assets/grafana-dashboard-microservices.png)\n\n##### **Microservices Logs**\n\nView real-time and historical logs from all microservices, aggregated using **Loki** (via Alloy agent).\n\n![Grafana Dashboard Microservices Logs](./assets/grafana-dashboard-microservices-logs.png)\n\n##### **PostgreSQL Overview**\n\nVisualizes internal PostgreSQL metrics (via `postgres_exporter`):\n\n- Query throughput and durations\n- Active connections and locks\n- Buffer cache, disk I/O performance\n\n![Grafana Dashboard PostgreSQL](./assets/grafana-dashboard-postgresql.png)\n\n##### **Redis Overview**\n\nTracks Redis instance metrics (via `redis_exporter`):\n\n- Memory usage\n- Cache hits/misses\n- Command rates\n- Keyspace statistics\n\n![Grafana Dashboard Redis](./assets/grafana-dashboard-redis.png)\n\n#### JAEGER TRACING\n\n**Access Jaeger UI**: Open **[http://localhost:16686](http://localhost:16686)**\n\n![Jaeger UI Search](./assets/jaeger-ui-search.png)\n![Jaeger UI Trace Detail](./assets/jaeger-ui-trace-detail.png)\n\n### SETUP - KUBERNETES\n\nOnce you've cloned the repository, you can use **Kind** and **Kubernetes** to set up and run all the services locally. A **Makefile** is included with commands to simplify the setup process, including cluster creation, image building, and service deployment.\n\nEnsure you have the following tools installed:\n\n- [Docker](https://docs.docker.com/engine/install)\n- [Kind](https://kind.sigs.k8s.io/docs/user/quick-start/#installation)\n- [kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl)\n- [kubectx](https://github.com/ahmetb/kubectx?tab=readme-ov-file#installation)\n- [cloud-provider-kind](https://github.com/kubernetes-sigs/cloud-provider-kind?tab=readme-ov-file#install)\n\nIf **make** is not installed on your system, install it using:\n\n- **Ubuntu/Debian:** `sudo apt install make`\n- **MacOS (Homebrew):** `brew install make`\n- **Windows (via Chocolatey):** `choco install make`\n\nClone all microservices:\n\n```bash\nmake clone\n```\n\nCreate the Kind Cluster:\n\n```bash\nmake kind-create-cluster\n```\n\nThis creates the Kubernetes cluster using Kind and sets up necessary components like namespaces, docker registry, and the metrics server. The metrics server is optional and can be used with **kubectl top** or [k9s](https://k9scli.io/topics/install/) to monitor resource usage of pods.\n\nBuild microservices images:\n\n```bash\nmake kind-build-images\n```\n\nPush all the built images to the local registry for use in Kind:\n\n```bash\nmake kind-push-images\n```\n\nDeploy NGINX Ingress Controller:\n\n```bash\nmake kind-deploy-nginx-ingress\n```\n\nThis configures ingress routing for external access. `cloud-provider-kind` functions as a load balancer for your local KIND cluster, similar to how cloud providers expose services. After obtaining the load balancer’s external IP, add it to your `/etc/hosts` file to access the API Gateway using the hostname defined in your Ingress (`microservices.local`).\n\nBefore deploying services, ensure that the **AWS**\\_\\* environment variables are encoded in Base64 and added to the **Secret.yaml** files of **encode-service**, **upload-service**, and **video-catalog-service**.\n\nDeploy Microservices and Datastores:\n\n```bash\nmake kind-deploy-services\n```\n\nThis applies Kubernetes manifests to deploy all microservices and required datastores. Once all pods are up and running, microservices can be accessed using the private IP obtained from the NGINX Ingress controller.\n\nTo delete the cluster and clean up resources:\n\n```bash\nmake kind-delete-cluster\n```\n\nThis removes the Kind cluster and stops the local Docker registry.\n\nTo remove only services:\n\n```bash\nmake kind-delete-services\n```\n\n### APIs\n\nCheckout the [**microservices-api-gateway**](https://github.com/SagarMaheshwary/microservices-api-gateway) repo for available apis listing, and Postman collection.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsagarmaheshwary%2Fmicroservices","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsagarmaheshwary%2Fmicroservices","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsagarmaheshwary%2Fmicroservices/lists"}