{"id":26276260,"url":"https://github.com/nasruddin/spring-boot-based-microservices","last_synced_at":"2025-10-03T13:26:47.856Z","repository":{"id":41894218,"uuid":"188917917","full_name":"Nasruddin/spring-boot-based-microservices","owner":"Nasruddin","description":"A modular, scalable Spring Boot microservices framework for managing courses and reviews. It features OAuth 2.0 authentication via Keycloak, API management with Spring Cloud Gateway, and observability using OTel, Grafana, Loki, Tempo, and Prometheus. MongoDB and PostgreSQL handle storage, with deployments via Docker Compose and Kubernetes.","archived":false,"fork":false,"pushed_at":"2025-03-12T05:18:21.000Z","size":10358,"stargazers_count":235,"open_issues_count":0,"forks_count":82,"subscribers_count":5,"default_branch":"master","last_synced_at":"2025-04-12T17:50:01.722Z","etag":null,"topics":["authentication","authorization","docker","docker-compose","jwt-auth","jwt-token","keycloak","kubernetes","loki","microservices","microservices-application","microservices-architecture","observability","openid-connect","prometheus","spring-boot","spring-cloud","spring-security","tempo"],"latest_commit_sha":null,"homepage":"http://javatab.wordpress.com","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Nasruddin.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2019-05-27T22:49:09.000Z","updated_at":"2025-04-10T20:16:12.000Z","dependencies_parsed_at":"2024-03-21T06:27:33.409Z","dependency_job_id":"87eb38c3-7406-4814-9705-0a8019e76920","html_url":"https://github.com/Nasruddin/spring-boot-based-microservices","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Nasruddin%2Fspring-boot-based-microservices","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Nasruddin%2Fspring-boot-based-microservices/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Nasruddin%2Fspring-boot-based-microservices/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Nasruddin%2Fspring-boot-based-microservices/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Nasruddin","download_url":"https://codeload.github.com/Nasruddin/spring-boot-based-microservices/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254592367,"owners_count":22097010,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["authentication","authorization","docker","docker-compose","jwt-auth","jwt-token","keycloak","kubernetes","loki","microservices","microservices-application","microservices-architecture","observability","openid-connect","prometheus","spring-boot","spring-cloud","spring-security","tempo"],"created_at":"2025-03-14T11:17:25.321Z","updated_at":"2025-10-03T13:26:47.760Z","avatar_url":"https://github.com/Nasruddin.png","language":"Java","readme":"# Table of Contents\n- [Background](#background)\n- [Architecture](#architecture)\n- [Prerequisites](#prerequisites)\n- [Run the application](#run-the-application)\n- [Check the endpoints](#check-the-endpoints)\n- [Keycloak](#keycloak)\n- [Observability](#observability)\n- [Guide to grafana](#guide-to-grafana)\n- [Verify the APIs](#verify-the-apis)\n- [Troubleshoot \u0026 Tips](#troubleshoot-and-tips)\n\n\n# Background\nA foundational framework for building **Spring Boot**-based microservices, designed for a **modular**, **scalable**, and **observable** system to manage **courses** and **reviews**. It incorporates **Spring Security** with **OAuth 2.0** via **Keycloak** for **authentication** and **Spring Cloud Gateway** as the **API gateway**. The architecture integrates a modern **observability stack**, including **OpenTelemetry (OTel)**, **Grafana**, **Loki**, **Tempo**, and **Prometheus**. **MongoDB** and **PostgreSQL** serve as **persistent storage** solutions. Deployment is supported through **Docker Compose** for **local environments** and **Kubernetes** for **scalable deployments**. The system utilizes **Spring Boot** and **Spring Cloud** to enable seamless **microservices communication**, **security**, and **observability**.\n\n# Container Diagram\n\n## Description\nThe **Course Management System** consists of multiple Spring Boot microservices, a gateway, databases, and external systems for authentication and observability, deployed via Docker Compose or Kubernetes (Tilt).\n\n## Containers\n| Container                | Technology              | Description                                      |\n|--------------------------|-------------------------|--------------------------------------------------|\n| **Gateway Service**      | Spring Cloud Gateway    | Routes requests to microservices, handles load balancing. |\n| **Course Service**       | Spring Boot             | Manages course data, interacts with Postgres.    |\n| **Review Service**       | Spring Boot             | Manages review data, interacts with MongoDB.     |\n| **Course Composite Service** | Spring Boot         | Aggregates data from Course and Review Services. |\n| **Postgres Database**    | PostgreSQL              | Stores course data for Course Service.           |\n| **MongoDB Database**     | MongoDB                 | Stores review data for Review Service.           |\n| **Keycloak**             | Keycloak                | External auth server for SSO and role-based access. |\n| **Grafana**              | Grafana                 | Visualization for metrics, logs, and traces.     |\n| **Loki**                 | Loki                    | Log aggregation system.                          |\n| **Tempo**                | Tempo                   | Distributed tracing system.                      |\n| **Fluent-bit**           | Fluent-bit              | Log forwarding agent.                            |\n| **Otel Collector**       | OpenTelemetry           | Collects and exports telemetry data.             |\n\n## Interactions\n- **Users/Admin** → **Gateway Service**: HTTP requests via browser/API client.\n- **Gateway Service** → **Keycloak**: Authenticates requests (OAuth2/JWT).\n- **Gateway Service** → **Course Service**: Routes course-related requests.\n- **Gateway Service** → **Review Service**: Routes review-related requests.\n- **Gateway Service** → **Course Composite Service**: Routes aggregate requests.\n- **Course Service** ↔ **Postgres Database**: CRUD operations for course data.\n- **Review Service** ↔ **MongoDB Database**: CRUD operations for review data.\n- **Course Composite Service** → **Course Service**: Fetches course data.\n- **Course Composite Service** → **Review Service**: Fetches review data.\n- **All Services** → **Fluent-bit**: Sends logs.\n- **All Services** → **Otel Collector**: Sends metrics and traces.\n- **Fluent-bit** → **Loki**: Forwards logs.\n- **Otel Collector** → **Tempo**: Sends traces.\n- **Otel Collector** → **Grafana**: Sends metrics.\n- **Grafana** ← **Loki**: Queries logs.\n- **Grafana** ← **Tempo**: Queries traces.\n\n# Architecture\n---\n### Level 1: System Context Diagram\n**Description**: High-level view of the system and its external actors.\n\n- **System**: Course Management System\n    - A microservices-based application for managing courses and reviews.\n- **Actors**:\n    - **User**: Browses courses and submits reviews (via browser/API).\n    - **Administrator**: Manages content and monitors system health.\n- **External Systems**:\n    - **Keycloak**: Authentication and authorization (OIDC/OAuth2).\n    - **Grafana Observability Stack**: Grafana, Loki, Tempo, Fluent-bit, and OpenTelemetry (Otel) for monitoring and logging.\n- **Interactions**:\n    - User → System: HTTP requests via Gateway.\n    - System → Keycloak: Authenticates users.\n    - System → Grafana Stack: Sends observability data.\n![System context](notes/images/context-level1.png)\n---\n### Level 2: Container Diagram\n**Description**: Breaks the system into deployable units.\n\n- **Containers**:\n    1. **API Gateway (Spring Cloud Gateway)**\n        - Tech: Spring Boot + Spring Cloud Gateway\n        - Role: Routes requests, enforces security.\n        - Interactions: User → Gateway → CourseComposite.\n    2. **CourseComposite (Aggregate Microservice)**\n        - Tech: Spring Boot\n        - Role: Aggregates Course and Review data.\n        - Interactions: Gateway → CourseComposite → Course/Review.\n    3. **Course (Core Microservice)**\n        - Tech: Spring Boot\n        - Role: Manages course data.\n        - Database: PostgreSQL\n        - Interactions: CourseComposite → Course → PostgreSQL.\n    4. **Review (Core Microservice)**\n        - Tech: Spring Boot\n        - Role: Manages review data.\n        - Database: MongoDB\n        - Interactions: CourseComposite → Review → MongoDB.\n    5. **Keycloak**\n        - Role: External auth provider.\n        - Interactions: Gateway → Keycloak.\n    6. **Observability Stack**:\n        - **Fluent-bit**: Log collection.\n        - **OpenTelemetry (OTel)**: Trace.\n        - **Loki**: Log storage.\n        - **Tempo**: Trace storage.\n        - **Grafana**: Visualization.\n        - Interactions: Microservices → Fluent-bit/OTel → Loki/Tempo → Grafana.\n![Container](notes/images/container-level2.png)\n---\n### Level 3: Component Diagram\n**Description**: Key components within containers.\n\n- **API Gateway**:\n    - Routing Component (Spring Cloud Gateway).\n    - Security Component (Keycloak integration).\n    - Observability Agent (OTel + Fluent-bit).\n- **CourseComposite**:\n    - Course Client (REST).\n    - Review Client (REST).\n    - Aggregation Logic.\n    - Observability Agent.\n- **Course**:\n    - Course Controller (REST).\n    - Course Service (Logic).\n    - Course Repository (JPA/PostgreSQL).\n    - Observability Agent.\n- **Review**:\n    - Review Controller (REST).\n    - Review Service (Logic).\n    - Review Repository (MongoDB).\n    - Observability Agent.\n\n![Component](notes/images/component-level3.png)\n---\n### Level 4: Deployment Notes\n**Description**: Two deployment setups.\n\n1. **Docker Compose Setup**:\n    - Containers: Gateway, CourseComposite, Course + PostgreSQL, Review + MongoDB, Keycloak, Observability Stack.\n    - Networking: Single Docker network.\n    - Config: `docker-compose.yml`.\n\n2. **Kubernetes Setup with Tilt**:\n    - Pods: Gateway, CourseComposite, Course + PostgreSQL, Review + MongoDB, Keycloak, Observability Pods.\n    - Resources: Ingress, Services, ConfigMaps/Secrets, PVCs.\n    - Tilt: Automates dev with live updates.\n    - Observability: Fluent-bit DaemonSet\n![Deployment](notes/images/deployment-level4.png)\n\n### Additional Notes\n- **Tech**: Spring Boot, Spring Cloud, JPA, MongoDB driver, OTel, Fluent-bit.\n- **Interactions**: REST/HTTP, JDBC, MongoDB protocol.\n- **Scalability**: Kubernetes supports replicas; Docker Compose for local dev.\n\n\n# Prerequisites\n\n- **Java 17+** (Recommended for Spring Boot 3.x)\n- **Maven 3.8+**\n- **Docker \u0026 Docker Compose** (for local containerized deployment)\n- **Minikube \u0026 Tilt** (for Kubernetes-based deployment)\n- **Httpie / cURL** (for API testing)\n- **Postman / Bruno**\n\n# Run the application\n\nYou can start the application using either **Docker (Docker Compose)** or **Kubernetes (Tilt)**.\n\nEnsure that your **Docker Engine** is running before proceeding.\n\n### Running with Docker or Individual Services on Host OS\n\n1. Start Persistent Services:\n```shell\n  cd docker \u0026\u0026 docker compose -f docker-compose-infra.yml up --build\n```\nThis command initializes **PostgreSQL** and **MongoDB**.\n\n2. Launch the Observability Stack:\n ```shell\n    cd docker \u0026\u0026 docker compose -f docker-compose-observability.yml up --build\n```\nThis set up **Grafana**, **Tempo**, **Loki**, **Fluent-bit**, **Prometheus**\n3. Start the Microservices\n```shell\n    sh run.sh docker\n```\nor \n```shell\n    sh run.sh\n```\nThis will start all microservices on their respective ports\n\n### Running with Tilt in Kubernetes env `(Minikube)`\n1. Let's Start minikube\n```shell\n    minikube start \\\n                --profile=microservice-deployment \\\n                --memory=4g \\\n                --cpus=4 \\\n                --disk-size=30g \\\n                --kubernetes-version=v1.27 \\\n                --driver=docker\n```\n```html\n😄  [microservice-deployment] minikube v1.35.0 on Darwin 15.3.1\n    ▪ MINIKUBE_ACTIVE_DOCKERD=microservice-deployment\n👉  Using Kubernetes 1.27.16 since patch version was unspecified\n✨  Using the docker driver based on user configuration\n📌  Using Docker Desktop driver with root privileges\n👍  Starting \"microservice-deployment\" primary control-plane node in \"microservice-deployment\" cluster\n🚜  Pulling base image v0.0.46 ...\n🔥  Creating docker container (CPUs=4, Memory=4096MB) ...\n🐳  Preparing Kubernetes v1.27.16 on Docker 27.4.1 ...\n    ▪ Generating certificates and keys ...\n    ▪ Booting up control plane ...\n    ▪ Configuring RBAC rules ...\n🔗  Configuring bridge CNI (Container Networking Interface) ...\n🔎  Verifying Kubernetes components...\n    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5\n🌟  Enabled addons: storage-provisioner, default-storageclass\n🏄  Done! kubectl is now configured to use \"microservice-deployment\" cluster and \"default\" namespace by default\n```\n2. Enable Ingress addon\n```shell\n    minikube addons enable ingress --profile microservice-deployment\n```\n```markdown\n💡  ingress is an addon maintained by Kubernetes. For any concerns contact minikube on GitHub.\n    You can view the list of minikube maintainers at: https://github.com/kubernetes/minikube/blob/master/OWNERS\n💡  After the addon is enabled, please run \"minikube tunnel\" and your ingress resources would be available at \"127.0.0.1\"\n    ▪ Using image registry.k8s.io/ingress-nginx/controller:v1.11.3\n    ▪ Using image registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.4.4\n    ▪ Using image registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.4.4\n🔎  Verifying ingress addon...\n🌟  The 'ingress' addon is enabled\n```\n3. Run this to make sure minikube can read images from local registry\n```shell\n    eval $(minikube -p microservice-deployment docker-env)\n```\n4. Let's build the images\n```shell\n    sh build-images.sh      \n```\n```html\nBuilding Docker images for Kubernetes using Minikube...\nBuilding course-composite-service...\n[+] Building 31.5s (14/14) FINISHED                                                                                                                                                                            docker:default\n =\u003e [internal] load build definition from Dockerfile                                                                                                                                                                     0.0s\n =\u003e =\u003e transferring dockerfile: 871B                                                                                                                                                                                     0.0s\n.\n.\n.\n.\ngateway-service built successfully!\nAll images built successfully!\n```\n5. List the images \n```shell\n    docker images\n```\nor\n```shell\n    docker image ls\n```\n\n```html\nREPOSITORY                                           TAG        IMAGE ID       CREATED         SIZE\ngateway-service                                      latest     2e07a4894e3d   3 minutes ago   339MB\nreview-service                                       latest     857bf2e92e8d   3 minutes ago   336MB\ncourse-service                                       latest     ad9a018bd4a3   3 minutes ago   359MB\ncourse-composite-service                             latest     5e1868bd77bc   3 minutes ago   328MB\nregistry.k8s.io/ingress-nginx/controller             \u003cnone\u003e     ee44bc236803   5 months ago    293MB \nregistry.k8s.io/ingress-nginx/kube-webhook-certgen   \u003cnone\u003e     a62eeff05ba5   5 months ago    63.8MB \nregistry.k8s.io/kube-apiserver                       v1.27.16   1113933272f1   7 months ago    123MB\nregistry.k8s.io/kube-controller-manager              v1.27.16   2db343b95a4c   7 months ago    115MB\nregistry.k8s.io/kube-scheduler                       v1.27.16   91ad8454afdd   7 months ago    57.7MB\nregistry.k8s.io/kube-proxy                           v1.27.16   ea1f910af975   7 months ago    79.9MB\nregistry.k8s.io/etcd                                 3.5.12-0   3861cfcd7c04   13 months ago   149MB\nregistry.k8s.io/coredns/coredns                      v1.10.1    ead0a4a53df8   2 years ago     53.6MB\nregistry.k8s.io/pause                                3.9        e6f181688397   2 years ago     744kB\ngcr.io/k8s-minikube/storage-provisioner              v5         6e38f40d628d   3 years ago     31.5MB\n```\n\nYou can also get Table view\n```shell\n    minikube image ls --format table --profile microservice-deployment\n```\n```html\n|----------------------------------------------------|----------|---------------|--------|\n|                       Image                        |   Tag    |   Image ID    |  Size  |\n|----------------------------------------------------|----------|---------------|--------|\n| docker.io/library/review-service                   | latest   | 857bf2e92e8d6 | 336MB  |\n| docker.io/library/course-composite-service         | latest   | 5e1868bd77bc2 | 328MB  |\n| registry.k8s.io/kube-controller-manager            | v1.27.16 | 2db343b95a4c2 | 115MB  |\n| registry.k8s.io/kube-scheduler                     | v1.27.16 | 91ad8454afddc | 57.7MB |\n| docker.io/library/gateway-service                  | latest   | 2e07a4894e3d5 | 339MB  |\n| docker.io/library/course-service                   | latest   | ad9a018bd4a3c | 359MB  |\n| registry.k8s.io/ingress-nginx/kube-webhook-certgen | \u003cnone\u003e   | a62eeff05ba51 | 63.8MB | \n| registry.k8s.io/kube-apiserver                     | v1.27.16 | 1113933272f1e | 123MB  | \n| registry.k8s.io/kube-proxy                         | v1.27.16 | ea1f910af975c | 79.9MB |\n| gcr.io/k8s-minikube/storage-provisioner            | v5       | 6e38f40d628db | 31.5MB |\n| registry.k8s.io/coredns/coredns                    | v1.10.1  | ead0a4a53df89 | 53.6MB |\n| registry.k8s.io/pause                              | 3.9      | e6f1816883972 | 744kB  |\n| registry.k8s.io/ingress-nginx/controller           | \u003cnone\u003e   | ee44bc2368033 | 293MB  |\n| registry.k8s.io/etcd                               | 3.5.12-0 | 3861cfcd7c04c | 149MB  |\n|----------------------------------------------------|----------|---------------|--------|\n```\n6. Let's start the Microservices\n```shell\n    Tilt up\n```\n```html\nTilt started on http://localhost:10350/\nv0.33.22, built 2025-01-03\n\n(space) to open the browser\n(s) to stream logs (--stream=true)\n(t) to open legacy terminal mode (--legacy=true)\n(ctrl-c) to exit\n```\n```shell\n    tilt get uiresources\n```\n```html\nNAME                       CREATED AT\ncourse-composite-service   2025-03-11T10:37:08Z\nprometheus                 2025-03-11T10:37:08Z\nfluent-bit                 2025-03-11T10:37:08Z\nuncategorized              2025-03-11T10:37:08Z\nreview-service             2025-03-11T10:37:08Z\ngateway-service            2025-03-11T10:37:08Z\nkeycloak                   2025-03-11T10:37:08Z\nreview-mongodb             2025-03-11T10:37:08Z\ntempo                      2025-03-11T10:37:08Z\nloki                       2025-03-11T10:37:08Z\ncourse-service             2025-03-11T10:37:08Z\ngrafana                    2025-03-11T10:37:08Z\ncourse-postgres            2025-03-11T10:37:08Z\n(Tiltfile)                 2025-03-11T10:37:08Z\n```\n```shell\n    kubectl get pods,svc,ingress\n```\n```markdown\nNAME                                            READY   STATUS              RESTARTS   AGE\npod/course-composite-service-7b57d566c4-6tz89   1/1     Running             0          85s\npod/course-postgres-5d748fd4cc-v72rq            0/1     ContainerCreating   0          2m43s\npod/course-service-76b45b858b-z6dwx             1/1     Running             0          83s\npod/fluent-bit-7f54dc97f-4zfxl                  0/1     ContainerCreating   0          2m43s\npod/gateway-service-588c7cff7b-f4gfl            1/1     Running             0          14s\npod/grafana-7d8cd9f95d-6qrcd                    0/1     ContainerCreating   0          2m43s\npod/keycloak-dd58c7c99-9gb6j                    0/1     ContainerCreating   0          2m43s\npod/loki-0                                      0/1     ContainerCreating   0          2m43s\npod/prometheus-74c59fb86c-792vs                 0/1     ContainerCreating   0          2m43s\npod/review-mongodb-695dc8799-4lpfm              0/1     ContainerCreating   0          2m43s\npod/review-service-9fc59c466-nkg9z              1/1     Running             0          84s\npod/tempo-54cb98d67f-kqzhv                      0/1     ContainerCreating   0          2m43s\n\nNAME                               TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE\nservice/course-composite-service   ClusterIP   10.105.126.22    \u003cnone\u003e        80/TCP                       86s\nservice/course-postgres            ClusterIP   10.97.180.213    \u003cnone\u003e        5432/TCP                     2m44s\nservice/course-service             ClusterIP   10.107.49.7      \u003cnone\u003e        80/TCP                       83s\nservice/fluent-bit                 ClusterIP   10.102.247.251   \u003cnone\u003e        24224/TCP                    2m43s\nservice/gateway-service            ClusterIP   10.96.187.185    \u003cnone\u003e        80/TCP                       14s\nservice/grafana                    ClusterIP   10.96.161.239    \u003cnone\u003e        3000/TCP                     2m43s\nservice/keycloak                   ClusterIP   10.109.159.232   \u003cnone\u003e        8080/TCP                     2m44s\nservice/kubernetes                 ClusterIP   10.96.0.1        \u003cnone\u003e        443/TCP                      4m36s\nservice/loki                       ClusterIP   10.100.62.50     \u003cnone\u003e        3100/TCP                     2m43s\nservice/prometheus                 ClusterIP   10.99.78.150     \u003cnone\u003e        9090/TCP                     2m43s\nservice/review-mongodb             ClusterIP   10.101.104.79    \u003cnone\u003e        27017/TCP                    2m44s\nservice/review-service             ClusterIP   10.101.51.125    \u003cnone\u003e        80/TCP                       84s\nservice/tempo                      ClusterIP   10.106.248.234   \u003cnone\u003e        4317/TCP,4318/TCP,3200/TCP   2m43s\n\nNAME                                           CLASS   HOSTS              ADDRESS        PORTS   AGE\ningress.networking.k8s.io/gateway-ingress      nginx   *                  192.168.49.2   80      2m44s\ningress.networking.k8s.io/grafana-ingress      nginx   grafana.local      192.168.49.2   80      2m44s\ningress.networking.k8s.io/keycloak-ingress     nginx   keycloak.local     192.168.49.2   80      2m44s\ningress.networking.k8s.io/prometheus-ingress   nginx   prometheus.local   192.168.49.2   80      2m44s\n```\n![Tilt web page](notes/images/tilt.png)\n---\n# Check the endpoints\n\u003e [!NOTE]\n\u003e On macOS and Windows, the Minikube ingress add-on doesn't support using the cluster's IP when running on Docker, so minikube tunnel --profile polar is required to expose the cluster locally via 127.0.0.1, similar to kubectl port-forward but for the entire cluster.\n\u003e```shell\n\u003e  minikube tunnel --profile microservice-deployment\n\u003e```\n\u003e Add below in your /etc/hosts\n\u003e ```shell\n\u003e   vi /etc/hosts\n\u003e```\n\u003e \n\u003e ```\n\u003e    127.0.0.1       grafana.local\n\u003e \n\u003e    127.0.0.1       keycloak.local\n\u003e \n\u003e    127.0.0.1       prometheus.local\n\u003e ```\n \n\n| **Components**      | **Docker**                             | **Kubernetes on Mac**                | Note                                           |\n|---------------------|----------------------------------------|--------------------------------------|------------------------------------------------|\n| **Gateway**         | http://localhost:9000                  | http://127.0.0.1:80                  | /course-aggregate/{courseid}/with-details      |\n| **CourseComposite** | http://localhost:8080/course-aggregate | http://127.0.0.1:80/course-aggregate |                                                |\n| **Course**          | http://localhost:8080/courses          | http://127.0.0.1:80/courses          |                                                |\n| **Review**          | http://localhost:8080/reviews          | http://127.0.0.1:80/reviews          |                                                |\n| **Grafana**         | http://localhost:3000                  | http://grafana.local                 | Add `127.0.0.1 grafana.local` in /etc/hosts    |\n| **Loki**            | http://loki:3100                       | http://loki:3100                     |                                                |\n| **Tempo**           | http://tempo:3200                      | http://tempo:3200                    |                                                |\n| **Fluent-bit**      | http://fluent-bit:24224                | http://fluent-bit:24224              | http on 4318 and grpc on 4317                  |\n| **Prometheus**      | http://localhost:9090                  | http://prometheus.local              | Add `127.0.0.1 prometheus.local` in /etc/hosts |\n| **Keycloak**        | http://localhost:8081                  | http://keycloak.local                | Add `127.0.0.1 keycloak.local` in /etc/hosts   |\n\n\u003e [!TIP]\n\u003e On Linux, Minikube runs as a native process directly on the host machine, rather than inside a virtual machine or a Docker container. This allows it to acquire a real, routable IP address that can be accessed from the host system without extra configuration.\n\u003e\n\u003e ```\n\u003e   $ minikube ip --profile microservice-deployment\n\u003e   192.154.19.8 \n\u003e ```\n\u003e \n\u003e Now, all the above tabular endpoints available at http://192.154.19.8/**\n\nAlso, please use **OpenAPI specs**, **bruno** or **postman** for API details. I will add Swagger/SpringDoc as when I get time!!\n\n# Keycloak\n### Head to Keycloak using above table ```http://localhost:8081/admin``` or ```http://keycloak.local/admin```\nCreate realm using `course-management-realm-realm.json` provided in the repo\n- Login to Keycloak → Go to http://localhost:8080/admin admin/admin\n- Create Realm → Click \"Create Realm\" in the top-left dropdown\n- Import Realm → Click \"Import\", select couse-management-realm-realm.json, and click \"Create\"\n- Verify → Check Clients, Users, Roles, and Mappers in the new realm\n\n![Keycloak web page](notes/images/keycloak.png)\n\n### Keycloak Users in `course-management-realm`\n\nWe have **3** users configured in **Keycloak** with the following credentials and roles:\n\n| Username    | Password  | Assigned Roles                                      |\n|------------|----------|------------------------------------------------------|\n| `nasruddin`  | `password` | `admin`, `guest`, `course-read`, `course-write`, `review-read`, `review-write` |\n| `courseuser` | `password` | `course-read`, `course-write`                     |\n| `reviewuser` | `password` | `review-read`, `review-write`                     |\n\n\u003e [!Notes]\n\u003e- **`nasruddin`** has **full access** to courses and reviews, along with admin privileges.\n\u003e- **`courseuser`** can only read and write courses.\n\u003e- **`reviewuser`** can only read and write reviews.\n\n\n\u003e [!TIP] Call the access token API to obtain a token, then visit jwt.io to inspect it.\n\u003e \n\u003e ```shell\n\u003e  curl -X POST http://localhost:8081/realms/course-management-realm/protocol/openid-connect/token  -d \"grant_type=password\"  -d \"client_id=course-app\"  -d \"client_secret=v1sCIPjANbvyJ87RsTkYeI9xHonDqZh7\"  -d \"username=nasruddin\"  -d \"password=password\" -d \"scope=openid roles\" | jq\n\u003e```\n\u003e```markdown\n\u003e{\n\u003e\"access_token\": \"eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICI0T1lYQVlZUXF6T2ZibE1wRjF0dmNvLW1UY2dEODVjai1Qak4xVnhuUExzIn0.eyJleHAiOjE3NDE3NzkxODYsImlhdCI6MTc0MTc0MzE4NiwianRpIjoiMTY0NGJlOTQtOTk3Ny00OTA4LWFiNWEtMzkwMzVmYmYzZGY0IiwiaXNzIjoiaHR0cDovL2xvY2FsaG9zdDo4MDgxL3JlYWxtcy9jb3Vyc2UtbWFuYWdlbWVudC1yZWFsbSIsImF1ZCI6InJldmlldy1hcHAiLCJzdWIiOiIzNzY2NmQ1Ni1jYTJkLTRkNDQtOTRmMS1kNDk4ZTdhZmVhOTUiLCJ0eXAiOiJCZWFyZXIiLCJhenAiOiJjb3Vyc2UtYXBwIiwic2lkIjoiYzk3NWQ2ZTgtODliOC00ZTE1LTg2N2YtMzdmNTZmYWQ3ZGNlIiwiYWNyIjoiMSIsImFsbG93ZWQtb3JpZ2lucyI6WyIvKiJdLCJyZWFsbV9hY2Nlc3MiOnsicm9sZXMiOlsiR1VFU1QiLCJBRE1JTiJdfSwicmVzb3VyY2VfYWNjZXNzIjp7InJldmlldy1hcHAiOnsicm9sZXMiOlsiUkVWSUVXLVJFQUQiLCJSRVZJRVctV1JJVEUiXX0sImNvdXJzZS1hcHAiOnsicm9sZXMiOlsiQ09VUlNFLVdSSVRFIiwiQ09VUlNFLVJFQUQiXX19LCJzY29wZSI6Im9wZW5pZCBlbWFpbCBwcm9maWxlIiwiZW1haWxfdmVyaWZpZWQiOmZhbHNlLCJuYW1lIjoiTmFzcnVkZGluIE5hc3J1ZGRpbiIsInByZWZlcnJlZF91c2VybmFtZSI6Im5hc3J1ZGRpbiIsImdpdmVuX25hbWUiOiJOYXNydWRkaW4iLCJmYW1pbHlfbmFtZSI6Ik5hc3J1ZGRpbiIsImVtYWlsIjoibmFzcnVkZGluQGdtYWlsLmNvbSJ9.E9k1reSQHmsidXaVYizlYs4ULMBjHE2hPzyGwDIQtM_ZszgqQzZH6CE5Y9fNdmN-ky4RLfvx3_C5DRQoDAn_PWLvlEFCsAVfWGopETHcd-NVm6rovbjuoBGrGPwZC9T49pzDEXTyudLEONnbHzXybhW88sTHbgj-8huS1tJFhjQ0rtROQHG5tK382z-bRxpo-Akzx9OP3W9YELf8V9TwW2sJ781WwOFeDP_k7EI8VbWdXJhEwKQJbE31roVk9PddK8_VUX4krCBMEUV6zzor9E3r7_OUTQ8-wnHrZrXBU8Sl2yO7SwUNb8l7zAzGf72vYS9cvF7ygIGA4_7EXIbIqw\",\n\u003e\"expires_in\": 36000,\n\u003e\"refresh_expires_in\": 1799,\n\u003e\"refresh_token\": \"eyJhbGciOiJIUzUxMiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICJhZjM3ZGFiMi1iMTQwLTRlY2EtYTNjNi05YmFkM2JkNzY0ODcifQ.eyJleHAiOjE3NDE3NDQ5ODYsImlhdCI6MTc0MTc0MzE4NiwianRpIjoiMTU0OTRkNDMtMDJlYS00ZGZjLWJiYTQtMDFhNWJkZmRkZjcyIiwiaXNzIjoiaHR0cDovL2xvY2FsaG9zdDo4MDgxL3JlYWxtcy9jb3Vyc2UtbWFuYWdlbWVudC1yZWFsbSIsImF1ZCI6Imh0dHA6Ly9sb2NhbGhvc3Q6ODA4MS9yZWFsbXMvY291cnNlLW1hbmFnZW1lbnQtcmVhbG0iLCJzdWIiOiIzNzY2NmQ1Ni1jYTJkLTRkNDQtOTRmMS1kNDk4ZTdhZmVhOTUiLCJ0eXAiOiJSZWZyZXNoIiwiYXpwIjoiY291cnNlLWFwcCIsInNpZCI6ImM5NzVkNmU4LTg5YjgtNGUxNS04NjdmLTM3ZjU2ZmFkN2RjZSIsInNjb3BlIjoib3BlbmlkIHdlYi1vcmlnaW5zIHJvbGVzIGFjciBiYXNpYyBlbWFpbCBwcm9maWxlIn0.WkDLs8k7EEAXjJwMUHK2XurjaXXY-0-y5br1JlNSUIyGNzVBUrfmfN8Te7ysWvSvGlhrO9k6ali6oqGiqxXTBA\",\n\u003e\"token_type\": \"Bearer\",\n\u003e\"id_token\": \"eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICI0T1lYQVlZUXF6T2ZibE1wRjF0dmNvLW1UY2dEODVjai1Qak4xVnhuUExzIn0.eyJleHAiOjE3NDE3NzkxODYsImlhdCI6MTc0MTc0MzE4NiwianRpIjoiNWNkODRkM2UtOWY3YS00YmYxLWIzYTAtMjU4MWY4NzJmODAzIiwiaXNzIjoiaHR0cDovL2xvY2FsaG9zdDo4MDgxL3JlYWxtcy9jb3Vyc2UtbWFuYWdlbWVudC1yZWFsbSIsImF1ZCI6ImNvdXJzZS1hcHAiLCJzdWIiOiIzNzY2NmQ1Ni1jYTJkLTRkNDQtOTRmMS1kNDk4ZTdhZmVhOTUiLCJ0eXAiOiJJRCIsImF6cCI6ImNvdXJzZS1hcHAiLCJzaWQiOiJjOTc1ZDZlOC04OWI4LTRlMTUtODY3Zi0zN2Y1NmZhZDdkY2UiLCJhdF9oYXNoIjoiTE9yZGRCcHlyTzZHTDlmU21tbkVnZyIsImFjciI6IjEiLCJyZXNvdXJjZV9hY2Nlc3MiOnsicmV2aWV3LWFwcCI6eyJyb2xlcyI6WyJSRVZJRVctUkVBRCIsIlJFVklFVy1XUklURSJdfSwiY291cnNlLWFwcCI6eyJyb2xlcyI6WyJDT1VSU0UtV1JJVEUiLCJDT1VSU0UtUkVBRCJdfX0sImVtYWlsX3ZlcmlmaWVkIjpmYWxzZSwicmVhbG1fYWNjZXNzIjp7InJvbGVzIjpbIkdVRVNUIiwiQURNSU4iXX0sIm5hbWUiOiJOYXNydWRkaW4gTmFzcnVkZGluIiwicHJlZmVycmVkX3VzZXJuYW1lIjoibmFzcnVkZGluIiwiZ2l2ZW5fbmFtZSI6Ik5hc3J1ZGRpbiIsImZhbWlseV9uYW1lIjoiTmFzcnVkZGluIiwiZW1haWwiOiJuYXNydWRkaW5AZ21haWwuY29tIn0.T2QhItWJXsACMZhsQ1HJcm8GGF60upkS4fq6QyHWsKi7A1wojhOk5qrcD3LF8tiDLIbBr1QHQQOnxY4LHYa0v13bsCS-HJrs90NaT679pHYyGVRsLbpE0U3xwqxtX0o0XrQIpuP8SB_YYcQ2JtxkxaGW3k9X8BwBvgL5GDAK-_C1tO3hBlIxAD6ecEv44G_FD4f6Zrc6h4EXV_TKOnrXfjUxWTUmvIYuAnBfHFqCv3Yx_RONqKoAlwh4J349i3GuYdwbtR97Q_TBDRqGSgfkYY7d-GrGScPzM-ao3lfOGSq8VBqQh4ZXCWix8hAj1IUSiRWF5Tr9m_SIBN8WAyq07A\",\n\u003e\"not-before-policy\": 0,\n\u003e\"session_state\": \"c975d6e8-89b8-4e15-867f-37f56fad7dce\",\n\u003e\"scope\": \"openid email profile\"\n\u003e}\n\u003e```\n\u003e ![Jwt web page](notes/images/jwt.png)\n---\n# Observability\n\n- **Prometheus**:\n    - Open the browser - http://prometheus.local and make sure all the 4 microservices health status is listed\n![Prometheus web page](notes/images/prometheus.png)\n\u003e [!TIP]\n\u003e Hit - http://127.0.0.1/actuator/prometheus in your browser and check all the metrics at one place\n\u003e ![prometheus metrics](notes/images/metrics.png)\n- **Fluent-bit**\n    - Make sure fluent bit is running and listening on 0.0.0.0:24224\n```shell\n    kubectl logs fluent-bit\n```\n```shell\n    docker logs fluent-bit\n```\n```markdown\n[2025/03/12 00:40:53] [ info] [input:forward:forward.0] listening on 0.0.0.0:24224\nlevel=info caller=out_grafana_loki.go:64 id=0 [flb-go]=\"Starting fluent-bit-go-loki\" version=\"(version=, branch=, revision=94e00299ec9b36ad97c147641566b6922268c54e-modified)\"\nlevel=info caller=out_grafana_loki.go:66 id=0 [flb-go]=\"provided parameter\" URL=http://loki:3100/loki/api/v1/push\nlevel=info caller=out_grafana_loki.go:67 id=0 [flb-go]=\"provided parameter\" TenantID=\nlevel=info caller=out_grafana_loki.go:68 id=0 [flb-go]=\"provided parameter\" BatchWait=1.000s\nlevel=info caller=out_grafana_loki.go:69 id=0 [flb-go]=\"provided parameter\" BatchSize=1001024\nlevel=info caller=out_grafana_loki.go:70 id=0 [flb-go]=\"provided parameter\" Timeout=10.000s\nlevel=info caller=out_grafana_loki.go:71 id=0 [flb-go]=\"provided parameter\" MinBackoff=0.500s\nlevel=info caller=out_grafana_loki.go:72 id=0 [flb-go]=\"provided parameter\" MaxBackoff=300.000s\nlevel=info caller=out_grafana_loki.go:73 id=0 [flb-go]=\"provided parameter\" MaxRetries=10\nlevel=info caller=out_grafana_loki.go:74 id=0 [flb-go]=\"provided parameter\" Labels=\"[job=fluent-bit]\"\nlevel=info caller=out_grafana_loki.go:75 id=0 [flb-go]=\"provided parameter\" LogLevel=info\nlevel=info caller=out_grafana_loki.go:76 id=0 [flb-go]=\"provided parameter\" AutoKubernetesLabels=false\nlevel=info caller=out_grafana_loki.go:77 id=0 [flb-go]=\"provided parameter\" RemoveKeys=\"[source container_id]\"\nlevel=info caller=out_grafana_loki.go:78 id=0 [flb-go]=\"provided parameter\" LabelKeys=[container_name]\nlevel=info caller=out_grafana_loki.go:79 id=0 [flb-go]=\"provided parameter\" LineFormat=0\nlevel=info caller=out_grafana_loki.go:80 id=0 [flb-go]=\"provided parameter\" DropSingleKey=true\nlevel=info caller=out_grafana_loki.go:81 id=0 [flb-go]=\"provided parameter\" LabelMapPath=map[]\nlevel=info caller=out_grafana_loki.go:82 id=0 [flb-go]=\"provided parameter\" Buffer=false\nlevel=info caller=out_grafana_loki.go:83 id=0 [flb-go]=\"provided parameter\" BufferType=dque\nlevel=info caller=out_grafana_loki.go:84 id=0 [flb-go]=\"provided parameter\" DqueDir=/tmp/flb-storage/loki\nlevel=info caller=out_grafana_loki.go:85 id=0 [flb-go]=\"provided parameter\" DqueSegmentSize=500\nlevel=info caller=out_grafana_loki.go:86 id=0 [flb-go]=\"provided parameter\" DqueSync=false\nlevel=info caller=out_grafana_loki.go:87 id=0 [flb-go]=\"provided parameter\" ca_file=\nlevel=info caller=out_grafana_loki.go:88 id=0 [flb-go]=\"provided parameter\" cert_file=\nlevel=info caller=out_grafana_loki.go:89 id=0 [flb-go]=\"provided parameter\" key_file=\nlevel=info caller=out_grafana_loki.go:90 id=0 [flb-go]=\"provided parameter\" insecure_skip_verify=false\n[2025/03/12 00:40:53] [ info] [sp] stream processor started\n```\n\n- **Tempo**\n    - Also, check tempo logs for it's availability.\n```shell\n    kubectl logs docker-tempo-1\n```\n```shell\n    docker logs docker-tempo-1\n```\n```markdown\nts=2025-03-12T00:15:15Z level=info msg=\"Starting GRPC server\" component=tempo endpoint=0.0.0.0:4317\nts=2025-03-12T00:15:15Z level=info msg=\"Starting HTTP server\" component=tempo endpoint=0.0.0.0:4318\nlevel=info ts=2025-03-12T00:15:15.886518261Z caller=tempodb.go:576 msg=\"polling enabled\" interval=5m0s blocklist_concurrency=50\nlevel=info ts=2025-03-12T00:15:15.893078036Z caller=wal.go:109 msg=\"beginning replay\" file=4b37865a-8f01-4d27-ad63-f90c86ee326b+single-tenant+vParquet4 size=4096\nlevel=info ts=2025-03-12T00:15:15.89624768Z caller=worker.go:249 msg=\"total worker concurrency updated\" totalConcurrency=20\nlevel=warn ts=2025-03-12T00:15:15.897771963Z caller=wal.go:115 msg=\"failed to replay block. removing.\" file=4b37865a-8f01-4d27-ad63-f90c86ee326b+single-tenant+vParquet4 err=\"error reading wal meta json: /var/tempo/wal/4b37865a-8f01-4d27-ad63-f90c86ee326b+single-tenant+vParquet4/meta.json: open /var/tempo/wal/4b37865a-8f01-4d27-ad63-f90c86ee326b+single-tenant+vParquet4/meta.json: no such file or directory\"\nlevel=info ts=2025-03-12T00:15:15.898190804Z caller=wal.go:109 msg=\"beginning replay\" file=724d6348-1e29-456e-920d-0bfa7b4e8bdd+single-tenant+vParquet4 size=4096\nlevel=info ts=2025-03-12T00:15:15.902527103Z caller=poller.go:287 msg=\"writing tenant index\" tenant=single-tenant metas=3 compactedMetas=2\nlevel=info ts=2025-03-12T00:15:15.917231924Z caller=poller.go:144 msg=\"blocklist poll complete\" seconds=0.023366227\nlevel=info ts=2025-03-12T00:15:15.917417761Z caller=compactor.go:159 msg=\"enabling compaction\"\nlevel=info ts=2025-03-12T00:15:15.919806486Z caller=tempodb.go:540 msg=\"compaction and retention enabled.\"\nlevel=info ts=2025-03-12T00:15:15.920241565Z caller=compactor.go:142 msg=\"starting compaction cycle\" tenantID=single-tenant offset=0\nlevel=info ts=2025-03-12T00:15:15.920400513Z caller=compactor.go:155 msg=\"compaction cycle complete. No more blocks to compact\" tenantID=single-tenant\nlevel=info ts=2025-03-12T00:15:16.414958877Z caller=wal.go:136 msg=\"replay complete\" file=724d6348-1e29-456e-920d-0bfa7b4e8bdd+single-tenant+vParquet4 duration=516.810746ms\nlevel=warn ts=2025-03-12T00:15:16.415046915Z caller=wal.go:99 msg=\"unowned file entry ignored during wal replay\" file=blocks err=null\nlevel=info ts=2025-03-12T00:15:16.421910578Z caller=ingester.go:448 msg=\"wal replay complete\"\nlevel=info ts=2025-03-12T00:15:16.422119251Z caller=ingester.go:462 msg=\"reloading local blocks\" tenants=0\nlevel=info ts=2025-03-12T00:15:16.42224564Z caller=lifecycler.go:677 msg=\"not loading tokens from file, tokens file path is empty\"\nlevel=info ts=2025-03-12T00:15:16.42315586Z caller=lifecycler.go:704 msg=\"instance not found in ring, adding with no tokens\" ring=ingester\nlevel=info ts=2025-03-12T00:15:16.423323314Z caller=app.go:205 msg=\"Tempo started\"\n```\n\n# Guide to Grafana\n### Accessing Loki, Tempo, Prometheus, and Dashboards in Grafana\n\n## Step 1: Log In to Grafana\n1. Open your browser and go to your Grafana instance (e.g., `http://localhost:3000`).\n2. Log in with your credentials (default: `admin`/`admin`). Update the password if prompted.\n\n---\n\n## Step 2: Add Data Sources (Loki, Tempo, Prometheus)\nConfigure Loki, Tempo, and Prometheus as data sources in Grafana.\n\n1. **Navigate to Connections**:\n    - From the left sidebar, hover over the **Connections** icon (plug symbol) and click **Data sources**.  \n      Or use the hamburger menu (☰) \u003e **Connections** \u003e **Data sources**.\n\n2. **Add a New Data Source**:\n    - Click **+ Add new data source** in the top-right corner.\n\n3. **Configure Prometheus**:\n    - Search for `Prometheus` and select it.\n    - Set **Name** (e.g., \"Prometheus\").\n    - Enter **URL** (e.g., `http://prometheus:9090`).\n    - Leave defaults unless specific settings (e.g., authentication) are needed.\n    - Click **Save \u0026 test**. Confirm \"Data source is working.\"\n\n4. **Configure Loki**:\n    - Search for `Loki` and select it.\n    - Set **Name** (e.g., \"Loki\").\n    - Enter **URL** (e.g., `http://loki:3100`).\n    - Optionally, set **Max lines** (e.g., 1000) under **Additional settings**.\n    - Click **Save \u0026 test**. Verify it works.\n\n5. **Configure Tempo**:\n    - Search for `Tempo` and select it.\n    - Set **Name** (e.g., \"Tempo\").\n    - Enter **URL** (e.g., `http://tempo:3200`).\n    - Optionally, link to Loki for trace-to-log correlation via **Derived Field** (e.g., match `traceID=(\\w+)`).\n    - Click **Save \u0026 test**. Ensure it’s operational.\n\n---\n\n## Step 3: Access via Explore\nUse the **Explore** view to query data directly from Loki, Tempo, and Prometheus.\n\n1. **Open Explore**:\n    - Click the **Explore** icon (compass) in the left sidebar, or use the hamburger menu (☰) \u003e **Explore**.\n\n2. **Select a Data Source**:\n    - Use the dropdown at the top to choose:\n        - **Prometheus**: Metrics (e.g., `rate(http_server_requests_seconds_count[5m])`).\n        - **Loki**: Logs (e.g., `{job=\"fluent-bit\"} |= \"error\"`).\n        - **Tempo**: Traces (e.g., search by trace ID or service).\n\n3. **Run Queries**:\n    - **Prometheus**: Enter a PromQL query and click **Run query**. View as graph or table.\n    - **Loki**: Use LogQL or the **Builder** tab to filter logs. Click **Run query** for logs or metrics.\n    - **Tempo**: Input a trace ID or use the **Search** tab (filter by service, duration, tags). View trace visualizations.\n\n4. **Switch Views**:\n    - Toggle between **Logs**, **Graph**, or **Traces** tabs above the results.\n\n---\n\n## Step 4: Access and Create Dashboards\nVisualize data from Loki, Tempo, and Prometheus using dashboards.\n\n1. **View Existing Dashboards**:\n    - Click the **Dashboards** icon (four-square grid) in the sidebar.\n    - Select **Browse** to see available dashboards (e.g., from `Spring Boot 3.x Statistic`).\n\n2. **Create a New Dashboard**:\n    - Go to **Dashboards** \u003e **+ New** \u003e **New dashboard**.\n    - Click **+ Add visualization**.\n    - Choose a data source (Prometheus, Loki, or Tempo).\n    - Add queries:\n        - **Prometheus**: Metric query (e.g., `http_server_requests_seconds_count`), visualize as Graph.\n        - **Loki**: Log query (e.g., `{job=\"fluent-bit\"}`), use **Logs** or **Time series**.\n        - **Tempo**: Grab a Trace ID or search query, visualize as a trace timeline.\n    - Customize panels and click **Apply**.\n\n3. **Save the Dashboard**:\n    - Click the **Save** icon (floppy disk) in the top-right.\n    - Name and save the dashboard.\n\n4. **Import Pre-Built Dashboards**:\n    - Go to **Dashboards** \u003e **+ New** \u003e **Import**.\n    - Upload JSON files provided in the directory /grafana-dashboard.\n    - Map data sources and import.\n---\n\n### Spring Boot Observability Dashboard\n![Observability Dashboard](/notes/images/observability.png)\n\n### Spring Boot Statistic Dashboard\n![Observability Dashboard](/notes/images/statistic.png)\n\n### Tempo Traces\n![Tempo trace](/notes/images/tempo1.png)\n![Tempo trace](/notes/images/tempo2.png)\n\n### Loki Logs\n![Loki Logs](/notes/images/loki.png)\n\n---\n# Verify the APIs\n\n## Accessing APIs with Bearer Token Authentication\n\n\n\n- `access-token`: `http://localhost:8081/realms/course-management-realm/protocol/openid-connect/token` or `http://keycloak.local/realms/course-management-realm/protocol/openid-connect/token`\n- `course-aggregate`: `http://9000/course-aggregate`\n- `courses`: `http://9000/courses`\n- `reviews`: `http://9000/reviews`\n\n---\n\n## Steps to Access APIs with Authorization\n\n## 1. Fetch Access Token\nYou need to get an access token from the **Access Point API**.\n\n### Request:\n\n```shell\n    curl -X POST http://localhost:8081/realms/course-management-realm/protocol/openid-connect/token  \\\n     -d \"grant_type=password\" \\\n     -d \"client_id=course-app\"  \\\n     -d \"client_secret=v1sCIPjANbvyJ87RsTkYeI9xHonDqZh7\" \\\n     -d \"username=nasruddin\"  \\\n     -d \"password=password\" \\\n     -d \"scope=openid roles\" | jq\n```\n```shell\n    http -f POST http://localhost:8081/realms/course-management-realm/protocol/openid-connect/token \\\n    grant_type=password \\\n    client_id=course-app \\\n    client_secret=v1sCIPjANbvyJ87RsTkYeI9xHonDqZh7 \\\n    username=nasruddin \\\n    password=password \\\n    scope=\"openid roles\" | jq\n```\n```markdown\n{\n    \"access_token\": \"eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICI0T1lYQVlZUXF6T2ZibE1wRjF0dmNvLW1UY2dEODVjai1Qak4xVnhuUExzIn0.eyJleHAiOjE3NDE3NzcxNDMsImlhdCI6MTc0MTc0MTE0MywianRpIjoiOTFjMjU1YTEtOGY2Zi00MDk2LTg3M2EtNGU2MWJiNTI3MmZmIiwiaXNzIjoiaHR0cDovL2xvY2FsaG9zdDo4MDgxL3JlYWxtcy9jb3Vyc2UtbWFuYWdlbWVudC1yZWFsbSIsImF1ZCI6InJldmlldy1hcHAiLCJzdWIiOiIzNzY2NmQ1Ni1jYTJkLTRkNDQtOTRmMS1kNDk4ZTdhZmVhOTUiLCJ0eXAiOiJCZWFyZXIiLCJhenAiOiJjb3Vyc2UtYXBwIiwic2lkIjoiNDEwZjE5ZDEtZTcwNi00MzQ0LWJmMGQtYjRhZDVhYzIzMmE1IiwiYWNyIjoiMSIsImFsbG93ZWQtb3JpZ2lucyI6WyIvKiJdLCJyZWFsbV9hY2Nlc3MiOnsicm9sZXMiOlsiR1VFU1QiLCJBRE1JTiJdfSwicmVzb3VyY2VfYWNjZXNzIjp7InJldmlldy1hcHAiOnsicm9sZXMiOlsiUkVWSUVXLVJFQUQiLCJSRVZJRVctV1JJVEUiXX0sImNvdXJzZS1hcHAiOnsicm9sZXMiOlsiQ09VUlNFLVdSSVRFIiwiQ09VUlNFLVJFQUQiXX19LCJzY29wZSI6Im9wZW5pZCBlbWFpbCBwcm9maWxlIiwiZW1haWxfdmVyaWZpZWQiOmZhbHNlLCJuYW1lIjoiTmFzcnVkZGluIE5hc3J1ZGRpbiIsInByZWZlcnJlZF91c2VybmFtZSI6Im5hc3J1ZGRpbiIsImdpdmVuX25hbWUiOiJOYXNydWRkaW4iLCJmYW1pbHlfbmFtZSI6Ik5hc3J1ZGRpbiIsImVtYWlsIjoibmFzcnVkZGluQGdtYWlsLmNvbSJ9.adXke6N_RB5bFZCet_ZoMo3Q8Xm-LP-iM2ahgSS4zGubHx4738-n0lzyG2Aa4grtLM9OXbLD_wcm7aXruckQOKzb2YbAJSUR1X3Ul63PQLngT3Qh7xkFRxKT0DK8eN9cJQ2iHndiSS17L3NUZ5lqf4NVBqhW0t2YjoZlTwNzDCQsQA1wCWRypW3AcGRuos-KSjTBLGhipJD2wo8REGf0vTnsX1kd-3HYtoRSA-P3p1p6WJmxfOxzwQyjemi-GBsRcS8R86F14MXfBPSuOc1pk2wumOlHxcE_sJLo5rVJM72YL95wSjJmuOrZ8Wx7TSk-InYSCxYOkm6_CajhTOYHaw\",\n    \"expires_in\": 36000,\n    \"refresh_expires_in\": 1800,\n    \"refresh_token\": \"eyJhbGciOiJIUzUxMiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICJhZjM3ZGFiMi1iMTQwLTRlY2EtYTNjNi05YmFkM2JkNzY0ODcifQ.eyJleHAiOjE3NDE3NDI5NDMsImlhdCI6MTc0MTc0MTE0MywianRpIjoiZjZlZmUwNTQtNGJkZC00MDNmLWJkNWYtYzU4MTdkNmMwM2IyIiwiaXNzIjoiaHR0cDovL2xvY2FsaG9zdDo4MDgxL3JlYWxtcy9jb3Vyc2UtbWFuYWdlbWVudC1yZWFsbSIsImF1ZCI6Imh0dHA6Ly9sb2NhbGhvc3Q6ODA4MS9yZWFsbXMvY291cnNlLW1hbmFnZW1lbnQtcmVhbG0iLCJzdWIiOiIzNzY2NmQ1Ni1jYTJkLTRkNDQtOTRmMS1kNDk4ZTdhZmVhOTUiLCJ0eXAiOiJSZWZyZXNoIiwiYXpwIjoiY291cnNlLWFwcCIsInNpZCI6IjQxMGYxOWQxLWU3MDYtNDM0NC1iZjBkLWI0YWQ1YWMyMzJhNSIsInNjb3BlIjoib3BlbmlkIHdlYi1vcmlnaW5zIHJvbGVzIGFjciBiYXNpYyBlbWFpbCBwcm9maWxlIn0.-aR1XC4o26s0nlORQV-MZas9_hXOqzevmNQjMVzIdyqWdO7r6cJC3O6jphNBXfsW1KRYa0hoojlVXuAOJVYNfg\",\n    \"token_type\": \"Bearer\",\n    \"id_token\": \"eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICI0T1lYQVlZUXF6T2ZibE1wRjF0dmNvLW1UY2dEODVjai1Qak4xVnhuUExzIn0.eyJleHAiOjE3NDE3NzcxNDMsImlhdCI6MTc0MTc0MTE0MywianRpIjoiNTA5OGU2NTMtYjgzMC00ZmVhLWE4ZDMtZDBlOWUyNDZhMzc0IiwiaXNzIjoiaHR0cDovL2xvY2FsaG9zdDo4MDgxL3JlYWxtcy9jb3Vyc2UtbWFuYWdlbWVudC1yZWFsbSIsImF1ZCI6ImNvdXJzZS1hcHAiLCJzdWIiOiIzNzY2NmQ1Ni1jYTJkLTRkNDQtOTRmMS1kNDk4ZTdhZmVhOTUiLCJ0eXAiOiJJRCIsImF6cCI6ImNvdXJzZS1hcHAiLCJzaWQiOiI0MTBmMTlkMS1lNzA2LTQzNDQtYmYwZC1iNGFkNWFjMjMyYTUiLCJhdF9oYXNoIjoiNFNWZjROYk1ldUY1bWdvWGlRd2UzdyIsImFjciI6IjEiLCJyZXNvdXJjZV9hY2Nlc3MiOnsicmV2aWV3LWFwcCI6eyJyb2xlcyI6WyJSRVZJRVctUkVBRCIsIlJFVklFVy1XUklURSJdfSwiY291cnNlLWFwcCI6eyJyb2xlcyI6WyJDT1VSU0UtV1JJVEUiLCJDT1VSU0UtUkVBRCJdfX0sImVtYWlsX3ZlcmlmaWVkIjpmYWxzZSwicmVhbG1fYWNjZXNzIjp7InJvbGVzIjpbIkdVRVNUIiwiQURNSU4iXX0sIm5hbWUiOiJOYXNydWRkaW4gTmFzcnVkZGluIiwicHJlZmVycmVkX3VzZXJuYW1lIjoibmFzcnVkZGluIiwiZ2l2ZW5fbmFtZSI6Ik5hc3J1ZGRpbiIsImZhbWlseV9uYW1lIjoiTmFzcnVkZGluIiwiZW1haWwiOiJuYXNydWRkaW5AZ21haWwuY29tIn0.UrhX9BfGUgNFpxRbk85_0GVbkU8PKDC1toWVNJbTT1LlADmsEk9miC8DUR-oSAgmyWsk4qptt_w22ESihmrfK8-lnVT_92g3tocxqNNymJFzbQdBx5U-v7_UOOEUNVkWgUAssMJ20KmhCEJwFAeUdHmv4sMeBYaZgihhTbz766V_S5zNuKRGCwkMt_OIGda7nYJ4aUI8jP5F4jViMpSYjDWEg2rKKRxbwIKVN3THDJt_Z3jfxh_GXq-5pR07S6De1koAEMqiHQSvqxvg2X3hsph7EBx5R8hoTDv-5nhdOZLNGBKCU3q74UmBvhwlEuXmJgRu3iLriJEVrm5qLnh1DQ\",\n    \"not-before-policy\": 0,\n    \"session_state\": \"410f19d1-e706-4344-bf0d-b4ad5ac232a5\",\n    \"scope\": \"openid email profile\"\n}\n```\n\n## 2. Use the Access Token in API Requests \nOnce you have the access token, include it in the Authorization header as a Bearer token when calling the other APIs. \n\n\u003e [!NOTE]\n\u003e All the below APIs are being accessed via gateway. However, you can also access them using their respective endpoints but make sure you append **api** in front of context. eg. *http://9001/api/courses* or *http://9002/api/reviews*.   \n\n### Access `course-aggregate` API:\n```shell\n     http :9000/course-aggregate/1/with-details \"Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICI0T1lYQVlZUXF6T2ZibE1wRjF0dmNvLW1UY2dEODVjai1Qak4xVnhuUExzIn0.eyJleHAiOjE3NDE3NzcyMDQsImlhdCI6MTc0MTc0MTIwNCwianRpIjoiOGUyMzg3YzEtMWEzOS00Yjg0LTlhMjAtYTg3M2ZlYTIzNDYxIiwiaXNzIjoiaHR0cDovL2xvY2FsaG9zdDo4MDgxL3JlYWxtcy9jb3Vyc2UtbWFuYWdlbWVudC1yZWFsbSIsImF1ZCI6InJldmlldy1hcHAiLCJzdWIiOiIzNzY2NmQ1Ni1jYTJkLTRkNDQtOTRmMS1kNDk4ZTdhZmVhOTUiLCJ0eXAiOiJCZWFyZXIiLCJhenAiOiJjb3Vyc2UtYXBwIiwic2lkIjoiOTJiZDBmZTQtNjhlYi00ZDViLTliNmEtMWMyMzMzZmNiOWRkIiwiYWNyIjoiMSIsImFsbG93ZWQtb3JpZ2lucyI6WyIvKiJdLCJyZWFsbV9hY2Nlc3MiOnsicm9sZXMiOlsiR1VFU1QiLCJBRE1JTiJdfSwicmVzb3VyY2VfYWNjZXNzIjp7InJldmlldy1hcHAiOnsicm9sZXMiOlsiUkVWSUVXLVJFQUQiLCJSRVZJRVctV1JJVEUiXX0sImNvdXJzZS1hcHAiOnsicm9sZXMiOlsiQ09VUlNFLVdSSVRFIiwiQ09VUlNFLVJFQUQiXX19LCJzY29wZSI6Im9wZW5pZCBlbWFpbCBwcm9maWxlIiwiZW1haWxfdmVyaWZpZWQiOmZhbHNlLCJuYW1lIjoiTmFzcnVkZGluIE5hc3J1ZGRpbiIsInByZWZlcnJlZF91c2VybmFtZSI6Im5hc3J1ZGRpbiIsImdpdmVuX25hbWUiOiJOYXNydWRkaW4iLCJmYW1pbHlfbmFtZSI6Ik5hc3J1ZGRpbiIsImVtYWlsIjoibmFzcnVkZGluQGdtYWlsLmNvbSJ9.DsifGnS890ljND6P4ibZnDV47K9snnbmitSYDzejt4OdR5Z4dykew2pYDAv9fOEYMAURVMQRJShs-eahNupiaIiC8QUnNWdL3cFZVX5VONS4DHhu9885yH8t6QB1RDLAjO0saK0S3kqgmR0cZIOh4Ps9kv2W4Zxq8UW25dXiRWvA-m5vozZ10x-9iT6-x_Vxr9do4oMVAT_q_S7qvIEy9EpMFPQa7RyxHToYhzpAJ2BHoJjSFZ26FaqvEtfRUTYUsH0uRgD88hEzIaMGcKLTwMox8MjD0cSe2WQVc2wzTvjJOw2086xLMGFYca8T-Wz-LlgaVYpp5AJ5OIM6gye35Q\"\n```\n```markdown\nHTTP/1.1 200 OK\nCache-Control: no-cache, no-store, max-age=0, must-revalidate\nContent-Length: 2150\nContent-Type: application/json\nExpires: 0\nPragma: no-cache\nReferrer-Policy: no-referrer\nX-Content-Type-Options: nosniff\nX-Frame-Options: DENY\nX-XSS-Protection: 0\n\n{\n    \"course\": {\n        \"author\": \"John Doe\",\n        \"id\": 1,\n        \"price\": 29.79,\n        \"publisher\": \"Whatsapp\",\n        \"title\": \"Microservices with Quarkus\"\n    },\n    \"reviews\": [\n        {\n            \"author\": \"John Doe\",\n            \"content\": \"Amazing book and loved reading it\",\n            \"courseId\": 1,\n            \"email\": \"abc@xyz.com\",\n            \"id\": \"67cfd9f2ab40f0347b1a3f15\"\n        },\n        {\n            \"author\": \"John Doe\",\n            \"content\": \"Amazing book and loved reading it\",\n            \"courseId\": 1,\n            \"email\": \"abc@xyz.com\",\n            \"id\": \"67cfd9f2ab40f0347b1a3f14\"\n        }\n    ]\n}\n```\n\n### Access `courses` API:\n```shell\n    http POST :9000/courses title=\"Microservices with Golang\" author=\"John Doe\" price:=29.79 publisher=\"GitHub\" \"Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICI0T1lYQVlZUXF6T2ZibE1wRjF0dmNvLW1UY2dEODVjai1Qak4xVnhuUExzIn0.eyJleHAiOjE3NDE3NzcyMDQsImlhdCI6MTc0MTc0MTIwNCwianRpIjoiOGUyMzg3YzEtMWEzOS00Yjg0LTlhMjAtYTg3M2ZlYTIzNDYxIiwiaXNzIjoiaHR0cDovL2xvY2FsaG9zdDo4MDgxL3JlYWxtcy9jb3Vyc2UtbWFuYWdlbWVudC1yZWFsbSIsImF1ZCI6InJldmlldy1hcHAiLCJzdWIiOiIzNzY2NmQ1Ni1jYTJkLTRkNDQtOTRmMS1kNDk4ZTdhZmVhOTUiLCJ0eXAiOiJCZWFyZXIiLCJhenAiOiJjb3Vyc2UtYXBwIiwic2lkIjoiOTJiZDBmZTQtNjhlYi00ZDViLTliNmEtMWMyMzMzZmNiOWRkIiwiYWNyIjoiMSIsImFsbG93ZWQtb3JpZ2lucyI6WyIvKiJdLCJyZWFsbV9hY2Nlc3MiOnsicm9sZXMiOlsiR1VFU1QiLCJBRE1JTiJdfSwicmVzb3VyY2VfYWNjZXNzIjp7InJldmlldy1hcHAiOnsicm9sZXMiOlsiUkVWSUVXLVJFQUQiLCJSRVZJRVctV1JJVEUiXX0sImNvdXJzZS1hcHAiOnsicm9sZXMiOlsiQ09VUlNFLVdSSVRFIiwiQ09VUlNFLVJFQUQiXX19LCJzY29wZSI6Im9wZW5pZCBlbWFpbCBwcm9maWxlIiwiZW1haWxfdmVyaWZpZWQiOmZhbHNlLCJuYW1lIjoiTmFzcnVkZGluIE5hc3J1ZGRpbiIsInByZWZlcnJlZF91c2VybmFtZSI6Im5hc3J1ZGRpbiIsImdpdmVuX25hbWUiOiJOYXNydWRkaW4iLCJmYW1pbHlfbmFtZSI6Ik5hc3J1ZGRpbiIsImVtYWlsIjoibmFzcnVkZGluQGdtYWlsLmNvbSJ9.DsifGnS890ljND6P4ibZnDV47K9snnbmitSYDzejt4OdR5Z4dykew2pYDAv9fOEYMAURVMQRJShs-eahNupiaIiC8QUnNWdL3cFZVX5VONS4DHhu9885yH8t6QB1RDLAjO0saK0S3kqgmR0cZIOh4Ps9kv2W4Zxq8UW25dXiRWvA-m5vozZ10x-9iT6-x_Vxr9do4oMVAT_q_S7qvIEy9EpMFPQa7RyxHToYhzpAJ2BHoJjSFZ26FaqvEtfRUTYUsH0uRgD88hEzIaMGcKLTwMox8MjD0cSe2WQVc2wzTvjJOw2086xLMGFYca8T-Wz-LlgaVYpp5AJ5OIM6gye35Q\"\n```\n```markdown\nHTTP/1.1 201 Created\nCache-Control: no-cache, no-store, max-age=0, must-revalidate\nContent-Length: 210\nContent-Type: application/json\nExpires: 0\nPragma: no-cache\nReferrer-Policy: no-referrer\nX-Content-Type-Options: nosniff\nX-Frame-Options: DENY\nX-XSS-Protection: 0\n\n{\n    \"author\": \"John Doe\",\n    \"createdDate\": \"2025-03-12T01:17:35.565586049Z\",\n    \"id\": 5,\n    \"lastModifiedDate\": \"2025-03-12T01:17:35.565586049Z\",\n    \"price\": 29.79,\n    \"publisher\": \"GitHub\",\n    \"title\": \"Microservices with Golang\",\n    \"version\": 0\n}\n```\n### Access `reviews` API:\n```shell\n    http POST :9000/reviews courseId:=1 author=\"John Doe\" content=\"Amazing book\"  email=\"abc@xyz.com\" \"Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICI0T1lYQVlZUXF6T2ZibE1wRjF0dmNvLW1UY2dEODVjai1Qak4xVnhuUExzIn0.eyJleHAiOjE3NDE3NzcyMDQsImlhdCI6MTc0MTc0MTIwNCwianRpIjoiOGUyMzg3YzEtMWEzOS00Yjg0LTlhMjAtYTg3M2ZlYTIzNDYxIiwiaXNzIjoiaHR0cDovL2xvY2FsaG9zdDo4MDgxL3JlYWxtcy9jb3Vyc2UtbWFuYWdlbWVudC1yZWFsbSIsImF1ZCI6InJldmlldy1hcHAiLCJzdWIiOiIzNzY2NmQ1Ni1jYTJkLTRkNDQtOTRmMS1kNDk4ZTdhZmVhOTUiLCJ0eXAiOiJCZWFyZXIiLCJhenAiOiJjb3Vyc2UtYXBwIiwic2lkIjoiOTJiZDBmZTQtNjhlYi00ZDViLTliNmEtMWMyMzMzZmNiOWRkIiwiYWNyIjoiMSIsImFsbG93ZWQtb3JpZ2lucyI6WyIvKiJdLCJyZWFsbV9hY2Nlc3MiOnsicm9sZXMiOlsiR1VFU1QiLCJBRE1JTiJdfSwicmVzb3VyY2VfYWNjZXNzIjp7InJldmlldy1hcHAiOnsicm9sZXMiOlsiUkVWSUVXLVJFQUQiLCJSRVZJRVctV1JJVEUiXX0sImNvdXJzZS1hcHAiOnsicm9sZXMiOlsiQ09VUlNFLVdSSVRFIiwiQ09VUlNFLVJFQUQiXX19LCJzY29wZSI6Im9wZW5pZCBlbWFpbCBwcm9maWxlIiwiZW1haWxfdmVyaWZpZWQiOmZhbHNlLCJuYW1lIjoiTmFzcnVkZGluIE5hc3J1ZGRpbiIsInByZWZlcnJlZF91c2VybmFtZSI6Im5hc3J1ZGRpbiIsImdpdmVuX25hbWUiOiJOYXNydWRkaW4iLCJmYW1pbHlfbmFtZSI6Ik5hc3J1ZGRpbiIsImVtYWlsIjoibmFzcnVkZGluQGdtYWlsLmNvbSJ9.DsifGnS890ljND6P4ibZnDV47K9snnbmitSYDzejt4OdR5Z4dykew2pYDAv9fOEYMAURVMQRJShs-eahNupiaIiC8QUnNWdL3cFZVX5VONS4DHhu9885yH8t6QB1RDLAjO0saK0S3kqgmR0cZIOh4Ps9kv2W4Zxq8UW25dXiRWvA-m5vozZ10x-9iT6-x_Vxr9do4oMVAT_q_S7qvIEy9EpMFPQa7RyxHToYhzpAJ2BHoJjSFZ26FaqvEtfRUTYUsH0uRgD88hEzIaMGcKLTwMox8MjD0cSe2WQVc2wzTvjJOw2086xLMGFYca8T-Wz-LlgaVYpp5AJ5OIM6gye35Q\"\n```\n```markdown\nHTTP/1.1 201 Created\nCache-Control: no-cache, no-store, max-age=0, must-revalidate\nContent-Length: 125\nContent-Type: application/json\nExpires: 0\nPragma: no-cache\nReferrer-Policy: no-referrer\nX-Content-Type-Options: nosniff\nX-Frame-Options: DENY\nX-XSS-Protection: 0\n\n{\n    \"author\": \"John Doe\",\n    \"content\": \"Amazing book\",\n    \"courseId\": 1,\n    \"email\": \"abc@xyz.com\",\n    \"id\": \"67d0e14a2bb2a161bb104cef\",\n    \"version\": 1\n}\n\n```\n---\n# Troubleshoot and Tips\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnasruddin%2Fspring-boot-based-microservices","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnasruddin%2Fspring-boot-based-microservices","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnasruddin%2Fspring-boot-based-microservices/lists"}