{"id":16407585,"url":"https://github.com/musabbozkurt/live-data-service","last_synced_at":"2025-10-26T17:31:49.035Z","repository":{"id":176736957,"uuid":"565215154","full_name":"musabbozkurt/live-data-service","owner":"musabbozkurt","description":"Prometheus, Grafana, Kafka, Zipkin, Redis and Elasticsearch Integrations with Spring Boot 3, Java 21 and Docker. Integration Tests with Testcontainers for Controller. Unit Tests for Controller, Service, and Mapper with Junit 5, and Mockito","archived":false,"fork":false,"pushed_at":"2025-01-26T11:58:19.000Z","size":375,"stargazers_count":5,"open_issues_count":1,"forks_count":2,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-01-26T12:25:08.381Z","etag":null,"topics":["docker-compose","elasticsearch","flyway","google-api-services-playintegrity","grafana","java-21","jsonplaceholder","junit5","kafka","kibana","micrometer","mockito","prometheus","redis","redisson","restclient","spring-boot-3","testcontainers","zipkin"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/musabbozkurt.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-11-12T17:37:22.000Z","updated_at":"2024-09-21T14:59:26.000Z","dependencies_parsed_at":null,"dependency_job_id":"cc858006-b5a8-4375-b287-98a916b80b5f","html_url":"https://github.com/musabbozkurt/live-data-service","commit_stats":null,"previous_names":["musabbozkurt/live-data-service"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/musabbozkurt%2Flive-data-service","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/musabbozkurt%2Flive-data-service/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/musabbozkurt%2Flive-data-service/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/musabbozkurt%2Flive-data-service/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/musabbozkurt","download_url":"https://codeload.github.com/musabbozkurt/live-data-service/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":238375298,"owners_count":19461578,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["docker-compose","elasticsearch","flyway","google-api-services-playintegrity","grafana","java-21","jsonplaceholder","junit5","kafka","kibana","micrometer","mockito","prometheus","redis","redisson","restclient","spring-boot-3","testcontainers","zipkin"],"created_at":"2024-10-11T06:14:17.805Z","updated_at":"2025-10-26T17:31:49.030Z","avatar_url":"https://github.com/musabbozkurt.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003c!-- TABLE OF CONTENTS --\u003e\n\u003cdetails open=\"open\"\u003e\n  \u003csummary\u003e\u003ch2 style=\"display: inline-block\"\u003eTable of Contents\u003c/h2\u003e\u003c/summary\u003e\n  \u003col\u003e\n    \u003cli\u003e\u003ca href=\"#Summary\"\u003eSummary\u003c/a\u003e\u003c/li\u003e\n    \u003cli\u003e\u003ca href=\"#Prerequisites\"\u003ePrerequisites and Installation\u003c/a\u003e\u003c/li\u003e\n    \u003cli\u003e\u003ca href=\"#Tech_Stack\"\u003eTech Stack\u003c/a\u003e\u003c/li\u003e\n    \u003cli\u003e\u003ca href=\"#How_To_Run_And_Test_Application\"\u003eHow To Run And Test Application\u003c/a\u003e\u003c/li\u003e\n    \u003cli\u003e\u003ca href=\"#How_To_Run_And_Test_Dockerfile\"\u003eHow To Run And Test Application with Dockerfile (OPTIONAL)\u003c/a\u003e\u003c/li\u003e\n    \u003cli\u003e\u003ca href=\"#How_To_Run_And_Test_Docker_Compose\"\u003eHow To Run And Test Application with docker-compose.yml (OPTIONAL)\u003c/a\u003e\u003c/li\u003e\n    \u003cli\u003e\u003ca href=\"#Redis\"\u003eRedis Commands\u003c/a\u003e\u003c/li\u003e\n    \u003cli\u003e\u003ca href=\"#Google_Play_Integrity_API\"\u003eGoogle Play Integrity API\u003c/a\u003e\u003c/li\u003e\n    \u003cli\u003e\u003ca href=\"#References\"\u003eReferences\u003c/a\u003e\u003c/li\u003e\n  \u003c/ol\u003e\n\u003c/details\u003e\n\n-------\n\n### Summary\n\n#### live-data-service is providing CRUD operations to process live events\n\n-------\n\n### Prerequisites\n\n- `Java 24+` needs to be installed `export JAVA_HOME=$(/usr/libexec/java_home -v 24)`\n- `Maven` needs to be installed\n- `Docker` needs to be installed\n- Install any Java IDE (`Eclipse`, `STS`, `Intellij` etc..) and ensure you are able to launch\n- Clone or checkout the project from version control system (`git`) and follow below steps\n\n-------\n\n### Tech_Stack\n\n- `Java 24`\n- `Spring Boot 3`\n- `H2 Database` (Default values are provided below)\n    - `Default H2 Database Url`: http://localhost:8080/h2-console\n    - `username`: `sa`\n    - `password`: `sa`\n    - `url`: `jdbc:h2:mem:MB_TEST;DB_CLOSE_DELAY=-1`\n- `Swagger`: http://localhost:8080/swagger-ui/index.html\n- `Flyway` for database migration\n- Centralize exception handling by `ControllerAdvice`\n- `Mapstruct` to map different type of objects to each other\n- `Micrometer` dependencies were added to track the logs easily\n- `Testcontainers` dependencies were added for integration tests\n- `docker-compose.yml` contains `Grafana`, `Prometheus` and `Zipkin` to track metrics, `Kafka` for event-driven\n  architecture, `Redis` for caching, `Elasticsearch`, `Kibana`, and `Logstash` for search, analytics engine and logs,\n  `PostgreSQL` and `MongoDB` for data storage.\n    - Kafka and its UI are included for message streaming, while the Elastic Stack (Elasticsearch, Logstash, Kibana) is\n      set up for log management and visualization.\n    - All services are connected through `bridge` networks, ensuring communication between containers.\n    - `Actuator`: http://localhost:8080/actuator\n    - `Kafka UI`: http://localhost:9999/\n    - `Grafana`\n        - Login Credentials\n            - Url: http://localhost:3000/\n            - Email or username: `admin`\n            - Password: `admin`\n        - Add `Prometheus` Datasource\n            - Url: http://host.docker.internal:9090\n        - Add Dashboards\n            - https://grafana.com/grafana/dashboards/4701-jvm-micrometer/\n            - https://grafana.com/grafana/dashboards/5373-micrometer-spring-throughput/\n    - `Prometheus`\n        - `Actuator`: http://localhost:8080/actuator/prometheus\n        - `Prometheus`: http://localhost:9090/graph\n    - `Zipkin UI`: http://localhost:9411\n    - `Elasticsearch`: http://localhost:9200/\n    - `Kibana`: http://localhost:5601/app/home#/\n        - Connect Spring Boot to `Elasticsearch` `(TESTED FOR SPRING BOOT 3)`\n        - Run Spring Boot project\n        - Open http://localhost:5601/app/home#/ --\u003e `Discover` --\u003e `Create data view` -\u003e `Name should be Index name` --\u003e\n          `Save date view to Kibana`\n            - ![img.png](src/main/resources/logstash/img.png)\n    - `Database credentials`\n        - `PostgreSQL`\n            - `url`: `jdbc:postgresql://localhost:5432/localdb`\n            - `username`: `test`\n            - `password`: `test`\n        - `MongoDB`\n            - `url`: `mongodb://localhost:27017/testdb?authSource=admin`\n            - `username`: `root`\n            - `password`: `example`\n\n- Test via `Postman` (OPTIONAL):\n    1. Import [Postman Collection](docs%2Funit_test_service.postman_collection.json)\n    2. Right-click the imported Postman Collection and click _**Run Collection**_ section.\n    3. On the right panel choose _**Functional**_ or _**Performance**_ section, edit _**Run configuration**_ and click\n       _**run**_ to test the application.\n\n-------\n\n### How_To_Run_And_Test_Application\n\n- Please follow the following steps, if you want to build and run Spring Boot Application\n\n```\n*** Run the application by following these steps.\n\n1 - cd live-data-service\n2 - docker-compose up -d\n3 - ./mvnw clean install or mvn clean install or mvn clean package \n4 - ./mvnw spring-boot:run or mvn spring-boot:run\n```\n\n-------\n\n### How_To_Run_And_Test_Dockerfile\n\n- Please follow the following steps, if you want to build and run Dockerfile\n\n```\n1 - cd live-data-service\n2 - docker-compose up -d\n3 - ./mvnw clean install or mvn clean install or mvn clean package --------THIS IS MUST---------\n4 - docker build -t mb/live-data-service .\n5 - docker run -p 8080:8080 mb/live-data-service\n```\n\n-------\n\n### How_To_Run_And_Test_Docker_Compose\n\n- Please follow the following steps, if you want to build and run `docker-compose.yml`\n- Remove `live-data-service` service comment in `services` section in `docker-compose.yml`\n- `Docker` -\u003e `Preferences` -\u003e `Resources` -\u003e `File sharing` -\u003e click add button and select `prometheus` folder under\n  the `/src/main/resources` or just add `/etc/prometheus` path -\u003e `Apply \u0026 Restart`\n    - ![img.png](img.png)\n\n```\n1 - cd live-data-service\n2 - ./mvnw clean install or mvn clean install or mvn clean package --------THIS IS MUST---------\n3 - docker build -t mb/live-data-service .\n4 - docker-compose up -d\n```\n\n-------\n\n### Redis\n\n- The following command returns all matched data by `'keyPattern:*'` pattern\n    - `redis-cli --scan --pattern 'keyPattern:*'`\n\n- The following command deletes all matched data by `'keyPattern:*'` pattern\n    - `redis-cli KEYS 'keyPattern:*' | xargs redis-cli DEL`\n\n- The following command finds `TYPE` in redis with `KEY`\n    - `TYPE key` -\u003e `TYPE xxx:hashedIdOrSomethingElse`\n\n- The following commands search by `TYPE`\n    - for `\"string\" TYPE`: `get key`\n    - for `\"hash\" TYPE`: `hgetall key`\n    - for `\"list\" TYPE`: `lrange key 0 -1`\n    - for `\"set\" TYPE`: `smembers key`\n    - for `\"zset\" TYPE`: `zrange key 0 -1 withScores`\n\n- `RedisInsight`: http://localhost:8001/\n\n-------\n\n### Google_Play_Integrity_API\n\n- [Setup Firebase](https://firebase.google.com/docs/admin/setup) or Just follow the following steps\n  to [Initialize the SDK, Create and Download Firebase Credentials JSON File](https://firebase.google.com/docs/admin/setup#initialize-sdk)\n    - After JSON file is downloaded, copy the content of downloaded file and replace `play-integrity-credentials.json`\n      content with that copied text\n    - Replace `application-name` value in application.yml file with `project_id` variable that is\n      in `play-integrity-credentials.json`\n    - [HELPFUL LINK BUT IT SHOULD NOT BE NEEDED](https://stackoverflow.com/a/40799378)\n\n-------\n\n### References\n\n- [Metrics Made Easy Via Spring Actuator, Docker, Prometheus, and Grafana](https://www.youtube.com/watch?v=Utv7MWgNTvI)\n- https://prometheus.io/docs/prometheus/latest/installation/#volumes-bind-mount\n- [Spring Boot Rest Controller Unit Test with @WebMvcTest](https://www.bezkoder.com/spring-boot-webmvctest/)\n- [Redis Commands](https://auth0.com/blog/introduction-to-redis-install-cli-commands-and-data-types/)\n- [Running RedisInsight using Docker Compose](https://collabnix.com/running-redisinsight-using-docker-compose/)\n- [Google Play Integrity API](https://developer.android.com/google/play/integrity)\n- [A Guide to Querydsl with JPA](https://www.baeldung.com/querydsl-with-jpa-tutorial)","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmusabbozkurt%2Flive-data-service","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmusabbozkurt%2Flive-data-service","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmusabbozkurt%2Flive-data-service/lists"}