{"id":15069066,"url":"https://github.com/obrunodelgado/camelboilerplate","last_synced_at":"2025-07-24T13:11:13.379Z","repository":{"id":156084954,"uuid":"348762242","full_name":"obrunodelgado/camelboilerplate","owner":"obrunodelgado","description":"A Spring Boot Camel boilerplate that aims to consume events from Apache Kafka, process it and send to a PostgreSQL database.","archived":false,"fork":false,"pushed_at":"2021-04-13T16:24:46.000Z","size":36,"stargazers_count":46,"open_issues_count":0,"forks_count":9,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-04-10T17:09:04.361Z","etag":null,"topics":["camel","camel-boilerplate","data-engineering","docker","flyway","java","java-11","junit5","kafka","migrations","postgresql","spring-boot","unit-testing"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/obrunodelgado.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-03-17T15:37:14.000Z","updated_at":"2024-02-29T01:01:08.000Z","dependencies_parsed_at":null,"dependency_job_id":"fc404847-718d-4583-abfd-9190889bd5e5","html_url":"https://github.com/obrunodelgado/camelboilerplate","commit_stats":null,"previous_names":[],"tags_count":0,"template":true,"template_full_name":null,"purl":"pkg:github/obrunodelgado/camelboilerplate","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/obrunodelgado%2Fcamelboilerplate","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/obrunodelgado%2Fcamelboilerplate/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/obrunodelgado%2Fcamelboilerplate/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/obrunodelgado%2Fcamelboilerplate/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/obrunodelgado","download_url":"https://codeload.github.com/obrunodelgado/camelboilerplate/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/obrunodelgado%2Fcamelboilerplate/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":266845403,"owners_count":23994062,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-24T02:00:09.469Z","response_time":99,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["camel","camel-boilerplate","data-engineering","docker","flyway","java","java-11","junit5","kafka","migrations","postgresql","spring-boot","unit-testing"],"created_at":"2024-09-25T01:40:17.956Z","updated_at":"2025-07-24T13:11:13.325Z","avatar_url":"https://github.com/obrunodelgado.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# SPRING-BOOT CAMEL BOILERPLATE\n\nThis is a Spring-Boot [Camel](https://camel.apache.org/manual/latest/faq/what-is-camel.html) \nApplication model that you can use as a reference to study or even to use in your company. \nIt contains a [Sample Route](https://github.com/obrunodelgado/camelboilerplate/blob/main/src/main/java/com/delgado/bruno/boilerplates/camel/routes/SampleRoute.java) that consumes events from a [Kafka](https://www.youtube.com/watch?v=06iRM1Ghr1k) \nTopic called *sample_topic*, process the data and saves it into a PostgreSQL database.\n\nIn this project you will find references for the following study topics:\n\n- Spring-Boot\n- Camel\n- Kafka\n- Docker\n- Docker-Compose\n- Schema Migrations\n- Unit Tests\n- Health Checks\n- Liveness and Readiness probes\n\n\n## Development Pre-Requisites\n- **IDE**: to develop and debug the application\n- **Java 11**: the programming language for this project is Java\n- **Maven**: used to manage packages and build the project\n- **Docker**: to avoid configuring development environment\n- **Docker-Compose**: to create the infrastructure boilerplate\n\n## Running the application\n\nThis project is configured to run using a \n[*docker-compose*](https://github.com/obrunodelgado/camelboilerplate/blob/main/docker-compose.yml) file or even at the \ncommand-line. The project is configured to use different configurations according to the environment which the project is being run.\n\n\u003e Running the project with Docker uses different environment variables than running\n\u003e it from the command-line.\n\nI highly recommend you to run this project using Docker instead of running locally otherwise you will need to guarantee \nthat you have a Kafka and a PostgreSQL instance running or locally or remotely so the application can connect to it and \ndo it's job. \n\n\u003e Running the application without Kafka and PostgreSQL instances will result in errors, because\n\u003e Flyway will try to execute the migrations and will not be capable of.\n\n## Running using Docker\n\nThis project contains a Dockerfile / Docker Compose that creates all the required infraestructure to develop and \ntest the project:\n\n- **Kafka**: event streaming platform;\n- **Zookeeper**: keeps track of status of the Kafka cluster nodes and it also keeps track of Kafka topics, partitions etc;\n- **PostgreSQL**: database used to load the data extracted from Kafka;\n- **Camel Boilerplate**: the application itself.\n\n**Start-Up Containers**  \n`$ docker-compose up`  \n**Shutdown Containers**  \n`$ docker-compose down`\n\nNote in the [`application-docker.properties`](https://github.com/obrunodelgado/camelboilerplate/blob/main/src/main/resources/application-docker.properties)\nthat we have specific configurations to run the project using \nDocker envinroment. More information in the Environment Variables topic.\n\n#### Build Note\n\nWhen you run the project using Docker, it compiles the source code and creates\na target folder containing the project's jar. After that you will not be able\nto compile the project using your IDE or your command line because the\nfolder's owner now is Docker's user.\n\nTo solve this problem, just delete the target folder using sudo or\nadministrative privileges when you want to run the application using your\nIDE or command line.\n\n## Running at local machine\n\nThis project contains a maven plugin to execute the application.\nIf you want to run it locally it is important that you guarantee\nthat there's a Kafka and a PostgreSQL running in localhost. \n\nYou also should your Operational System's hosts file - that generally is located in `/etc/hosts` - and\nadd the following line to it.\n\n    127.0.0.1   kafka\n\nAfter this you just need to run the following maven command:\n\n    $ mvn clean install spring-boot:run -Dspring-boot.run.arguments=--database.password=postgres -Dspring-boot.run.profiles=local\n\nThe `-Dspring-boot.run` command-line argument is defining a system variable called `database.password` avoiding to expose \nproduction credentials in the repository. The `-Dspring-boot.run.profiles` tells Spring-Boot that it must read specific \nlocal configurations from the `application-local.properties` file.\n\n\u003e If you have a Kafka and a PostgreSQL instance running somewhere else, you should change the `application-local.properties` \n\u003e to point to it's hosts. Don't forget to change the `database.password` value in the command-line.\n\n## Sending events to Kafka\nTo be able to send events to Kafka, you can connect into it's container and send events using the CLI.\nTo do so, go into your terminal with the containers running and type the following commands to connect into the\nKafka's container:\n\n    $ docker exec -it camel_boilerplate_kafka sh\n    $ cd /opt/kafka/bin\n\nInside Kafka's bin folder there's a sh script named `kafka-console.producer.sh`.\nThis script allows you to create a producer and send events to the running instance on Kafka. Run the following command\nto connect into the topic *sample_topic*.\n\n    $ kafka-console-producer.sh --topic sample_topic --bootstrap-server localhost:9092\n\nYou should see your cursor after a `\u003e` char if you've been connected successfully. Now everything that you type and\nhit enter inside the console will produce a new event in the Kafka broker. You can use the following data as event\nmodel to test the application:\n\n**Sample Event**\n\n    {\"name\":\"Product\",\"quantity\":12,\"price\":12.5}\n\n## Environment Variables\nAll real applications contains different configurations for different environments. In other words you will always have \ndifferent connection strings for the Production and Local environment, for example. At Spring it is \ncalled *[profiles](https://docs.spring.io/spring-boot/docs/1.2.0.M1/reference/html/boot-features-profiles.html#boot-features-profiles)*.\n\nIn our case we need to set different hosts to run the project using the Docker environment and the local environment. \nThis happens because each module of this application will run in 4 different containers, one for the application, one \nfor the Zookeeper, one for the Kafka and another one for the PostgreSQL. Each of these containers \"have it's own \nlocalhost definitions\", meaning that localhost for the application if one host that is differente for Kafka.\n\nIn other words, when you are running the application using a Docker container and you configure it to connect into \nKafka using a localhost definition it will not be able to connect to Kafka because there's not a Kafka running at the \nApplication container because Kafka is in another container, another 'virtual machine'.\n\nTo solve this problem we need to tell the Application that Kafka is hosted at [`camel_boilerplate_kafka`](https://github.com/obrunodelgado/camelboilerplate/blob/main/docker-compose.yml#L15) - Kafka's \ncontainer name - and the PostgreSQL is hosted at [`camel_boilerplate_postgres`](https://github.com/obrunodelgado/camelboilerplate/blob/main/docker-compose.yml#L31) - PostgreSQL container name.  These hosts \nare defined at the `docker-compose.yml` file.\n\nSo if we want to run the application in different environments we need to configure it to know how to manage this \ndifferent environments. In the project we have 3 properties  files:\n\n- [**application.properties**](https://github.com/obrunodelgado/camelboilerplate/blob/main/src/main/resources/application.properties):  properties that have the same values for any environment;\n- [**application-docker.properties**](https://github.com/obrunodelgado/camelboilerplate/blob/main/src/main/resources/application-docker.properties): properties that have specific configuration values for Docker environment;\n- [**application-local.properties**](https://github.com/obrunodelgado/camelboilerplate/blob/main/src/main/resources/application-local.properties): properties that have specific configuration values for Local environment;\n\nThe next step is to tell the application which property file it should consider other than the `application.properties` \nfile and in this case we do it using a special command-line argument `-Dspring-boot.run.profiles={environment}` \nhaving the `{environment}` part replaced by `docker` or `local` string. For example: `-Dspring-boot.run.profiles=docker` \nor `-Dspring-boot.run.profiles=local`.\n\nA last consideration to have about Environment Variables is that I ommited the database password value from any \nconfiguration file because you should not save credentials in your code repository for security reasons. That's why we \nare passing the command-line argument `-Dspring-boot.run.arguments=--database.password=postgres` when we execute the \napplication: we are telling to Spring-Boot that the value for `database.password` property is `postgres`\n\n\u003e While defining credentials will open a security issue for anyone who have access to your code repository, defining \n\u003e credentials in the command-line or event using Operational System variables will expose your credentials to anyone \n\u003e who has access to the server machine. There's better approaches to secure your application but it would make this \n\u003e Camel sample even more complex.\n\n\n## Health Checks\n\nThis project is configured with some endpoints in order to inform the state of the application.\n\nBasically we have this endpoints:\n\n- **/actuator**: a summary of the available health endpoints\n- **/actuator/health**: general overview of the health status of the project and it's components.\n- **/actuator/health/liveness**: tells whether the internal state is valid. If Liveness is broken, this means that the application itself is in a failed state and cannot recover from it.\n- **/actuator/health/readiness**: tells whether the application is ready to accept requests. It will be defined \n  as out of service if Kafka goes down.\n- **/actuator/metrics**: a list of all the available metrics of the application\n- **/actuator/metrics/{metric_name}**:  metric details\n\nA health application should return the following json when you make a GET request to `/actuator/health`:\n\n    {\n       \"status\":\"UP\",\n       \"components\":{\n          \"camelHealth\":{\n             \"status\":\"UP\",\n             \"details\":{\n                \"name\":\"camel-health-check\",\n                \"context\":\"UP\",\n                \"route:sample\":\"UP\"\n             }\n          },\n          \"db\":{\n             \"status\":\"UP\",\n             \"details\":{\n                \"database\":\"PostgreSQL\",\n                \"validationQuery\":\"isValid()\"\n             }\n          },\n          \"diskSpace\":{\n             \"status\":\"UP\",\n             \"details\":{\n                \"total\":254356226048,\n                \"free\":170402906112,\n                \"threshold\":10485760,\n                \"exists\":true\n             }\n          },\n          \"kafka\":{\n             \"status\":\"UP\"\n          },\n          \"livenessState\":{\n             \"status\":\"UP\"\n          },\n          \"ping\":{\n             \"status\":\"UP\"\n          },\n          \"readinessState\":{\n             \"status\":\"UP\"\n          }\n       },\n       \"groups\":[\n          \"liveness\",\n          \"readiness\"\n       ]\n    }\n\nIf Kafka or the database goes offline, the kafka property in the health json should looks like this:\n\n    \"db\": {\n      \"status\": \"DOWN\",\n      \"details\": {\n        \"database\": \"PostgreSQL\",\n        \"validationQuery\": \"isValid()\"\n      }\n    },\n    \"kafka\": {\n      \"status\": \"DOWN\",\n      \"details\": {\n        \"Reason\": \"Kafka may be offline\"\n      }\n    }\n\nAs soon as Kafka goes online again, the application will recover it's state.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fobrunodelgado%2Fcamelboilerplate","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fobrunodelgado%2Fcamelboilerplate","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fobrunodelgado%2Fcamelboilerplate/lists"}