{"id":40355221,"url":"https://github.com/mgorav/apache-camel-aws-eis","last_synced_at":"2026-01-20T10:01:37.777Z","repository":{"id":167333209,"uuid":"179508724","full_name":"mgorav/apache-camel-aws-eis","owner":"mgorav","description":"Spring Boot + Apache Camel EIA using AWS S3, REST APIs, AWS Kinesis, Apache Kafka --\u003e Data Engineering EIA --\u003e Data Engineering As Routes --\u003e Router Pattern","archived":false,"fork":false,"pushed_at":"2019-04-27T10:39:35.000Z","size":5002,"stargazers_count":3,"open_issues_count":1,"forks_count":2,"subscribers_count":1,"default_branch":"master","last_synced_at":"2024-01-27T02:30:47.476Z","etag":null,"topics":["apache-camel","apache-kafka","aws-kinesis","aws-s3","camel-spring","eia","eip","spring-boot"],"latest_commit_sha":null,"homepage":"https://gonnect.org","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mgorav.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2019-04-04T14:01:02.000Z","updated_at":"2021-02-04T18:50:28.000Z","dependencies_parsed_at":"2023-05-25T11:45:39.145Z","dependency_job_id":null,"html_url":"https://github.com/mgorav/apache-camel-aws-eis","commit_stats":{"total_commits":51,"total_committers":1,"mean_commits":51.0,"dds":0.0,"last_synced_commit":"77ce5ecf44bcfc8948ff97db1fe0d4cd38a43d42"},"previous_names":["mgorav/apache-camel-aws-eis"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/mgorav/apache-camel-aws-eis","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mgorav%2Fapache-camel-aws-eis","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mgorav%2Fapache-camel-aws-eis/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mgorav%2Fapache-camel-aws-eis/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mgorav%2Fapache-camel-aws-eis/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mgorav","download_url":"https://codeload.github.com/mgorav/apache-camel-aws-eis/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mgorav%2Fapache-camel-aws-eis/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28601284,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-20T09:39:28.479Z","status":"ssl_error","status_checked_at":"2026-01-20T09:38:10.511Z","response_time":117,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apache-camel","apache-kafka","aws-kinesis","aws-s3","camel-spring","eia","eip","spring-boot"],"created_at":"2026-01-20T10:01:29.259Z","updated_at":"2026-01-20T10:01:37.734Z","avatar_url":"https://github.com/mgorav.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Router Pattern In Data/Software Engineering\n\nWhat is common between Navigation System \u0026 Data Engineering - \"_**movement from A to B with location transparency**_\". **Location transparency** means that \"A\" is unaware of \"B\". A navigation system handles \"location transparency\" by defining \"routes\" i.e. how to go from \"A\" to \"B\" by picking up the correct highway and as a driver \"_I don't want to know how_\"\n\nCan we apply same principle to \"Data Engineering\". The answer is  \"Yes\". Location transparency is achieved by implementation of \"**Router Pattern**\". _The Router pattern has been recognized as an excellent way to accomplish Enterprise Application Integration (EAI)._ A router is a component  that connects its consumer  to one of multiple output strategies (as enunciated in the strategy design pattern). This pattern is also one of the most powerful design pattern for the \"micro-services architecture\" as it can transform an application that is monolithic, non modular, non configurable among other bad things into a thing of beauty/piece of art. \n\n![alt text](./navigation.png)\n\nCheck out my Github project which shows:\n1. Moving data to \"Data Lake - AWS S3\"\n2. Moving data to \"Data Pipeline - AWS Kinesis \u0026 Apache Kafka\"\n3. Moving API data to \"Data Pipeline - AWS Kinesis\" \n4. Content Based Routing (CBR) to \"Data Pipeline Apache Kafka\"\n![alt text](./cbrwithroutingandenrichment.png)\n5. REST POST to \"Data Pipeline - AWS Kinesis\"\n\nIt also demonstrates opinionated way of AWS based development without AWS account  but the same software will run on AWS cloud with \"no fuss\".\n\n## EIA Patterns demonstrated\n\nFollowing EIA are implemented using Spring Boot \u0026 Apache Camel:\n1. Route to Spring bean\n2. Route to AWS S3\n3. Route to AWS Kinesis from REST end point\n4. Route to Apache Kafka\n5. Content based routing\n\n![alt text](./routes.png)\n\nLet's checkout one of routes which moved REST API data to AWS Kinesis:\n```java_holder_method_tree\n  restConfiguration().host(\"localhost\").port(4001);\n\n        from(\"timer:hi?period={{timer.period}}\")\n                .setHeader(\"id\", simple(\"${random(1,3)}\"))\n                .to(\"rest:get:cars/{id}\")\n                .log(\"[going to Kinesis]\"+\"${body}\")\n                .setHeader(KinesisConstants.PARTITION_KEY,simple(\"1\"))\n                .setHeader(KinesisConstants.SHARD_ID, simple(\"1\"))\n                .to(\"aws-kinesis://mykinesisstream?amazonKinesisClient=#amazonKinesisClient\")\n                .to(\"log:out?showAll=true\")\n                .log(\"Completed Writing to Kinesis\");\n```\nTo run this project you can setup/emulate AWS locally on you laptop by following below steps. It also comes with docker image of Apache Kafka + Zookeeper\n\n### Step 1: Create *python* virtual environment\n```bash\npython3 -m virtualenv localstackenv\n```\n\n### Step 2: Activate virtual environment\n```bash\nsource localstackenv/bin/activate   \n```\n\n### Step 3: Install AWS Local stack\n```bash\npip install localstack    \n```\n### Step4: Start localstack\n\n```bash\nlocalstack start --docker\n```\n\n### Step 4: Start kafka with zookeeper\n```bash\ndocker-compose up\n```\n\n\n## Play time\nJust run the Spring Boot application - \"**SpringBootCamelApplication**\" from IDE or maven \u0026 observe the logs.\n```bash\n2019-04-06 11:50:50.010  INFO 22172 --- [ucer[TestTopic]] route2                                   : \n{\n  \"orderNumber\": 1,\n  \"country\": \"US\",\n  \"amount\": 100,\n  \"items\": [\n    {\n      \"itemId\" : 123,\n      \"itemCost\": 33,\n      \"itemQty\": 12\n    }\n  ]\n}\n2019-04-06 11:50:50.013  INFO 22172 --- [umer[TestTopic]] FromKafka                                : consumed message from Kafka\n2019-04-06 11:50:50.013  INFO 22172 --- [umer[TestTopic]] FromKafka                                : Hello from Gonnect\n2019-04-06 11:50:50.036  INFO 22172 --- [umer[TestTopic]] FromKafka                                : consumed message from Kafka\n2019-04-06 11:50:50.036  INFO 22172 --- [umer[TestTopic]] FromKafka                                : \n{\n  \"orderNumber\": 1,\n  \"country\": \"NL\",\n  \"amount\": 100,\n  \"items\": [\n    {\n      \"itemId\" : 123,\n      \"itemCost\": 33,\n      \"itemQty\": 12\n    }\n  ]\n}\n\n.... ...... ........\n\n```\n\n### Route Information\n\n```bash\ncurl -XGET -s http://localhost:4001/actuator/camelroutes\n```\n```json\n[\n  {\n    \"id\": \"route1\",\n    \"uptime\": \"9.780 seconds\",\n    \"uptimeMillis\": 9781,\n    \"properties\": {\n      \"parent\": \"64beb2b7\",\n      \"rest\": \"false\",\n      \"description\": null,\n      \"id\": \"route1\"\n    },\n    \"status\": \"Started\"\n  },\n  {\n    \"id\": \"RandomTextGeneratorRoute\",\n    \"uptime\": \"9.780 seconds\",\n    \"uptimeMillis\": 9780,\n    \"properties\": {\n      \"parent\": \"4e4b7abd\",\n      \"rest\": \"false\",\n      \"description\": null,\n      \"id\": \"RandomTextGeneratorRoute\"\n    },\n    \"status\": \"Started\"\n  },\n  {\n    \"id\": \"FromKafka\",\n    \"uptime\": \"9.736 seconds\",\n    \"uptimeMillis\": 9736,\n    \"properties\": {\n      \"parent\": \"730cd2d0\",\n      \"rest\": \"false\",\n      \"description\": null,\n      \"id\": \"FromKafka\"\n    },\n    \"status\": \"Started\"\n  },\n  {\n    \"id\": \"kafkaStartWithPartitioner\",\n    \"group\": \"kafka-route-group\",\n    \"uptime\": \"9.735 seconds\",\n    \"uptimeMillis\": 9735,\n    \"properties\": {\n      \"parent\": \"f10d3e4\",\n      \"rest\": \"false\",\n      \"description\": null,\n      \"id\": \"kafkaStartWithPartitioner\",\n      \"group\": \"kafka-route-group\"\n    },\n    \"status\": \"Started\"\n  },\n  {\n    \"id\": \"route2\",\n    \"uptime\": \"9.734 seconds\",\n    \"uptimeMillis\": 9734,\n    \"properties\": {\n      \"parent\": \"38e4f7b\",\n      \"rest\": \"false\",\n      \"description\": null,\n      \"id\": \"route2\"\n    },\n    \"status\": \"Started\"\n  },\n  {\n    \"id\": \"hello\",\n    \"group\": \"hello-group\",\n    \"uptime\": \"9.734 seconds\",\n    \"uptimeMillis\": 9734,\n    \"properties\": {\n      \"parent\": \"1915ce41\",\n      \"rest\": \"false\",\n      \"description\": null,\n      \"id\": \"hello\",\n      \"group\": \"hello-group\"\n    },\n    \"status\": \"Started\"\n  }\n]\n```\n\n```bash\ncurl -XGET -s http://localhost:4001/actuator/camelroutes/{id}/detail\n```\n**NOTE**:  _id = route1_ \n```json\n{\n  \"id\": \"route1\",\n  \"uptime\": \"2 minutes\",\n  \"uptimeMillis\": 164554,\n  \"properties\": {\n    \"parent\": \"64beb2b7\",\n    \"rest\": \"false\",\n    \"description\": null,\n    \"id\": \"route1\"\n  },\n  \"status\": \"Started\",\n  \"details\": {\n    \"deltaProcessingTime\": -4,\n    \"exchangesInflight\": 0,\n    \"exchangesTotal\": 82,\n    \"externalRedeliveries\": 0,\n    \"failuresHandled\": 0,\n    \"firstExchangeCompletedExchangeId\": \"ID-APMGJGH67551C6-1554502425015-0-3\",\n    \"firstExchangeCompletedTimestamp\": \"2019-04-05T22:13:49.139+0000\",\n    \"lastExchangeCompletedExchangeId\": \"ID-APMGJGH67551C6-1554502425015-0-590\",\n    \"lastExchangeCompletedTimestamp\": \"2019-04-05T22:16:30.879+0000\",\n    \"lastProcessingTime\": 10,\n    \"maxProcessingTime\": 408,\n    \"meanProcessingTime\": 21,\n    \"minProcessingTime\": 8,\n    \"redeliveries\": 0,\n    \"totalProcessingTime\": 1769,\n    \"hasRouteController\": false\n  }\n}\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmgorav%2Fapache-camel-aws-eis","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmgorav%2Fapache-camel-aws-eis","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmgorav%2Fapache-camel-aws-eis/lists"}