{"id":24874595,"url":"https://github.com/fejesa/camel-and-testcontainers","last_synced_at":"2026-03-17T05:57:39.574Z","repository":{"id":274941170,"uuid":"895141226","full_name":"fejesa/camel-and-testcontainers","owner":"fejesa","description":"Apache Camel integration with Testcontainers for testing","archived":false,"fork":false,"pushed_at":"2025-02-21T06:59:09.000Z","size":56,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-13T11:09:48.970Z","etag":null,"topics":["camel","java","testcontainers"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/fejesa.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-11-27T16:24:06.000Z","updated_at":"2025-02-21T06:59:12.000Z","dependencies_parsed_at":"2025-01-30T09:39:25.221Z","dependency_job_id":null,"html_url":"https://github.com/fejesa/camel-and-testcontainers","commit_stats":null,"previous_names":["fejesa/camel-and-testcontainers"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fejesa%2Fcamel-and-testcontainers","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fejesa%2Fcamel-and-testcontainers/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fejesa%2Fcamel-and-testcontainers/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fejesa%2Fcamel-and-testcontainers/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/fejesa","download_url":"https://codeload.github.com/fejesa/camel-and-testcontainers/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248703199,"owners_count":21148118,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["camel","java","testcontainers"],"created_at":"2025-02-01T07:27:55.153Z","updated_at":"2026-03-17T05:57:34.549Z","avatar_url":"https://github.com/fejesa.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Apache Camel with Testcontainers\n## Introduction\n[Apache Camel](https://camel.apache.org/) is a robust integration framework built on well-known Enterprise Integration Patterns (EIPs). It offers a straightforward API to connect diverse systems and applications.\n\nTesting integration routes is vital to ensure seamless functionality, but it often involves external dependencies like databases, web services, or message brokers, making it challenging. [Testcontainers](https://testcontainers.com/) addresses this problem by providing lightweight, disposable [Docker](https://www.docker.com/) containers for such dependencies.\n\nThis project demonstrates how to combine Apache Camel with Testcontainers to test integration routes effectively.\n\n## Use Case\nImagine a messaging platform for universities to communicate with applicants. Messages are stored in a database, and at the end of each year, they must be archived. This process involves:\n\n* **Reading records** from the active database.\n* **Transforming data** as needed.\n* **Storing the transformed data** into archive database.\n\nThe integration route implementing this process needs thorough testing to ensure reliability. Using **Apache Camel** and **Testcontainers**, we can achieve this seamlessly.\n\n## Architecture\nThis application operates as a command-line tool, executable manually or via a scheduled cron job. The workflow involves the following steps:\n\n1. The [Timer Component](https://camel.apache.org/components/4.8.x/timer-component.html) triggers the process.\n2. Data is extracted from the source database using the [JDBC component](https://camel.apache.org/components/4.8.x/jdbc-component.html).\n3. Extracted data is transformed or extended.\n4. Transformed data is loaded into the target database using JDBC component.\n\n![Route components](docs/camel-route.png)\n\nOnce the process completes, the application exits. Both the source and target databases use [PostgreSQL](https://www.postgresql.org/).\n\n## Implementation\nFor database interaction, the **Camel JDBC** component is used. This component sends SQL queries in the message body to the source database, processes the results, and returns them in the message body.\n\nHere’s a simplified route implementation for the archiving process:\n```java\nfrom(\"timer://load-applicant\")\n    .routeId(\"applicant-route\")\n    .setBody()\n        .simple(getSqlTemplate(\"source-query.sql\"))\n    .to(\"jdbc:source\")\n    .split(body())\n    .process(this::transform)\n        .setHeaders(getJdbcParameters(\"target-query.sql\"))\n        .setBody(constant(getSqlTemplate(\"target-query.sql\")))\n    .to(\"jdbc:target\");\n```\n* **Timer**: Triggers the process.\n* **JDBC Source**: Reads data from the source database.\n* **Split**: Breaks out the list of records into a series of individual ones, each containing data related to one item; it uses the [Split EIP](https://camel.apache.org/components/4.8.x/eips/split-eip.html).\n* **Transform**: Processes and modifies the data; it uses the [Message Translator EIP](https://camel.apache.org/components/4.8.x/eips/message-translator.html).\n* **JDBC Target**: Writes data to the target database.\n\n**Note**: SQL commands can include raw SQL or parameterized SQL templates.\n\nHow the data is exchanged between components is crucial. The Camel API uses Message Exchange Patterns (MEP) to define messaging styles, such as one-way or request-reply.\nThe Camel Exchange API provides methods to access messages:\n* `getIn()` retrieves the IN message.\n* `getOut()` retrieves the OUT message (if applicable).\n\nIn our case the message is a list of records, which is then split and processed individually, and the exchange of a record looks like this:\n```java\nvoid transform(Exchange exchange) {\n    Map\u003cString, Object\u003e sourceData = exchange.getIn().getBody(Map.class);\n    sourceData.put(\"archiving_time\", LocalDateTime.now());\n    sourceData.put(\"year\", year);\n}\n```\nwhere the `sourceData` is a map of the read fields, and we extend it with additional fields like `archiving_time` and `year`.\n\n## Quarkus Integration\nThis project leverages [Quarkus](https://quarkus.io/), which simplifies integration with Apache Camel. \n\n## Testing\nTesting is achieved using `QuarkusTestResource`, with implementations wrapping [PostgreSQLContainer](https://java.testcontainers.org/modules/databases/postgres/) from Testcontainers. The container’s lifecycle is managed automatically. For example the source database container creation and initialization is done as follows:\n```java\npublic PostgresSourceDatabaseTestResource() {\n    var dockerImageName = DockerImageName.parse(\"postgres\").withTag(\"15.0\");\n    container = new PostgreSQLContainer\u003c\u003e(dockerImageName)\n        .withExposedPorts(getDbPort())\n        .withDatabaseName(getDbName())\n        .withUsername(getDbUser())\n        .withPassword(getDbPassword())\n        .withClasspathResourceMapping(\"init-source-db.sql\", \"/docker-entrypoint-initdb.d/init-source-db.sql\", BindMode.READ_ONLY)\n        .withLogConsumer(new Slf4jLogConsumer(logger)).waitingFor(Wait.forListeningPort());\n}\n\npublic Map\u003cString, String\u003e start() {\n    container.start();\n    return Map.of(\"quarkus.datasource.source.jdbc.url\", container.getJdbcUrl(), \"timer.period\", \"100\", \"timer.delay\", \"0\");\n}\n\npublic void stop() {\n    if (container != null) {\n        container.stop();\n    }\n}\n```\nBy annotating any test class within the module with `@QuarkusTestResource(PostgresSourceDatabaseTestResource.class)`, the container starts automatically before tests execute. The same approach is used for the target database. For example testing the database connections:\n```java\n@QuarkusTest\n@QuarkusTestResource(PostgresSourceDatabaseTestResource.class)\n@QuarkusTestResource(PostgresTargetDatabaseTestResource.class)\nclass DatabaseConnectionTesterTest {\n\n    @Inject\n    DatabaseConnectionTester databaseConnectionTester;\n\n    @Test\n    void checkConnectionTestRoutesEndpoint() {\n        var routeDefinitions = databaseConnectionTester.getRouteCollection().getRoutes();\n        assertThat(routeDefinitions.stream()\n            .map(RouteDefinition::getEndpointUrl)\n            .anyMatch(\"timer:source-database-test?delay=-1\u0026repeatCount=1\"::equals)).isTrue();\n        assertThat(routeDefinitions.stream()\n            .map(RouteDefinition::getEndpointUrl)\n            .anyMatch(\"timer:target-database-test?delay=-1\u0026repeatCount=1\"::equals)).isTrue();\n    }\n}\n```\nwhere both the source and target databases were started automatically before the test execution, and connections to them were tested.\n\n## Configuration\nDatabases connection properties are set in the `application.properties` file.\n\n## Building the project\n### Prerequisites\n- Java 21\n- Maven 3\n- Docker\n\n### Build and Test\nRun the following command to build the project and execute tests:\n```shell\nmvn clean verify\n```\n\n# Limitations\nCurrent implementation does not support batch processing, and it is executed in a single transaction.\nThis can lead not to load all data if the data set is too large. To mitigate this, the data set can be split into smaller chunks. For example the message result data can be split by the ID:\n```sql\nSELECT m.id, m.a_id, a.app_id, d.code d_code, u.code u_code, m.subject, m.text_msg,\n       m.read_time, m.creation_time\nFROM APPLICANT_MESSAGE m, APPLICANT a, UNIVERSITY u, DEPARTMENT d\nWHERE a.id = m.a_id AND a.u_id = u.id AND a.d_id = d.id\nORDER BY m.id\nLIMIT 10000 OFFSET 0\n```\nIncrease the `OFFSET` incrementally (e.g., by 10,000) until all data is processed.\n\nThis project demonstrates a practical approach to testing and implementing integration routes using Apache Camel and Testcontainers, offering scalability and reliability for modern integration needs.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffejesa%2Fcamel-and-testcontainers","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffejesa%2Fcamel-and-testcontainers","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffejesa%2Fcamel-and-testcontainers/lists"}