{"id":26158684,"url":"https://github.com/jvalue/outboxer-postgres2rabbitmq","last_synced_at":"2025-04-14T10:13:03.613Z","repository":{"id":54459108,"uuid":"329554976","full_name":"jvalue/outboxer-postgres2rabbitmq","owner":"jvalue","description":"Transactional Outbox Pattern for Microservices using Postgres and RabbitMQ","archived":false,"fork":false,"pushed_at":"2021-02-16T15:41:48.000Z","size":146,"stargazers_count":4,"open_issues_count":2,"forks_count":4,"subscribers_count":5,"default_branch":"main","last_synced_at":"2025-04-14T10:12:59.442Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jvalue.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-01-14T08:38:13.000Z","updated_at":"2023-12-13T19:33:18.000Z","dependencies_parsed_at":"2022-08-13T16:20:17.506Z","dependency_job_id":null,"html_url":"https://github.com/jvalue/outboxer-postgres2rabbitmq","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jvalue%2Foutboxer-postgres2rabbitmq","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jvalue%2Foutboxer-postgres2rabbitmq/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jvalue%2Foutboxer-postgres2rabbitmq/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jvalue%2Foutboxer-postgres2rabbitmq/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jvalue","download_url":"https://codeload.github.com/jvalue/outboxer-postgres2rabbitmq/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248860168,"owners_count":21173342,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-03-11T10:59:19.798Z","updated_at":"2025-04-14T10:13:03.583Z","avatar_url":"https://github.com/jvalue.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# outboxer-postgres2rabbitmq\n\nThis service allows to implement the [transactional outbox pattern](https://microservices.io/patterns/data/transactional-outbox.html).\nIt reads the changes from an outbox table in a PostgreSQL database using [debezium](https://debezium.io) and publishes \nthe events to RabbitMQ.\n\n## Usage\nOutboxer runs inside a docker container. You can build the docker image from the `Dockerfile` located in the root \ndirectory.\n\nAs a running PostgreSQL and RabbitMQ instance is needed, it is best to use docker-compose. You can find a reference \n`docker-compose.yml` file in the `integration-test` folder.\n\n## Configuration\nOutboxer can either be configured via a `outboxer.properties` file or via environment variables. The following \nproperties are supported:\n\n- `database.hostname` The database host name\n- `database.port` The database port\n- `database.user` The database username\n- `database.password` The database password\n- `database.dbname` The database name\n- `database.server.name` The unique name that identifies this debezium PostgreSQL connector\n- `table.include.list` The name of the outbox table\n- `offset.storage.file.filename` The file where outboxer should store its current offset. The offset is a location in \n  Postgres write-ahead log. Debezium uses this offset to track which records have been processed. After a restart \n  outboxer uses the stored offset as the resume point where to start processing records. If outboxer runs inside a \n  container, this file should be storage outside the container (for example in a docker volume). This ensures that the\n  file does survive container restarts.\n- `publisher.amqp.url` The url to the AMQP broker\n- `publisher.amqp.exchange` The AMQP exchange on which the events should be published.\n- `publisher.amqp.retries` How often a failed event publication should be retried\n- `publisher.amqp.retry.delay` The delay before performing retry\n- `transforms.outbox.table.field.event.id` The name of the column containing the unique event id\n- `transforms.outbox.table.field.event.routing_key` The name of the column containing the event routing key\n- `transforms.outbox.table.field.event.payload` The name of the column containing the event payload\n\nNote: all properties can also be passed via environment variables, but they must be prefixed with `outboxer.` and can be\nwritten in `UPPER_UNDERSCORE` format (e.g. `OUTBOXER_DATABASE_HOSTNAME`).\n\n## Architecture\nThe outboxer service consists of three main components:\n- `Outboxer` This is the main component which clues everything together. It is responsible for configuring, starting and\n  stopping the DebeziumEngine.\n- `AmqpPublisher` This a `DebeziumEngine.ChangeConsumer` that gets called by the DebeziumEngine to handle the change \n  records. From each change record it will create an AMQP message and publishes it to the configured AMQP exchange.\n- `OutboxTableTransform` This transformation extracts the unique event id, the routing key, and the payload from the raw\n  change records. It does also discard update and delete change records because once an event is published (e.g. added \n  to the outbox table) it should neither be changed nor deleted.\n\n## Resilience\nOutboxer is very resilience to any kind of failure. The offset in the write-ahead log of the last processed event is \nstored in a file (see `offset.storage.file.filename` property). Outboxer does restore the current offset from this file\nafter a restart and will only process events that have happened after this offset. Because a new offset can not be\nwritten to disk atomically after the event has been processed, outboxer can only provide at least once delivery for the\nevents. Therefore, idempotent event consumer should be used, so duplicate events do not cause any issues.\n\nIn the following a few common failure scenarios are described in more detail:\n- **PostgreSQL becomes unavailable**: If the connection the database is broken, debezium will determine the error kind. \n  If it is a retriable error, debezium will automatically wait some time (configurable with the \n  `retriable.restart.connector.wait.ms` property) before restarting the database connector. If the restart fails, or a\n  non retriable error was thrown initially outboxer will terminate. The operator is then responsible to resolve the\n  issue and start outboxer again.\n- **AMQP broker becomes unavailable**: If the publication of an event is failing, retries are performed as configured\n  with the `publisher.amqp.retries` and `publisher.amqp.retry.delay` properties. If all retries are failing, outboxer\n  will terminate. Then the operator is responsible to resolve the issue and restart outboxer.\n- **Outboxer crashes**: If outboxer is restarted after a crash, it will read the stored offset and will start processing\n  events that have happened after the current offset.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjvalue%2Foutboxer-postgres2rabbitmq","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjvalue%2Foutboxer-postgres2rabbitmq","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjvalue%2Foutboxer-postgres2rabbitmq/lists"}