{"id":15019205,"url":"https://github.com/lorenzoranucci/tor","last_synced_at":"2025-10-24T07:30:47.634Z","repository":{"id":63177155,"uuid":"564777084","full_name":"lorenzoranucci/tor","owner":"lorenzoranucci","description":"A stateless app to read events from MySQL binlog and write on Kafka using the Transactional Outbox Pattern.","archived":false,"fork":false,"pushed_at":"2022-12-19T22:27:31.000Z","size":204,"stargazers_count":11,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-01-31T00:25:27.804Z","etag":null,"topics":["binlog","cdc","change-data-capture","cqrs","debezium","event-driven","event-driven-architecture","kafka","microservice","mysql","outbox","outbox-pattern","redis","transactional-outbox","transactional-outbox-pattern"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lorenzoranucci.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-11-11T13:25:51.000Z","updated_at":"2024-01-12T18:46:12.000Z","dependencies_parsed_at":"2023-01-29T23:01:33.845Z","dependency_job_id":null,"html_url":"https://github.com/lorenzoranucci/tor","commit_stats":null,"previous_names":[],"tags_count":15,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lorenzoranucci%2Ftor","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lorenzoranucci%2Ftor/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lorenzoranucci%2Ftor/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lorenzoranucci%2Ftor/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lorenzoranucci","download_url":"https://codeload.github.com/lorenzoranucci/tor/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":237932055,"owners_count":19389560,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["binlog","cdc","change-data-capture","cqrs","debezium","event-driven","event-driven-architecture","kafka","microservice","mysql","outbox","outbox-pattern","redis","transactional-outbox","transactional-outbox-pattern"],"created_at":"2024-09-24T19:53:09.653Z","updated_at":"2025-10-24T07:30:42.265Z","avatar_url":"https://github.com/lorenzoranucci.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Tor\n\nTor (Transactional Outbox Router) it's a CDC (Change Data Capture) service that reads events\nfrom an outbox table using the MySQL binlog and writes them on Kafka preserving order.\n\nIt avoids reprocessing the same binlog entries on restart, persisting state on Redis.\n\nTor is inspired by\n[Debezium Outbox Event Router](https://debezium.io/documentation/reference/1.9/transformations/outbox-event-router.html)\nand [airbnb/SpinalTap](https://github.com/airbnb/SpinalTap).\nIt is designed to be lightweight, simple, modular and easy to install as a container.\n\n## Use case\n\nYou have a service that persists state changes on a database (MySQL) and you want other services that do not share\nthe same process/memory to be notified of the state change so they can react accordingly.\nThis pattern is known as [**Pub/Sub**](https://en.wikipedia.org/wiki/Publish%E2%80%93subscribe_pattern), where\nnotifications are called messages or events, those who publish them are called Publishers or Dispatchers, and those who\nconsume the messages are known as Subscribers or Consumers.\n\nGuarantees:\n\n- Message publishing must be **eventually consistent** with respect to the state change itself, and no\n  change is lost.\n- Messages must be delivered in the **same order** that the state changes occurred.\n  Delivering messages to a log-based stream broker (e.g., Kafka), ensures subscribers to process them in\n  order and it avoids concurrency issues (by partition).\n- Messages may be **duplicated** and processing by subscribers must be **idempotent**.\n\nExamples:\n\n- You want to build a **microservice** system that communicates in an **event-driven** (asynchronous) manner,\n  and you want a reliable and consistent communication system that avoids **out-of-order messages** and message loss\n  problems due to **dual-writes** .\n- You want to implement **CQRS** (or simply cache and denormalization) by building read-models asynchronously\n  and consistently.\n- You want to **decouple** your monolithic code so that you communicate through a message-broker instead of through\n  procedure calls, databases, or RPCs.\n\nIf you want to go deep on the topic we recommend reading\nthis [article](https://martin.kleppmann.com/2015/05/27/logs-for-data-infrastructure.html) by Martin Kleppmann or his\namazing book [Designing\nData-Intensive Applications](https://www.amazon.com/Designing-Data-Intensive-Applications-Reliable-Maintainable/dp/1449373321)\n.\n\n## Modules\n\nTor is composed of several modules so it can be extensible and make dependencies footprint minimal.\n\n- `router`: contains the core of Tor. It is based\n  on  [github.com/go-mysql-org/go-mysql](https://github.com/go-mysql-org/go-mysql).\n- `adapters`: contains the adapters with which `router` can be built to run a tor app.\n    - `kafka`: an event dispatcher for Kafka.\n    - `redis`: a state handler for Redis.\n- `example`: contains examples of tor apps.\n    - `tor`: an example instance of `router` app using `kafka` and `redis` adapters.\n    - `api-server`: an example api-server implementing a business logic, persisting state and producing events.\n- `.devenv`: developing and running examples locally.\n  It uses Go Workspaces, so every change applied to a module is reflected automatically without the need of\n  using `replace` or pseudo-versions.\n\n## Run example\n\nSet up the system:\n```shell\nmake up\n```\n\nInvoke api-server API:\n```shell\ncurl --request PUT 'localhost:8080/29fd8c5b-6b44-48b6-98a3-c1c601eaae26'\n\ncurl --request PUT 'localhost:8080/5c0e9be6-ea69-452d-9824-7f55b544f2e3'\n\ncurl --request DELETE 'localhost:8080/29fd8c5b-6b44-48b6-98a3-c1c601eaae26'\n```\n\nUse [plumber](https://github.com/batchcorp/plumber) or similar to see events written in the Kafka topic:\n\n```shell\nplumber read kafka --address=localhost:9093 --topics=outbox_topic -f\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Florenzoranucci%2Ftor","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Florenzoranucci%2Ftor","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Florenzoranucci%2Ftor/lists"}