{"id":13577477,"url":"https://github.com/eugene-khyst/postgresql-event-sourcing","last_synced_at":"2025-05-16T06:00:22.817Z","repository":{"id":40678677,"uuid":"361487149","full_name":"eugene-khyst/postgresql-event-sourcing","owner":"eugene-khyst","description":"A reference implementation of an event-sourced system that uses PostgreSQL as an event store built with Spring Boot. Fork the repository and use it as a template for your projects. Or clone the repository and run end-to-end tests to see how everything works together.","archived":false,"fork":false,"pushed_at":"2024-06-10T09:26:10.000Z","size":1684,"stargazers_count":1173,"open_issues_count":4,"forks_count":105,"subscribers_count":14,"default_branch":"main","last_synced_at":"2025-04-08T15:11:31.348Z","etag":null,"topics":["cqrs","ddd","event-driven","event-driven-architecture","event-sourcing","event-store","eventsourcing","postgres","postgres-notifications","postgresql","solution-architecture"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/eugene-khyst.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-04-25T16:59:29.000Z","updated_at":"2025-04-06T09:53:14.000Z","dependencies_parsed_at":"2024-09-25T00:19:15.667Z","dependency_job_id":null,"html_url":"https://github.com/eugene-khyst/postgresql-event-sourcing","commit_stats":null,"previous_names":["eugene-khyst/postgresql-event-sourcing"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eugene-khyst%2Fpostgresql-event-sourcing","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eugene-khyst%2Fpostgresql-event-sourcing/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eugene-khyst%2Fpostgresql-event-sourcing/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eugene-khyst%2Fpostgresql-event-sourcing/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/eugene-khyst","download_url":"https://codeload.github.com/eugene-khyst/postgresql-event-sourcing/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254478160,"owners_count":22077675,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cqrs","ddd","event-driven","event-driven-architecture","event-sourcing","event-store","eventsourcing","postgres","postgres-notifications","postgresql","solution-architecture"],"created_at":"2024-08-01T15:01:21.806Z","updated_at":"2025-05-16T06:00:20.460Z","avatar_url":"https://github.com/eugene-khyst.png","language":"Java","funding_links":[],"categories":["Java","开发框架","postgresql","\u003ca name=\"Java\"\u003e\u003c/a\u003eJava"],"sub_categories":[],"readme":"# \u003ca id=\"0\"\u003e\u003c/a\u003eEvent Sourcing with PostgreSQL\n\n- [Introduction](#1)\n- [Example domain](#2)\n- [Event sourcing and CQRS basics](#3)\n  - [State-oriented persistence](#3-1)\n  - [Event sourcing](#3-2)\n  - [Snapshotting](#3-3)\n  - [Querying the data](#3-4)\n  - [CQRS](#3-5)\n  - [Event handlers](#3-6)\n  - [Domain events vs Integration events](#3-7)\n  - [Advantages of CQRS](#3-8)\n  - [Advantages of event sourcing](#3-9)\n- [Solution architecture](#4)\n  - [Component diagram](#4-1)\n  - [ER diagram](#4-2)\n  - [Optimistic concurrency control](#4-3)\n  - [Snapshotting](#4-4)\n  - [Loading any revision of the aggregate](#4-5)\n  - [Synchronously updating projections](#4-6)\n  - [Asynchronously sending integration events to a message broker](#4-7)\n    - [Transactional outbox using transaction ID](#4-7-1)\n    - [Transactional outbox using table-level lock](#4-7-2)\n    - [Database polling](#4-7-3)\n    - [Listen/Notify as an alternative to database polling](#4-7-4)\n  - [Adding new asynchronous event handlers](#4-8)\n  - [Drawbacks](#4-9)\n- [Project structure](#5)\n  - [Gradle subprojects](#5-1)\n  - [Database schema migrations](#5-2)\n  - [Class diagrams](#5-3)\n    - [Class diagram of the domain model](#5-3-1)\n    - [Class diagram of the projections](#5-3-2)\n    - [Class diagram of the service layer](#5-3-3)\n- [How to adapt it to your domain?](#6)\n- [How to run the sample?](#7)\n\n\u003c!-- Table of contents is made with https://github.com/eugene-khyst/md-toc-cli --\u003e\n\n## \u003ca id=\"1\"\u003e\u003c/a\u003eIntroduction\n\nUsually, our applications operate with the current state of a domain object.\nBut sometimes, we need to know the entire history of the domain object changes.\nFor example, we want to know how an order got into its current state.\n\nThe audit trail (also called the audit log) is a chronological record of the history\nand details of the actions that affected the system.\nAn audit trail may be a regulatory or business requirement.\n\nWe can store all changes to the domain object state as a sequence of events in an append-only event stream.\nThus, event streams will contain an entire history of changes.\nBut how can we be sure that this history is authentic and error-free?\nWe can use event streams as a primary source of truth in a system.\nTo get the current state of an object, we have to replay all events in the order of occurrence.\nThis pattern is called event sourcing. The database for storing event streams is called an event store.\nEvent sourcing provides a complete and accurate record of all changes made to a system.\nEvent sourcing is an industry standard for implementing audit trail.\n\nThere are specialized databases for event sourcing.\nDeveloper Advocates working for the companies behind these specialized databases said\nyou shouldn't implement event sourcing with traditional relational or document-oriented databases.\nIs this true or just a marketing ploy?\n\nSpecialized databases for event sourcing are convenient and provide the necessary functionality out of the box.\nBut PostgreSQL, the world's most advanced open-source database, is also suitable for event sourcing.\nYou can use PostgreSQL as an event store without additional frameworks or extensions\ninstead of setting up and maintaining a separate specialized database for event sourcing.\n\nThis repository provides a reference implementation of an event-sourced system \nthat uses PostgreSQL as an event store built with Spring Boot.\n[Fork](https://github.com/eugene-khyst/postgresql-event-sourcing/fork) the repository and use it as a template for your projects.\nOr clone the repository and run end-to-end tests to see how everything works together.\n\n![PostgreSQL Logo](img/potgresql-logo.png)\n\nSee also\n\n* [Event Sourcing with EventStoreDB](https://github.com/eugene-khyst/eventstoredb-event-sourcing)\n* [Event Sourcing with Kafka and ksqlDB](https://github.com/eugene-khyst/ksqldb-event-souring)\n\n## \u003ca id=\"2\"\u003e\u003c/a\u003eExample domain\n\nThis sample uses a simplified domain model of the ride-hailing system.\n\n* A rider can place an order for a ride along a route specifying a price.\n* A rider can edit an order price to pay more instead of waiting in cases of very high demand.\n* A driver can accept an order.\n* A driver can complete previously accepted order.\n* An order can be canceled before completion.\n\n![Domain use case diagram](img/domain-1.svg)\n\n![Domain state diagram](img/domain-2.svg)\n\n## \u003ca id=\"3\"\u003e\u003c/a\u003eEvent sourcing and CQRS basics\n\n### \u003ca id=\"3-1\"\u003e\u003c/a\u003eState-oriented persistence\n\nState-oriented persistence (CRUD) applications store only the latest version of an entity.\nDatabase records present entities.\nWhen an entity is updated, the corresponding database record gets updated too.\nSQL `INSERT`, `UPDATE` and `DELETE` statements are used.\n\n![State-oriented persistence](img/state-oriented-persistence.svg)\n\n### \u003ca id=\"3-2\"\u003e\u003c/a\u003eEvent sourcing\n\nEvent sourcing applications persist the state of an entity as a sequence of immutable state-changing events.\n\n![Event sourcing](img/event-sourcing-1.svg)\n\nWhenever the state of an entity changes, a new event is appended to the list of events.\nOnly SQL `INSERT` statements are used.\nEvents are immutables, so SQL `UPDATE` and `DELETE` statements are not used.\n\n![Event sourcing](img/event-sourcing-2.svg)\n\nThe current state of an entity can be restored by replaying all its events.\n\nEvent sourcing is closely related to domain-driven design (DDD) and shares some terminology.\n\nAn entity in event sourcing is called an **aggregate**.\n\nA sequence of events for the same aggregate is called a **stream**.\n\nEvent sourcing is best suited for short-living entities with a small total number of\nevents (e.g., orders).\n\nRestoring the state of the short-living entity by replaying all its events doesn't have any\nperformance impact. Thus, no optimizations for restoring state are required for short-living\nentities.\n\nFor endlessly stored entities (e.g., users, bank accounts) with thousands of events restoring state\nby replaying all events is not optimal, and snapshotting should be considered.\n\n### \u003ca id=\"3-3\"\u003e\u003c/a\u003eSnapshotting\n\nSnapshotting is an optimization technique where a snapshot of the aggregate's state is also saved,\nso an application can restore the current state of the aggregate from the snapshot rather than from\nall the events (potentially thousands).\n\nOn every *nth* event, make an aggregate snapshot by storing an aggregate state and its version.\n\nTo restore an aggregate state:\n\n1. first read the latest snapshot,\n2. then read events forward from the original stream starting from the version pointed by the snapshot.\n\n![Snapshotting in event souring](img/event-sourcing-snapshotting.svg)\n\n### \u003ca id=\"3-4\"\u003e\u003c/a\u003eQuerying the data\n\nIt's easy to find an aggregate by ID, but other queries are difficult.\nSince aggregates are stored as append-only lists of immutable events,\nquerying the data using SQL, as we used to, is impossible.\nTo find an aggregate by some field, we need to first read all the events and replay them to restore all the aggregates.\n\nTo bring back all the querying power a relational database has to offer,\nwe can create a dedicated read model derived from the event stream.\n\nThe event stream is the write model and the primary source of truth.\n\nThe read model is a \"denormalized\" view of the write model, allowing faster and more convenient querying.\nRead models are projections of the system state.\nTherefore, read models are also known as **projections**.\n\nProjections provide a view of data for a single aggregate type\nor perform aggregations and combine data from multiple aggregate types.\n\nThat's where CQRS comes in handy.\n\n### \u003ca id=\"3-5\"\u003e\u003c/a\u003eCQRS\n\nCommand-query responsibility segregation (CQRS) stands for segregating the responsibility between\ncommands (write requests) and queries (read requests).\nThe write requests and the read requests are processed by different handlers.\n\nA command generates zero or more events or results in an error.\n\n![CQRS](img/cqrs-1.svg)\n\nCQRS is a self-sufficient architectural pattern and doesn't require event sourcing.\nBut in practice, event sourcing is usually used in conjunction with CQRS.\nEvent store is used as a write database, and SQL or NoSQL database as a read database.\n\n![CQRS with event sourcing](img/cqrs-2.svg)\n\n### \u003ca id=\"3-6\"\u003e\u003c/a\u003eEvent handlers\n\nCommands generate events.\nEvent processing is done by **event handlers**.\nAs a part of event processing, we may need to update projections,\nsend a message to a message broker, or make an API call.\n\nThere are two types of event handlers: **synchronous** and **asynchronous**.\n\nStoring the write model and read model in the same database allows for transactional updates of the read model.\nEach time we append a new event, the projection is updated **synchronously** in the same transaction.\nThe projection is **consistent** with the event stream.\n\nWhen an event handler communicates with an external system or middleware (e.g., sends a message to Kafka),\nit should run **asynchronously** after the transaction updating the write model.\nAsynchronous execution leads to **eventual consistency**.\n\nCommunication with external systems should not occur in the same transaction updating the write model.\nThe external system call may succeed, but the transaction will later be rolled back, resulting in an inconsistency.\n\nAnyway, distributed systems should be designed with eventual consistency in mind.\n\n### \u003ca id=\"3-7\"\u003e\u003c/a\u003eDomain events vs Integration events\n\nEvents in event sourcing are **domain events**.\nThe domain event is a part of a bounded context and should not be used \"as-is\" for integration with other bounded\ncontexts.\n\nFor communication between bounded contexts **integration events** are used.\nThe integration event represents the current state of an aggregate, not just changes to the aggregate as a domain event.\n\n### \u003ca id=\"3-8\"\u003e\u003c/a\u003eAdvantages of CQRS\n\n* Independent scaling of the read and write databases.\n* Optimized data schema for the read database (e.g. the read databases can be denormalized).\n* Simpler queries (e.g. complex `JOIN` operations can be avoided).\n\n### \u003ca id=\"3-9\"\u003e\u003c/a\u003eAdvantages of event sourcing\n\n* A true history of the system (audit and traceability).\n  An industry standard for implementing audit trail.\n* Ability to put the system in any prior state (e.g. for debugging).\n* New read-side projections can be created as needed (later) from events.\n  It allows responding to future needs and new requirements.\n\n## \u003ca id=\"4\"\u003e\u003c/a\u003eSolution architecture\n\nPostgreSQL can be used as an event store.\nIt will natively support appending events, concurrency control and reading events.\nSubscribing on events requires additional implementation.\n\n### \u003ca id=\"4-1\"\u003e\u003c/a\u003eComponent diagram\n\n![PostgreSQL event store component diagram](img/postgresql-event-sourcing.svg)\n\n### \u003ca id=\"4-2\"\u003e\u003c/a\u003eER diagram\n\n![PostgreSQL event store ER diagram](img/er-diagram.svg)\n\nEvents are stored in the `ES_EVENT` table.\n\n### \u003ca id=\"4-3\"\u003e\u003c/a\u003eOptimistic concurrency control\n\nLatest aggregate version is stored in the `ES_AGGREGATE` table.\nVersion checking is used for optimistic concurrency control.\nVersion checking uses version numbers to detect conflicting updates (and to prevent lost updates).\n\nAppending an event operation consists of 2 SQL statements in a single transaction:\n\n1. check the actual and expected version match and increment the version\n    ```sql\n    UPDATE ES_AGGREGATE\n       SET VERSION = :newVersion\n     WHERE ID = :aggregateId\n       AND VERSION = :expectedVersion\n    ```\n2. insert new event\n    ```sql\n    INSERT INTO ES_EVENT (TRANSACTION_ID, AGGREGATE_ID, VERSION, EVENT_TYPE, JSON_DATA)\n    VALUES(pg_current_xact_id(), :aggregateId, :version, :eventType, :jsonObj::json)\n    RETURNING ID, TRANSACTION_ID::text, EVENT_TYPE, JSON_DATA\n    ```\n   `pg_current_xact_id()` returns the current transaction's ID. The need for this will be explained later.\n\n![Optimistic Concurrency Control](img/event-sourcing-concurrency.svg)\n\n### \u003ca id=\"4-4\"\u003e\u003c/a\u003eSnapshotting\n\nOn every *nth* event insert an aggregate state (snapshot) to the `ES_AGGREGATE_SNAPSHOT` table specifying the version\n\n```sql\nINSERT INTO ES_AGGREGATE_SNAPSHOT (AGGREGATE_ID, VERSION, JSON_DATA)\nVALUES (:aggregateId, :version, :jsonObj::json)\n```\n\nSnapshotting for an aggregate type can be disabled and configured in the [`application.yml`](src/main/resources/application.yml)\n\n```yaml\nevent-sourcing:\n  snapshotting:\n    # com.example.eventsourcing.domain.AggregateType\n    ORDER:\n      enabled: true\n      # Create a snapshot on every nth event\n      nth-event: 10\n```\n\n### \u003ca id=\"4-5\"\u003e\u003c/a\u003eLoading any revision of the aggregate\n\nTo restore any revision of the aggregate:\n\n1. first read the latest value of the snapshot\n    ```sql\n    SELECT a.AGGREGATE_TYPE,\n           s.JSON_DATA\n      FROM ES_AGGREGATE_SNAPSHOT s\n      JOIN ES_AGGREGATE a ON a.ID = s.AGGREGATE_ID\n     WHERE s.AGGREGATE_ID = :aggregateId\n       AND (:version IS NULL OR s.VERSION \u003c= :version)\n     ORDER BY s.VERSION DESC\n     LIMIT 1\n    ```\n2. then read forward from the event stream from the revision the snapshot points to\n    ```sql\n    SELECT ID,\n           TRANSACTION_ID::text,\n           EVENT_TYPE,\n           JSON_DATA\n      FROM ES_EVENT\n     WHERE AGGREGATE_ID = :aggregateId\n       AND (:fromVersion IS NULL OR VERSION \u003e :fromVersion)\n       AND (:toVersion IS NULL OR VERSION \u003c= :toVersion)\n     ORDER BY VERSION ASC\n    ```\n\n### \u003ca id=\"4-6\"\u003e\u003c/a\u003eSynchronously updating projections\n\nUsing PostgreSQL as an event store and a read database allows for transactional updates of the read model.\nEach time we append a new event, the projection is updated synchronously in the same transaction.\nIt's a big advantage because sometimes consistency is not so easy to achieve.\n\nYou can't get consistent projections when a separate database is used as an event store.\n\n### \u003ca id=\"4-7\"\u003e\u003c/a\u003eAsynchronously sending integration events to a message broker\n\nIntegration events should be sent asynchronously after the transaction updating the write model.\n\nPostgreSQL doesn't allow subscribing on changes, so the solution is a Transactional Outbox pattern.\nA service that uses a database inserts events into an outbox table as part of the local transaction.\nA separate process publishes the events inserted into database to a message broker.\n\n![Transactional outbox pattern](img/transactional-outbox-1.svg)\n\nWe may have multiple asynchronous event handlers or so-called subscriptions.\nThe subscription concept is required to keep track of delivered events separately for different event handlers.\nThe last event processed by the event handler (subscription) is stored in the separate table `ES_EVENT_SUBSCRIPTION`.\nNew events are processed by polling the `ES_EVENT` table.\n\n![Transactional outbox pattern with subscriptions](img/transactional-outbox-2.svg)\n\nSince multiple instances of this application can run in parallel,\nwe need to ensure that any processing only affects the event once.\nWe don't want more than one event handler instance to handle the same event.\n\nThis is achieved by acquiring locks on the rows of the `ES_EVENT_SUBSCRIPTION` table.\nWe lock the row (`SELECT FOR UPDATE`) of the currently processed subscription.\n\nTo not hang other backend instances, we want to skip already locked rows (`SELECT FOR UPDATE SKIP LOCKED`)\nand lock the \"next\" subscription.\nIt allows multiple backend instances to select different subscriptions that do not overlap.\nThis way, we improve availability and scalability.\n\nThe event subscription processor polls `ES_EVENT_SUBSCRIPTION` table every second (interval is configurable) for new events\nand processes them:\n\n1. read the last transaction ID and event ID processed by the subscription and acquire lock\n    ```sql\n    SELECT LAST_TRANSACTION_ID::text,\n           LAST_EVENT_ID\n      FROM ES_EVENT_SUBSCRIPTION\n     WHERE SUBSCRIPTION_NAME = :subscriptionName\n       FOR UPDATE SKIP LOCKED\n    ```\n2. read new events\n    ```sql\n    SELECT e.ID,\n           e.TRANSACTION_ID::text,\n           e.EVENT_TYPE,\n           e.JSON_DATA\n      FROM ES_EVENT e\n      JOIN ES_AGGREGATE a on a.ID = e.AGGREGATE_ID\n     WHERE a.AGGREGATE_TYPE = :aggregateType\n       AND (e.TRANSACTION_ID, e.ID) \u003e (:lastProcessedTransactionId::xid8, :lastProcessedEventId)\n       AND e.TRANSACTION_ID \u003c pg_snapshot_xmin(pg_current_snapshot())\n     ORDER BY e.TRANSACTION_ID ASC, e.ID ASC\n    ```\n    A comparison like `(a, b) \u003e (c, d)` is a row comparison and is equivalent to `a \u003e c OR (a = c AND b \u003e d)`.\n3. update the last transaction ID and event ID processed by the subscription\n    ```sql\n    UPDATE ES_EVENT_SUBSCRIPTION\n       SET LAST_TRANSACTION_ID = :lastProcessedTransactionId::xid8,\n           LAST_EVENT_ID = :lastProcessedEventId\n     WHERE SUBSCRIPTION_NAME = :subscriptionName\n    ```\n\n#### \u003ca id=\"4-7-1\"\u003e\u003c/a\u003eTransactional outbox using transaction ID\n\nUsing only the event ID to track events processed by the subscription is unreliable \nand can result in lost events.\n\nThe `ID` column of the `ES_EVENT` table is of type `BIGSERIAL`.\nIt's a notational convenience for creating ID columns having their default values assigned from a `SEQUENCE` generator.\n\nPostgreSQL sequences can't be rolled back.\n`SELECT nextval('ES_EVENT_ID_SEQ')` increments and returns the sequence value.\nEven if the transaction is not yet committed, the new sequence value becomes visible to other transactions. \n\nIf transaction #2 started after transaction #1 but committed first,\nthe event subscription processor can read the events created by transaction #2, update the last processed event ID,\nand thus lose the events created by transaction #1.\n\n![PostgreSQL naive transactional outbox](img/postgresql-naive-outbox.svg)\n\nWe use transaction ID with event ID to build a reliable PostgreSQL polling mechanism that doesn't lose events.\n\nEach event is supplemented with the current transaction ID.\n`pg_current_xact_id()` returns the current transaction's ID of type `xid8`.\n`xid8` values increase strictly monotonically and cannot be reused in the lifetime of a database cluster.\n\nThe latest event that is \"safe\" to process is right before the `xmin` of the current snapshot. \n`pg_current_snapshot()` returns a current snapshot, a data structure showing which transaction IDs are now in-progress.\n`pg_snapshot_xmin(pg_snapshot)` returns the `xmin` of a snapshot.\n`xmin` is the lowest transaction ID that was still active. \nAll transaction IDs less than `xmin` are either committed and visible, or rolled back.\n\nEven if transaction #2 started after transaction #1 and committed first,\nthe events it created won't be read by the event subscription processor until transaction #1 is committed.\n\n![PostgreSQL reliable transactional outbox using transaction ID](img/postgresql-reliable-outbox.svg)\n\n\u003e **NOTE**  \n\u003e The transaction ID solution is used by default as it is non-blocking.\n\n#### \u003ca id=\"4-7-2\"\u003e\u003c/a\u003eTransactional outbox using table-level lock\n\nWith the transaction ID solution, event subscription processor doesn't wait for in-progress transactions to complete.\nEvents created by already committed transactions will not be available for processing \nwhile transactions started earlier are still in-progress. \nThese events will be processed immediately after these earlier transactions have completed.\n\nAn alternative solution is to use PostgreSQL explicit locking to make event subscription processor wait for in-progress transactions.\n\nBefore reading new events, the event subscription processor\n* gets the most recently issued ID sequence number,\n* very briefly locks the table for writes to wait for all pending writes to complete.\n\nThe most recently issued `ES_EVENT_ID_SEQ` sequence number is obtained using the `pg_sequence_last_value` function:\n`SELECT pg_sequence_last_value('ES_EVENT_ID_SEQ')`.\n\nEvents are created with the command `INSERT INTO ES_EVENT...` that acquires the `ROW EXCLUSIVE` lock mode on `ES_EVENT` table.\n`ROW EXCLUSIVE (RowExclusiveLock)` lock mode is acquired by any command that modifies data in a table.\n\nThe command `LOCK ES_EVENT IN SHARE ROW EXCLUSIVE MODE` acquires the `SHARE ROW EXCLUSIVE` lock mode on `ES_EVENT` table.\n`SHARE ROW EXCLUSIVE (ShareRowExclusiveLock)` mode protects a table against concurrent data changes, \nand is self-exclusive so that only one session can hold it at a time. \n\n`SHARE ROW EXCLUSIVE` lock must be acquired in a separate transaction (`Propagation.REQUIRES_NEW`).\nThe transaction must contain only this command and commit quickly to release the lock, so writes can resume.\n\nWhen the lock is acquired and released, it means \nthat there are no more uncommitted writes with an ID less than or equal to the ID returned by `pg_sequence_last_value`.\n\n![PostgreSQL reliable transactional outbox using table-level lock](img/postgresql-reliable-outbox-with-lock.svg)\n\n#### \u003ca id=\"4-7-3\"\u003e\u003c/a\u003eDatabase polling\n\nTo get new events from the `ES_EVENT` table, the application has to poll the database.\nThe shorter the polling period, the shorter the delay between persisting a new event and processing it by the subscription. \nBut the lag is inevitable. If the polling period is 1 second, then the lag is at most 1 second.\n\nThe polling mechanism implementation [ScheduledEventSubscriptionProcessor](src/main/java/com/example/eventsourcing/service/ScheduledEventSubscriptionProcessor.java)\nuses a Spring annotation [@Scheduled](https://docs.spring.io/spring-framework/docs/current/javadoc-api/org/springframework/scheduling/annotation/Scheduled.html) to poll database with a fixed period.\n\nThe polling event subscription processing can be enabled and configured in the [`application.yml`](src/main/resources/application.yml)\n\n```yaml\nevent-sourcing:\n  subscriptions: polling # Enable database polling subscription processing\n  polling-subscriptions:\n    polling-initial-delay: PT1S\n    polling-interval: PT1S\n```\n\n#### \u003ca id=\"4-7-4\"\u003e\u003c/a\u003eListen/Notify as an alternative to database polling\n\nTo reduce the lag associated with database polling, the polling period can be set to a very low value, \nsuch as 1 second.\nBut this means that there will be 3600 database queries per hour and 86400 per day, even if there are no new events. \n\nPostgreSQL `LISTEN` and `NOTIFY` feature can be used instead of polling.\nThis mechanism allows for sending asynchronous notifications across database connections.\nNotifications are not sent directly from the application, \nbut via the database [trigger](src/main/resources/db/migration/V2__notify_trigger.sql) on a table.\n\nTo use this functionality an unshared [PgConnection](https://jdbc.postgresql.org/documentation/publicapi/org/postgresql/jdbc/PgConnection.html) \nwhich remains open is required.\nThe long-lived dedicated JDBC `Connection` for receiving notifications has to be created using the `DriverManager` API, \ninstead of getting from a pooled `DataSource`.\n\nPostgreSQL JDBC driver can't receive asynchronous notifications \nand must poll the backend to check if any notifications were issued. \nA timeout can be given to the poll function `getNotifications(int timeoutMillis)`, \nbut then the execution of statements from other threads will block.\nWhen `timeoutMillis` = 0, blocks forever or until at least one notification has been received. \nIt means that notification is delivered almost immediately, without a lag.\nIf more than one notification is about to be received, these will be returned in one batch.\n\nThis solution  significantly reduces the number of issued queries \nand also solves the lag problem that the polling solution suffers from.\n\nThe Listen/Notify mechanism implementation [PostgresChannelEventSubscriptionProcessor](src/main/java/com/example/eventsourcing/service/PostgresChannelEventSubscriptionProcessor.java)\nis inspired by the Spring Integration class [PostgresChannelMessageTableSubscriber](https://github.com/spring-projects/spring-integration/blob/v6.0.0/spring-integration-jdbc/src/main/java/org/springframework/integration/jdbc/channel/PostgresChannelMessageTableSubscriber.java).\n\nThe Listen/Notify event subscription processing can be enabled in the [`application.yml`](src/main/resources/application.yml)\n\n```yaml\nevent-sourcing:\n  subscriptions: postgres-channel # Enable Listen/Notify event subscription processing\n```\n\n\u003e **NOTE**  \n\u003e The Listen/Notify mechanism is used by default as it is more efficient.\n\n### \u003ca id=\"4-8\"\u003e\u003c/a\u003eAdding new asynchronous event handlers\n\nAfter restarting the backend, existing subscriptions will only process new events after the last processed event \nand not everything from the first one.\n\n\u003e **WARNING**  \n\u003e Critical content demanding immediate user attention due to potential risks.\nNew subscriptions (event handlers) in the first poll will read and process all events.\nBe careful, if there are too many events, they may take a long time to process.\n\n### \u003ca id=\"4-9\"\u003e\u003c/a\u003eDrawbacks\n\nUsing PostgreSQL as an event store has a lot of advantages, but there are also drawbacks.\n\n1. **Asynchronous event handlers can process the same event more than once.** \n   It might crash after processing an event but before recording the fact that it has done so. \n   When it restarts, it will then process the same event again (e.g., send an integration event).\n   Integration events are delivered with **at-least-once** delivery guarantee.\n   The exactly-once delivery guarantee is hard to achieve due to a dual-write.\n   A dual-write describes a situation when you need to atomically update the database and publish messages\n   without two-phase commit (2PC).\n   Consumers of integration events should be idempotent and filter duplicates and unordered events.\n2. The asynchronous event handling results in the **eventual consistency between the write model and sent integration events**.\n   The polling database table for new events with a fixed delay introduces a full consistency lag \n   greater than or equal to the interval between polls (1 second by default).\n3. **A long-running transaction in the same database will effectively \"pause\" all event handlers.** \n   `pg_snapshot_xmin(pg_snapshot)` will return the ID of this long-running transaction\n   and events created by all later transactions will be read by the event subscription processor \n   only after this long-running transaction is committed.\n\n## \u003ca id=\"5\"\u003e\u003c/a\u003eProject structure\n\n### \u003ca id=\"5-1\"\u003e\u003c/a\u003eGradle subprojects\n\nThis reference implementation can be easily extended to comply with your domain model.\n\nEvent sourcing related code and application specific code are located in separate Gradle subprojects:\n* [`postgresql-event-sourcing-core`](postgresql-event-sourcing-core): event sourcing and PostgreSQL related code, a shared library, `eventsourcing.postgresql` package,\n* [`event-sourcing-app`](event-sourcing-app): application specific code, a simplified ride-hailing sample, `com.example.eventsourcing` package.\n\n`event-sourcing-app` depends on `postgresql-event-sourcing-core`:\n```groovy\ndependencies {\n    implementation project(':postgresql-event-sourcing-core')\n}\n```\n\n### \u003ca id=\"5-2\"\u003e\u003c/a\u003eDatabase schema migrations\n\nEvent sourcing related database schema migrations:\n* [V1__eventsourcing_tables.sql](event-sourcing-app/src/main/resources/db/migration/V1__eventsourcing_tables.sql)\n* [V2__notify_trigger.sql](event-sourcing-app/src/main/resources/db/migration/V2__notify_trigger.sql)\n\nApplication specific projections database schema migration:\n* [V3__projection_tables.sql](event-sourcing-app/src/main/resources/db/migration/V3__projection_tables.sql)\n\n### \u003ca id=\"5-3\"\u003e\u003c/a\u003eClass diagrams\n\n#### \u003ca id=\"5-3-1\"\u003e\u003c/a\u003eClass diagram of the domain model\n\n![Class diagram of the domain model](img/class-domain.svg)\n\n#### \u003ca id=\"5-3-2\"\u003e\u003c/a\u003eClass diagram of the projections\n\n![Class diagram of the projections](img/class-projection.svg)\n\n#### \u003ca id=\"5-3-3\"\u003e\u003c/a\u003eClass diagram of the service layer\n\n![Class diagram of the service layer](img/class-service.svg)\n\n## \u003ca id=\"6\"\u003e\u003c/a\u003eHow to adapt it to your domain?\n\nTo adapt this sample to your domain model, make changes to `event-sourcing-app` subproject.\nNo changes to `postgresql-event-sourcing-core` subproject are required.\n\n## \u003ca id=\"7\"\u003e\u003c/a\u003eHow to run the sample?\n\n1. Download \u0026 install [SDKMAN!](https://sdkman.io/install).\n\n2. Install JDK 21\n    ```bash\n    sdk list java\n    sdk install java 21-tem\n    ```\n\n3. Install [Docker](https://docs.docker.com/engine/install/)\n   and [Docker Compose](https://docs.docker.com/compose/install/).\n\n4. Build Java project and Docker image\n    ```bash\n    ./gradlew clean build jibDockerBuild -i\n    ```\n\n5. Run PostgreSQL, Kafka and event-sourcing-app\n    ```bash\n    docker compose --env-file gradle.properties up -d --scale event-sourcing-app=2\n    # wait a few minutes\n    ```\n\n6. Follow the logs of the application\n    ```bash\n    docker compose logs -f event-sourcing-app\n    ```\n   \n7. Run E2E tests and see the output\n    ```bash\n    E2E_TESTING=true ./gradlew clean test -i\n    ```\n\n8. Explore the database using the Adminer database management tool at http://localhost:8181.\n   Find the database name, user, and password in the [docker-compose.yml](docker-compose.yml).\n   \nYou can also manually call the REST API endpoints.\n\n1. Install [curl](https://curl.se/) and [jq](https://stedolan.github.io/jq/)\n    ```bash\n    sudo apt install curl jq\n    ```\n\n2. Place new order\n    ```bash\n    ORDER_ID=$(curl -s -X POST http://localhost:8080/orders -d '{\"riderId\":\"63770803-38f4-4594-aec2-4c74918f7165\",\"price\":\"123.45\",\"route\":[{\"address\":\"Kyiv, 17A Polyarna Street\",\"lat\":50.51980052414157,\"lon\":30.467197278948536},{\"address\":\"Kyiv, 18V Novokostyantynivska Street\",\"lat\":50.48509161169076,\"lon\":30.485170724431292}]}' -H 'Content-Type: application/json' | jq -r .orderId)\n    ```\n\n3. Get the placed order\n    ```bash\n    curl -s -X GET http://localhost:8080/orders/$ORDER_ID | jq\n    ```\n\n4. Accept the order\n    ```bash\n    curl -s -X PUT http://localhost:8080/orders/$ORDER_ID -d '{\"status\":\"ACCEPTED\",\"driverId\":\"2c068a1a-9263-433f-a70b-067d51b98378\"}' -H 'Content-Type: application/json'\n    ```\n\n5. Get the accepted order\n    ```bash\n    curl -s -X GET http://localhost:8080/orders/$ORDER_ID | jq\n    ```\n\n6. Complete the order\n    ```bash\n    curl -s -X PUT http://localhost:8080/orders/$ORDER_ID -d '{\"status\":\"COMPLETED\"}' -H 'Content-Type: application/json'\n    ```\n\n7. Get the completed order\n    ```bash\n    curl -s -X GET http://localhost:8080/orders/$ORDER_ID | jq\n    ```\n\n8. Try to cancel a completed order to simulate business rule violation\n    ```bash\n    curl -s -X PUT http://localhost:8080/orders/$ORDER_ID -d '{\"status\":\"CANCELLED\"}' -H 'Content-Type: application/json' | jq\n    ```\n\n9. Print integration events\n    ```bash\n    docker compose exec kafka /bin/kafka-console-consumer --bootstrap-server localhost:9092 --topic order-events --from-beginning --property print.key=true --timeout-ms 10000\n    ```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Feugene-khyst%2Fpostgresql-event-sourcing","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Feugene-khyst%2Fpostgresql-event-sourcing","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Feugene-khyst%2Fpostgresql-event-sourcing/lists"}