{"id":15637608,"url":"https://github.com/krasserm/akka-stream-eventsourcing","last_synced_at":"2025-07-30T10:44:49.373Z","repository":{"id":138233498,"uuid":"93136729","full_name":"krasserm/akka-stream-eventsourcing","owner":"krasserm","description":"Event sourcing for Akka Streams","archived":false,"fork":false,"pushed_at":"2017-09-27T14:41:46.000Z","size":60,"stargazers_count":100,"open_issues_count":4,"forks_count":16,"subscribers_count":21,"default_branch":"master","last_synced_at":"2025-03-30T13:23:32.487Z","etag":null,"topics":["akka-persistence","akka-streams","apache-kafka","event-sourcing","functional-programming","reactive-programming","scala"],"latest_commit_sha":null,"homepage":"","language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/krasserm.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-06-02T06:59:10.000Z","updated_at":"2025-01-13T14:54:45.000Z","dependencies_parsed_at":null,"dependency_job_id":"8d532365-ecdf-4bb9-ac6d-1d08fb8ada0e","html_url":"https://github.com/krasserm/akka-stream-eventsourcing","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/krasserm%2Fakka-stream-eventsourcing","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/krasserm%2Fakka-stream-eventsourcing/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/krasserm%2Fakka-stream-eventsourcing/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/krasserm%2Fakka-stream-eventsourcing/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/krasserm","download_url":"https://codeload.github.com/krasserm/akka-stream-eventsourcing/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251652402,"owners_count":21621940,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["akka-persistence","akka-streams","apache-kafka","event-sourcing","functional-programming","reactive-programming","scala"],"created_at":"2024-10-03T11:12:17.620Z","updated_at":"2025-04-30T06:23:35.935Z","avatar_url":"https://github.com/krasserm.png","language":"Scala","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Event sourcing for Akka Streams\n\n[![Gitter](https://badges.gitter.im/Join%20Chat.svg)](https://gitter.im/akka-stream-eventsourcing/Lobby?utm_source=badge\u0026utm_medium=badge\u0026utm_campaign=pr-badge)\n[![Build Status](https://travis-ci.org/krasserm/akka-stream-eventsourcing.svg?branch=master)](https://travis-ci.org/krasserm/akka-stream-eventsourcing)\n\n## General concept\n\nThis project brings to [Akka Streams](http://doc.akka.io/docs/akka/2.5.2/scala/stream/index.html) what [Akka Persistence](http://doc.akka.io/docs/akka/2.5.2/scala/persistence.html) brings to [Akka Actors](http://doc.akka.io/docs/akka/2.5.2/scala/actors.html): persistence via event sourcing. It provides a stateful [`EventSourcing`](https://github.com/krasserm/akka-stream-eventsourcing/blob/master/src/main/scala/com/github/krasserm/ases/EventSourcing.scala) graph stage of type `BidiFlow[REQ, E, E, RES, _]` (simplified) which models the event sourcing message flow. An `EventSourcing` stage therefore \n\n- consumes and validates a request (command or query)\n- produces events (derived from a command and internal state) to be written to an event log\n- consumes written events for updating internal state\n- produces a response after internal state update\n\nAn `EventSourcing` stage maintains current state and uses a request handler for validating requests and generating events and responses:\n  \n```scala\n  import com.github.krasserm.ases.EventSourcing._\n\n  type RequestHandler[S, E, REQ, RES] = (S, REQ) =\u003e Emission[S, E, RES]\n```\n\nRequest handler input is current state and a request, output is an instruction to emit events and/or a response. `emit` instructs the `EventSourcing` stage to emit `events` and call a `responseFactory` with current state after all emitted events have been written and applied to current state:   \n\n```scala\n  def emit[S, E, RES](events: Seq[E], responseFactory: S =\u003e RES): Emission[S, E, RES]\n```\n\n`respond` instructs the `EventSourcing` stage to emit a `response` immediately without emitting any events: \n\n```scala\n  def respond[S, E, RES](response: RES): Emission[S, E, RES]\n```\n\nMethods `emit` and `respond` themselves are side-effect free. They only generate `Emission` values that are interpreted by the `EventSourcing` stage. \n\nFor updating internal state the `EventSourcing` stage uses an event handler:\n  \n```scala\n  type EventHandler[S, E] = (S, E) =\u003e S\n```\n\nEvent handler input is current state and a written event, output is updated state. Given definitions of emitter id, initial state, a request handler and an event handler, an `EventSourcing` stage can be constructed with:\n   \n```scala\n  import akka.stream.scaladsl.BidiFlow\n  import com.github.krasserm.ases.EventSourcing\n\n  def emitterId: String  \n  def initialState: S\n  def requestHandler: RequestHandler[S, E, REQ, RES]\n  def eventHandler: EventHandler[S, E]\n\n  def eventSourcingStage: BidiFlow[REQ, E, E, RES, _] =\n    EventSourcing(emitterId, initialState, requestHandler, eventHandler)\n```\n   \nEvent logs are modeled as `Flow[E, E, _]`. This project provides event log implementations that can use [Akka Persistence journals](http://doc.akka.io/docs/akka/2.5.2/scala/persistence.html#storage-plugins) or [Apache Kafka](http://kafka.apache.org/) as storage backends. [Eventuate event logs](http://rbmhtechnology.github.io/eventuate/architecture.html#event-logs) as storage backends will be supported later. The following example uses the Akka Persistence in-memory journal as storage backend:    \n   \n```scala\n  import akka.stream.scaladsl.Flow\n  import com.github.krasserm.ases.log.AkkaPersistenceEventLog\n\n  val provider: AkkaPersistenceEventLog = \n    new AkkaPersistenceEventLog(journalId = \"akka.persistence.journal.inmem\")\n\n  def persistenceId: String = \n    emitterId\n\n  def eventLog: Flow[E, E, _] =\n    provider.flow[E](persistenceId)\n```\n\nAfter materialization, an event log emits replayed events to its output port before emitting newly written events that have been received from its input port. To allow `eventSourcingStage` to use `eventLog` for writing and reading events it must `join` the event log:\n\n```scala\n  def requestProcessor: Flow[REQ, RES, _] =\n    eventSourcingStage.join(eventLog)\n```\n\nThe result is a stateful, event-sourced request processor of type `Flow[REQ, RES, _]` that processes a request stream. It will only demand new requests from upstream if there is downstream demand for responses and events. Any slowdown in response processing or event writing will back-pressure upstream request producers.\n\nIn the same way as `PersistentActor`s in Akka Persistence, request processors provide a consistency boundary around internal state but additionally provide type safety and back-pressure for the whole event sourcing message flow.\n\nThe examples presented in this section are a bit simplified for better readability. Take a look at section *Event logging protocols* and the tests (e.g. [`EventSourcingSpec`](https://github.com/krasserm/akka-stream-eventsourcing/blob/master/src/test/scala/com/github/krasserm/ases/EventSourcingSpec.scala)) for further details.\n\n## Dynamic request processor loading\n\nWhen applying domain-driven design, consistency boundaries are often around so-called *aggregates*. A request processor that manages a single aggregate should be dynamically loaded and recovered when the first request targeted at that aggregate arrives.\n \nThis can be achieved with a [`Router`](https://github.com/krasserm/akka-stream-eventsourcing/blob/master/src/main/scala/com/github/krasserm/ases/Router.scala). A router is configured with a function that extracts the aggregate id from a request and another function that creates a request processor from an aggregate id:   \n\n```scala\n  import com.github.krasserm.ases.Router\n\n  trait Aggregate[A] {\n    def aggregateId(a: A): String\n  }\n\n  def eventSourcingStage(aggregateId: String): BidiFlow[REQ, E, E, RES, _] =\n    EventSourcing(aggregateId, initialState, requestHandler, eventHandler)\n\n  def requestProcessor(aggregateId: String): Flow[REQ, RES, _] =\n    eventSourcingStage(aggregateId).join(provider.flow[E](aggregateId))\n\n  def requestRouter(implicit agg: Aggregate[REQ]): Flow[REQ, RES, _] = {\n    Router(req =\u003e agg.aggregateId(req), (aggregateId: String) =\u003e requestProcessor(aggregateId))\n  }\n```\n\nA running example is in [`RequestRoutingSpec`](https://github.com/krasserm/akka-stream-eventsourcing/blob/master/src/test/scala/com/github/krasserm/ases/RequestRoutingSpec.scala). Dynamic unloading of request processors (e.g. using an LRU policy) is not supported yet as this requires special support from Akka Streams.\n\n## Request processor collaboration (microservices)\n\nIn contrast to `PersistentActor`s, `EventSourcing` stages can form a group by sharing an event log. Within a group, events emitted by one member can be consumed by all members in the group i.e. stages communicate via broadcast using the ordering guarantees of the underlying event log implementation. This feature can be used to implement event-sourced microservices that collaborate via events over a shared event log. \n\nThe following example defines a `requestProcessor` method that creates request processors that use a Kafka topic partition as shared event log. Request processors created with this method form a group whose members collaborate over the shared `kafkaTopicPartition`. All members consume events in the same order as a topic partition provides total ordering:\n\n```scala\n  import com.github.krasserm.ases.log.KafkaEventLog\n  import org.apache.kafka.common.TopicPartition\n\n  def kafkaHost: String\n  def kafkaPort: Int\n  def kafkaTopicPartition: TopicPartition\n\n  val provider = new KafkaEventLog(kafkaHost, kafkaPort)\n\n  def requestProcessor(emitterId: String): Flow[REQ, RES, _] =\n    EventSourcing(emitterId, initialState, requestHandler, eventHandler)\n      .join(provider.flow(kafkaTopicPartition))\n  \n  // A group of request processors    \n  val requestProcessor1 = requestProcessor(\"processor1\")\n  val requestProcessor2 = requestProcessor(\"processor2\")\n  // ...\n```\n\nYou can find a running example in [`EventCollaborationSpec`](https://github.com/krasserm/akka-stream-eventsourcing/blob/master/src/test/scala/com/github/krasserm/ases/EventCollaborationSpec.scala) (a replicated event-sourced counter). Collaboration of request processors is comparable to collaboration of `EventsourcedActor`s in [Eventuate](http://rbmhtechnology.github.io/eventuate/) (see [event collaboration](http://rbmhtechnology.github.io/eventuate/architecture.html#event-collaboration) for details).\n\n## Handler switching\n\nApplications can switch request and event handlers as a function of current state by defining request and event handler *providers*: \n\n```scala\n  def emitterId: String\n  def initialState: S\n  def requestHandlerProvider: S =\u003e RequestHandler[S, E, REQ, RES]\n  def eventHandlerProvider: S =\u003e EventHandler[S, E]\n\n  def eventSourcingStage: BidiFlow[REQ, E, E, RES, _] =\n    EventSourcing(emitterId, initialState, requestHandlerProvider, eventHandlerProvider)\n```\n   \nA request handler provider is called with current state for each request, an event handler provider is called with current state for each written event. This feature will be used later to implement a state machine DSL on top of the current handler API.\n\n## Event logging protocols\n \nThe examples so far modeled event logs as `Flow[E, E, _]` and `EventSourcing` stages as `BidiFlow[REQ, E, E, RES, _]` where `E` is the type of a domain event. This is not sufficient for real-world use cases. Further event metadata are required: \n\n- Events emitted by an `EventSourcing` stage must contain the id of the emitting stage (`emitterId`). This is needed, for example, by consumers on the query side of a CQRS application to distinguish emitters when consuming events from an aggregated or shared event log.\n- Events emitted by an event log must additionally contain the sequence number of the written event. Sequence numbers are required to track event processing progress, for example.\n- An event log must also signal to an `EventSourcing` stage when event replay has been completed. Only after successful recovery, an `EventSourcing` stage is allowed to signal demand for new requests to upstream producers.\n\nFor these reasons, the current implementation uses \n\n```scala\nFlow[Emitted[E], Delivery[Durable[E]], _]\n```\n\nas type for event logs and \n\n```scala\nBidiFlow[REQ, Emitted[E], Delivery[Durable[E]], RES, _]\n```\n\nas type for `EventSourcing` stages. `Emitted`, `Durable` and `Delivery` are defined in [`Protocol.scala`](https://github.com/krasserm/akka-stream-eventsourcing/blob/master/src/main/scala/com/github/krasserm/ases/Protocol.scala). These protocol definitions are still preliminary and expected to change. For example, an extension to `Emitted` could support the emission of event batches for atomic batch writes.\n \n## Project status\n\nIn its current state, the project is a prototype that demonstrates the basic ideas how event sourcing and event collaboration could be added to Akka Streams. It can be used for experiments but is not yet ready for production. The prototype should serve as basis for further discussions and evolve to a potential later contribution to Akka, if there is enough interest in the community.    \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkrasserm%2Fakka-stream-eventsourcing","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkrasserm%2Fakka-stream-eventsourcing","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkrasserm%2Fakka-stream-eventsourcing/lists"}