{"id":19714854,"url":"https://github.com/dm3/everest","last_synced_at":"2025-06-20T07:35:56.874Z","repository":{"id":57714027,"uuid":"113448787","full_name":"dm3/everest","owner":"dm3","description":"Clojure/Postgres-based event store","archived":false,"fork":false,"pushed_at":"2019-01-25T12:38:58.000Z","size":44,"stargazers_count":3,"open_issues_count":3,"forks_count":2,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-05-25T00:43:51.683Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Clojure","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dm3.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-12-07T12:33:49.000Z","updated_at":"2019-06-14T06:21:44.000Z","dependencies_parsed_at":"2022-09-26T21:30:57.241Z","dependency_job_id":null,"html_url":"https://github.com/dm3/everest","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/dm3/everest","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dm3%2Feverest","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dm3%2Feverest/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dm3%2Feverest/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dm3%2Feverest/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dm3","download_url":"https://codeload.github.com/dm3/everest/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dm3%2Feverest/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":260465889,"owners_count":23013448,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-11T22:36:20.471Z","updated_at":"2025-06-20T07:35:51.862Z","avatar_url":"https://github.com/dm3.png","language":"Clojure","funding_links":[],"categories":[],"sub_categories":[],"readme":"# everest\n\nA Clojure/Postgres-based event store/library/toolkit.\n\nWhen you should use it?\n\n* You don't need to process terabytes of events (Postgres will deal\n  perfectly fine with tables that fit in RAM)\n* You are OK with what Postgres offers for HA (getting simpler with each release)\n* You believe in event-based integration\n* You are a fan of event sourcing/event driven integration\n\nAlternative Event Store implementations:\n\n* [Rill](https://github.com/rill-event-sourcing/rill) - also based on Postgres,\n  opinionated\n* [NEventStore](https://github.com/NEventStore) - if you prefer .NET\n\nAlternatives if Postgres doesn't cut your needs:\n\n* [Kafka](http://kafka.apache.org/)\n* [GEventStore](https://geteventstore.com/)\n\n## TODO\n\n* Tests with embedded Postgres\n* More examples\n    - use with Component\n    - auto load handlers\n    - event sourcing\n    - recover from failure\n    - transactional handlers\n    - complicated handler state\n\n## Design\n\nEverest is split into several modules:\n\n* Core `[everest]`\n  - core event store protocols\n  - in-memory event store implementation\n  - implementation-agnostic subscription mechanism\n* PG `[everest.module/pg]` - relational event store implementation using\n  Postgres\n* PG Schema `[everest.module/pg-schema]` - schema for the JDBC module with\n  Flyway-based migrations\n* JSON `[everest.module/json]` - Cheshire (Jackson)-based store middleware for\n  serialization/deserialization into JSON\n* DSL `[everest.module/dsl]` - experimental DSL to compose event handlers\n\n### Event Store\n\nEverest event store interface is highly influenced by Greg Young's excellent\n[EventStore](https://geteventstore.com/). It supports the following operations:\n\n* `read-event` to read a single event\n* `read-stream-forward` to read all events in the given stream starting from\n  the given position\n* `read-stream-backward` to read all events in the given stream starting from\n  the given position towards the beginning\n* `delete-stream!` to delete the given stream and all of the events\n* `append-events!` to append a number of events to the given stream\n\nThere is a special `:all` stream which returns all of the events in the store.\nIt is not possible to delete or append to the `:all` stream.\n\nDelete and append operations expect an `expected-version` argument which will\ncheck the following:\n\n* if `:everest.stream.version/any` - no check is performed\n* if `:everest.stream.version/not-present` - the stream should not exist\n* if a number - the last event in the stream should be at the given position\n\n### Subscriptions\n\nOne great advantage of having the events is being able to react to them as they\nappear. Everest event store supports subscriptions through the\n`everest.store/subscribe` function which returns a stream of events for the\ngiven stream starting right after the provided position.\n\n### Store and Subscription Middleware\n\nEverest supports a concept of store middleware which allows wrapping the events\ncoming in and out of the event store with custom functions. This fits the\nClojure mindset very well and is very convenient for pluggable event\nserialization, validation and coercion, discussed further below.\n\n### Event Serialization\n\nEverest provides a JSON serialization middleware,\n`everest.json.store.middleware/wrap-cheshire-json`, which will serialize the\nevents going into the store and deserialize ones coming out.  It plays\nespecially well with Postgres' *json/jsonb* database types.\n\nWe can create our own middleware easily by wrapping e.g.\n[Nippy](https://github.com/ptaoussanis/nippy) or [Avro](https://avro.apache.org/).\n\n### Event Schemas and Versioning\n\nA common need when working with data in a dynamic language is making sure data\nfollows the specification. Clojure has a de-facto standard for dynamic\nvalidation and coercion: [clojure.spec](https://clojure.org/guides/spec).\nThere's no module in Everest to provide Spec-based event validation, but it\nwould be quite easy to implement!\n\nFurthermore, events evolve. Although a contract for an event shouldn't change\ndrastically, it may change in a non-compatible way which could require\nmigration. Obviously, we should try to evolve the contracts in a\nbackwards-compatible manner. If that isn't possible, we need to implement\ncustom logic which will migrate the events to the newest versions upon loading.\n\nWe can choose from the following migration/versioning strategies:\n\n1. Do nothing. Handle upgrades in the application code of aggregates,\n   projections and processes.\n2. Define upcasters in the components which own the events.  Projections and\n   processes in other components will have to work with old event versions.\n3. Upcasters in contract libraries. Package event definitions into separate\n   libraries and include them as dependencies in consuming projects.  This will\n   ensure the upcasters will run every time the events are read.\n4. Upcaster migrations. Write an upcaster in the component which owns the event\n   and run it on the event store to update the events. This will work in\n   Postgres, but only for upcasters that do not split the event into multiple\n   ones while upcasting. These kinds of upcasters would have to append more\n   events which would get propagated to the subscriptions. This could\n   theoretically be dealt with by adding some sort of metadata which would be\n   used in subscriptions to filter out such upcasted events.\n\nIf you need to break your contracts - solution #3 is preferable.\n\nFor more insight into event versioning:\n\n* https://abdullin.com/post/event-sourcing-versioning/\n* http://files.movereem.nl/2017saner-eventsourcing.pdf\n* https://leanpub.com/esversioning/read\n\nCurrently Everest doesn't provide any abstractions to deal with upcasting, as\nthe scenarios for upcasting are quite different, e.g.:\n\n* Upcast an event by supplying a default value based on the `version` defined\n  in its metadata\n* Upcast an event by looking up data in the external sources\n* Upcast an event by capturing the state seen earlier in the stream\n* Split an event into multiple events\n\n### Event Handlers\n\nOnce events are in the store, we want to read and react to them. This is\nachieved by subscribing to the new events using the subscription mechanism\nprovided by Everest. However, just getting a hold of an event stream is too low\nlevel. Usually we want to do one of the few basic things:\n\n* Process a number of events and write the results into some other representation\n* Raise more events in response to processed events\n* Interact with the outside world in response to processed events\n* Wrap the above interactions into a state machine which reacts to events (also\n  known as a *Process Manager*)\n\nAn event handler will most surely want to track its position in the event\nstream so that it doesn't have to process all of the events every time it\nstarts up. Position tracking is also much simpler to reason about if the position\nis saved atomically with the result of processing the event. This way\nidempotency is guaranteed and there's no trouble in performing a live back up\nof the position together with the result.\n\nThus, an event handler will want to access an event together with some kind of\ncontext, be it a [component](https://github.com/stuartsierra/component) or just\na map holding the handler's state. Also, you will probably want to run the\nhandlers in such a manner so as not to interfere with the workings of other\nhandlers. If a handler is slow or failing, it should not affect the other\nhandlers.\n\nAll of the cross-cutting concerns that affect many handlers should be expressed\nas *event-handler middleware*. An event is just a map which can be `assoc`ed\nwith additional stuff at any point in the middleware chain. The same way it works\nwith [Ring](https://github.com/ring-clojure/ring):\n\n* event ~= request\n* event handler = [Ring handler](https://github.com/ring-clojure/ring/wiki/Concepts#handlers)\n* event handler middleware = [Ring handler middleware](https://github.com/ring-clojure/ring/wiki/Concepts#middleware)\n\nAlthough the concept of an event handler is very simple, handling some of the\ncommon cases is quite tricky and Everest aims to provide support for some of\nthe patterns.\n\n#### Projections and Other Simple Event Handlers\n\nProjections are the simplest type of event handlers. They consume an event and\nproduce a result - storing it either in a persistent storage or memory - then\nwait for another event.\n\nEverest provides some generic projection middleware:\n\n* `everest.handler.middleware/wrap-position` - tracks the current position of\n  the handler\n* `everest.handler.middleware/wrap-state` - tracks the state returned from the\n  handler\n\nThese middlewares can be parameterized with the\n`:handler.middleware/state-store` which controls where the handler state will\nbe stored. Everest provides a JDBC-based setup for position/state tracking.\n\nPosition updates only:\n\n```clojure\n(def handler-state-store (memory.handler.store/create))\n\n(everest.handler.middleware/wrap-position\n  (fn [event]\n    (println \"Hi: \" (:name event)))\n  {:handler.middleware/state-store handler-state-store})\n```\n\nPosition and state updates:\n\n```clojure\n(everest.handler.middleware/wrap-state\n  (fn [event]\n    (update event :handler/state (fnil inc 0)))\n  {:handler.middleware/state-store handler-state-store\n   :handler.middleware/state-key :handler/state}) ;; default value\n```\n\nThe above functionality depends on the JDBC-specific middleware:\n\n* `wrap-connection` - opens the connection, attaches it to the event and\n  executes the handler\n* `wrap-tx` - runs the wrapped handler inside of the transaction\n\nSo you would actually use it like so:\n\n```clojure\n(-\u003e event-handler\n    (wrap-state {:get-state (get-fn 'HandlerId)\n                 :upsert-state (upsert-fn 'HandlerType 'HandlerId)})\n    (wrap-tx)\n    (wrap-connection {:db dbspec}))\n```\n\nOf course, clojure data cannot be stored in the JDBC-backed storage without\nserialization. Serialization is also dealt with via middleware, which we'll\ntouch upon a bit later.\n\n## License\n\nCopyright © 2017 Vadim Platonov\n\nDistributed under the MIT License.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdm3%2Feverest","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdm3%2Feverest","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdm3%2Feverest/lists"}