{"id":27362576,"url":"https://github.com/askmeagain/event-meshinery","last_synced_at":"2025-04-13T03:24:20.558Z","repository":{"id":39645087,"uuid":"400630290","full_name":"AskMeAgain/Event-Meshinery","owner":"AskMeAgain","description":"A new Framework for asynchronous signaling of processes via Kafka/Mysql/anything.","archived":false,"fork":false,"pushed_at":"2025-04-11T19:15:48.000Z","size":1918,"stargazers_count":9,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-04-11T19:51:47.985Z","etag":null,"topics":["kafka","mysql","rest","spring-boot","stream-processing"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/AskMeAgain.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2021-08-27T20:41:05.000Z","updated_at":"2025-03-25T19:33:41.000Z","dependencies_parsed_at":"2023-12-31T19:19:39.289Z","dependency_job_id":"43260239-8ecf-41fa-92e8-5efbfde68ac1","html_url":"https://github.com/AskMeAgain/Event-Meshinery","commit_stats":null,"previous_names":[],"tags_count":6,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AskMeAgain%2FEvent-Meshinery","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AskMeAgain%2FEvent-Meshinery/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AskMeAgain%2FEvent-Meshinery/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AskMeAgain%2FEvent-Meshinery/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/AskMeAgain","download_url":"https://codeload.github.com/AskMeAgain/Event-Meshinery/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248658519,"owners_count":21140958,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["kafka","mysql","rest","spring-boot","stream-processing"],"created_at":"2025-04-13T03:24:15.861Z","updated_at":"2025-04-13T03:24:20.543Z","avatar_url":"https://github.com/AskMeAgain.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Event Meshinery\n\n## Table of contents\n\n\u003c!-- toc --\u003e\n- [Description](#description)\n- [Motivation ](#motivation-)\n- [Advantages of Event-Meshinery ](#advantages-of-event-meshinery-)\n- [Module Structure ](#module-structure-)\n- [Architecture ](#architecture-)\n  - [Meshinery Processors ](#meshinery-processors-)\n  - [MeshineryTasks ](#meshinerytasks-)\n  - [DataContext ](#datacontext-)\n  - [Round Robin Scheduler ](#round-robin-scheduler-)\n  - [Connectors](#connectors)\n- [On Failure ](#on-failure-)\n  - [Replays of a DataContext](#replays-of-a-datacontext)\n  - [Exception Handling ](#exception-handling-)\n- [Logging](#logging)\n- [Monitoring](#monitoring)\n- [Drawing Graphs](#drawing-graphs)\n  - [Pictures](#pictures)\n  - [Mermaid.js](#mermaidjs)\n- [Getting started](#getting-started)\n- [Roadmap](#roadmap)\n\u003c!-- /toc --\u003e\n\n## Description\n\nThis framework was originally written to replace KafkaStreams in a specific usecase, but you can use this framework\nwithout Kafka.\n\nThe framework is a state store independent **signaling** framework and designed to easily structure **long running**,\n**multi step** or **long delay heavy** processing tasks in a transparent and safe way.\n\nIt is used as a way to **signal** the next processing step in your application, with transparent code and without hidden\nbehaviour. You describe the event and the resulting processing task, the framework will make sure that the work gets\ndone.\n\nIt can connect any event/signal imaginable with any processing task and any state store, all with an **asynchronous**\napi to make sure that your events are processed the moment they happen.\n\n**Event-Meshinery assumes that the restricting resource is time/network io and **not** processing power or throughput.**\n\n## Motivation \u003ca name=\"Motivation\"\u003e\u003c/a\u003e\n\nDoing long running (blocking) procedures (rest calls for example)\nvia [Kafka Streams](https://kafka.apache.org/documentation/streams/) represents a challenge:\n\n**If you block a partition with a long running call, then you cannot process any other messages from this partition\nuntil the processing is unblocked.**\n\nThis means that you can only scale in Kafka Streams as far as your Kafka Cluster (Partition count) allows:\nIf your Kafka Cluster has 32 Partitions per topic, you can only have a max number of 32 running threads and can only run\n32 stream processor/message processing in parallel (for this topic).\n\nTo solve this problem, the Event-Meshinery framework removes a guarantee:\n\n**Messages in a partition are not processed in order, but processed as they arrive.**\n\nThis is possible if your events are completely independent of each other and the order of events in a single\ntopic/partition is not important.\n\nConfluent recognized this need and created\nthe [parallel consumer](https://www.confluent.io/blog/introducing-confluent-parallel-message-processing-client/), but\nthis one only works in a Kafka only environment. The moment you need to bridge your signals out of Kafka (using a\ndifferent state store), you are on your own. This framework is exactly for this usecase: signaling, but in a state store\nindependent way.\n\n## Advantages of Event-Meshinery \u003ca name=\"Advantages\"\u003e\u003c/a\u003e\n\n* Structure your code in a really transparent way by having a **state store independent api** and separating the\n  business layer from the underlying implementation layer. One look at a task definition tells you exactly WHAT happens\n  WHEN.\n* You have complete **asynchronous processing** via Java Futures without the annoying thread handling\n* This framework can be integrate into any existing state store and even connect different ones: Kafka, Mysql etc.\n* A simple api you are already familiar with: Consume-Process-Produce\n* Easily integrated in your existing development environment, by utilizing the existing state store: You have a MysqlDb?\n  Use the mysql connector. You other team uses Kafka and you need to bridge a little bit of data? Add the existing\n  KafkaConnector.\n* Create a complete [event diagram](https://github.com/AskMeAgain/Event-Meshinery/wiki/Draw) to display your events and how they interact with\n  each other, completely automated.\n* You can resume a process in case of error and you will start exactly where you left off (within bounds).\n* Automatic Prometheus Monitoring integration of all your tasks and their respective task queues.\n* Complete **Spring** integration: 1-3 Annotations start everything, you only need to define the business logic and wire\n  it together.\n\n## Module Structure \u003ca name=\"Module-Structure\"\u003e\u003c/a\u003e\n\n* [meshinery-core](https://github.com/AskMeAgain/Event-Meshinery/wiki/Core) contains the scheduler and everything basic you need. You only need\n  this to start. This library exposes the basic api on which the other packages depend on.\n    * [meshinery-core-spring](https://github.com/AskMeAgain/Event-Meshinery/wiki/Core-Spring) contains the **Spring** AutoConfiguration\n      for the core library, like starting the Scheduler automatically and providing some utility hooks\n* [meshinery-monitoring](https://github.com/AskMeAgain/Event-Meshinery/wiki/Monitoring) contains a prometheus monitoring solution\n    * [meshinery-monitoring-spring](https://github.com/AskMeAgain/Event-Meshinery/wiki/Monitoring-Spring) contains the **Spring**\n      AutoConfiguration of the monitoring\n* [meshinery-draw](https://github.com/AskMeAgain/Event-Meshinery/wiki/Draw) contains the MeshineryDrawer class, which takes MeshineryTasks and\n  draws system diagrams for multiple sources: Pictures (PNG,JPG) and Mermaid\n    * [meshinery-draw-spring](https://github.com/AskMeAgain/Event-Meshinery/wiki/Draw-Spring) contains a **Spring** AutoConfiguration of\n      the Drawing with Endpoints\n* [meshinery-connectors-mysql](https://github.com/AskMeAgain/Event-Meshinery/wiki/Mysql) has the Mysql state store\n  integration\n    * [meshinery-connectors-mysql-spring](modules/connectors/mysql/meshinery-mysql-connector-spring/mysql-spring.md) has\n      the Spring AutoConfiguration for Mysql\n* [meshinery-connectors-postgres](https://github.com/AskMeAgain/Event-Meshinery/wiki/Postgres) has the Postgres state store\n  integration\n    * [meshinery-connectors-postgres-spring](modules/connectors/postgres/meshinery-postgres-connector-spring/postgres-spring.md) has\n      the Spring AutoConfiguration for Postgres\n* [meshinery-connectors-kafka](https://github.com/AskMeAgain/Event-Meshinery/wiki/Kafka) has the Kafka state store\n  integration\n    * [meshinery-connectors-kafka-spring](https://github.com/AskMeAgain/Event-Meshinery/wiki/Kafka-Spring)\n      has the Spring AutoConfiguration for Kafka\n\n## Architecture \u003ca name=\"Architecture\"\u003e\u003c/a\u003e\n\n[Detailed architecture documentation](https://github.com/AskMeAgain/Event-Meshinery/wiki/Meshinery-Core-Architecture)\n\nThe general building blocks of this framework consist of 5 ideas:\n\n* [MeshineryProcessor](#Processor)\n* [MeshineryTask](#Task)\n* [DataContext](#Context)\n* [RoundRobinScheduler](#Scheduler)\n* [Input/OutputSources](#Sources)\n\n### Meshinery Processors \u003ca name=\"Processor\"\u003e\u003c/a\u003e\n\n[Detailed Documentation](https://github.com/AskMeAgain/Event-Meshinery/wiki/Meshinery-Processor)\n\nMeshinery Processors define the actual business work, like doing restcalls, calculating user information etc. They take\nin a DataContext and a thread Executor and return a **CompletableFuture**.\n\n    public class LongRunningRestcallProcessor implements MeshineryProcessor\u003cTestContext, TestContext\u003e {\n    \n        @Override\n        public CompletableFuture\u003cTestContext\u003e processAsync(TestContext context, Executor executor) {\n            return CompletableFuture.supplyAsync(() -\u003e {\n            \n                  log.info(\"Starting Request\");\n                  thisIsASuperLongRestCall();\n                  log.info(\"Finished Request\");\n            \n                  return context;\n                }, executor); //running on this thread executor\n        }\n    }\n\n### MeshineryTasks \u003ca name=\"Task\"\u003e\u003c/a\u003e\n\n[Detailed Documentation](https://github.com/AskMeAgain/Event-Meshinery/wiki/Meshinery-Task)\n\nA MeshineryTask describes a single **business** unit of work, which consists of an input source, a list of processors to\nsolve a part of the business logic and one or multiple output calls, which trigger itself other events.\n\nAn input source takes an eventkey, which gets fed to the inputsource to produce data. This data is then given to the\nprocessors and multiple output sources, which spawn more events.\n\n    var meshineryTask = MeshineryTask.\u003cString, TestContext\u003ebuilder()\n        .read(\"state-a\", executorService) //Input state \u0026 thread config\n        .taskName(\"cool task name\") //Task Name for logging\n        .defaultOutputSource(outputSource) //Kafka connection \n        .process(processorA) //Processing step\n        .write(\"event-b\") //Event \"event-b\" put to Kafka topic \"event-b\" with the result of processorA\n        .process(processorB) //Another Processing step\n        .write(\"event-c\") //Event \"event-c\" put to Kafka topic \"event-c with the result of processorB\n\nA task can have any amount of processors and sub processing (via processors). This allows you to include some logic on\nhow the pipeline should react. **The goal is that each tasks describes exactly WHAT processor and WHEN a processor is\nexecuted.** This allows for super transparent code which allows you to argue about the execution on a higher level. The\nspecific implementation of the processors and the underlying state store is not important when arguing about the\nbusiness case.\n\n### DataContext \u003ca name=\"Context\"\u003e\u003c/a\u003e\n\n[Detailed Documentation](https://github.com/AskMeAgain/Event-Meshinery/wiki/Meshinery-DataContext)\n\nA MeshineryTask has a dataContext assigned, which is basically just the input and output class type in\nsources/processors.\n\n    var task = MeshineryTaskFactory.\u003cString, TestContext\u003ebuilder() //here the DataContext is TestContext\n        .inputSource(inputSource) //this source can read all TestContext from the state store\n        .outputSource(defaultOutput) //this output source writes TestContext back to the state store\n        .read(INPUT_KEY, executor)\n        .process(testContextProcessor) //this processor gets a TestContext as input and returns a TestContext as output\n        .write(OUTPUT_KEY); //writing the TestContext to the state store, which triggers other events\n\nThe idea here is that multiple Tasks all use the same dataContext, but enrich the data by putting their result\nadditively to the context. You dont need to handle millions of dtos, just 1 for each Business Case.\n\nIf you add another task at the end of the processing pipeline, you just have access to all the data which got processed\nbefore.\n\n**Your state stores contain a log on how the processing went from step to step.**\n\n### Round Robin Scheduler \u003ca name=\"Scheduler\"\u003e\u003c/a\u003e\n\n[Detailed Documentation](https://github.com/AskMeAgain/Event-Meshinery/wiki/Meshinery-Scheduler)\n\nThe RoundRobinScheduler takes a list of tasks, creates small \"work packages\" (called TaskRuns)\nbased on each task, and executes them on all available threads. The scheduler has alot of configurations and can run in\na continuous way or stop processing when all inputsources are exhausted.\n\n    RoundRobinScheduler.builder()\n        .task(task)\n        [..]\n        .backpressureLimit(100)\n        .batchJob(true)\n        .buildAndStart();\n\n### Connectors\n\n[Detailed Documentation](https://github.com/AskMeAgain/Event-Meshinery/wiki/Meshinery-Connector)\n\nThere are Input and OutputSources and both form a MeshineryConnector. InputSources provide the data which gets passed to\nprocessors. OutputSources write the data to state stores and trigger one or more new events (by the respective\nInputSource). A MeshineryConnector implements both interfaces to connect a single event with input and outputs.\n\nMost of the time a signaling source can implement both Input and Output, like in mysql you can write data and read this\nexact data back again in different parts of your application. But sometimes this is not the case, for example if you\nreceive data from a rest api, you can read this data, but you cannot write this data back to the original source.\nExample is\nthe [CronInputSource](modules/meshinery-core/src/main/java/io/github/askmeagain/meshinery/core/source/CronInputSource.java)\n, which triggers based on a cron.\n\nA Source describes a connection to a state store and takes an event-key as input/output, which is passed to the state\nstore to read/write data to specific logically separated parts of the store. For example in Kafka an event-key would\nresult in a new topic, in mysql just a different column in a table. Each State Store implements the event-key lookup\ndifferently, but you can imagine these as different states of the data/processing.\n\nTechnically there can only be a single InputSource definition on a MeshineryTask, but you can combine multiple input\nsources to a single InputSource for joins for example. There can be any amount of OutputSources.\n\nHere \"result_topic\" and \"input_topic\" are event-keys and passed to the Source:\n\n    var task = MeshineryTaskFactory.\u003cString, TestContext\u003ebuilder() \n        .inputSource(inputSource) //this is a kafka input source for example\n        .defaultOutputSource(defaultOutput) //this is a kafka output source for example\n        .read(INPUT_KEY, executor) //reading from kafka topic\n        .process(testContextProcessor) //processing etc\n        .write(\"result_topic\"); //writing event to \"result_topic\"\n\nObviously, you can mix and match these sources and even write your own. They only implement a single interface function\n\nCurrently supported are the following state sources:\n\n* [Mysql](https://github.com/AskMeAgain/Event-Meshinery/wiki/Mysql)\n* [Postgres](https://github.com/AskMeAgain/Event-Meshinery/wiki/Postgres)\n* [Kafka](https://github.com/AskMeAgain/Event-Meshinery/wiki/Kafka)\n* [Memory](https://github.com/AskMeAgain/Event-Meshinery/wiki/Meshinery-Connector#utility-sources)\n\nAnd the following Utility Source:\n\n* [Cron](https://github.com/AskMeAgain/Event-Meshinery/wiki/Meshinery-Connector#utility-sources)\n* [Signaling Source](https://github.com/AskMeAgain/Event-Meshinery/wiki/Meshinery-Connector#utility-sources)\n* [InnerJoin Source](https://github.com/AskMeAgain/Event-Meshinery/wiki/Meshinery-Connector#utility-sources)\n\n## On Failure \u003ca name=\"Failure\"\u003e\u003c/a\u003e\n\nThis framework works with the at-most-once guarantee, which means that a state transition is only looked at once, since\nit assumes that in case of a failure a use case specific error correction procedure needs to be called. If a processing\nrequest results in an error and you want to resume this process, you just need to replay the message, which triggers the\nprocessing again, via the\nprovided [TaskReplayFactory](modules/meshinery-core/src/main/java/io/github/askmeagain/meshinery/core/task/TaskReplayFactory.java)\n\nEach InputSource gives you an easy way of replaying a single event, which feeds the event back into the scheduler to\nwork on.\n\n### Replays of a DataContext\n\nThe core library includes\na [TaskReplayFactory](modules/meshinery-core/src/main/java/io/github/askmeagain/meshinery/core/task/TaskReplayFactory.java)\n, which allows you to \"inject\"\nany concrete DataContext into any task, just by specifying a Taskname and providing the data as string. You can do this\nfor error correction or manual triggering of tasks (although a memory source would be more elegant here).\n\nThis TaskReplayFactory can run (A)synchronous and is available as an endpoint in\nthe [meshinery-core-spring](https://github.com/AskMeAgain/Event-Meshinery/wiki/Core-Spring) package.\n\n### Exception Handling \u003ca name=\"ExceptionHandling\"\u003e\u003c/a\u003e\n\nYou can handle exceptions which happen **inside** a completable future (in a processor), by setting a new error handler.\nThe default behaviour is that null is returned, which will then just stop the execution of this single event, by the\nRoundRobingScheduler. You can throw here hard, turn off the scheduler, do some rest/db calls and other stuff.\n\n    var task = MeshineryTaskFactory.\u003cString, TestContext\u003ebuilder()\n      [..]\n      .read(KEY, executor)\n      .process(new Processor())\n      .exceptionHandler(exception -\u003e {\n        log.info(\"Error Handling\"); //we add an additional log message\n        return new TestContext(); //we return a new default value\n      });\n\n## Logging\n\nThis Framework already does the hard work with logging: Setting up the MDC for each thread correctly. Each log message\nin **each** processor, **even in threads created by CompletableFuture.runAsync()**, you will have a correct MDC value **\nautomatically** of:\n\n* \"task.name\" -\u003e taskName\n* \"task.id\" -\u003e ContextId\n\n## Monitoring\n\n* [Detailed Documentation](https://github.com/AskMeAgain/Event-Meshinery/wiki/Monitoring)\n* [Spring Integration](https://github.com/AskMeAgain/Event-Meshinery/wiki/Monitoring-Spring)\n\nThe Monitoring package adds a basic monitoring solution. It\nuses [prometheus/client_java](https://github.com/prometheus/client_java)\npackage to expose metrics in a format compatible with prometheus. All metrics are written to an internal Prometheus\nregistry which can be shown via rest (already done in the meshinery-monitoring-spring package)\nand easily expanded by your needs.\n\n## Drawing Graphs\n\n* [Detailed Documentation](https://github.com/AskMeAgain/Event-Meshinery/wiki/Draw)\n* [Spring Integration](https://github.com/AskMeAgain/Event-Meshinery/wiki/Draw-Spring)\n\nSince this framework provides a single way of defining tasks, we can use this to draw diagrams\nvia [GraphStream](https://graphstream-project.org/). These diagrams are rendered based on the actual\nimplementation/connection of tasks and can be styled as you wish. Such a diagram can give you an easy way to argue about\nthe actual topology of the application and is generated completely **automatically**.\n\n### Pictures\n\nA picture can be generated of the actual topology layer.\n\n![example-png-graph](modules/meshinery-monitoring/example-graph.png)\n\n### Mermaid.js\n\nThere is also a [Mermaid](https://mermaid-js.github.io/mermaid/#/) implementation which can be hooked\ninto [Jeremy Branhams Diagram panel](https://grafana.com/grafana/plugins/jdbranham-diagram-panel/)\nplugin to provide a real time overview of the system and all its metrics in [Grafana](https://grafana.com/).\nThe [meshinery-draw-spring](modules/meshinery-draw-spring/draw-spring.md) package provides an endpoint which can be\npassed into the plugin to display the topology directly, but you can easily implement this by yourself.\n\n![example-mermaid-diagram](modules/meshinery-monitoring/grafana-graph.png)\n\n## Getting started\n\nCheckout the [Getting Started](https://github.com/AskMeAgain/Event-Meshinery/wiki/Getting-Started) wiki page\n\n## Roadmap\n\nThe following things are planned (not in order)\n\n* Quarkus/Micronaut integration\n* Sharding Possibilities in InputSources\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faskmeagain%2Fevent-meshinery","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Faskmeagain%2Fevent-meshinery","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faskmeagain%2Fevent-meshinery/lists"}