{"id":20751869,"url":"https://github.com/frankhaugen/frank.pulseflow","last_synced_at":"2025-03-11T14:34:09.667Z","repository":{"id":211661345,"uuid":"729678206","full_name":"frankhaugen/Frank.PulseFlow","owner":"frankhaugen","description":"An in memory messaging system for data-event-driven architecture focused on giving simple interfaces","archived":false,"fork":false,"pushed_at":"2024-03-03T13:09:26.000Z","size":187,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-02-20T07:39:46.642Z","etag":null,"topics":["channel","csharp","event-driven","flow","memory","messaging","pub-sub","pulse"],"latest_commit_sha":null,"homepage":"","language":"C#","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/frankhaugen.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-12-10T01:44:09.000Z","updated_at":"2024-02-20T20:49:40.000Z","dependencies_parsed_at":"2024-01-14T14:53:58.745Z","dependency_job_id":"1e5a7a91-846c-4ebd-90c2-54ac238109a3","html_url":"https://github.com/frankhaugen/Frank.PulseFlow","commit_stats":null,"previous_names":["frankhaugen/frank.pulseflow"],"tags_count":11,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/frankhaugen%2FFrank.PulseFlow","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/frankhaugen%2FFrank.PulseFlow/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/frankhaugen%2FFrank.PulseFlow/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/frankhaugen%2FFrank.PulseFlow/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/frankhaugen","download_url":"https://codeload.github.com/frankhaugen/Frank.PulseFlow/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243052521,"owners_count":20228400,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["channel","csharp","event-driven","flow","memory","messaging","pub-sub","pulse"],"created_at":"2024-11-17T08:38:47.443Z","updated_at":"2025-03-11T14:34:09.645Z","avatar_url":"https://github.com/frankhaugen.png","language":"C#","funding_links":[],"categories":[],"sub_categories":[],"readme":"# PulseFlow (Local Messaging With Channels)\n\nPulseFlow Local Messaging is a lightweight, high-performance messaging system that enables seamless communication, and thread-safe data transfer between different parts of an application. It's designed to be simple, flexible, and scalable, allowing for easy integration into any system architecture.\n\n___\n[![GitHub License](https://img.shields.io/github/license/frankhaugen/Frank.PulseFlow)](LICENSE)\n[![NuGet](https://img.shields.io/nuget/v/Frank.PulseFlow.svg)](https://www.nuget.org/packages/Frank.PulseFlow)\n[![NuGet](https://img.shields.io/nuget/dt/Frank.PulseFlow.svg)](https://www.nuget.org/packages/Frank.PulseFlow)\n\n![GitHub contributors](https://img.shields.io/github/contributors/frankhaugen/Frank.PulseFlow)\n![GitHub Release Date - Published_At](https://img.shields.io/github/release-date/frankhaugen/Frank.PulseFlow)\n![GitHub last commit](https://img.shields.io/github/last-commit/frankhaugen/Frank.PulseFlow)\n![GitHub commit activity](https://img.shields.io/github/commit-activity/m/frankhaugen/Frank.PulseFlow)\n![GitHub pull requests](https://img.shields.io/github/issues-pr/frankhaugen/Frank.PulseFlow)\n![GitHub issues](https://img.shields.io/github/issues/frankhaugen/Frank.PulseFlow)\n![GitHub closed issues](https://img.shields.io/github/issues-closed/frankhaugen/Frank.PulseFlow)\n___\n\n## Table of Contents\n\n- [Overview](#overview)\n  - [Key Features](#key-features)\n  - [Illustration](#illustration)\n  - [Use Cases](#use-cases)\n    - [Example 1 - Web API](#example-1---web-api)\n    - [Example 2 - Priority Queue](#example-2---priority-queue)\n    - [Example 3 - Complex Data Categorization](#example-3---complex-data-categorization)\n- [Getting Started](#getting-started)\n  - [Installation](#installation)\n- [Concepts](#concepts)\n  - [Nexus](#nexus)\n  - [Conduit](#conduit)\n  - [Pulse](#pulse)\n  - [Flow](#flow)\n___\n\n## Acknowledgements\n\nThis is a very thin abstraction layer on top of [System.Threading.Channels](https://www.nuget.org/packages/System.Threading.Channels), which is a NuGet package that gets shipped along with every release of .NET. \nThe reason for this abstraction layer is to make it easier to use `System.Threading.Channels` in a Dependency Injection scenario, and to make it easier to use `System.Threading.Channels` in a thread-safe manner.\nThanks to the .NET team for making `System.Threading.Channels`, and in particular [Stephen Toub](https://github.com/stephentoub) for his [blog post](https://devblogs.microsoft.com/dotnet/an-introduction-to-system-threading-channels/) on `System.Threading.Channels`, and as part of the rest of the team.\n\n## Overview\n\nPulseFlow Local Messaging is a lightweight, high-performance messaging system that enables seamless communication\nbetween different parts of an application. It's designed to be simple, flexible, and scalable, allowing for easy\nintegration into any system architecture.\n\nThis library does have a dependency on `Frank.Channels.DependencyInjection`, which is a simple registration of `System.\nThreading.Channels.Channel\u003cT\u003e` in the Dependency Injection container. This is done to make it easier to use `System.\nThreading.Channels` in a Dependency Injection scenario, and to make it easier to use `System.Threading.Channels` in a \nthread-safe manner. This includes a `ChannelWriter\u003cT\u003e` and a `ChannelReader\u003cT\u003e` that are a thread-safe singlteton, and can \nbe called directly from the Dependency Injection container without having to call `Channel\u003cT\u003e`, and then `Channel\u003cT\u003e.Writer` or \n`Channel\u003cT\u003e.Reader` directly. This is done to make it easier to use `System.Threading.Channels` in a Dependency Injection \nbehind the scenes in PulseFlow. This will also make it possible to \"intercept\" the `ChannelWriter\u003cT\u003e` and `ChannelReader\u003cT\u003e` \nfor debugging, logging, or other purposes.\n\n### Key Features\n\n- **Lightweight**: PulseFlow is a lightweight messaging system, with a small footprint and minimal resource\n  requirements.\n- **High Performance**: It's designed for high performance and scalability, capable of handling a vast volume of\n  messages simultaneously.\n- **Asynchronous Communication**: PulseFlow supports asynchronous data flow, allowing for non-blocking and concurrent\n  message transmissions.\n- **Flexible and Adaptable**: It's flexible and adaptable, capable of handling various types of messages and adapting\n  its processing logic based on the nature and requirements of each message.\n- **Simple and Easy to Use**: PulseFlow is simple and easy to use, with a straightforward API and minimal configuration\n  requirements. This is assuming Dependency Injection is used. If not, then it's a bit more complicated, and not supported at this time.\n- **Thread-Safe**: It's thread-safe, ensuring that messages are processed in a safe and reliable manner.\n- **Open Source**: PulseFlow is open-source, with the source code available on GitHub.\n- **Cross-Platform**: It's cross-platform, supporting multiple operating systems and platforms.\n- **Supports multiple consumers**: PulseFlow supports multiple consumers, thereby allowing for parallel processing of\n  messages. This is particularly useful in scenarios where there's a need to save messages to some form of audit trail or log.\n\n### Illustration\n\n```mermaid\ngraph TB\n    subgraph \"Transmission\"\n        EmailPulse[Email : IPulse] --\u003e|transmitted via| Conduit[IConduit]\n        FtpPulse[FTP : IPulse] --\u003e|transmitted via| Conduit[IConduit]\n        ApiPulse[API : IPulse] --\u003e|transmitted via| Conduit[IConduit]\n    end\n    subgraph \"Delivery\"\n        Conduit --\u003e|delivered to| Channel[Channel\u003cIPulse\u003cT\u003e\u003e]\n    end\n    subgraph \"Consumption and Routing\"\n        Channel --\u003e|consumed and routed by| Nexus[Nexus]\n        Nexus --\u003e|typeof==Email| EmailFlow[EmailFlow : IFlow]\n        Nexus --\u003e|typeof==API| Flow[FtpAndApiFlow : IFlow]\n        Nexus --\u003e|typeof==FTP| Flow[FtpAndApiFlow : IFlow]\n        Nexus --\u003e|typeof==?| AuditFlow[AuditFlow : IFlow]\n        AuditFlow --\u003e|audit trail by| ILogger[ILogger]\n    end\n```\n\nIn this Mermaid diagram:\n- **IPulse** is the interface for the Pulse.\n- **IConduit** is the interface for the Conduit, which is the pathway through which messages are transmitted.\n- **Nexus** is the central processing service, which handles the pulse messages and routes them to their respective destinations.\n- **IFlow** is the interface for the a flow, which is the mechanism that handles/consumes the pulse messages.\n- **ILogger** is the interface for the generic logger in dotnet.\n\nWhen you include this in a GitHub Markdown file, GitHub will render the Mermaid diagram as a visual graph. Remember to remove the extra backticks (```) in the beginning and end when adding this to your README.\n\n### Use Cases\n\nPulseFlow is a general-purpose messaging system that can be used in a wide variety of applications. It's particularly\nuseful in scenarios where there's a need for asynchronous communication between different parts of the system.\n\n\n#### Example 1 - Web API\n\nA use case is when you need multiple threads to do some non-multiple-thread-safe work, like saving to a log-file. Example: You have a web API and you want to save a all requests' route, method and user to a log-file. You can use PulseFlow to do this in a thread-safe manner, because PulseFlow's `IConduit` and `IChannel` are thread-safe. You can have multiple threads saving to the same log-file, and PulseFlow will ensure that the log-file is not written to by multiple threads at the same time, and that the writing happens in the order the messages are received.\n\n```mermaid\ngraph TB\n    subgraph Transmission\n        ApiPulse1[ApiMessage : IPulse] --\u003e|transmitted via| Conduit[IConduit]\n        ApiPulse2[ApiMessage : IPulse] --\u003e|transmitted via| Conduit[IConduit]\n        ApiPulse3[ApiMessage : IPulse] --\u003e|transmitted via| Conduit[IConduit]\n        ApiPulse4[ApiMessage : IPulse] --\u003e|transmitted via| Conduit[IConduit]\n        ApiPulse5[ApiMessage : IPulse] --\u003e|transmitted via| Conduit[IConduit]\n        ApiPulse6[ApiMessage : IPulse] --\u003e|transmitted via| Conduit[IConduit]\n    end\n    subgraph Delivery\n        Conduit --\u003e|delivered to| IChannel[Channel]\n    end\n    subgraph Consumption and Routing\n        IChannel --\u003e|consumed and routed by| Nexus[Nexus]\n        Nexus --\u003e|typeof==API| Flow[FileLoggerFlow : IFlow]\n    end\n```\n\n#### Example 2 - Priority Queue\n\nAnother use case is when you need to prioritize messages. You might have some user-input that needs to be processed, and you want to prioritize some messages over others. You can use PulseFlow to do this,whithout having to worry about thread-safety. You can have multiple threads processing the messages, and PulseFlow will ensure that the messages are processed in the order they are received. And so if you have a different type for each priority, you can have multiple threads processing each priority, \nand so when you have 10-15 \"normal\" messages, your single \"premium\" message will get processed at in its own thread.\n\n```mermaid\ngraph LR\n    PrioritizedWork[StandardMessage : IPulse] --\u003e Conduit[IConduit]\n    PrioritizedWork[StandardMessage : IPulse] --\u003e|standard| Conduit[IConduit]\n    PrioritizedWork[StandardMessage : IPulse] --\u003e Conduit[IConduit]\n    PrioritizedWork[PremiumMessage : IPulse] --\u003e|premium| Conduit[IConduit]\n    Conduit --\u003e|delivered to| IChannel[Channel]\n    IChannel --\u003e Nexus[Nexus]\n    \n    subgraph parallel processing\n        Nexus --\u003e StandardFlow[StandardFlow : IFlow]\n        Nexus --\u003e|typeof==StandardMessage| StandardFlow[StandardFlow : IFlow]\n        Nexus --\u003e StandardFlow[StandardFlow : IFlow]\n        Nexus --\u003e|typeof==PremiumMessage| PremiumFlow[PremiumFlow : IFlow]\n    end      \n    \n    StandardFlow --\u003e SomeResource[SomeResource]\n    PremiumFlow --\u003e SomeResource[SomeResource]\n```\n\n#### Example 3 - Complex Data Categorization\n\nIf you have layered data, and you want to categorize the data based on some of the layers, you can use PulseFlow to do this. Suppesed you have a data ingest system and all you get is a byte-array, and what you want is to parse the bytes to data that could represent an invoice. This can be done in \"classical\" logic structures, but this can mean large if/else or switch statements, and this can be hard to maintain or change. With PulseFlow you can get a more modular structure, where each step is a separate \nclass, and you can easily add new steps, or change the order of the steps, or even add parallel steps. And you can do this without having to worry about thread-safety unless you want to through access to some shared resource instead of using the `IFlow`s to pass data between the steps, e.g. if you write to a log-file from different steps. (This is not recommended, but it is possible. Correct way is to use `IFlow`s to pass data between steps, and then have a separate `IFlow` that writes to the log-file.)\n\n```mermaid\ngraph TB\n    File[Bytes : IPulse] --\u003e |PulseFlow| X1[IdentifierFlow]\n    X1 -.-\u003e|.xml| XmlFlow[XmlFlow]\n    X1 -.-\u003e|.json| JsonFlow[JsonFlow]\n    X1 -.-\u003e|.csv| CsvFlow[CsvFlow]\n\n    XmlFlow --\u003e|PulseFlow| XDocumentFlow\n    XDocumentFlow --\u003e|PulseFlow| XDocumentIdentifer\n    XDocumentIdentifer --\u003e|PEPPOL-PulseFlow| PeppolFlow --\u003e|UBL-PulseFlow| UblFlow\n    XDocumentIdentifer --\u003e|UBL-PulseFlow| UblFlow\n    \n    UblFlow --\u003e|PulseFlow| UblIdentifer\n    UblIdentifer --\u003e|PulseFlow| UblInvoiceFlow --\u003e|PulseFlow| InvoiceFlow\n    UblIdentifer --\u003e|PulseFlow| UblCreditNoteFlow\n    UblIdentifer --\u003e|PulseFlow| UblDebitNoteFlow --\u003e|PulseFlow| InvoiceFlow\n    UblIdentifer --\u003e|PulseFlow| UblReminderFlow --\u003e|PulseFlow| InvoiceFlow\n    \n    JsonFlow --\u003e|PulseFlow| JsonIdentifer\n    JsonIdentifer --\u003e|PulseFlow| JsonInvoiceFlow\n    JsonInvoiceFlow --\u003e|PulseFlow| InvoiceFlow\n    InvoiceFlow --\u003e|PulseFlow| InvoicePaymentFlow\n    \n    subgraph Pulse_Flow\n        subgraph Sender\n            PulseA -.-\u003e|Send| IConduit[IConduit]\n            PulseB -.-\u003e|Send| IConduit[IConduit]\n            PulseC -.-\u003e|Send| IConduit[IConduit]\n        end\n        IConduit --\u003e IChannel[Channel]\n        IChannel --\u003e Nexus[Nexus]\n        Nexus --\u003e FlowX[IFlow]\n        Nexus --\u003e FlowY[IFlow]\n    end\n```\n\n## Getting Started\n\nThis section provides a quick guide on how to get started with PulseFlow Local Messaging.\n\n### Installation\n\nPulseFlow is available as a NuGet package, which can be installed using the following command:\n\n```bash\ndotnet add package Frank.PulseFlow\n```\n\nOnce installed, you can start using PulseFlow by adding the following using statement to your code:\n\n```csharp\nusing Frank.PulseFlow;\n```\n\n### Basic Usage\n\nThe following code snippet shows a basic example of how to use PulseFlow:\n\n```csharp\npublic class Program\n{\n    public static async Task Main(string[] args)\n    {\n        IHostBuilder builder = Host.CreateDefaultBuilder();\n        builder.ConfigureServices((context, services) =\u003e\n        {\n            services.AddPulseFlow(messagingBuilder =\u003e\n            {\n                messagingBuilder.AddFlow\u003cTextPulseFlow\u003e();\n            });\n            services.AddHostedService\u003cTestingService\u003e();\n        });\n        IHost app = builder.Build();\n        await app.RunAsync();\n    }\n}\n\npublic class TextFlow : IFlow\n{\n    private readonly ILogger\u003cTextFlow\u003e _logger;\n    public TextFlow(ILogger\u003cTextFlow\u003e logger) =\u003e _logger = logger;\n    public async Task HandleAsync(IPulse message, CancellationToken cancellationToken)\n    {\n        if (message is TextPulse textMessage)\n            _logger.LogInformation(\"Received text message: {Text}\", textMessage.Text);\n        await Task.CompletedTask;\n    }\n    public bool CanHandle(Type pulseType) =\u003e pulseType == typeof(TextPulse);\n}\n\npublic class TestingService : BackgroundService\n{\n    private readonly IConduit _messenger;\n    public TestingService(IConduit messenger) =\u003e _messenger = messenger;\n    protected override async Task ExecuteAsync(CancellationToken stoppingToken)\n    {\n        await Task.Delay(2000, stoppingToken);\n        while (!stoppingToken.IsCancellationRequested)\n        {\n            await _messenger.SendAsync(new TextPulse { Id = Guid.NewGuid(), Text = \"Hello World\" });\n            await Task.Delay(1000, stoppingToken);\n        }\n    }\n}\n\npublic class TextPulse : BasePulse\n{\n    public string Text { get; set; }\n}\n```\n\n## Concepts\n\nThis section provides an in-depth explanation of the key concepts and components within the system: Nexus, Conduit,\nPulse, and PulseFlow. Understanding these concepts is crucial for grasping how the system operates and interacts with\ndata.\n\n### Nexus\n\nThe **Nexus** is the central hub of our messaging system, analogous to a neural network's core. It serves as the primary\nprocessing service, where all data messages, or 'Pulses', are received, interpreted, and routed to their respective\ndestinations.\n\n- **Role**: Nexus acts as the orchestrator within the system, managing the flow of messages and ensuring that each one\n  is processed according to predefined rules and logic.\n- **Functionality**: It handles various tasks like message validation, transformation, and decision-making on how and\n  where messages should be directed post-processing.\n- **Scalability and Performance**: Designed for high performance and scalability, Nexus can handle a vast volume of\n  messages simultaneously, ensuring minimal latency and high throughput in data processing.\n\n### Conduit\n\nThe **Conduit** represents the pathway through which messages, or 'Pulses', are transmitted within the system. It's the\nmessenger that ensures the delivery of data from one point to another.\n\n- **Mechanism**: Conduit facilitates the smooth and efficient transport of messages across different parts of the\n  system.\n- **Reliability and Integrity**: Ensuring data integrity, Conduit maintains the fidelity of the messages as they\n  traverse through various processes.\n- **Asynchronous Communication**: It supports asynchronous data flow, allowing for non-blocking and concurrent message\n  transmissions, which is key for a responsive and efficient system.\n\n### Pulse\n\n**Pulse** is the term used to describe the individual units of data or messages that flow through the system.\n\n- **Data Encapsulation**: Each Pulse is a packet of information, encapsulating the necessary data in a well-defined\n  format.\n- **Types and Variability**: Pulses can vary in type and structure, ranging from simple text messages to complex data\n  structures, each tailored to carry specific information relevant to its intended process.\n- **Lifecycle**: The lifecycle of a Pulse includes its creation, transmission through the Conduit, processing in the\n  Nexus, and final delivery or action as dictated by the system's logic.\n\n### Flow\n\n**Flow** is the sophisticated mechanism responsible for handling and manipulating the Pulses as they move through\nthe system.\n\n- **Message Handling**: It's specifically designed to process each Pulse, applying necessary transformations, routing,\n  and any other required operations.\n- **Flexibility and Adaptability**: PulseFlow is adept at handling various types of Pulses, capable of adapting its\n  processing logic based on the nature and requirements of each message.\n- **Integration Point**: Acting as a key integration point within the system, it ensures that Pulses are managed\n  efficiently and effectively, readying them for their next phase in the data journey.\n\n## License\n\nThis project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.\n\n## Contributing\n\nContributions are welcome! Please see [CONTRIBUTING.md](CONTRIBUTING.md) for more details.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffrankhaugen%2Ffrank.pulseflow","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffrankhaugen%2Ffrank.pulseflow","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffrankhaugen%2Ffrank.pulseflow/lists"}