{"id":16686627,"url":"https://github.com/houseofcat/rabbitmq.dataflows","last_synced_at":"2025-03-21T18:33:15.839Z","repository":{"id":173398276,"uuid":"283627127","full_name":"houseofcat/RabbitMQ.Dataflows","owner":"houseofcat","description":"A set of libraries for rapidly developing RabbitMQ workflows/pipelines.","archived":false,"fork":false,"pushed_at":"2024-05-01T17:12:11.000Z","size":2095,"stargazers_count":10,"open_issues_count":0,"forks_count":2,"subscribers_count":0,"default_branch":"main","last_synced_at":"2024-05-01T18:09:42.083Z","etag":null,"topics":["csharp","data","dataflow","dataflows","net","netcore","pipelines","queue","queueing","rabbit","rabbitmq","tpl","workflow","workflows"],"latest_commit_sha":null,"homepage":"","language":"C#","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/houseofcat.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":"FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":["houseofcat"]}},"created_at":"2020-07-30T00:01:18.000Z","updated_at":"2024-05-02T20:34:37.452Z","dependencies_parsed_at":null,"dependency_job_id":"d872b002-6d46-4474-8dbd-831551b69ba8","html_url":"https://github.com/houseofcat/RabbitMQ.Dataflows","commit_stats":{"total_commits":643,"total_committers":6,"mean_commits":"107.16666666666667","dds":0.1508553654743391,"last_synced_commit":"4dd8ccefcc199d810c58bd8b19dbd60d8fec0838"},"previous_names":["houseofcat/tesseract","houseofcat/library","houseofcat/rabbitdataflows","houseofcat/rabbitmq.dataflows"],"tags_count":41,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/houseofcat%2FRabbitMQ.Dataflows","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/houseofcat%2FRabbitMQ.Dataflows/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/houseofcat%2FRabbitMQ.Dataflows/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/houseofcat%2FRabbitMQ.Dataflows/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/houseofcat","download_url":"https://codeload.github.com/houseofcat/RabbitMQ.Dataflows/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244694784,"owners_count":20494729,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["csharp","data","dataflow","dataflows","net","netcore","pipelines","queue","queueing","rabbit","rabbitmq","tpl","workflow","workflows"],"created_at":"2024-10-12T15:06:27.413Z","updated_at":"2025-03-21T18:33:15.515Z","avatar_url":"https://github.com/houseofcat.png","language":"C#","readme":"# RabbitMQ \u0026 Dataflows\n\nA RabbitMQ library of `.NET` tools to help rapidly develop well performant services or\nto just help manage durable connectivity with the `RabbitMQ.Client`!  \n\n## Status\n\nUpdated: `05/03/2024`  \nTest Server: `Windows 11 Pro (v10.0.22631 [Build 22631])`  \nRabbitMQ Server: `v3.13.2`  \nErlang: `v26.2.5`  \n\n![Release](https://img.shields.io/github/v/release/houseofcat/RabbitMQ.Dataflows)\n[![build](https://github.com/houseofcat/RabbitMQ.Dataflows/workflows/build/badge.svg)](https://github.com/houseofcat/RabbitMQ.Dataflows/actions/workflows/build.yml)  \n[![Codacy Badge1](https://app.codacy.com/project/badge/Grade/2ac2a6f51a8c4efd88135bcb835e3a97)](https://app.codacy.com/gh/houseofcat/RabbitMQ.Dataflows/dashboard?utm_source=gh\u0026utm_medium=referral\u0026utm_content=\u0026utm_campaign=Badge_grade)   \n[![Codacy Badge2](https://app.codacy.com/project/badge/Coverage/2ac2a6f51a8c4efd88135bcb835e3a97)](https://app.codacy.com/gh/houseofcat/RabbitMQ.Dataflows/dashboard?utm_source=gh\u0026utm_medium=referral\u0026utm_content=\u0026utm_campaign=Badge_coverage)\n\n## Why Make A RabbitMQ Powered Dataflow?  \n\n`Dataflows` have configurable concurrency, serialization, monitoring, compression, and\nencryption all as first class citizens. This paradigm allows developers to just focus on\nthe important stuff - getting work done!\n\nHere are some features ready with `RabbitMQ.Dataflows`.\n\n### Workflows\n\n * Pipelines (v1) are light weight workflow-esque function-as-step executor.\n   * All steps process in the order provided allowing you to still control order of execution.  \n   * Configurable parallelism and concurrency built-in.\n   * No automatic OpenTelemetry.\n   * No automatic exception catching/async error handling (thrown exceptions kill Pipelines).\n * Dataflows (v2) are the more complex, yet complete, workflow class.\n   * All steps process in the order provided allowing you to still control order of execution.  \n   * Configurable parallelism and concurrency built-in.\n   * Async processing, batch processing, consumer cloning and more!  \n   * Async error handling (simplify functional error handling by allowing functions to throw).\n   * AutoPublish (to the next Queue) functionality built-in.\n   * OpenTelemetry with native distributed tracing for Publish/Consumer. \n\n### Built-Ins\n\n* A RabbitMQ `IConnectionPool` and `IChannelPool` (connection durability).  \n* Supports `ILogger\u003cT\u003e` via LogHelpers static singleton. \n* Contracted `IWorkState` simplifies functional generic returns and integration.  \n* Has `ISerializationProvider` support for `System.Text.Json` and `MessagePack`.\n  * Easy to write your own providers. \n* Has `ICompressionProvider` built-in support for Gzip, Deflate, Brotli, and LZ4.\n* Has `IEncryptionProvider` built-in support for AesGcm and BouncyCastle AesGcm.\n* RecyclableMemoryStream options for `ICompressionProvider` and `IEncryptionProvider`.\n* Publishers AutoPublish seamless encryption/decryption steps (adds headers too).  \n* Consumers allow seamless compression/decompression steps (uses headers).  \n* Publisher/Consumers have built-in `OpenTelemetry` support.\n\n### Testing\n\n* All built-in steps will have integration tests that should remove concerns from end-user \ndeveloper.   \n* Future case will include much more complex abstract UnitTesting as time allows.  \n* The developer should only need to unit test their functional business code.  \n\n## Guides, Code, and Examples\n\n * Getting started with *RabbitMQ.Dataflows* [ConnectionPool](https://github.com/houseofcat/RabbitMQ.Dataflows/blob/main/guides/rabbitmq/ConnectionPools.md).\n * Getting started with *RabbitMQ.Dataflows* [ChannelPool](https://github.com/houseofcat/RabbitMQ.Dataflows/blob/main/guides/rabbitmq/ChannelPools.md).\n * Getting started with *RabbitMQ.Dataflows* [ChannelPool and BasicPublish](https://github.com/houseofcat/RabbitMQ.Dataflows/blob/main/guides/rabbitmq/BasicPublish.md).\n * Getting started with *RabbitMQ.Dataflows* [ChannelPool and BasicGet](https://github.com/houseofcat/RabbitMQ.Dataflows/blob/main/guides/rabbitmq/BasicGet.md).\n * Getting started with *RabbitMQ.Dataflows* [ChannelPool and BasicConsume](https://github.com/houseofcat/RabbitMQ.Dataflows/blob/main/guides/rabbitmq/BasicConsume.md).\n * Getting started with *RabbitMQ.Dataflows* [Serialization](https://github.com/houseofcat/RabbitMQ.Dataflows/blob/main/guides/rabbitmq/Serialization.md).  \n * Getting started with *RabbitMQ.Dataflows* [Publisher](https://github.com/houseofcat/RabbitMQ.Dataflows/blob/main/guides/rabbitmq/Publisher.md).  \n * Getting started with *RabbitMQ.Dataflows* [AutoPublisher](https://github.com/houseofcat/RabbitMQ.Dataflows/blob/main/guides/rabbitmq/AutoPublisher.md).   \n * Getting started with *RabbitMQ.Dataflows* [Consumer](https://github.com/houseofcat/RabbitMQ.Dataflows/blob/main/guides/rabbitmq/Consumer.md).  \n * ***Getting started with RabbitMQ.Dataflows [ConsumerDataflow](https://github.com/houseofcat/RabbitMQ.Dataflows/blob/main/guides/rabbitmq/ConsumerDataflow.md)***.  \n\nMore to come and visual aids can be found towards the bottom!\n\nYou can also find various library examples inside the `tests/UnitTests` or the `tests/RabbitMQ.Console.Test` project.\n\n# Main RabbitMQ Library\n\n## HouseofCat.RabbitMQ\n[![NuGet](https://img.shields.io/nuget/v/HouseofCat.RabbitMQ.svg)](https://www.nuget.org/packages/HouseofCat.RabbitMQ/)  \n[![NuGet](https://img.shields.io/nuget/dt/HouseofCat.RabbitMQ.svg)](https://www.nuget.org/packages/HouseofCat.RabbitMQ/)  \n\nA library that focuses on RabbitMQ connection and channel management to create fault tolerant Publishers and Consumers.  \nFormerly called CookedRabbit.Core/Tesseract.\n\n# DataFlow Library\n\n## HouseofCat.Dataflows\n[![NuGet](https://img.shields.io/nuget/v/HouseofCat.Dataflows.svg)](https://www.nuget.org/packages/HouseofCat.Dataflows/)  \n[![NuGet](https://img.shields.io/nuget/dt/HouseofCat.Dataflows.svg)](https://www.nuget.org/packages/HouseofCat.Dataflows/)  \n\nA library that provides the base magic Dataflows for RabbitMQ.Dataflows. \n\n * Custom TPL Block - ChannelBock used as a Channel-based `BufferBlock\u003cTIn\u003e`\n * Has DataFlowEngine and ChannelBlockEngine.\n * Has Pipelines (Dataflow alternative).\n\n\n# Core Productivity Libraries\nThese libraries are here to help you build powerful Dataflows for your messages.\n\n## HouseofCat.Serialization\n[![NuGet](https://img.shields.io/nuget/v/HouseofCat.Serialization.svg)](https://www.nuget.org/packages/HouseofCat.Serialization/)  \n[![NuGet](https://img.shields.io/nuget/dt/HouseofCat.Serialization.svg)](https://www.nuget.org/packages/HouseofCat.Serialization/)  \n\nA library that has a collection of .NET `ISerializationProvider` or the interface to make your own.  \n * Supports MessagePack and System.Text.Json and Newtonsoft.Json.  \n\n## HouseofCat.Compression\n[![NuGet](https://img.shields.io/nuget/v/HouseofCat.Compression.svg)](https://www.nuget.org/packages/HouseofCat.Compression/)  \n[![NuGet](https://img.shields.io/nuget/dt/HouseofCat.Compression.svg)](https://www.nuget.org/packages/HouseofCat.Compression/)  \n\nA library that has a collection of .NET `ICompressionProvider` or the interface to make your own.\n\n * Supports LZ4, Gzip, Brotli, and Deflate.  \n * Supports RecyclableMemoryStream variants. \n\n## HouseofCat.Hashing\n[![NuGet](https://img.shields.io/nuget/v/HouseofCat.Hashing.svg)](https://www.nuget.org/packages/HouseofCat.Hashing/)  \n[![NuGet](https://img.shields.io/nuget/dt/HouseofCat.Hashing.svg)](https://www.nuget.org/packages/HouseofCat.Hashing/)  \n\nA library that focuses on implementing hashing.  \n\n * Supports Argon2.\n\n## HouseofCat.Encryption\n[![NuGet](https://img.shields.io/nuget/v/HouseofCat.Encryption.svg)](https://www.nuget.org/packages/HouseofCat.Encryption/)  \n[![NuGet](https://img.shields.io/nuget/dt/HouseofCat.Encryption.svg)](https://www.nuget.org/packages/HouseofCat.Encryption/)  \n\nA library that provides encryption contracts and the base `AesGCM`/`AesCBC` .NET `IEncryptionProvider` as\nwell as the interface to make your own.  \n\n * Supports AesCbc via CryptoStream (good for encrypted file/memorystreams).  \n * Supports .NET AesGcm 128, 192, 256 (non-streams).  \n * Supports BouncyCastle AesGcm 128/192/256.  \n * Supports RecyclableMemoryStream variants.  \n\n## HouseofCat.Utilities\n[![NuGet](https://img.shields.io/nuget/v/HouseofCat.Utilities.svg)](https://www.nuget.org/packages/HouseofCat.Utilities/)  \n[![NuGet](https://img.shields.io/nuget/dt/HouseofCat.Utilities.svg)](https://www.nuget.org/packages/HouseofCat.Utilities/)  \n\nA library that focuses on general purpose re-usable functions and extensions that simplify the coding experience.  \n\n# Non-Critical Library Integrations\n\n## HouseofCat.Data\n[![NuGet](https://img.shields.io/nuget/v/HouseofCat.Data.svg)](https://www.nuget.org/packages/HouseofCat.Data/)  \n[![NuGet](https://img.shields.io/nuget/dt/HouseofCat.Data.svg)](https://www.nuget.org/packages/HouseofCat.Data/)  \n\nA library that provides the provides helper classes for data manipulation and transformation. \n\nAlso provides Database abstractions, a simple Dapper integration, and SqlKata integration SQL query generation.\n\n### Database Connection Factory Support\n * System.Data.SqlClient\n * Microsoft.Data.SqlClient\n * MySql.Data.MySqlClient\n * Npgsq\n * MySql.Data\n * Oracle\n * SQLite\n\n## Visual Aids\n\nI have included a high level Excalidraw file with screenshots below. Be sure to open the file though\nfor better quality and more up-to-date information as I will continue enhancing it over time. The goal\nisn't to have everything in here, but I myself understand the flow of a library with the addition of\na visual aid. There is nothing truly special about this library, it's just the basics, but there are\na lot of moving parts when you step back.\n\n### RabbitService - Start Life Cycle\n![Rabbit Service Start](guides/rabbitmq/misc/rabbit_service_start.png)\n\n### Consumer - Start Life Cycle\n![Rabbit Service Consumer Start](guides/rabbitmq/misc/rabbit_service_consumer_start.png)\n\n### ConsumerDataflow - Data Flow\n![Consumer Dataflow Flow](guides/rabbitmq/misc/consumer_dataflow_flow.png)\n\n# [HouseofCat.io](https://houseofcat.io)","funding_links":["https://github.com/sponsors/houseofcat"],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhouseofcat%2Frabbitmq.dataflows","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhouseofcat%2Frabbitmq.dataflows","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhouseofcat%2Frabbitmq.dataflows/lists"}