https://github.com/houseofcat/rabbitmq.dataflows
A set of libraries for rapidly developing RabbitMQ workflows/pipelines.
https://github.com/houseofcat/rabbitmq.dataflows
csharp data dataflow dataflows net netcore pipelines queue queueing rabbit rabbitmq tpl workflow workflows
Last synced: 7 months ago
JSON representation
A set of libraries for rapidly developing RabbitMQ workflows/pipelines.
- Host: GitHub
- URL: https://github.com/houseofcat/rabbitmq.dataflows
- Owner: houseofcat
- License: mit
- Created: 2020-07-30T00:01:18.000Z (about 5 years ago)
- Default Branch: main
- Last Pushed: 2024-05-01T17:12:11.000Z (over 1 year ago)
- Last Synced: 2024-05-01T18:09:42.083Z (over 1 year ago)
- Topics: csharp, data, dataflow, dataflows, net, netcore, pipelines, queue, queueing, rabbit, rabbitmq, tpl, workflow, workflows
- Language: C#
- Homepage:
- Size: 2 MB
- Stars: 10
- Watchers: 0
- Forks: 2
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- Funding: FUNDING.yml
- License: LICENSE
Awesome Lists containing this project
README
# RabbitMQ & Dataflows
A RabbitMQ library of `.NET` tools to help rapidly develop well performant services or
to just help manage durable connectivity with the `RabbitMQ.Client`!
## Status
Updated: `05/03/2024`
Test Server: `Windows 11 Pro (v10.0.22631 [Build 22631])`
RabbitMQ Server: `v3.13.2`
Erlang: `v26.2.5`
[](https://github.com/houseofcat/RabbitMQ.Dataflows/actions/workflows/build.yml)
[](https://app.codacy.com/gh/houseofcat/RabbitMQ.Dataflows/dashboard?utm_source=gh&utm_medium=referral&utm_content=&utm_campaign=Badge_grade)
[](https://app.codacy.com/gh/houseofcat/RabbitMQ.Dataflows/dashboard?utm_source=gh&utm_medium=referral&utm_content=&utm_campaign=Badge_coverage)
## Why Make A RabbitMQ Powered Dataflow?
`Dataflows` have configurable concurrency, serialization, monitoring, compression, and
encryption all as first class citizens. This paradigm allows developers to just focus on
the important stuff - getting work done!
Here are some features ready with `RabbitMQ.Dataflows`.
### Workflows
* Pipelines (v1) are light weight workflow-esque function-as-step executor.
* All steps process in the order provided allowing you to still control order of execution.
* Configurable parallelism and concurrency built-in.
* No automatic OpenTelemetry.
* No automatic exception catching/async error handling (thrown exceptions kill Pipelines).
* Dataflows (v2) are the more complex, yet complete, workflow class.
* All steps process in the order provided allowing you to still control order of execution.
* Configurable parallelism and concurrency built-in.
* Async processing, batch processing, consumer cloning and more!
* Async error handling (simplify functional error handling by allowing functions to throw).
* AutoPublish (to the next Queue) functionality built-in.
* OpenTelemetry with native distributed tracing for Publish/Consumer.
### Built-Ins
* A RabbitMQ `IConnectionPool` and `IChannelPool` (connection durability).
* Supports `ILogger` via LogHelpers static singleton.
* Contracted `IWorkState` simplifies functional generic returns and integration.
* Has `ISerializationProvider` support for `System.Text.Json` and `MessagePack`.
* Easy to write your own providers.
* Has `ICompressionProvider` built-in support for Gzip, Deflate, Brotli, and LZ4.
* Has `IEncryptionProvider` built-in support for AesGcm and BouncyCastle AesGcm.
* RecyclableMemoryStream options for `ICompressionProvider` and `IEncryptionProvider`.
* Publishers AutoPublish seamless encryption/decryption steps (adds headers too).
* Consumers allow seamless compression/decompression steps (uses headers).
* Publisher/Consumers have built-in `OpenTelemetry` support.
### Testing
* All built-in steps will have integration tests that should remove concerns from end-user
developer.
* Future case will include much more complex abstract UnitTesting as time allows.
* The developer should only need to unit test their functional business code.
## Guides, Code, and Examples
* Getting started with *RabbitMQ.Dataflows* [ConnectionPool](https://github.com/houseofcat/RabbitMQ.Dataflows/blob/main/guides/rabbitmq/ConnectionPools.md).
* Getting started with *RabbitMQ.Dataflows* [ChannelPool](https://github.com/houseofcat/RabbitMQ.Dataflows/blob/main/guides/rabbitmq/ChannelPools.md).
* Getting started with *RabbitMQ.Dataflows* [ChannelPool and BasicPublish](https://github.com/houseofcat/RabbitMQ.Dataflows/blob/main/guides/rabbitmq/BasicPublish.md).
* Getting started with *RabbitMQ.Dataflows* [ChannelPool and BasicGet](https://github.com/houseofcat/RabbitMQ.Dataflows/blob/main/guides/rabbitmq/BasicGet.md).
* Getting started with *RabbitMQ.Dataflows* [ChannelPool and BasicConsume](https://github.com/houseofcat/RabbitMQ.Dataflows/blob/main/guides/rabbitmq/BasicConsume.md).
* Getting started with *RabbitMQ.Dataflows* [Serialization](https://github.com/houseofcat/RabbitMQ.Dataflows/blob/main/guides/rabbitmq/Serialization.md).
* Getting started with *RabbitMQ.Dataflows* [Publisher](https://github.com/houseofcat/RabbitMQ.Dataflows/blob/main/guides/rabbitmq/Publisher.md).
* Getting started with *RabbitMQ.Dataflows* [AutoPublisher](https://github.com/houseofcat/RabbitMQ.Dataflows/blob/main/guides/rabbitmq/AutoPublisher.md).
* Getting started with *RabbitMQ.Dataflows* [Consumer](https://github.com/houseofcat/RabbitMQ.Dataflows/blob/main/guides/rabbitmq/Consumer.md).
* ***Getting started with RabbitMQ.Dataflows [ConsumerDataflow](https://github.com/houseofcat/RabbitMQ.Dataflows/blob/main/guides/rabbitmq/ConsumerDataflow.md)***.
More to come and visual aids can be found towards the bottom!
You can also find various library examples inside the `tests/UnitTests` or the `tests/RabbitMQ.Console.Test` project.
# Main RabbitMQ Library
## HouseofCat.RabbitMQ
[](https://www.nuget.org/packages/HouseofCat.RabbitMQ/)
[](https://www.nuget.org/packages/HouseofCat.RabbitMQ/)
A library that focuses on RabbitMQ connection and channel management to create fault tolerant Publishers and Consumers.
Formerly called CookedRabbit.Core/Tesseract.
# DataFlow Library
## HouseofCat.Dataflows
[](https://www.nuget.org/packages/HouseofCat.Dataflows/)
[](https://www.nuget.org/packages/HouseofCat.Dataflows/)
A library that provides the base magic Dataflows for RabbitMQ.Dataflows.
* Custom TPL Block - ChannelBock used as a Channel-based `BufferBlock`
* Has DataFlowEngine and ChannelBlockEngine.
* Has Pipelines (Dataflow alternative).
# Core Productivity Libraries
These libraries are here to help you build powerful Dataflows for your messages.
## HouseofCat.Serialization
[](https://www.nuget.org/packages/HouseofCat.Serialization/)
[](https://www.nuget.org/packages/HouseofCat.Serialization/)
A library that has a collection of .NET `ISerializationProvider` or the interface to make your own.
* Supports MessagePack and System.Text.Json and Newtonsoft.Json.
## HouseofCat.Compression
[](https://www.nuget.org/packages/HouseofCat.Compression/)
[](https://www.nuget.org/packages/HouseofCat.Compression/)
A library that has a collection of .NET `ICompressionProvider` or the interface to make your own.
* Supports LZ4, Gzip, Brotli, and Deflate.
* Supports RecyclableMemoryStream variants.
## HouseofCat.Hashing
[](https://www.nuget.org/packages/HouseofCat.Hashing/)
[](https://www.nuget.org/packages/HouseofCat.Hashing/)
A library that focuses on implementing hashing.
* Supports Argon2.
## HouseofCat.Encryption
[](https://www.nuget.org/packages/HouseofCat.Encryption/)
[](https://www.nuget.org/packages/HouseofCat.Encryption/)
A library that provides encryption contracts and the base `AesGCM`/`AesCBC` .NET `IEncryptionProvider` as
well as the interface to make your own.
* Supports AesCbc via CryptoStream (good for encrypted file/memorystreams).
* Supports .NET AesGcm 128, 192, 256 (non-streams).
* Supports BouncyCastle AesGcm 128/192/256.
* Supports RecyclableMemoryStream variants.
## HouseofCat.Utilities
[](https://www.nuget.org/packages/HouseofCat.Utilities/)
[](https://www.nuget.org/packages/HouseofCat.Utilities/)
A library that focuses on general purpose re-usable functions and extensions that simplify the coding experience.
# Non-Critical Library Integrations
## HouseofCat.Data
[](https://www.nuget.org/packages/HouseofCat.Data/)
[](https://www.nuget.org/packages/HouseofCat.Data/)
A library that provides the provides helper classes for data manipulation and transformation.
Also provides Database abstractions, a simple Dapper integration, and SqlKata integration SQL query generation.
### Database Connection Factory Support
* System.Data.SqlClient
* Microsoft.Data.SqlClient
* MySql.Data.MySqlClient
* Npgsq
* MySql.Data
* Oracle
* SQLite
## Visual Aids
I have included a high level Excalidraw file with screenshots below. Be sure to open the file though
for better quality and more up-to-date information as I will continue enhancing it over time. The goal
isn't to have everything in here, but I myself understand the flow of a library with the addition of
a visual aid. There is nothing truly special about this library, it's just the basics, but there are
a lot of moving parts when you step back.
### RabbitService - Start Life Cycle
### Consumer - Start Life Cycle
### ConsumerDataflow - Data Flow
# [HouseofCat.io](https://houseofcat.io)