{"id":20736869,"url":"https://github.com/eoincampbell/qluent","last_synced_at":"2025-04-24T01:41:38.755Z","repository":{"id":147036411,"uuid":"129434053","full_name":"eoincampbell/Qluent","owner":"eoincampbell","description":"Qluent provides a very simple Fluent API and wrapper classes around the Microsoft Azure Storage SDK","archived":false,"fork":false,"pushed_at":"2018-04-22T19:50:03.000Z","size":111,"stargazers_count":3,"open_issues_count":0,"forks_count":1,"subscribers_count":4,"default_branch":"master","last_synced_at":"2024-03-29T05:23:22.454Z","etag":null,"topics":["asynchronous","azure","azure-sdk","azure-storage","azure-storage-queue","consumer","csharp-library","dequeue","message-count","message-poison","message-queue","message-visibility","messaging","netcore2","netframework","pop","producer-consumer","receive-messages","serialization","storage-queue"],"latest_commit_sha":null,"homepage":"http://www.trycatch.me","language":"C#","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/eoincampbell.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-04-13T17:30:02.000Z","updated_at":"2019-01-23T13:55:06.000Z","dependencies_parsed_at":null,"dependency_job_id":"b1a9e508-75fe-489e-9bfa-f8a0797c5f98","html_url":"https://github.com/eoincampbell/Qluent","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eoincampbell%2FQluent","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eoincampbell%2FQluent/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eoincampbell%2FQluent/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eoincampbell%2FQluent/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/eoincampbell","download_url":"https://codeload.github.com/eoincampbell/Qluent/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250545650,"owners_count":21448246,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["asynchronous","azure","azure-sdk","azure-storage","azure-storage-queue","consumer","csharp-library","dequeue","message-count","message-poison","message-queue","message-visibility","messaging","netcore2","netframework","pop","producer-consumer","receive-messages","serialization","storage-queue"],"created_at":"2024-11-17T06:12:11.011Z","updated_at":"2025-04-24T01:41:38.741Z","avatar_url":"https://github.com/eoincampbell.png","language":"C#","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Qluent\n\n[![Nuget Stable][nuget-stable-badge]][nuget-stable-url]\n[![Nuget Beta][nuget-beta-badge]][nuget-beta-url]\n[![master][appveyor-master-badge]][appveyor-master-url]\n[![release][appveyor-release-badge]][appveyor-release-url]\n\n[nuget-stable-badge]: https://img.shields.io/badge/nuget--stable-1.0.0.35-blue.svg\n[nuget-stable-url]: https://www.nuget.org/packages/Qluent/\n\n[nuget-beta-badge]: https://img.shields.io/badge/nuget--beta-0.2.0.26--beta-orange.svg\n[nuget-beta-url]: https://www.nuget.org/packages/Qluent/\n\n[appveyor-master-badge]: https://ci.appveyor.com/api/projects/status/5uwjfc79j458m4ju/branch/master?svg=true\u0026passingText=master%20passing\u0026pendingText=master%20building\u0026failingText=master%20failing\n[appveyor-master-url]: https://ci.appveyor.com/project/eoincampbell/qluent/branch/master\n\n[appveyor-release-badge]: https://ci.appveyor.com/api/projects/status/5uwjfc79j458m4ju/branch/master?svg=true\u0026passingText=release%20v0.2.0%20passing\u0026pendingText=release%20v0.2.0%20building\u0026failingText=release%20v0.2.0%20failing\n[appveyor-release-url]: https://ci.appveyor.com/project/eoincampbell/qluent/branch/release/v0.2.0\n\n---\n\n***Qluent*** is a ***Fluent Queue Client***\n\nQluent provides a very simple Fluent API and wrapper classes around the Microsoft \nAzure Storage SDK, allowing you to interact with storage queues using \nstrongly typed objects like this.\n\n```csharp\nvar q = await Builder\n    .CreateAQueueOf\u003cTask\u003e()\n    .UsingStorageQueue(\"my-test-queue\")\n    .BuildAsync();\n    \nawait q.PushAsync(new Task());\n\nvar consumer = Builder\n    .CreateAConsumerFor\u003cTask\u003e()\n    .UsingQueue(q)\n    .ThatHandlesMessagesUsing((msg) =\u003e { Console.WriteLine($\"Processing {msg.Value.TaskId}\"); return true; })\n    .Build();\n\nawait consumer.Start()\n```\n\n---\n\n## Documentation\n\n - [Working with Queues](#working-with-queues)\n   - [Creating a Queue](#creating-a-queue)\n   - [Basic Operations](#basic-operations)\n   - [Sending Messages](#sending-messages)\n   - [Receiving Messages](#receiving-messages)\n   - [Receiving Messages and Controlling Deletion](#receiving-messages-and-controlling-deletion)\n - [Working with Queue Consumers](#working-with-queue-consumers)\n   - [Creating a Consumer](#creating-a-consumer)\n   - [Processing Messages](#processing-messages)\n   - [Handling Exceptions](#handling-exceptions)\n   - [Consumer Settings](#consumer-settings)\n   - [Logging](#Logging)\n - [Advanced Features](#advanced-features)\n   - [Message Visibility](#message-visibility)\n   - [Handling Poison Messages](#handling-poison-messages)\n   - [Customising Serialization](#customising-serialization)\n   - [Asynchronous Model](#asynchronous-model)\n - [Background](#background)\n   - [Why do I need this?](#why-do-i-need-this)\n   - [Why did you build this?](#why-did-you-build-this)\n   - [What is this not?](#what-is-this-not)\n   - [Todo List](#todo-list)\n\n---\n\n## Working with Queues\n\n\n### Creating a Queue\n\nBy default the builder will create a queue connected to development storage.\n\n```csharp\nvar q = await Builder\n    .CreateAQueueOf\u003cPerson\u003e()\n    .UsingStorageQueue(\"my-test-queue\")\n    .BuildAsync();\n```\n\nOr you can explicitly provide a connection string to a specific storage account.\n\n```csharp\nvar q = await Builder\n    .CreateAQueueOf\u003cPerson\u003e()\n    .ConnectedToAccount(\"UseDevelopmentStorage=true\")\n    .UsingStorageQueue(\"my-test-queue\")\n    .BuildAsync();\n```\n\n---\n\n### Basic Operations\n\nYou can clear all messages from a queue.\n\n```csharp\nawait q.PurgeAsync();\n```\n\nYou can check the approximate message count on a queue.\n\n```csharp\nvar count = await q.CountAsync()\n```\n\n---\n\n### Sending Messages\n\nQueues are created for a specific type. You can push an object of \nthat type directly to the queue.\n\n```csharp\nvar q = await Builder\n    .CreateAQueueOf\u003cPerson\u003e()\n    .UsingStorageQueue(\"my-test-queue\")\n    .BuildAsync(); \n    \nvar person = new Person(\"John\");\nawait q.PushAsync(person);\n``` \n\nYou can also push an `IEnumerable\u003cT\u003e` of messages to the queue.\n\n```csharp\nList\u003cPerson\u003e people = new List\u003cPerson\u003e();\nawait q.PushAsync(people);\n```\n\n---\n\n### Receiving Messages\n\nYou can directly Pop an object off the queue. This will dequeue the \n`CloudQueueMessage`, attempt to deserialize it and if deserialization succeeds\nremove it from the queue and return it to you.\n\nIf deserialization fails, the default behavior is to throw an exception. \nThis will result in the message's dequeue count increasing, and it\nreappearing on the queue after it's visibility timeout expires.\n\nSee: [Handling Poison Messages] for more info.\n\n```csharp\nvar q = await Builder\n    .CreateAQueueOf\u003cPerson\u003e()\n    .UsingStorageQueue(\"my-test-queue\")\n    .BuildAsync(); \n    \nvar person = await q.PopAsync();\n``` \n\nIf you don't want to remove the object from the queue, you can peek at it instead.\n\n```csharp\nvar person = await q.PeekAsync();\n``` \n\nYou can also Peek or Pop multiple messages at a time by passing a message count.\n\n```csharp\nIEnumerable\u003cPerson\u003e peekedPeople = await q.PeekAsync(5);\n\nIEnumerable\u003cPerson\u003e poppedPeople = await q.PopAsync(5);\n```\n\n---\n\n### Receiving Messages \u0026 Controlling Deletion\n\nThe Azure SDK supports a two phase dequeue process. First, the message is \nreceived from the queue. Second, the message is deleted from the queue. \nThis allows a consumer to attempt processing in between these two steps, \nand if processing fails, the client can abort the operation and the message \nwill appear on the queue again after it's visibilty timeout expires.\n\nThe previous `PopAsync` methods perform both Get \u0026 Delete operations in one.\n\nIf more control is required, by the consumer, Qluent also provides:\n - `GetAsync`, which returns a `IMessage\u003cT\u003e` wrapper object including the underlying message Id \u0026 PopReceipt\n - `DeleteAsync`, which accepts a `IMessage\u003cT\u003e`\n\n```csharp\nvar q = await Builder\n    .CreateAQueueOf\u003cPerson\u003e()\n    .UsingStorageQueue(\"my-test-queue\")\n    .BuildAsync(); \n\nvar wrappedPerson = await q.GetAsync();\n\ntry\n{    \n    //attempt to process wrappedPerson.Value;\n    await q.DeleteAsync(wrappedPerson);\n}\ncatch(Exception ex){ ... }\n```\n\n---\n\n## Working with Queue Consumers\n\nQluent provides a simple message consumer which you can provide with a \nqueue and configure to handle your messages. This allows you to focus on\nthe message processing without worrying about the dispatcher/polling logic.\n\n---\n\n### Creating a Consumer\n\nTo create a consumer, you first build a queue, and then build a consumer using that queue.\n\n```csharp\n//Create Queue\nvar consumerQueue = await Builder\n    .CreateAQueueOf\u003cJob\u003e()\n    .UsingStorageQueue(\"my-job-queue\")\n    .BuildAsync();\n\n//Create Consumer\nvar consumer = Builder\n    .CreateAMessageConsumerFor\u003cJob\u003e()\n    .UsingQueue(consumerQueue)\n    ...\n    .Build();\n```\n\nTo run the consumer, you call the `Start()` method. The polling loop can be terminated by triggering a cancellation token.\n\n```csharp\nvar cancellationTokenSource = new CancellationTokenSource();\nawait consumer.Start(cancellationTokenSource.Token);\n\n//later\ncancellationTokenSource.Cancel();\n``` \n\n---\n\n### Processing Messages\n\nThe message consumer supports 2 handlers for processing your messages.\n\n1. A message handler which describes how to process your messages\n2. A failed message handler which describes what to do when the message handler fails\n\n```csharp\nvar consumer = Builder\n    .CreateAMessageConsumerFor\u003cJob\u003e()\n    .UsingQueue(consumerQueue)\n    .ThatHandlesMessagesUsing(HandleMessage)\n    .AndHandlesFailedMessagesUsing(HandleFailure)\n    .Build();\n``` \n\nThe message handlers can be passed to the fluent api as either\n\n - a `Func\u003cIMessage\u003cT\u003e, bool\u003e` \n - an implementation of `Qluent.Consumers.Handlers.IMessageHandler\u003cT\u003e`\n\n---\n\n### Handling Exceptions\n\nIn the event the main message handler throws an exception, \na special handler is available for post-processing messages.\n\n```csharp\nvar consumer = Builder\n    .CreateAMessageConsumerFor\u003cJob\u003e()\n    .UsingQueue(consumerQueue)\n    ...\n    .AndHandlesExceptionsUsing(HandleException)\n    .Build();\n``` \n\nThe message exception handler can be passed to the fluent api as either\n\n - a `Func\u003cIMessage\u003cT\u003e, Exception, bool\u003e` \n - a implementation of `Qluent.Consumers.Handlers.IMessageExceptionHandler\u003cT\u003e`\n\n---\n\n### Consumer Settings\n\nThe consumer supports a number of other settings as well. \n\nYou can specify a unique Id for your consumer. This is useful for logging if you intend to \nhost multiple consumers in the same app.\n\n```csharp\nvar consumer = Builder\n    .CreateAMessageConsumerFor\u003cJob\u003e()\n    .UsingQueue(consumerQueue)\n    .WithAnIdOf(\"my-custom-id\")\n    ...\n    .Build();\n```\n\nYou can specify whether or not an unhandled exception should kill or cause your app to continue.\nCan be used in conjunction with the exception handling behavior of a queue to bubble and trap serialization exceptions.\n\n```csharp\nvar consumer = Builder\n    .CreateAMessageConsumerFor\u003cJob\u003e()\n    .UsingQueue(consumerQueue)\n    .AndHandlesExceptions(By.Continuing)\n    ...\n    .Build();\n```\n\nWhile polling an empty queue the Consumer can be configured how often to re-poll while waiting. \nBy default the consumer will requery the queue every 5 seconds.\n\nYou can override this behavior by providing a custom `Qluent.Consumers.Policies.IMessageConsumerQueuePollingPolicy`\n\nThe library provides 2 built in policies\n\n - `SetIntervalQueuePollingPolicy` which specifies a set number of seconds between polls\n - `BackOffQueuePolingPolicy` which backs off the poling frequency by doubling the interval from 1 second to 60 seconds               \n\n```csharp\nvar consumer = Builder\n    .CreateAMessageConsumerFor\u003cJob\u003e()\n    .UsingQueue(consumerQueue)\n    ...\n    .WithAQueuePolingPolicyOf(new SetIntervalQueuePolingPolicy(5))\n    .Build();\n```\n\n---\n\n### Logging\n\nThe consumer supports very basic logging using NLog.\n\nSimply configure NLog as you typically would (specifying targets, formats and filter rules) and the consumer will output \nmessages on progress including startup, shutdown, handler execution \u0026 exception messages.\n\n---\n\n## Advanced Features\n\nThe `IAzureStorageQueue` Builder supports a number of other advanced features that let\nyou control the more common aspects of StorageQueues without having to dive into the SDK.\n\n---\n\n### Message Visibility\n\nYou can provide a number of settings to override the various message visbility \nand time to live settings.\n\nYou can set a delay time before the message appears to consumers on the queue.\n\n```csharp\nvar q = await Builder\n    .CreateAQueueOf\u003cPerson\u003e()\n    .UsingStorageQueue(\"my-test-queue\")\n    .ThatDelaysMessageVisibilityAfterEnqueuingFor(TimeSpan.FromMinutes(1))\n    .BuildAsync();\n``` \n\nYou can specify the duration that a message remains invisible for after it's \nbeen dequeued, useful in combination with handling poison messages.\n\n```csharp\nvar q = await Builder\n    .CreateAQueueOf\u003cPerson\u003e()\n    .UsingStorageQueue(\"my-test-queue\")\n    .ThatKeepsMessagesInvisibleAfterDequeuingFor(TimeSpan.FromMinutes(1))\n    .BuildAsync();\n``` \n\nYou can specify the duration that message will remain alive on the queue if \nno consumers dequeue them.\n\n```csharp\nvar q = await Builder\n    .CreateAQueueOf\u003cPerson\u003e()\n    .UsingStorageQueue(\"my-test-queue\")\n    .ThatSetsAMessageTTLOf(TimeSpan.FromDays(1))\n    .BuildAsync();\n```\n\n---\n\n### Handling Poison Messages\n\nWhen a message is removed from a Storage Queue, Qluent will attempt \nto deserialize it into the `\u003cT\u003e` you specified.\n\nIt is possible that deserialization might fail for a number of reasons. e.g. \nAn unexpected/corrupted message may have been added to the queue which you cannot parse.\n\nYou can control how many times the library will attempt to dequeue and deserialize \nfor you before it considers the message poison. Once considered poisonly, you can optionally\nchoose to route it to another queue for analysis/later processing.\n\n```csharp\nvar jobQueue = await Builder\n    .CreateAQueueOf\u003cPerson\u003e()\n    .UsingStorageQueue(\"my-test-queue\")\n    .ThatConsidersMessagesPoisonAfter(3)\n    .AndSendsPoisonMessagesTo(\"my-poison-queue\")\n    .BuildAsync();\n```\n\nYou can specify what should happen when a poison message is detected. The default behavior \nis to throw an exception each time the message fails to deserialize. \nYou can override that behavior by specifying that exceptions should be swallowed. \nIn this case, the Pop/Peek method will return null (or will remove the null result \nfrom an `IEnumerable\u003cT\u003e`).\n\n\n```csharp\nvar jobQueue = await Builder\n    .CreateAQueueOf\u003cJob\u003e()\n    .UsingStorageQueue(\"my-test-queue\")\n    .ThatConsidersMessagesPoisonAfter(3)\n    .AndSendsPoisonMessagesTo(\"my-poison-queue\")\n    .AndHandlesExceptionsOnPoisonMessages(By.SwallowingExceptions)\n    .BuildAsync();\n```\n\n---\n\n### Asynchronous Model\n\nThe library is built against the .NET Standard 2.0 to target both .NET Framework \n\u0026 .NET Core. All Operations are asynchronous and support a method overload to pass\na cancellation token.\n\n```csharp\nvar person = await q.PopAsync();\n\nCancellationToken ct = new CancellationToken(false);\nvar person = await q.PopAsync(ct);\n```\n\nWithin the library all asynchronous calls are postpended with a call to `.ConfigureAwait(false)`.\n\nDuring queue creation the library will perform an async operation to create the queue \nif it doesn't exist. Therefore the builder (and previous examples) provide an awaitable\n`BuildAsync()` method. However if you need to create your queues in a non async manner\ne.g. in a DI Container/Bootstrapper you can use the non async `Build` method.\n\n```csharp\nvar q = Builder\n    .CreateAQueueOf\u003cPerson\u003e()\n    .ConnectedToAccount(\"UseDevelopmentStorage=true\")\n    .UsingStorageQueue(\"my-test-queue\")\n    .Build();\n```\n\n---\n\n### Customising Serialization\n\nBy default Qluent will serialize your entities to Json Strings using \n[Json.NET](https://www.newtonsoft.com/json).\nSerialization is performed using the default `JsonConvert` utility.\n\nFor scenarios, where your client does not control both ends of the queue, you may have \nto deal with messages that have been serialized differently. \n\nTo support this, Qluent allows your to pass your own custom binary or string serializer.\n\nTo create a custom binary serializer, implement the interface `Qluent.Serialization.IBinaryMessageSerializer\u003cT\u003e`\nThis will serialize/deserialize your message to a `byte[]` and push/pop it to the queue as bytes.\n```csharp\nvar q = Builder\n    .CreateAQueueOf\u003cPerson\u003e()\n    .ConnectedToAccount(\"UseDevelopmentStorage=true\")\n    .UsingStorageQueue(\"my-test-queue\")\n    .WithACustomSerializer(new CustomBinarySerializer())\n    .Build();\n``` \n\nTo create a custom binary serializer, implement the interface `Qluent.Serialization.IStringMessageSerializer\u003cT\u003e`\nThis will serialize/deserialize your message to a `string` and push/pop it to the queue as string content.\n\n```csharp\nvar q = Builder\n    .CreateAQueueOf\u003cPerson\u003e()\n    .ConnectedToAccount(\"UseDevelopmentStorage=true\")\n    .UsingStorageQueue(\"my-test-queue\")\n    .WithACustomSerializer(new CustomStringSerializer())\n    .Build();\n``` \n\n---\n\n## Background\n\n### Why do I need this?\n\nThere's a lot of ceremony involved when using the SDK for Azure Storage Queues. \nCreate an account, create a client, create a queue reference, make sure it exists,\nobject serialization/deserialization etc...\n\n```csharp\nvar storageAccount = CloudStorageAccount.Parse(\"UseDevelopmentStorage=true;\");\nvar queueClient = storageAccount.CreateCloudQueueClient();\nvar queue = queueClient.GetQueueReference(\"myqueue\");\nqueue.CreateIfNotExists();\n\nvar person = new Person(\"John\");\nvar serializedPerson = JsonConvert.Serialize(person)\nvar message = new CloudQueueMessage(serializedPerson); \nqueue.AddMessage(message);\n\nvar result = queue.GetMessage();\nvar deserializedPerson = JsonConvert.Deserialize\u003cPerson\u003e(result.AsString);\nqueue.DeleteMessage(result);\n```\n\nI'm also not a fan of the architectural decision in the SDK to leave settings like\nmessage visbility up to the developer to decide on at the call site. If you're \ngoing to create your queues and access them via a DI framework, I'd prefer to \ncentralize/standardize these settings at queue creation.\n\n---\n\n### Why did you build this?\n\nThis project was borne out frustration with a several different things. \n\nFirstly, my team and I have recently been working with Azure Durable Functions. They are still quite immature \nand given their static nature and the lack of support for DI frameworks like Autofac\nI found we were having to either write a lot of boiler plate code or rely on the built\nin binding/trigger capabilities which were lacking in certain amounts of control/capability. \nAfter seeing yet-another-queue-wrapper being written, I wanted to consolidate our approach.\n\nSecondly, we have been wrestling with a legacy bug for a while now, which was the result of a predecessors\ndecision to take a dependency on a nuget package for which support had long since waned and for which\nthe source code was no longer available. After eventually decompiling and picking through some enterprise-fizz-buzz\nqueue code, the problem was discovered. \n\nThirdly, I needed to scratch an itch :smirk:. I wanted to test out a number of things including building a fluent builder pattern, \nbuilding a net standard 2.0 library that could be consumed by both NetFramework and NetCore20.\n\n---\n\n### What is this not?\n\nThis is not an Enterprise Service Bus. It is a simple wrapper around Azure Storage \nQueues to make working with them a little easier.\n\nThere are lots of complicated things you may find yourself doing in a \ndistributed environment. Complex Retry Policies; complicated routing paths; \nPub/Sub models involving topics and queues; the list goes on.\n\nIf you find yourself needing to do something complex like this, then perhaps you should \nbe looking at a different technology stack (Azure Service Bus, Event Hubs, Event Grid, \nKafka, NService Bus, Mulesoft etc...)\n\n---\n\n### Todo List\n\n- ~~Interface based Refactoring~~\n- ~~Document calls properly~~\n- ~~Support Cancellation Tokens so that they can be passed through.~~\n- ~~Support Pop Receipts so that the consumer can decide how to handle messages~~\n- ~~Write up the docs around message visibility when the above is done~~\n- ~~.NET Core Tests~~\n- ~~Remove the TransactionScope stuff \u0026 stick it in an experimental branch~~\n- ~~Build a Sample Message Consumer~~\n  - ~~Test Harness~~\n  - Unit Tests\n  - ~~Documentation~~\n  - ~~XMLDoc Comment the Public API~~\n- Logging\n  - ~~NLog~~\n  - Serilog\n- ~~Big Documentation Tidy up~~\n- ~~Nuget Packages~~\n- ~~AppVeyor Setup~~","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Feoincampbell%2Fqluent","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Feoincampbell%2Fqluent","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Feoincampbell%2Fqluent/lists"}