{"id":19100180,"url":"https://github.com/blehnen/DotNetWorkQueue","last_synced_at":"2025-04-18T17:32:59.541Z","repository":{"id":41344746,"uuid":"45283749","full_name":"blehnen/DotNetWorkQueue","owner":"blehnen","description":"A work queue for dot.net with SQL server, SQLite, Redis and PostGreSQL backends","archived":false,"fork":false,"pushed_at":"2024-08-20T15:19:32.000Z","size":8152,"stargazers_count":97,"open_issues_count":0,"forks_count":14,"subscribers_count":8,"default_branch":"master","last_synced_at":"2024-10-29T15:34:06.191Z","etag":null,"topics":["dotnet","dotnet-core","job-scheduler","redis-queue","workqueue"],"latest_commit_sha":null,"homepage":"","language":"C#","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/blehnen.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"License.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2015-10-31T00:59:27.000Z","updated_at":"2024-10-02T14:53:59.000Z","dependencies_parsed_at":"2023-01-25T16:16:25.148Z","dependency_job_id":"79d797f5-7333-4097-954d-614ca649ee5b","html_url":"https://github.com/blehnen/DotNetWorkQueue","commit_stats":null,"previous_names":[],"tags_count":39,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/blehnen%2FDotNetWorkQueue","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/blehnen%2FDotNetWorkQueue/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/blehnen%2FDotNetWorkQueue/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/blehnen%2FDotNetWorkQueue/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/blehnen","download_url":"https://codeload.github.com/blehnen/DotNetWorkQueue/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":223478761,"owners_count":17151891,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dotnet","dotnet-core","job-scheduler","redis-queue","workqueue"],"created_at":"2024-11-09T03:52:32.901Z","updated_at":"2024-11-09T03:53:15.742Z","avatar_url":"https://github.com/blehnen.png","language":"C#","funding_links":[],"categories":["dotnet"],"sub_categories":[],"readme":"DotNetWorkQueue\r\n=========\r\n\r\n[![License LGPLv2.1](https://img.shields.io/badge/license-LGPLv2.1-green.svg)](http://www.gnu.org/licenses/lgpl-2.1.html)\r\n[![Build status](https://ci.appveyor.com/api/projects/status/vqqq9m0j9xodbfof/branch/master?svg=true)](https://ci.appveyor.com/project/blehnen/dotnetworkqueue/branch/master)\r\n[![Coverity status](https://scan.coverity.com/projects/10126/badge.svg)](https://scan.coverity.com/projects/blehnen-dotnetworkqueue)\r\n[![codecov](https://codecov.io/gh/blehnen/DotNetWorkQueue/branch/master/graph/badge.svg?token=E23UZ6U9CU)](https://codecov.io/gh/blehnen/DotNetWorkQueue)\r\n\r\nA producer / distributed consumer library for dot net applications. Dot net 4.6.2, 4.7.2, 4.8, 6.0, 8.0 and Dot net standard 2.0 are supported\r\n\r\nHigh level features\r\n\r\n* Queue / De-queue POCO for distributed processing\r\n* Queue / Process LINQ statements; compiled or dyanamic (expressed as string)\r\n* Re-occurring job scheduler\r\n\r\nSee the [Wiki](https://github.com/blehnen/DotNetWorkQueue/wiki) for more indepth documention\r\n\r\nInstallation\r\n-------------\r\n\r\nBase\r\n* NuGet package [DotNetWorkQueue](https://www.nuget.org/packages/DotNetWorkQueue/)\r\n\r\nTransports\r\n\r\n* NuGet package [DotNetWorkQueue.Transport.Redis](https://www.nuget.org/packages/DotNetWorkQueue.Transport.Redis/)\r\n* NuGet package [DotNetWorkQueue.Transport.SqlServer](https://www.nuget.org/packages/DotNetWorkQueue.Transport.SqlServer/)\r\n* NuGet package [DotNetWorkQueue.Transport.SQLite](https://www.nuget.org/packages/DotNetWorkQueue.Transport.SQLite/)\r\n* NuGet package [DotNetWorkQueue.Transport.PostgreSQL](https://www.nuget.org/packages/DotNetWorkQueue.Transport.PostgreSQL/)\r\n* NuGet package [DotNetWorkQueue.Transport.LiteDb](https://www.nuget.org/packages/DotNetWorkQueue.Transport.LiteDb/)\r\n\r\nMetrics\r\n\r\n* NuGet package [DotNetWorkQueue.AppMetrics](https://www.nuget.org/packages/DotNetWorkQueue.AppMetrics/)\r\n\r\nDifferences between versions\r\n------------\r\n\r\nDot net standard 2.0 / 6.0 / 8.0 are missing the following features from the full framework versions\r\n\r\n- No support for aborting threads when stopping the consumer queues\r\n- No support for dynamic linq statements\r\n\r\nUsage - POCO\r\n------\r\n\r\n[**Producer - Sql server**]\r\n\r\n\r\nhttps://github.com/blehnen/DotNetWorkQueue.Samples/blob/master/Source/Samples/SQLServer/SQLServerProducer/Program.cs\r\n\r\n[**Producer - SQLite**]\r\n\r\nhttps://github.com/blehnen/DotNetWorkQueue.Samples/blob/master/Source/Samples/SQLite/SQLiteProducer/Program.cs\r\n\r\n[**Producer - Redis**]\r\n\r\nhttps://github.com/blehnen/DotNetWorkQueue.Samples/blob/master/Source/Samples/Redis/RedisProducer/Program.cs\r\n\r\n[**Producer - PostgreSQL**]\r\n\r\nhttps://github.com/blehnen/DotNetWorkQueue.Samples/blob/master/Source/Samples/PostgreSQL/PostgreSQLProducer/Program.cs\r\n\r\n\r\n[**Producer - LiteDb**]\r\n\r\nhttps://github.com/blehnen/DotNetWorkQueue.Samples/blob/master/Source/Samples/LiteDb/LiteDbProducer/Program.cs\r\n\r\n\r\n[**Consumer - Sql server**]\r\n\r\nhttps://github.com/blehnen/DotNetWorkQueue.Samples/blob/master/Source/Samples/SQLServer/SQLServerConsumer/Program.cs\r\n\r\n[**Consumer - SQLite**]\r\n\r\nhttps://github.com/blehnen/DotNetWorkQueue.Samples/blob/master/Source/Samples/SQLite/SQLiteConsumer/Program.cs\r\n\r\n[**Consumer - Redis**]\r\n\r\nhttps://github.com/blehnen/DotNetWorkQueue.Samples/blob/master/Source/Samples/Redis/RedisConsumer/Program.cs\r\n\r\n[**Consumer - PostgreSQL**]\r\n\r\nhttps://github.com/blehnen/DotNetWorkQueue.Samples/blob/master/Source/Samples/PostgreSQL/PostGreSQLConsumer/Program.cs\r\n\r\n[**Consumer - LiteDb**]\r\n\r\nhttps://github.com/blehnen/DotNetWorkQueue.Samples/blob/master/Source/Samples/LiteDb/LiteDbConsumer/Program.cs\r\n\r\nUsage - Linq Expression\r\n------\r\n\r\nYou can choose to send Linq expressions to be executed instead. This has some advantages, as the producers and consumers are generic; they no longer need to be message specific. The below examples are not transport specifc and assume that any queue creation steps have already been performed.\r\n\r\nNOTE: It's possbile for a producer to queue up work that a consumer cannot process. In order for a consumer to execute the Linq statement, all types must be resolvable. For dynamic statements, it's also possible to queue up work that doesn't compile due to syntax errors. That won't be discovered until the consumer dequeues the work.\r\n\r\n####Example#####\r\n\r\n[**Producer**]\r\nNOTE - if passing in the message or worker notifications as arguments to dynamic linq, you must cast them. The internal compiler treats those as objects. You can see this syntax in the examples below. That's not nessasry if using standard Linq expressions.\r\n\r\n```csharp\r\nMessage\r\n(IReceivedMessage\u003cMessageExpression\u003e)\r\n\r\nWorkerNotification\r\n(IWorkerNotification)\r\n```\r\n\r\nhttps://github.com/blehnen/DotNetWorkQueue.Samples/blob/master/Source/Samples/SQLite/SQLiteProducerLinq/Program.cs\r\n\r\nIf you are passing value types, you will need to parse them. Here is an example.\r\nThe Guid and the int are both inside string literials and parsed via the built in dot.net methods.\r\n\r\n```csharp\r\nvar id = Guid.NewGuid();\r\nvar runTime = 200;\r\n$\"(message, workerNotification) =\u003e StandardTesting.Run(new Guid(\\\"{id}\\\"), int.Parse(\\\"{runTime}\\\"))\"\r\n```\r\n\r\nThis will produce a linq expression that can be compiled and executed by the consumer, assuming that it can resolve all of the types.\r\n\r\n\r\n[**Consumer**]\r\nThe consumer is generic; it can process any linq expression. However, it must be able to resolve all types that the linq expression uses. You may need to wire up an assembly resolver if your DLL's cannot be located.\r\n\r\nhttps://msdn.microsoft.com/en-us/library/system.appdomain.assemblyresolve(v=vs.110).aspx\r\n\r\nhttps://github.com/blehnen/DotNetWorkQueue.Samples/blob/master/Source/Samples/SQLite/SQLiteConsumerLinq/Program.cs\r\n\r\nThe above queue will process all Linq statements sent to the specified connection / queue.\r\n\r\n[**Considerations**]\r\n\r\nNo sandboxing or checking for risky commands is performed. For instance, the below statement will cause your consumer host to exit.\r\n\r\n```csharp\r\n\"(message, workerNotification) =\u003e Environment.Exit(0)\"\r\n```\r\n\r\nIf you decide to allow configuration files to define dyanmic Linq statements (or if you cannot trust the producer), you should consider running the consumer in an application domain sandbox. Otherwise, the only thing stopping a command like the following from executing would be O/S user permissions.\r\n\r\n```csharp\r\n\"(message, workerNotification) =\u003e System.IO.Directory.Delete(@\"C:\\Windows\\, true)\"\r\n```\r\n\r\nUsage - Job Scheduler\r\n------\r\n\r\nJobs may be scheduled using [Schyntax ](https://github.com/schyntax/cs-schyntax) format. The scheduler and consumers are seperate; schedulers don't process any work, they queue it for processing by a consumer.  The standard LINQ consumers are used to process work enqueued by a scheduler / schedule.\r\n\r\nAny LINQ statement that a linq producer supports can be scheduled using the scheduler.\r\n\r\nMultiple schedulers with the same schedule may be ran if needed for redundancy. However, it's important that the clocks on the machines are in sync, or that the same time provider is injected into the schedulers and consumers. See the WIKI for more information on this.\r\n\r\nGenerally speaking, you may get funny results if you are using multiple machines and the clocks are not in sync. The server based transports tend to provide solutions for this if you can't sync the clocks of the local machines; see the WIKI.\r\n\r\nSee [Schyntax ](https://github.com/schyntax/schyntax) for event scheduling format.\r\n\r\n[**Scheduler**]\r\n\r\nThe scheduler and container must be kept in scope until you are done scheduling work or shutting down. No work will be queued if the scheduler is disposed or falls out of scope.\r\n\r\nhttps://github.com/blehnen/DotNetWorkQueue.Samples/blob/master/Source/Samples/SQLite/SQliteScheduler/Program.cs\r\n\r\nTo consume / process scheduled jobs, a [Linq Consumer ](https://github.com/blehnen/DotNetWorkQueue/wiki/ConsumerLinq) is used\r\n\r\nhttps://github.com/blehnen/DotNetWorkQueue.Samples/blob/master/Source/Samples/SQLite/SQLiteSchedulerConsumer/Program.cs\r\n\r\n---------------------\r\n[**Samples**](https://github.com/blehnen/DotNetWorkQueue.Samples)\r\n\r\n[**More examples**](https://github.com/blehnen/DotNetWorkQueue.Examples/tree/master/Source/Examples)\r\n------\r\n\r\nBuilding the source\r\n---------------------\r\n\r\nYou'll need VS2022 (any version) and you'll also need to install the dot net core 2.0/6.0 SDKs\r\n\r\nAll references are either in NuGet or the \\lib folder - building from Visual studio should restore any needed files.\r\n\r\nLicense\r\n--------\r\nCopyright � 2015-2024 Brian Lehnen\r\n\r\nThis program is free software: you can redistribute it and/or modify\r\nit under the terms of the GNU Lesser General Public License as published by\r\nthe Free Software Foundation, either version 2.1 of the License, or\r\n(at your option) any later version.\r\n\r\nThis program is distributed in the hope that it will be useful,\r\nbut WITHOUT ANY WARRANTY; without even the implied warranty of\r\nMERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the\r\nGNU Lesser General Public License for more details.\r\n\r\nYou should have received a copy of the GNU Lesser General Public License\r\nalong with this program.  If not, see [http://www.gnu.org/licenses/](http://www.gnu.org/licenses).\r\n\r\n3rd party Libraries\r\n--------\r\n\r\nThis library uses multiple 3rd party libaries, listed below.\r\n\r\n[**DotNetWorkQueue**]\r\n\r\n* [LibLog ](https://github.com/damianh/LibLog)\r\n\r\n* [NetFX-Guard ](http://netfx.codeplex.com/)\r\n\r\n* [SimpleInjector ](https://simpleinjector.org/index.html)\r\n\r\n* [Microsoft.IO.RecyclableMemoryStream ](https://github.com/Microsoft/Microsoft.IO.RecyclableMemoryStream)\r\n\r\n* [Newtonsoft.Json ](http://www.newtonsoft.com/json)\r\n\r\n* [JpLabs.DynamicCode ](http://jp-labs.blogspot.com/2008/11/dynamic-lambda-expressions-using.html)\r\n\r\n* [JsonNet.PrivateSetterContractResolvers ](https://github.com/danielwertheim/jsonnet-privatesetterscontractresolvers)\r\n\r\n* [Expression-JSON-Serializer ](https://github.com/blehnen/expression-json-serializer)\r\n\r\n* [Schtick](https://github.com/schyntax/cs-schtick)\r\n\r\n* [Schyntax ](https://github.com/blehnen/cs-schyntax)\r\n\r\n* [Polly ](https://github.com/App-vNext/Polly)\r\n\r\n[**DotNetWorkQueue.Transport.Redis**]\r\n\r\n* [GuerrillaNTP ](https://github.com/blehnen/GuerrillaNtp)\r\n\r\n* [MsgPack-CLI ](https://github.com/msgpack/msgpack-cli)\r\n\r\n* [StackExchange.Redis ](https://github.com/StackExchange/StackExchange.Redis)\r\n\r\n\r\n[**DotNetWorkQueue.Transport.SqlServer**]\r\n\r\n* None\r\n\r\n[**DotNetWorkQueue.Transport.SQLite**]\r\n\r\n* [System.Data.SQLite ](https://www.sqlite.org/)\r\n\r\n[**DotNetWorkQueue.Transport.SQLite.Microsoft**]\r\n\r\n* [Microsoft.Data.Sqlite ](https://github.com/aspnet/Microsoft.Data.Sqlite)\r\n\r\n[**DotNetWorkQueue.Transport.PostgreSQL**]\r\n\r\n* [Npgsql ](http://www.npgsql.org/)\r\n\r\n[**DotNetWorkQueue.Transport.LiteDb**]\r\n\r\n* [LiteDb ](https://www.litedb.org/)\r\n\r\n[**DotNetWorkQueue.AppMetrics**]\r\n\r\n* [AppMetrics ](https://github.com/AppMetrics/AppMetrics)\r\n\r\n[**Unit / Integration Tests**]\r\n\r\n* [AutoFixture ](https://github.com/AutoFixture/AutoFixture)\r\n\r\n* [CompareNetObjects ](http://comparenetobjects.codeplex.com/)\r\n\r\n* [FluentAssertions ](http://www.fluentassertions.com/)\r\n\r\n* [nsubstitute ](http://nsubstitute.github.io/)\r\n\r\n* [ObjectFiller ](http://objectfiller.net/)\r\n\r\n* [Xunit ](https://github.com/xunit/xunit)\r\n\r\n##### Developed with:\r\n\r\n\u003cimg src=\"https://resources.jetbrains.com/storage/products/company/brand/logos/ReSharper_icon.png\" width=\"48\"\u003e\r\n\u003cimg src=\"https://resources.jetbrains.com/storage/products/company/brand/logos/dotCover_icon.png\" width=\"48\"\u003e\r\n\u003cimg src=\"https://resources.jetbrains.com/storage/products/company/brand/logos/dotTrace_icon.png\" width=\"48\"\u003e\r\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fblehnen%2FDotNetWorkQueue","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fblehnen%2FDotNetWorkQueue","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fblehnen%2FDotNetWorkQueue/lists"}