{"id":17008720,"url":"https://github.com/i-e-b/diskqueue","last_synced_at":"2025-04-09T06:11:06.357Z","repository":{"id":7154826,"uuid":"8453767","full_name":"i-e-b/DiskQueue","owner":"i-e-b","description":"A robust, thread-safe, and multi-process persistent queue library for dotnet","archived":false,"fork":false,"pushed_at":"2025-03-27T16:16:26.000Z","size":1120,"stargazers_count":116,"open_issues_count":2,"forks_count":37,"subscribers_count":10,"default_branch":"master","last_synced_at":"2025-03-27T17:25:28.841Z","etag":null,"topics":["c-sharp","disk","fast","mono","multi-process","net-standard-2","persistent-storage","production-ready","queue","storage-location","thread","transaction"],"latest_commit_sha":null,"homepage":"","language":"C#","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/i-e-b.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2013-02-27T10:30:14.000Z","updated_at":"2025-03-27T16:16:30.000Z","dependencies_parsed_at":"2024-10-27T12:49:17.109Z","dependency_job_id":"426e7ac7-f705-4b5f-b58c-3d56704d217d","html_url":"https://github.com/i-e-b/DiskQueue","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/i-e-b%2FDiskQueue","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/i-e-b%2FDiskQueue/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/i-e-b%2FDiskQueue/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/i-e-b%2FDiskQueue/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/i-e-b","download_url":"https://codeload.github.com/i-e-b/DiskQueue/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247987285,"owners_count":21028895,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["c-sharp","disk","fast","mono","multi-process","net-standard-2","persistent-storage","production-ready","queue","storage-location","thread","transaction"],"created_at":"2024-10-14T05:29:06.154Z","updated_at":"2025-04-09T06:11:06.352Z","avatar_url":"https://github.com/i-e-b.png","language":"C#","readme":"DiskQueue\n=========\n\n* https://www.nuget.org/packages/DiskQueue\n* https://github.com/i-e-b/DiskQueue\n\nA robust, thread-safe, and multi-process persistent queue.\n\nBased very heavily on  http://ayende.com/blog/3479/rhino-queues-storage-disk\n\nRequirements and Environment\n----------------------------\n\n- Works on dotnet standard 2.0 platforms\n- Requires access to filesystem storage\n- Only partial support for \"Trim Self-Contained\" deployments and executables (see below)\n\nThe file system is used to hold locks, so any bug in your file system may cause\nissues with DiskQueue -- although it tries to work around them.\n\nThanks to\n---------\n\nThese kind folks have helped in the development of DiskQueue\n\n- Tom Halter https://github.com/thalter\n- Niklas Rydén https://github.com/nikryden\n- Stefan Dascalu https://github.com/stefandascalu64\n\nBasic Usage\n-----------\n\n - `PersistentQueue.WaitFor(...)` is the main entry point. This will attempt to gain an exclusive lock\n   on the given storage location. On first use, a directory will be created with the required files\n   inside it.\n - This queue object can be shared among threads. Each thread should call `OpenSession()` to get its \n   own session object.\n - Both `IPersistentQueue`s and `IPersistentQueueSession`s should be wrapped in `using()` clauses, or otherwise\n   disposed of properly. Failure to do this will result in lock contention -- you will get errors that the queue\n   is still in use.\n   \nThere is also a generic-typed `PersistentQueue\u003cT\u003e(...);` which will handle the serialisation and deserialization of\nelements in the queue, as long at the type is decorated with `[Serializable]`. You can also inject your own `ISerializationStrategy\u003cT\u003e` \ninto your `PersistentQueueSession\u003cT\u003e` if you wish to have more granular control over Serialization/Deserialization, or if you wish to \nuse your own serializer (e.g Json.NET).\n\nUse `new PersistentQueue\u003cT\u003e(...)` in place of `new PersistentQueue(...)`\nor `PersistentQueue.WaitFor\u003cT\u003e(...)` in place of `PersistentQueue.WaitFor(...)` in any of the examples below.\n\nNote: `BinaryFormatter` was removed from the default serializer. See https://learn.microsoft.com/en-us/dotnet/standard/serialization/binaryformatter-security-guide.\n\nExample\n-------\nQueue on one thread, consume on another; retry some exceptions.\n\n**Note** this is one queue being shared between two sessions. You should not open two queue instances for one storage location at once.\n\n```csharp\nIPersistentQueue queue = new PersistentQueue(\"queue_a\");\nvar t1 = new Thread(() =\u003e\n{\n\twhile (HaveWork())\n\t{\n\t\tusing (var session = queue.OpenSession())\n\t\t{\n\t\t\tsession.Enqueue(NextWorkItem());\n\t\t\tsession.Flush();\n\t\t}\n\t}\n});\nvar t2 = new Thread(()=\u003e {\n\twhile (true) {\n\t\tusing (var session = queue.OpenSession()) {\n\t\t\tvar data = session.Dequeue();\n\t\t\tif (data == null) {Thread.Sleep(100); continue;}\n\t\t\t\n\t\t\ttry {\n\t\t\t\tMaybeDoWork(data)\n\t\t\t\tsession.Flush();\n\t\t\t} catch (RetryException) {\n\t\t\t\tcontinue;\n\t\t\t} catch {\n\t\t\t\tsession.Flush();\n\t\t\t}\n\t\t}\n\t}\n});\n\nt1.Start();\nt2.Start();\n```\n\nExample\n-------\nBatch up a load of work and have another thread work through it.\n```csharp\nIPersistentQueue queue = new PersistentQueue(\"batchQueue\");\nvar worker = new Thread(()=\u003e {\n\tusing (var session = queue.OpenSession()) {\n\t\tbyte[] data;\n\t\twhile ((data = session.Dequeue()) != null) {\n\t\t\tMaybeDoWork(data)\n\t\t\tsession.Flush();\n\t\t}\n\t}\n});\n\nusing (var session = queue.OpenSession()) {\n\tforeach (var item in LoadsOfStuff()) {\n\t\tsession.Enqueue(item);\n\t}\n\tsession.Flush();\n}\n\nworker.IsBackground = true; // anything not complete when we close will be left on the queue for next time.\nworker.Start();\n```\n\nTransactions\n------------\nEach session is a transaction. Any Enqueues or Dequeues will be rolled back when the session is disposed unless\nyou call `session.Flush()`. Data will only be visible between threads once it has been flushed.\nEach flush incurs a performance penalty. By default, each flush is persisted to disk before continuing. You \ncan get more speed at a safety cost by setting `queue.ParanoidFlushing = false;`\n\nData loss and transaction truncation\n------------------------------------\nBy default, DiskQueue will silently discard transaction blocks that have been truncated; it will throw an `InvalidOperationException`\nwhen transaction block markers are overwritten (this happens if more than one process is using the queue by mistake. It can also happen with some kinds of disk corruption).\nIf you construct your queue with `throwOnConflict: false`, all recoverable transaction errors will be silently truncated. This should only be used when\nuptime is more important than data consistency.\n\n```\nusing (var queue = new PersistentQueue(path, Constants._32Megabytes, throwOnConflict: false)) {\n    . . .\n}\n```\n\nGlobal default settings\n-----------------------\nEach instance of a `PersistentQueue` has it's own settings for flush levels and corruption behaviour. You can set these individually after creating an instance,\nor globally with `PersistentQueue.DefaultSettings`. Default settings are applied to all queue instances in the same process created *after* the setting is changed.\n\nFor example, if performance is more important than crash safety:\n```csharp\nPersistentQueue.DefaultSettings.ParanoidFlushing = false;\nPersistentQueue.DefaultSettings.TrimTransactionLogOnDispose = false;\n```\n\nOr if up-time is more important than detecting corruption early (often the case for embedded systems):\n```csharp\nPersistentQueue.DefaultSettings.AllowTruncatedEntries = true;\nPersistentQueue.DefaultSettings.ParanoidFlushing = true;\n```\n\nInternal Logging\n----------------\n\nSome internal warnings and non-critical errors are logged through `PersistentQueue.Log`.\nThis defaults to `System.Console.WriteLine`, but can be replace with any `Action\u003cstring\u003e`\n\nRemoving or resetting queues\n----------------------------\n\nQueues create a directory and set of files for storage. You can remove all files for a queue with the `HardDelete` method.\nIf you give true as the reset parameter, the directory will be written again.\n\nThis WILL delete ANY AND ALL files inside the queue directory. You should not call this method in normal use.\nIf you start a queue with the same path as an existing directory, this method will delete the entire directory, not just\nthe queue files.\n\n```csharp\nvar subject = new PersistentQueue(\"queue_a\");\nsubject.HardDelete(true); // wipe any existing data and start again\n```\n\nMulti-Process Usage\n-------------------\nEach `IPersistentQueue` gives exclusive access to the storage until it is disposed.\nThere is a static helper method `PersistentQueue.WaitFor(\"path\", TimeSpan...)` which will wait to gain access until\nother processes release the lock or the timeout expires.\nIf each process uses the lock for a short time and wait long enough, they can share a storage location.\n\nE.g.\n```csharp\n...\nvoid AddToQueue(byte[] data) {\n\tThread.Sleep(150);\n\tusing (var queue = PersistentQueue.WaitFor(SharedStorage, TimeSpan.FromSeconds(30)))\n\tusing (var session = queue.OpenSession()) {\n\t\tsession.Enqueue(data);\n\t\tsession.Flush();\n\t}\n}\n\nbyte[] ReadQueue() {\n\tThread.Sleep(150);\n\tusing (var queue = PersistentQueue.WaitFor(SharedStorage, TimeSpan.FromSeconds(30)))\n\tusing (var session = queue.OpenSession()) {\n\t\tvar data = session.Dequeue();\n\t\tsession.Flush();\n\t\treturn data;\n\t}\n}\n...\n\n```\n\nCross-process Locking\n---------------------\n\nDiskQueue tries very hard to make sure the lock files are managed correctly.\nYou can use this as an inter-process lock if required. Simply open a session to\nacquire the lock, and dispose of the session to release it.\n\n\nIf you need the transaction semantics of sessions across multiple processes, try a more robust solution like https://github.com/i-e-b/SevenDigital.Messaging\n\nTrim self-contained deployments and executables\n-----------------------------------------------\n\nThe default serialiser for typed queues is **NOT** supported under \"trimmed self-contained\" projects.\n\nYou can either use the raw byte-based interfaces and do your own serialisation, or create and use a custom `ISerializationStrategy\u003cT\u003e` for your types.\n\nSee\n- https://learn.microsoft.com/en-us/dotnet/core/deploying/trimming/incompatibilities#reflection-based-serializers\n- https://github.com/i-e-b/DiskQueue/issues/35\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fi-e-b%2Fdiskqueue","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fi-e-b%2Fdiskqueue","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fi-e-b%2Fdiskqueue/lists"}