{"id":46362037,"url":"https://github.com/soenneker/soenneker.deduplication.slidingwindow","last_synced_at":"2026-04-24T04:03:34.229Z","repository":{"id":341833527,"uuid":"1171746215","full_name":"soenneker/soenneker.deduplication.slidingwindow","owner":"soenneker","description":"High-performance sliding-window deduplication for .NET.","archived":false,"fork":false,"pushed_at":"2026-04-21T05:32:42.000Z","size":98,"stargazers_count":0,"open_issues_count":1,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-04-21T07:37:08.851Z","etag":null,"topics":["auto-expire","concurrency","csharp","de-dupe","dedupe","deduplication","dotnet","object","set","slidingwindow","slidingwindowdedupe","threadsafe"],"latest_commit_sha":null,"homepage":"https://soenneker.com","language":"C#","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/soenneker.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":".github/CONTRIBUTING.md","funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":".github/CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":".github/SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null},"funding":{"github":"soenneker","thanks_dev":"soenneker"}},"created_at":"2026-03-03T15:07:31.000Z","updated_at":"2026-04-21T05:32:02.000Z","dependencies_parsed_at":"2026-03-08T04:01:08.473Z","dependency_job_id":null,"html_url":"https://github.com/soenneker/soenneker.deduplication.slidingwindow","commit_stats":null,"previous_names":["soenneker/soenneker.deduplication.slidingwindow"],"tags_count":42,"template":false,"template_full_name":null,"purl":"pkg:github/soenneker/soenneker.deduplication.slidingwindow","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/soenneker%2Fsoenneker.deduplication.slidingwindow","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/soenneker%2Fsoenneker.deduplication.slidingwindow/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/soenneker%2Fsoenneker.deduplication.slidingwindow/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/soenneker%2Fsoenneker.deduplication.slidingwindow/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/soenneker","download_url":"https://codeload.github.com/soenneker/soenneker.deduplication.slidingwindow/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/soenneker%2Fsoenneker.deduplication.slidingwindow/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32162611,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-22T17:06:48.269Z","status":"online","status_checked_at":"2026-04-23T02:00:06.710Z","response_time":53,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["auto-expire","concurrency","csharp","de-dupe","dedupe","deduplication","dotnet","object","set","slidingwindow","slidingwindowdedupe","threadsafe"],"created_at":"2026-03-05T01:17:15.150Z","updated_at":"2026-04-24T04:03:34.193Z","avatar_url":"https://github.com/soenneker.png","language":"C#","funding_links":["https://github.com/sponsors/soenneker","https://thanks.dev/soenneker"],"categories":[],"sub_categories":[],"readme":"[![](https://img.shields.io/nuget/v/soenneker.deduplication.slidingwindow.svg?style=for-the-badge)](https://www.nuget.org/packages/soenneker.deduplication.slidingwindow/)\n[![](https://img.shields.io/github/actions/workflow/status/soenneker/soenneker.deduplication.slidingwindow/publish-package.yml?style=for-the-badge)](https://github.com/soenneker/soenneker.deduplication.slidingwindow/actions/workflows/publish-package.yml)\n[![](https://img.shields.io/nuget/dt/soenneker.deduplication.slidingwindow.svg?style=for-the-badge)](https://www.nuget.org/packages/soenneker.deduplication.slidingwindow/)\n[![](https://img.shields.io/github/actions/workflow/status/soenneker/soenneker.deduplication.slidingwindow/codeql.yml?label=CodeQL\u0026style=for-the-badge)](https://github.com/soenneker/soenneker.deduplication.slidingwindow/actions/workflows/codeql.yml)\n\n# ![](https://user-images.githubusercontent.com/4441470/224455560-91ed3ee7-f510-4041-a8d2-3fc093025112.png) Soenneker.Deduplication.SlidingWindow\n\n### High-performance sliding-window deduplication for .NET.\n\n## Installation\n\n```bash\ndotnet add package Soenneker.Deduplication.SlidingWindow\n```\n\n---\n\n# Overview\n\n`Soenneker.Deduplication.SlidingWindow` provides a **thread-safe sliding time window deduplication utility** designed for extremely high throughput workloads.\n\nIt allows you to efficiently determine whether a value has been **seen recently within a time window** without storing the original input values.\n\nInputs are hashed using **XXH3 (XxHash3)** and only the resulting `ulong` is stored internally, keeping memory usage low while maintaining high performance.\n\nTypical usage pattern:\n\n* First time value appears → **`TryMarkSeen()` returns `true`**\n* Value appears again within the window → **returns `false`**\n* Value appears after the window expires → **returns `true` again**\n\n---\n\n# Key Features\n\n* **Sliding time window deduplication**\n* **Thread-safe concurrent access**\n* **High-throughput design**\n* **Allocation-free span APIs**\n* **XXH3 hashing for speed**\n* **UTF8 + UTF16 support**\n* **Optional hashing seed**\n* **Async disposal support**\n\nInternally it uses a **bucketed concurrent set with rotating expiration**, allowing expired entries to fall out automatically as the window advances.\n\n---\n\n# Quick Start\n\n```csharp\nusing Soenneker.Deduplication.SlidingWindow;\n\nvar dedupe = new SlidingWindowXxHashDedupe(\n    window: TimeSpan.FromMinutes(5),\n    rotationInterval: TimeSpan.FromSeconds(10)\n);\n\nif (dedupe.TryMarkSeen(\"user:123\"))\n{\n    // First occurrence in the last 5 minutes\n}\nelse\n{\n    // Duplicate within the window\n}\n```\n\nAfter the window expires, the same value will again return `true`.\n\n---\n\n# API\n\n## TryMarkSeen\n\nChecks if the value was seen recently and records it if not.\n\n```csharp\nbool added = dedupe.TryMarkSeen(\"value\");\nbool added2 = dedupe.TryMarkSeen(\"value\".AsSpan());\nbool added3 = dedupe.TryMarkSeenUtf8(utf8Bytes);\n```\n\nReturn value:\n\n| Result  | Meaning                                        |\n| ------- | ---------------------------------------------- |\n| `true`  | Value was not seen recently and was added      |\n| `false` | Value already exists within the sliding window |\n\n---\n\n## Contains\n\nChecks if a value exists within the current window.\n\n```csharp\nbool exists = dedupe.Contains(\"value\");\nbool exists2 = dedupe.Contains(\"value\".AsSpan());\nbool exists3 = dedupe.ContainsUtf8(utf8Bytes);\n```\n\nThese methods are **pure checks** and do not modify the set.\n\n---\n\n## TryRemove\n\nManually removes a value if present.\n\n```csharp\nbool removed = dedupe.TryRemove(\"value\");\nbool removed2 = dedupe.TryRemove(\"value\".AsSpan());\nbool removed3 = dedupe.TryRemoveUtf8(utf8Bytes);\n```\n\n---\n\n## Count\n\nApproximate number of items currently in the window.\n\n```csharp\nint count = dedupe.Count;\n```\n\nThis value is intended for **diagnostics/monitoring**, not strict accounting.\n\n---\n\n# Configuration\n\n```csharp\nvar dedupe = new SlidingWindowXxHashDedupe(\n    window: TimeSpan.FromMinutes(10),\n    rotationInterval: TimeSpan.FromSeconds(30),\n    capacityHint: 100_000,\n    seed: 12345\n);\n```\n\n| Parameter          | Description                              |\n| ------------------ | ---------------------------------------- |\n| `window`           | Total deduplication duration             |\n| `rotationInterval` | How frequently buckets rotate            |\n| `capacityHint`     | Optional size hint to reduce resizing    |\n| `seed`             | Optional XXH3 seed for hash partitioning |\n\n### Window behavior\n\nThe sliding window works by **rotating buckets** at the specified `rotationInterval`.\n\nExample:\n\n```\nwindow = 10 minutes\nrotationInterval = 30 seconds\n```\n\nResults in ~20 rotating buckets.\n\nExpired buckets are automatically cleared as the window advances.\n\n---\n\n# Memory Efficiency\n\nValues are stored as **64-bit hashes** instead of full strings.\n\nExample approximate memory usage:\n\n| Entries | Approx Memory |\n| ------- | ------------- |\n| 1,000   | ~8 KB         |\n| 10,000  | ~80 KB        |\n| 100,000 | ~800 KB       |\n\nActual usage depends on dictionary overhead.\n\n---\n\n# Hashing \u0026 Collisions\n\nInputs are deduplicated using **64-bit XXH3 hashes**.\n\nThis provides extremely fast hashing with a very low collision probability.\n\nHowever, collisions are theoretically possible since only hashes are stored.\nFor most event deduplication, ingestion pipelines, and telemetry scenarios, this is more than sufficient.\n\n---\n\n# Disposal\n\n`SlidingWindowXxHashDedupe` maintains an internal background rotation timer and therefore supports disposal.\n\n```csharp\ndedupe.Dispose();\n```\n\nor\n\n```csharp\nawait dedupe.DisposeAsync();\n```\n\nDisposing stops the internal rotation loop and releases resources.\n\n---\n\n# When to Use\n\nIdeal for:\n\n* Event stream deduplication\n* Message processing pipelines\n* API request suppression\n* Preventing duplicate webhook processing\n* Temporary ID or phone number dedupe\n* High-volume ingestion systems\n\n---\n\n# When Not to Use\n\nNot recommended if:\n\n* You require **permanent deduplication**\n* You need **exact storage of original values**\n* Collision risk must be absolutely zero\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsoenneker%2Fsoenneker.deduplication.slidingwindow","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsoenneker%2Fsoenneker.deduplication.slidingwindow","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsoenneker%2Fsoenneker.deduplication.slidingwindow/lists"}