{"id":46793540,"url":"https://github.com/soenneker/soenneker.deduplication.bounded","last_synced_at":"2026-06-06T02:03:14.311Z","repository":{"id":342306409,"uuid":"1173593979","full_name":"soenneker/soenneker.deduplication.bounded","owner":"soenneker","description":"A thread-safe high-performance bounded size deduplication utility for .NET.","archived":false,"fork":false,"pushed_at":"2026-05-29T02:28:41.000Z","size":124,"stargazers_count":0,"open_issues_count":1,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-05-29T04:16:38.206Z","etag":null,"topics":["bounded","boundeddedupe","csharp","dedupe","deduplication","dotnet","max","maxsize","object","size"],"latest_commit_sha":null,"homepage":"https://soenneker.com","language":"C#","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/soenneker.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":".github/CONTRIBUTING.md","funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":".github/CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":".github/SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null},"funding":{"github":"soenneker","buy_me_a_coffee":"soenneker","thanks_dev":"soenneker","ko_fi":"soenneker","patreon":"soenneker"}},"created_at":"2026-03-05T14:40:15.000Z","updated_at":"2026-05-22T10:08:23.000Z","dependencies_parsed_at":"2026-03-15T18:01:38.733Z","dependency_job_id":null,"html_url":"https://github.com/soenneker/soenneker.deduplication.bounded","commit_stats":null,"previous_names":["soenneker/soenneker.deduplication.bounded"],"tags_count":35,"template":false,"template_full_name":null,"purl":"pkg:github/soenneker/soenneker.deduplication.bounded","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/soenneker%2Fsoenneker.deduplication.bounded","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/soenneker%2Fsoenneker.deduplication.bounded/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/soenneker%2Fsoenneker.deduplication.bounded/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/soenneker%2Fsoenneker.deduplication.bounded/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/soenneker","download_url":"https://codeload.github.com/soenneker/soenneker.deduplication.bounded/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/soenneker%2Fsoenneker.deduplication.bounded/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33966639,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-06T02:00:07.033Z","response_time":107,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bounded","boundeddedupe","csharp","dedupe","deduplication","dotnet","max","maxsize","object","size"],"created_at":"2026-03-10T03:05:13.574Z","updated_at":"2026-06-06T02:03:14.272Z","avatar_url":"https://github.com/soenneker.png","language":"C#","funding_links":["https://github.com/sponsors/soenneker","https://buymeacoffee.com/soenneker","https://thanks.dev/soenneker","https://ko-fi.com/soenneker","https://patreon.com/soenneker"],"categories":[],"sub_categories":[],"readme":"[![](https://img.shields.io/nuget/v/soenneker.deduplication.bounded.svg?style=for-the-badge)](https://www.nuget.org/packages/soenneker.deduplication.bounded/)\n[![](https://img.shields.io/github/actions/workflow/status/soenneker/soenneker.deduplication.bounded/publish-package.yml?style=for-the-badge)](https://github.com/soenneker/soenneker.deduplication.bounded/actions/workflows/publish-package.yml)\n[![](https://img.shields.io/nuget/dt/soenneker.deduplication.bounded.svg?style=for-the-badge)](https://www.nuget.org/packages/soenneker.deduplication.bounded/)\n[![](https://img.shields.io/github/actions/workflow/status/soenneker/soenneker.deduplication.bounded/codeql.yml?label=CodeQL\u0026style=for-the-badge)](https://github.com/soenneker/soenneker.deduplication.bounded/actions/workflows/codeql.yml)\n\n# ![](https://user-images.githubusercontent.com/4441470/224455560-91ed3ee7-f510-4041-a8d2-3fc093025112.png) Soenneker.Deduplication.Bounded\n\n### A thread-safe high-performance bounded size deduplication utility for .NET.\n\n## Installation\n\n```bash\ndotnet add package Soenneker.Deduplication.Bounded\n```\n\n## What it does\n\n`Soenneker.Deduplication.Bounded` provides a fast, thread-safe *“seen set”* for deduplication with a **maximum size**.\n\nYou call `TryMarkSeen(...)` with an input value:\n\n* **Returns `true`** if this value has **not** been seen before (it was added)\n* **Returns `false`** if it has already been seen (already exists)\n\nInternally it hashes your input to a `ulong` using **XXH3 (XxHash3)** and stores only the hash in a bounded concurrent set. That means it’s very memory efficient and avoids storing original strings/byte arrays.\n\n## Key characteristics\n\n* **Bounded size**: targets `MaxSize` and opportunistically trims under contention (best-effort, not strict)\n* **Thread-safe**: safe to use concurrently from many threads\n* **High-throughput**: stores `ulong` hashes instead of strings\n* **Span-friendly**: avoids allocations via `ReadOnlySpan\u003cchar\u003e` and `ReadOnlySpan\u003cbyte\u003e`\n* **Optional hashing seed**: lets you rotate/partition hash space if desired\n* **Diagnostics-friendly**: exposes an approximate `Count`\n\n## Quick start\n\n```csharp\nusing Soenneker.Deduplication.Bounded;\n\nvar dedupe = new BoundedDedupe(maxSize: 250_000);\n\n// returns true the first time\nif (dedupe.TryMarkSeen(\"user:123\"))\n{\n    // process first occurrence\n}\n\n// returns false on repeats\nif (!dedupe.TryMarkSeen(\"user:123\"))\n{\n    // duplicate\n}\n```\n\n## API\n\n### TryMarkSeen\n\nUse these for the fast “check + add” operation.\n\n```csharp\nbool added = dedupe.TryMarkSeen(\"some string\");\nbool added2 = dedupe.TryMarkSeen(\"some string\".AsSpan());\nbool added3 = dedupe.TryMarkSeenUtf8(utf8Bytes);\n```\n\n### Contains\n\nPure membership checks (no mutation).\n\n```csharp\nbool exists = dedupe.Contains(\"some string\");\nbool exists2 = dedupe.Contains(\"some string\".AsSpan());\nbool exists3 = dedupe.ContainsUtf8(utf8Bytes);\n```\n\n### TryRemove\n\nRemoves an entry if present.\n\n```csharp\nbool removed = dedupe.TryRemove(\"some string\");\nbool removed2 = dedupe.TryRemove(\"some string\".AsSpan());\nbool removed3 = dedupe.TryRemoveUtf8(utf8Bytes);\n```\n\n### Properties\n\n```csharp\nint max = dedupe.MaxSize;\nint approx = dedupe.Count; // approximate; good for diagnostics/telemetry\n```\n\n## Configuration\n\n```csharp\nvar dedupe = new BoundedDedupe(\n    maxSize: 250_000,\n    capacityHint: 300_000,           // optional, reduces resizing\n    seed: 0,                         // optional XXH3 seed\n    trimBatchSize: 64,               // work chunk size when trimming\n    trimStartOveragePercent: 5,      // begin trimming after +5% over MaxSize\n    maxTrimWorkPerCall: 4096,        // caps trimming effort per write\n    resyncAfterNoProgress: 8,        // resync count if trimming stalls\n    queueOverageFactor: 4            // internal queue sizing multiplier\n);\n```\n\n### Notes on trimming / bounded behavior\n\nThis is **not a strict LRU** and does **not guarantee exact eviction order**. Under heavy contention it may temporarily exceed `MaxSize`, then trims opportunistically during subsequent writes.\n\nThis design is intentional: it favors throughput and low contention over perfect eviction accuracy.\n\n## Hashing \u0026 collisions\n\nInputs are deduped by their **64-bit XXH3 hash** (`ulong`). Like all hashing-based dedupe approaches, there is a theoretical possibility of collisions (different inputs producing the same hash). For most dedupe/telemetry/rate-limit style workloads, a 64-bit hash is typically more than sufficient.\n\nIf collision risk is unacceptable for your use case, you should store full keys (or use a stronger scheme), at higher memory cost.\n\n## When to use this\n\n* Deduping inbound events/messages by ID for a fixed memory budget\n* “Seen recently” protection in high-volume ingestion pipelines\n* De-duplicating phone numbers / emails / identifiers without storing raw values\n* Fast in-memory suppression lists\n\n## When not to use this\n\n* You need **exact** dedupe of raw strings (no collision tolerance)\n* You need strict FIFO/LRU eviction ordering guarantees\n* You need time-window expiration semantics (use a sliding window approach instead)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsoenneker%2Fsoenneker.deduplication.bounded","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsoenneker%2Fsoenneker.deduplication.bounded","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsoenneker%2Fsoenneker.deduplication.bounded/lists"}