{"id":21041727,"url":"https://github.com/quantumplation/watermark","last_synced_at":"2026-07-03T13:04:21.963Z","repository":{"id":35059288,"uuid":"201833828","full_name":"Quantumplation/watermark","owner":"Quantumplation","description":"A simple watermarking set","archived":false,"fork":false,"pushed_at":"2022-09-12T02:06:58.000Z","size":21,"stargazers_count":1,"open_issues_count":3,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-06-23T23:13:40.252Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Quantumplation.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-08-12T01:07:48.000Z","updated_at":"2020-08-08T19:38:42.000Z","dependencies_parsed_at":"2022-08-08T04:15:42.062Z","dependency_job_id":null,"html_url":"https://github.com/Quantumplation/watermark","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/Quantumplation/watermark","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Quantumplation%2Fwatermark","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Quantumplation%2Fwatermark/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Quantumplation%2Fwatermark/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Quantumplation%2Fwatermark/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Quantumplation","download_url":"https://codeload.github.com/Quantumplation/watermark/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Quantumplation%2Fwatermark/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":262266711,"owners_count":23284747,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-19T13:56:00.152Z","updated_at":"2026-07-03T13:04:21.925Z","avatar_url":"https://github.com/Quantumplation.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![Crates.io](https://img.shields.io/crates/v/watermark.svg)](https://crates.io/crates/watermark)\n\n# Watermark\n\nA simple watermarking set.\n\nA watermarking set holds any integer values, and supports two operations:\n\n - insert(element: T)\n   - Inserts an item into the set\n - contains(element: T)\n   - Checks whether an item has previously been added to the set\n\nA watermark set works best when the \"inserts\" *all* happen, and happen \"mostly\"\nin order. For example, when keeping track of which message IDs have been seen.\n\n## Example\n\nTo make a simple idempotent data processor:\n\n```rust\nlet mut ws = watermark::WatermarkSet::default();\nfor message in message_bus {\n  if !ws.contains(message.id) {\n    ws.insert(message.id);\n    // Do some work with message.data\n  }\n}\n```\n\n## Operation\n\nInternally, a watermarking set contains a \"watermark\" and a bitvector of\n\"recently added\" items.  The watermark guarantees that all items below\nthat number have been seen, and the recently added items handles everything\nelse.  This means that if all elements eventually get added, memory usage\nis kept very low and membership tests are very very cheap.\n\n## Performance\n\nThis crate comes with some simple benchmarks that you can run for yourself.\nHere are the results, compared to a hash set, that I got on my machine:\n\n```\nSpecs:\n - AMD Ryzen 7 3700x 8-Core Processor\n - 2x16gb DDR4 3200\n```\n\n```\nWatermarkSet Insert/In Order\n                        time:   [7.6194 us 7.8101 us 7.9793 us]\n                        thrpt:  [125.32 Melem/s 128.04 Melem/s 131.24 Melem/s]\n\nHashSet Insert/In Order time:   [40.319 us 40.340 us 40.372 us]\n                        thrpt:  [24.770 Melem/s 24.789 Melem/s 24.802 Melem/s]\nFound 8 outliers among 100 measurements (8.00%)\n  1 (1.00%) low severe\n  2 (2.00%) low mild\n  3 (3.00%) high mild\n  2 (2.00%) high severe\n\nWatermarkSet Insert/Out of Order\n                        time:   [9.1897 us 9.4983 us 9.7602 us]\n                        thrpt:  [102.46 Melem/s 105.28 Melem/s 108.82 Melem/s]\n\nHashSet Insert/Out of Order\n                        time:   [26.327 us 26.342 us 26.355 us]\n                        thrpt:  [37.943 Melem/s 37.962 Melem/s 37.983 Melem/s]\nFound 1 outliers among 100 measurements (1.00%)\n  1 (1.00%) high mild\n\nWatermarkSet Contains/Aligned\n                        time:   [242.04 ns 242.25 ns 242.47 ns]\n                        thrpt:  [2.6396 Gelem/s 2.6419 Gelem/s 2.6442 Gelem/s]\nFound 1 outliers among 100 measurements (1.00%)\n  1 (1.00%) high severe\n\nHashSet Contains/Aligned\n                        time:   [7.6920 us 7.7382 us 7.8144 us]\n                        thrpt:  [81.900 Melem/s 82.706 Melem/s 83.204 Melem/s]\nFound 1 outliers among 100 measurements (1.00%)\n  1 (1.00%) high severe\n\nWatermarkSet Contains/Unaligned\n                        time:   [290.21 ns 290.39 ns 290.56 ns]\n                        thrpt:  [2.2026 Gelem/s 2.2039 Gelem/s 2.2053 Gelem/s]\nFound 4 outliers among 100 measurements (4.00%)\n  2 (2.00%) low mild\n  1 (1.00%) high mild\n  1 (1.00%) high severe\n\nHashSet Contains/Unaligned\n                        time:   [7.9000 us 7.9021 us 7.9043 us]\n                        thrpt:  [80.969 Melem/s 80.991 Melem/s 81.012 Melem/s]\nFound 5 outliers among 100 measurements (5.00%)\n  1 (1.00%) low mild\n  3 (3.00%) high mild\n  1 (1.00%) high severe\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fquantumplation%2Fwatermark","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fquantumplation%2Fwatermark","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fquantumplation%2Fwatermark/lists"}