{"id":22835190,"url":"https://github.com/jchristn/watsondedupe","last_synced_at":"2026-02-28T18:01:53.659Z","repository":{"id":76080599,"uuid":"80537426","full_name":"jchristn/WatsonDedupe","owner":"jchristn","description":"Self-contained C# library for data deduplication using Sqlite","archived":false,"fork":false,"pushed_at":"2023-04-07T18:22:35.000Z","size":3537,"stargazers_count":36,"open_issues_count":0,"forks_count":5,"subscribers_count":8,"default_branch":"master","last_synced_at":"2025-08-22T06:46:12.479Z","etag":null,"topics":["chunk","chunk-data","chunk-key","compress","compression","data-deduplication","dedupe","deduplication","duplicate-data","nuget","sqlite-database","storage"],"latest_commit_sha":null,"homepage":null,"language":"C#","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jchristn.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":".github/FUNDING.yml","license":"LICENSE.md","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null},"funding":{"github":["jchristn"],"custom":["https://paypal.me/joelchristner"]}},"created_at":"2017-01-31T16:16:58.000Z","updated_at":"2024-12-12T09:54:26.000Z","dependencies_parsed_at":null,"dependency_job_id":"5111772c-3551-41d6-b330-87d8c6184c56","html_url":"https://github.com/jchristn/WatsonDedupe","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"purl":"pkg:github/jchristn/WatsonDedupe","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jchristn%2FWatsonDedupe","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jchristn%2FWatsonDedupe/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jchristn%2FWatsonDedupe/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jchristn%2FWatsonDedupe/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jchristn","download_url":"https://codeload.github.com/jchristn/WatsonDedupe/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jchristn%2FWatsonDedupe/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29946463,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-28T17:57:52.716Z","status":"ssl_error","status_checked_at":"2026-02-28T17:57:31.974Z","response_time":90,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["chunk","chunk-data","chunk-key","compress","compression","data-deduplication","dedupe","deduplication","duplicate-data","nuget","sqlite-database","storage"],"created_at":"2024-12-12T22:08:38.722Z","updated_at":"2026-02-28T18:01:53.645Z","avatar_url":"https://github.com/jchristn.png","language":"C#","funding_links":["https://github.com/sponsors/jchristn","https://paypal.me/joelchristner"],"categories":[],"sub_categories":[],"readme":"![alt tag](https://github.com/jchristn/watsondedupe/blob/master/assets/watson.ico)\n\n# Watson Deduplication Library\n\n[![NuGet Version](https://img.shields.io/nuget/v/WatsonDedupe.svg?style=flat)](https://www.nuget.org/packages/WatsonDedupe/) [![NuGet](https://img.shields.io/nuget/dt/WatsonDedupe.svg)](https://www.nuget.org/packages/WatsonDedupe) \n\nSelf-contained C# library for data deduplication targeted to .NET Core, .NET Standard, and .NET Framework.\n\n![alt tag](https://github.com/jchristn/WatsonDedupe/blob/master/assets/diagram_half.png)\n\n## New in v2.0.x\n\n- Major internal refactor and breaking changes\n- Consolidated settings, callbacks, and statistics into new objects\n- Paginated enumeration including prefix-based search\n- Refactored many APIs for simplification\n- Internal table structure extended\n- TryGet APIs\n- Additional properties on ```DedupeObject``` including compressed length and number of chunks\n\n## Help and Support\n\nPlease contact me or file an issue here if you encounter any problems with the library or have suggestions! \n\n## Working with Sqlite on .NET Framework\n\nSqlite has shown to have issues with .NET Framework.  You may need to clone and build, then copy the ```runtimes``` directory from the ```bin/debug/netcoreapp#.#``` directory into your working directory.  You may also need to download and add sqlite3.dll to your project manually.  Set it to copy to output always.\n\n## Under the Hood\n\nThe Watson Dedupe library will take an incoming byte array or stream (which you assign a unique key) and utilize a sliding window performing MD5 calculations over the data in the window to identify breakpoints in the data (this is called 'chunking').  Each chunk of data is assigned a chunk key (based on the SHA256 of the data).  MD5 is used to dynamically identify chunk boundaries, and when a chunk boundary is identified, SHA256 is used to assign a unique key to each chunk.  \n\nTables in a database (Sqlite by default, or, bring your own by implementing the ```DbProvider``` class) are maintained to indicate which objects (```dedupeobject``` table) map to which chunks (```dedupechunk``` table) and their ordering/position (```dedupeobjmap``` table).  Chunks are stored as flat files in a directory you specify using a sanitized version of the chunk key as the filename.  \n\nOn retrieval, the object map is retrieved from the index, the appropriate chunks are retrieved, and the object is reconstructed.  As long as the chunk data is consistent across analyzed data sets, identical chunk keys will be created, meaning duplicate data chunks are only stored once.  Further, each chunk key has a separate associated reference count to ensure that chunks are not garbage collected when a referencing object is deleted should another object also hold that reference.\n \n## Test App \n\nRefer to the Test project which will help you exercise DedupeLibrary.\n\nFor an example of how to use WatsonDedupe using your own database, refer to the ```Test.External``` project along with the sample implementation of the ```WatsonDedupe.Database.DbProvider``` class found in ```Database.cs```.\n\n## CLI\n\nRefer to the CLI project which provide a binary that can be used in a shell or terminal window to interact with an index for object storage, retrieval, removal, and statistics.  \n\n## Library Example\n\nTo use WatsonDedupe, instantiate the ```DedupeSettings``` class and the ```DedupeCallbacks``` class.  ```DedupeSettings``` dictates the inner working of the chunk identification algorithn, and ```DedupeCallbacks``` defines the functions in your application that are invoked for writing, reading, and deleting chunk data.  In this way, your application dictates how chunk data is stored, accessed, and managed.\n\nThe example below is from the ```SampleApp``` project and shows how to use WatsonDedupe with a managed internal Sqlite database.\n \n```csharp\nusing System;\nusing System.IO;\nusing WatsonDedupe;\n\nnamespace SampleApp\n{\n    class Program\n    { \n        static void Main(string[] args)\n        {\n            // Create chunk directory\n            if (!Directory.Exists(\"chunks\")) Directory.CreateDirectory(\"chunks\");\n\n            // Define settings, callbacks, and initialize\n            DedupeSettings  settings  = new DedupeSettings(32768, 262144, 2048, 2);\n            DedupeCallbacks callbacks = new DedupeCallbacks(WriteChunk, ReadChunk, DeleteChunk);\n            DedupeLibrary   dedupe    = new DedupeLibrary(\"test.db\", settings, callbacks);\n\n            // Store objects in the index\n            dedupe.Write(\"kjv1\", File.ReadAllBytes(\"samplefiles/kjv.txt\"));\n            dedupe.Write(\"kjv2\", File.ReadAllBytes(\"samplefiles/kjv.txt\"));\n            dedupe.Write(\"kjv3\", File.ReadAllBytes(\"samplefiles/kjv.txt\"));\n\n            // Check existence and retrieve an object from the index\n            if (dedupe.Exists(\"kjv2\")) Console.WriteLine(\"Exists\");\n            DedupeObject obj = dedupe.Get(\"kjv1\");\n\n            // List all objects\n            Console.WriteLine(dedupe.ListObjects().ToTabularString());\n\n            // Display index statistics\n            Console.WriteLine(dedupe.IndexStats().ToString());\n\n            // Delete an object from the index\n            dedupe.Delete(\"kjv1\");\n        }\n\n        // Called during store operations, consider using FileStream with FileOptions.WriteThrough to ensure crash consistency\n        static void WriteChunk(DedupeChunk data)\n        {\n            File.WriteAllBytes(\"Chunks\\\\\" + data.Key, data.Data); \n        }\n\n        // Called during read operations\n        static byte[] ReadChunk(string key)\n        {\n            return File.ReadAllBytes(\"Chunks\\\\\" + key);\n        }\n\n        // Called during delete operations\n        static void DeleteChunk(string key)\n        {\n            File.Delete(\"Chunks\\\\\" + key); \n        }\n    }\n}\n```\n\n## CLI Example\n\nExample taken using the ```Cli``` project.\n\n```\n// Initialize new index\n\u003e dedupe test.idx create --params=32768,262144,2048,2\n\n// Store an object\n\u003e dedupe test.idx write --chunks=chunks --key=obj1 \u003c obj1.txt\n\n// Retrieve an object\n\u003e dedupe test.idx get --chunks=chunks --key=obj1 \u003e obj1_new.txt\n\n// Delete an object\n\u003e dedupe test.idx del --chunks=chunks --key=obj1\n\n// List objects\n\u003e dedupe test.idx list\n\n// Check object existence\n\u003e dedupe test.idx exists --key=obj1\n\n// Gather index stats\n\u003e dedupe test.idx stats\n```\n\n## Index Settings\n\nFour parameters are used when creating the index: minimum chunk size, maximum chunk size, shift count, and boundary check bytes.  They are defined as follows:\n\n- Minimum chunk size: the smallest amount of data that can be considered a chunk of data\n- Maximum chunk size: the largest amount of data that can be considered a chunk of data\n- Shift count: the number of bytes to move the sliding window while evaluating data in the window for a chunk boundary\n- Boundary check bytes: the number of bytes in the MD5 hash of the data in the sliding window that is evaluated to identify a chunk boundary\n\nThe index parameters should be set in such a way to balance performance vs the frequency with which duplicate data is identified.  With smaller values, it is more likely the library will find repeated data patterns, but processing and storage will take longer and create more chunk records and files.  With larger values, the opposite is true.\n\nIn some cases, it is assumed that duplicate data will not be found within a file and that the library should only be used to identify large chunks of redundant data across larger files (for instance, copies of files with minor changes or backup files).  In such cases, use large values for the index settings.\n\nIn other cases, where it is assumed that duplicate data will be found within a file and across files, smaller values can be used.\n\nRecommended settings for most environments (min, max, shift, boundary):\n\n- For small file environments, use 2048, 16384, 128, and 2\n- For large file environments, use 32768, 262144, 512, and 2 \n\n## External Databases\n\nExternal databases can be used with WatsonDedupe if you implement the ```WatsonDedupe.Database.DbProvider``` class.  You are free to use whatever table and column names you wish, since your code will be responsible for issuing queries and returning results using the WatsonDedupe classes.  Refer to the ```Test.External``` project for an example.\n\n## Version History\n\nPlease refer to CHANGELOG.md.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjchristn%2Fwatsondedupe","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjchristn%2Fwatsondedupe","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjchristn%2Fwatsondedupe/lists"}