{"id":18426377,"url":"https://github.com/jordicorbilla/duplicatechecker","last_synced_at":"2025-04-13T18:54:20.328Z","repository":{"id":74790973,"uuid":"109577389","full_name":"JordiCorbilla/DuplicateChecker","owner":"JordiCorbilla","description":"☑️ .Net service that allows you to check duplicate rows on a sql table using Levenshtein distance","archived":false,"fork":false,"pushed_at":"2020-11-13T09:46:13.000Z","size":55,"stargazers_count":2,"open_issues_count":0,"forks_count":1,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-02-16T08:26:49.173Z","etag":null,"topics":["duplicates","levenshtein-distance"],"latest_commit_sha":null,"homepage":"","language":"C#","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/JordiCorbilla.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-11-05T12:26:06.000Z","updated_at":"2021-09-27T17:18:35.000Z","dependencies_parsed_at":null,"dependency_job_id":"c759cfc5-278b-4e7b-beda-b2f6e69e39c2","html_url":"https://github.com/JordiCorbilla/DuplicateChecker","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JordiCorbilla%2FDuplicateChecker","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JordiCorbilla%2FDuplicateChecker/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JordiCorbilla%2FDuplicateChecker/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JordiCorbilla%2FDuplicateChecker/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/JordiCorbilla","download_url":"https://codeload.github.com/JordiCorbilla/DuplicateChecker/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248766302,"owners_count":21158297,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["duplicates","levenshtein-distance"],"created_at":"2024-11-06T05:07:48.840Z","updated_at":"2025-04-13T18:54:20.308Z","avatar_url":"https://github.com/JordiCorbilla.png","language":"C#","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Duplicate Checker\n.Net service that allows you to check duplicated rows in a sql table.\n\nThe concept is very straight forward. You have a table that contains x number of rows and you would like to know if there are any duplicates on it. By saying duplicate I mean exact duplicates or \"almost\" duplicates. For example if you have the name \"John Snow\" and \"John Snow.\" you can see that they are actually the same but one contains one additional character. To solve this problem I will rely on the Levenshtein distance to tell me how \"similar\" those two texts are and then provide the list of similarities aka duplicates.\n\nThis service will allow you to:\n- Run the duplicate checker on a particular SQL server table where a row would be inspected and a List\u003cstring\u003e of items would be returned with the duplicates.\n - The service will run several threads to try to use as many CPU cycles as possible. The complexity of the whole solution is O(n2) as it requires to run all rows against all of them: M x N, where M is a single row and N is the size of the table - 1 row.\n\nFrom the sample list below, the results are as follows:\n`````c#\npublic List\u003cCar\u003e Get()\n{\n    List\u003cCar\u003e list = new List\u003cCar\u003e();\n\n    list.Add(new Car { Id = 1, Name = \"Caballero\" });\n    list.Add(new Car { Id = 2, Name = \"Cabrio\" });\n    list.Add(new Car { Id = 3, Name = \"Cabriolet\" });\n    list.Add(new Car { Id = 4, Name = \"Calais\" });\n    list.Add(new Car { Id = 5, Name = \"Camargue\" });\n    list.Add(new Car { Id = 6, Name = \"Camaro\" });\n    list.Add(new Car { Id = 7, Name = \"Camry\" });\n    list.Add(new Car { Id = 8, Name = \"Capri\" });\n    list.Add(new Car { Id = 9, Name = \"Caprice\" });\n    list.Add(new Car { Id = 10, Name = \"Caravan\" });\n    list.Add(new Car { Id = 11, Name = \"Caravana\" });\n    list.Add(new Car { Id = 12, Name = \"Caravelle\" });\n    list.Add(new Car { Id = 13, Name = \"Caravella\" });\n    list.Add(new Car { Id = 14, Name = \"Carry\" });\n    list.Add(new Car { Id = 15, Name = \"Catera\" });\n    list.Add(new Car { Id = 16, Name = \"Cattera\" });\n    list.Add(new Car { Id = 17, Name = \"Cavalier\" });\n    list.Add(new Car { Id = 18, Name = \"Cayenne\" });\n    list.Add(new Car { Id = 19, Name = \"Celebrity\" });\n    list.Add(new Car { Id = 20, Name = \"Celica\" });\n    list.Add(new Car { Id = 21, Name = \"Century\" });\n    list.Add(new Car { Id = 22, Name = \"Challenger\" });\n    list.Add(new Car { Id = 23, Name = \"Champ\" });\n    list.Add(new Car { Id = 24, Name = \"Charade\" });\n    list.Add(new Car { Id = 25, Name = \"Charade\" });\n    return list;\n}\n`````\nOutput:\n\n`````\nDuplicates\nCharade, Charade, 0, exact\n\nClose Fit\nCabrio, Cabriolet, 3, closefit\nCapri, Caprice, 2, closefit\nCaravan, Caravana, 1, closefit\nCaravelle, Caravella, 1, closefit\nCatera, Cattera, 1, closefit\n\nSimilar\nCaballero, Cavalier, 3, similar\nCabrio, Capri, 2, similar\nCamargue, Camaro, 3, similar\nCamaro, Camry, 2, similar\nCamry, Carry, 1, similar\nCaravana, Caravella, 3, similar\n`````\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjordicorbilla%2Fduplicatechecker","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjordicorbilla%2Fduplicatechecker","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjordicorbilla%2Fduplicatechecker/lists"}