{"id":27248846,"url":"https://github.com/bakdata/dedupe","last_synced_at":"2025-04-10T23:48:16.935Z","repository":{"id":33650230,"uuid":"154974836","full_name":"bakdata/dedupe","owner":"bakdata","description":"Java DSL for (online) deduplication","archived":false,"fork":false,"pushed_at":"2025-01-22T10:38:30.000Z","size":1064,"stargazers_count":20,"open_issues_count":2,"forks_count":2,"subscribers_count":7,"default_branch":"master","last_synced_at":"2025-04-10T23:48:10.852Z","etag":null,"topics":["data-cleaning","data-cleansing","deduplication","duplicate-detection","duplicate-removal"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bakdata.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-10-27T14:41:11.000Z","updated_at":"2025-01-22T10:38:32.000Z","dependencies_parsed_at":"2024-02-07T09:42:18.198Z","dependency_job_id":"7fb9584b-7f62-4ad6-98b9-04fde613b213","html_url":"https://github.com/bakdata/dedupe","commit_stats":null,"previous_names":[],"tags_count":9,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bakdata%2Fdedupe","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bakdata%2Fdedupe/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bakdata%2Fdedupe/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bakdata%2Fdedupe/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bakdata","download_url":"https://codeload.github.com/bakdata/dedupe/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248317727,"owners_count":21083528,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-cleaning","data-cleansing","deduplication","duplicate-detection","duplicate-removal"],"created_at":"2025-04-10T23:48:16.527Z","updated_at":"2025-04-10T23:48:16.924Z","avatar_url":"https://github.com/bakdata.png","language":"Java","funding_links":[],"categories":["数据科学"],"sub_categories":["计算机视觉"],"readme":"[![Build Status](https://dev.azure.com/bakdata/public/_apis/build/status/bakdata.dedupe?branchName=master)](https://dev.azure.com/bakdata/public/_build/latest?definitionId=4\u0026branchName=master)\n[![Sonarcloud status](https://sonarcloud.io/api/project_badges/measure?project=com.bakdata.dedupe%3Adedupe\u0026metric=alert_status)](https://sonarcloud.io/dashboard?id=com.bakdata.dedupe%3Adedupe)\n[![Code coverage](https://sonarcloud.io/api/project_badges/measure?project=com.bakdata.dedupe%3Adedupe\u0026metric=coverage)](https://sonarcloud.io/dashboard?id=com.bakdata.dedupe%3Adedupe)\n[![Maven](https://img.shields.io/maven-central/v/com.bakdata.dedupe/core.svg)](https://search.maven.org/search?q=g:com.bakdata.dedupe\u0026core=gav)\n\nJava DSL for (online) deduplication\n===================================\n\nDeduplication is a notorious configuration-heavy process with many competing algorithms and very domain-dependent details. \nThis project aims to provide a concise Java DSL that allows developers to focus on these details without having to switch between languages and frameworks.\n\nIn particular, this project provides the following:\n- A set of interfaces and base classes to model typical stages and (intermediate) outputs of a deduplication process.\n- A type-safe configuration DSL that relies on builder patterns to provide an easy and intuitive way to plug the different parts together.\n- A pure java implementation such that each part can be extended or fully customized with arbitrary complex user code.\n- Implementations for common algorithms for each stage (to be expanded).\n- Wrapper for or reimplementation of common similarity measures.\n- A focus on online algorithms that continuously deduplicate a stream of input records.\n- Examples for different domains (to be expanded).\n\n## Getting Started\n\nYou can add dedupe via Maven Central.\n\n#### Gradle\n```gradle\ncompile group: 'com.bakdata.dedupe', name: 'common', version: '1.1.0'\n```\n\n#### Maven\n```xml\n\u003cdependency\u003e\n    \u003cgroupId\u003ecom.bakdata.dedupe\u003c/groupId\u003e\n    \u003cartifactId\u003ecommon\u003c/artifactId\u003e\n    \u003cversion\u003e1.1.0\u003c/version\u003e\n\u003c/dependency\u003e\n```\n\nFor other build tools or versions, refer to the [latest version in MvnRepository](https://mvnrepository.com/artifact/com.bakdata.dedupe/common/latest).\n\n# Using the framework #\n\nFor a reference, please refer to the [full javadoc](https://bakdata.github.io/dedupe/javadoc/). \n\nIn this section, we cover the different configuration stages. Ultimately, we want to receive a *deduplication* instance where we can feed in the records and receive the results. In this section, we configure an online, pair-based deduplication. The runnable code can be found in [examples](examples).\n\nA pair-base deduplication consists of a [*duplicate detection*](#Duplicate-detection) that finds duplicate clusters and a [*fusion*](#Fusion) that provides a consisted representation of the found clusters.\n```java\n// create deduplication, parts are explained later\nOnlineDeduplication\u003cPerson\u003e deduplication = \n        FusingOnlineDuplicateDetection.\u003cLong, Person\u003ebuilder()\n                .duplicateDetection(new PersonDuplicateDetection())\n                .fusion(new PersonFusion())\n                .build();\n\n// apply it to a list of customers\nList\u003cPerson\u003e customers = ...;\nfor(Person customer: customers) {\n    final Person fusedPerson = deduplication.deduplicate(customer);    \n    // store fused person\n}\n```\n\n## Duplicate detection ##\n\nThe pair-base duplicate detection in turn has a [*candidate selection*](#Candidate-selection) that chooses promising pairs to limit search space, a [*classifier*](#Candidate-classification) that labels pairs as duplicate or non-duplicate, and [*clustering*](#Clustering) that consolidates all pairs into plausible clusters.\n\n```java\nOnlineDuplicateDetection\u003cLong, Person\u003e duplicateDetection = \n        OnlinePairBasedDuplicateDetection.\u003cLong, Person\u003ebuilder()\n                .classifier(new PersonClassifier())\n                .candidateSelection(new PersonCandidateSelection())\n                .clustering(new PersonClustering())\n                .build();\n```\n\n## Candidate selection ##\nThe different parts are configured in the following. First, we define the candidate selection for a person.\n\n```java\n// configure candidate selection with 3 passes (= 3 sorting keys)\nOnlineCandidateSelection\u003cPerson\u003e candidateSelection = OnlineSortedNeighborhoodMethod.\u003cPerson\u003ebuilder()\n    .defaultWindowSize(WINDOW_SIZE)\n    .sortingKey(new SortingKey\u003c\u003e(\"First name+Last name\",\n            person -\u003e CompositeValue.of(normalizeName(person.getFirstName()), normalizeName(person.getLastName()))))\n    .sortingKey(new SortingKey\u003c\u003e(\"Last name+First name\",\n            person -\u003e CompositeValue.of(normalizeName(person.getLastName()), normalizeName(person.getFirstName()))))\n    .sortingKey(new SortingKey\u003c\u003e(\"Bday+Last name\",\n            person -\u003e CompositeValue.of(person.getBirthDate(), normalizeName(person.getLastName()))))\n    .build();\n\nprivate static String normalizeName(String value) {\n    // split umlauts into canonicals\n    return java.text.Normalizer.normalize(value.toLowerCase(), java.text.Normalizer.Form.NFD)\n            // remove everything in braces\n            .replaceAll(\"\\\\(.*?\\\\)\", \"\")\n            // remove all non-alphanumericals\n            .replaceAll(\"[^\\\\p{Alnum}]\", \"\");\n}\n```\n\nThe chosen sorted neighborhood uses sorting keys to sort the input and compare all records within a given window. In this case, we perform 3 passes with different sorting keys. Each pass uses the same window size but they can also be configured individually.\n\nThe sorting keys can be of arbitrary, comparable data type. The framework provides *CompositeValue* when the sorting key consists of several parts (which is recommended to resolve ties in the first part of the key). The normalizeName function is a custom UDF specific for this domain.\n\n## Candidate classification ##\n\nThe output of the candidate selection is a list of candidate pairs, which is fed into the candidate classificationResult.\n\n```java\nimport static com.bakdata.dedupe.similarity.CommonSimilarityMeasures.*;\nClassifier\u003cPerson\u003e personClassifier = RuleBasedClassifier.\u003cPerson\u003ebuilder()\n    .negativeRule(\"Different social security number\", inequality().of(Person::getSSN))\n    .positiveRule(\"Default\", CommonSimilarityMeasures.\u003cPerson\u003eweightedAverage()\n        .add(10, Person::getSSN, equality())\n        .add(2, Person::getFirstName, max(levenshtein().cutoff(.5f), jaroWinkler()))\n        .add(2, Person::getLastName, max(equality().of(beiderMorse()), jaroWinkler()))\n        .build()\n        .scaleWithThreshold(.9f))\n    .build();\n```\n\nThe classificationResult DSL heavy relies on static factory functions in *CommonSimilarityMeasures* and can be easily extended by custom base similarity measures.\n\nThe used rule-based classificationResult applies a list of rules until a rule triggers and thus determines the classificationResult.\n\nIn this case, a negative rule first checks if the two persons of the candidate pair have a different social security number (SSN) if present. \n\nThe next rule performs a weighted average of 3 different feature similarities. If SSN is given, it is highly weighted and almost suffices for a positive or negative classificationResult. Additionally, first and last name contribute to the classificationResult.\n\n## Clustering ##\n\nTo consolidate a list of duplicate pairs into a consistent cluster, an additional clustering needs to be performed. Often times, only transitive closure is applied, which is also available in this framework. However, for high precision use cases, transitive closure is not enough.\n\n```java\nRefineCluster\u003cLong, Person\u003e refineCluster = RefineCluster.\u003cLong, Person\u003ebuilder()\n        .classifier(new PersonClassifier())\n        .clusterIdGenerator(ClusterIdGenerators.longGenerator())\n        .build();\n\nClustering\u003cLong, Person\u003e refinedTransitiveClosure = RefinedTransitiveClosure.\u003cLong, Person, String\u003ebuilder()\n        .refineCluster(this.refineCluster)\n        .idExtractor(Person::getId)\n        .build();\n\n@Delegate\nClustering\u003cLong, Person\u003e clustering = ConsistentClustering.\u003cLong, Person, String\u003ebuilder()\n        .clustering(this.refinedTransitiveClosure)\n        .idExtractor(Person::getId)\n        .build();\n```\n\nHere, we configure a *refining transitive closure* strategy that in particular checks for explicit negative classifications inside the transitive closure such that duplicate clusters are split into more plausible subclusters.\n\nFinally, the wrapping *consistent clustering* ensure that IDs remain stable in the long run of an online deduplication.\n\n## Fusion ##\n\nUltimately, we want to receive a duplicate-free dataset. Hence, we need to configure the fusion.\n\n```java\nConflictResolution\u003cPerson, Person\u003e personMerge = ConflictResolutions.merge(Person::new)\n    .field(Person::getId, Person::setId).with(min())\n    .field(Person::getFirstName, Person::setFirstName).with(longest()).then(vote())\n    .field(Person::getLastName, Person::setLastName).correspondingToPrevious()\n    .field(Person::getSSN, Person::setSSN).with(assumeEqualValue())\n    .build();\nFusion\u003cPerson\u003e personFusion = ConflictResolutionFusion.\u003cPerson\u003ebuilder()\n    .sourceExtractor(Person::getSource)\n    .lastModifiedExtractor(Person::getLastModified)\n    .rootResolution(personMerge)\n    .build();\n```\n\nThe shown conflict resolution approach, reconciles each field in a recursive manner by choosing the respective value according to a list of conflict resolution functions. Each function reduces the list of candidate values until hopefully only one value remains. If not, the fusion fails and has to be manually resolved.\n\nThe conflict resolution may also use source preferences, source confidence scores, and timestamps to find the best value.\n\n# Maintenance #\n\n## Structure ##\n\nThe project consists of three parts\n- **core**: Provides the basic interfaces and data structures of the various deduplication stages.\n- **common**: Implements common similarities and algorithms.\n- **examples**: Showcases different domains.\n\n## Building \u0026 executing tests ##\n\nThis project requires Java 11. To build and execute all tests, please run\n```bash\n./gradlew build\n```\n\nFor IDEs, import the project (or the build.gradle.ktsgit a file) as a gradle project. The project makes heavily use of [Lombok](http://projectlombok.org/), so make sure you have the appropriate IDE plugin and enabled annotation preprocessing. \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbakdata%2Fdedupe","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbakdata%2Fdedupe","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbakdata%2Fdedupe/lists"}