{"id":41202586,"url":"https://github.com/databricks-industry-solutions/auto-data-linkage","last_synced_at":"2026-01-22T21:30:08.673Z","repository":{"id":65863809,"uuid":"586834887","full_name":"databricks-industry-solutions/auto-data-linkage","owner":"databricks-industry-solutions","description":"Low effort linking and easy de-duplication. Databricks ARC provides a simple, automated, lakehouse integrated entity resolution solution for intra and inter data linking.","archived":false,"fork":false,"pushed_at":"2024-10-28T22:28:28.000Z","size":6422,"stargazers_count":51,"open_issues_count":8,"forks_count":22,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-09-25T07:27:40.314Z","etag":null,"topics":["entity-resolution"],"latest_commit_sha":null,"homepage":"https://databricks-industry-solutions.github.io/auto-data-linkage/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/databricks-industry-solutions.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-01-09T10:45:37.000Z","updated_at":"2025-09-25T02:53:30.000Z","dependencies_parsed_at":"2023-10-11T14:46:00.355Z","dependency_job_id":"e6128eb6-08d0-43a7-80ba-2c4586299bb3","html_url":"https://github.com/databricks-industry-solutions/auto-data-linkage","commit_stats":null,"previous_names":["databricks-industry-solutions/splink-public-sector-hackathon"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/databricks-industry-solutions/auto-data-linkage","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/databricks-industry-solutions%2Fauto-data-linkage","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/databricks-industry-solutions%2Fauto-data-linkage/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/databricks-industry-solutions%2Fauto-data-linkage/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/databricks-industry-solutions%2Fauto-data-linkage/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/databricks-industry-solutions","download_url":"https://codeload.github.com/databricks-industry-solutions/auto-data-linkage/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/databricks-industry-solutions%2Fauto-data-linkage/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28671719,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-22T20:48:19.482Z","status":"ssl_error","status_checked_at":"2026-01-22T20:48:14.968Z","response_time":144,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["entity-resolution"],"created_at":"2026-01-22T21:30:07.562Z","updated_at":"2026-01-22T21:30:08.667Z","avatar_url":"https://github.com/databricks-industry-solutions.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ARC has not been under development since early 2022. It is highly recommended you do not use this tool, and is left here purely as a resource for those wishing to explore automatic linking with Splink. It is recommended that you instead use Splink directly. \n\n\n# Databricks ARC\n\nWelcome to the Databricks ARC Github page.\n\n## Installation\n\nThe package can be installed with pip:\n\n```bash\n%pip install databricks-arc\n```\n\n## Databricks Runtime Requirements\n\nARC requires DBR 12.2 LTS ML\n\n## Project Description\n\nDatabricks ARC (Automated Record Connector) is a solution accelerator by Databricks that performs highly scalable probabilistic data de-duplication \nand linking without the requirement for any labelled data or subject matter expertise in entity resolution.\n\nDe-duplication and linking are 2 sides of the same coin; de-duplication will find records *within* a dataset which represent the same entity, \nwhilst linking will find records *across* 2 datasets which represent the same entity. De-deduplication is key requirement for implementing a Master Data Management strategy;\nfor example, to provide a Single Customer View by consolidating different data silos. Linking is also a key part, by bringing together different fragments of information\nto build a holistic representation of an entity. \n\nTo illustrate with an example, this table requires de-duplicating\n\n|**First Name**|**Surname**|**Address Line 1**|**Address Line 2**|**Address Line 3**|**Post Code**|**DoB**|\n|--------------|-----------|------------------|------------------|------------------|-------------|-------|\n|Jo|Blogs|123 Fake Street|Real Town|Real County|\n\nARC's linking engine is the UK Ministry of Justice's open-sourced entity resolution package, [Splink](https://github.com/moj-analytical-services/splink). It builds on the technology of Splink by removing the need to manually provide parameters to calibrate an unsupervised de-duplication task, which require both a deep understanding of entity resolution and good knowledge of the dataset itself. The way in which ARC achieves this is detailed in the table below:\n\n| **Parameter**           | **Splink**                                                                                                                                            | **ARC**                                                                                                                                                                                 |\n|-------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|\n| Prior match probability | User to provide SQL-like statements for “deterministic rules” and a recall score for Splink to estimate the prior probability that two records match. | Automatically set prior probability to $$\\frac{1}{N}$$.                                                                                                                                                  |\n| Training rules          | User to provide SQL-like statements for a series of rules which trains the m probability values for each column.                                      | Automatically generate training rule statements such that each column is trained.                                                                                                       |\n| Comparisons             | User to provide distance functions and thresholds for each column to compare.                                                                         | Automatically optimise multi-level parameter space for functions and thresholds.                                                                                                        |\n| Blocking rules          | User to provide SQL-like statements to determine the possible comparison space and reduce the number of pairs to compare.                             | User to provide a parameter for the maximum number of pairs they’re willing to compare; Arc identifies all possible blocking rules within that boundary and optimises for the best one. |\n\n\n### Parameter Optimisation\n\nArc uses Hyperopt (http://hyperopt.github.io/hyperopt/) to perform a Bayesian search to find the optimal settings, where optimality is defined as minimising the entropy of the data after clustering and standardising cluster record values. The intuition here is that as we are linking different representations of the same entity together (e.g. Facebook == Fakebook), then standardising data values within a cluster will reduce the total number of data values in the dataset.\n\nTo achieve this, Arc optimises for a custom information gain metric which it calculates based on the clusters of duplicates that Splink predicts. Intuitively, it is based on the reduction in entropy when the data is split into its clusters. The higher the reduction in entropy in the predicted clusters of duplicates predicted, the better the model is doing. Mathematically, we define the metric as follows:\n\nLet the number of clusters in the matched subset of the data be *c*.\n\nLet the maximum number of unique values in any column in the original dataset be *u*.\n\nThen the \"scaled\" entropy of column *k*, *N* unique values with probability *P* is\n\n$$E_{s,k} = -\\Sigma_{i}^{N} P_{i} \\log_{c}(P_{i})$$\n\nThen the \"adjusted\" entropy of column *k*, *N* unique values with probability *P* is\n\n$$E_{a,k} = -\\Sigma_{i}^{N} P_{i} \\log_{u}(P_{i})$$\n\nThe scaled information gain is\n\n$$I_{s} = \\Sigma_{k}^{K} E_{s,k} - E'_{s,k}$$\n\nand the adjusted information gain is\n\n$$I_{a} = \\Sigma_{k}^{K} E_{a,k} - E'_{a,k}$$\n\nwhere *E* is the mean entropy of the individual clusters predicted.\n\nThe metric to optimise for is:\n\n$$I_{s}^{I_{a}}$$\n\n\n## Getting Started\n\nLoad a Spark DataFrame of data to be deduplicated:\n\n```python\ndata = spark.read.table(\"my_catalog.my_schema.my_duplicated_data\")\n```\n\nAfter installation, import and enable ARC:\n\n```python\nimport arc\nfrom arc.autolinker import AutoLinker\n\narc.enable_arc()\n```\n\nInitialise an instance of the `AutoLinker` class:\n\n```python\nautolinker = AutoLinker()\n```\n\nRun unsupervised de-duplication:\n\n```python\nautolinker.auto_link(\n  data=data,                                                         # Spark DataFrame of data to deduplicate\n)\n```\n\nAccess clustered DataFrame - predicted duplicates will share the same `cluster_id`:\n\n```python\nclusters = autolinker.best_clusters_at_threshold()\n```\n\nUse Splink's built-in visualisers and dashboards:\n\n```python\nautolinker.cluster_viewer()\n```\nFor a more in-depth walkthrough please see the included notebooks\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatabricks-industry-solutions%2Fauto-data-linkage","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdatabricks-industry-solutions%2Fauto-data-linkage","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatabricks-industry-solutions%2Fauto-data-linkage/lists"}