{"id":13453855,"url":"https://github.com/moj-analytical-services/splink","last_synced_at":"2025-05-13T16:07:30.757Z","repository":{"id":36968802,"uuid":"223420284","full_name":"moj-analytical-services/splink","owner":"moj-analytical-services","description":"Fast, accurate and scalable probabilistic data linkage with support for multiple SQL backends","archived":false,"fork":false,"pushed_at":"2025-04-22T10:11:56.000Z","size":103079,"stargazers_count":1561,"open_issues_count":196,"forks_count":172,"subscribers_count":18,"default_branch":"master","last_synced_at":"2025-04-22T11:23:39.049Z","etag":null,"topics":["data-matching","data-science","deduplicate-data","deduplication","duckdb","em-algorithm","entity-resolution","fuzzy-matching","record-linkage","spark","uk-gov-data-science"],"latest_commit_sha":null,"homepage":"https://moj-analytical-services.github.io/splink/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/moj-analytical-services.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2019-11-22T14:27:33.000Z","updated_at":"2025-04-22T10:10:53.000Z","dependencies_parsed_at":"2023-01-17T10:01:28.136Z","dependency_job_id":"6bb9d3fb-8392-4115-bc5b-d456ecd7d5ee","html_url":"https://github.com/moj-analytical-services/splink","commit_stats":{"total_commits":7251,"total_committers":77,"mean_commits":94.16883116883118,"dds":0.6314991035719211,"last_synced_commit":"b7319930a821962d2ea4498846a6697d03c074e9"},"previous_names":[],"tags_count":124,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/moj-analytical-services%2Fsplink","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/moj-analytical-services%2Fsplink/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/moj-analytical-services%2Fsplink/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/moj-analytical-services%2Fsplink/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/moj-analytical-services","download_url":"https://codeload.github.com/moj-analytical-services/splink/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250529906,"owners_count":21445692,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-matching","data-science","deduplicate-data","deduplication","duckdb","em-algorithm","entity-resolution","fuzzy-matching","record-linkage","spark","uk-gov-data-science"],"created_at":"2024-07-31T08:00:48.520Z","updated_at":"2025-04-23T23:22:18.409Z","avatar_url":"https://github.com/moj-analytical-services.png","language":"Python","funding_links":[],"categories":["Libraries Powered by DuckDB","JavaScript","NLP","spark",":hammer: Frameworks","Software","Open-Source Software"],"sub_categories":["Web Clients","Clustering /","End-to-End Entity Resolution"],"readme":"\u003cp align=\"center\"\u003e\n\u003cimg src=\"https://user-images.githubusercontent.com/7570107/85285114-3969ac00-b488-11ea-88ff-5fca1b34af1f.png\" alt=\"Splink Logo\" height=\"150px\"\u003e\n\u003c/p\u003e\n\n[![pypi](https://img.shields.io/github/v/release/moj-analytical-services/splink?include_prereleases)](https://pypi.org/project/splink/#history)\n[![Downloads](https://static.pepy.tech/badge/splink/month)](https://pepy.tech/project/splink)\n[![Documentation](https://img.shields.io/badge/API-documentation-blue)](https://moj-analytical-services.github.io/splink/)\n\n\n\u003e [!IMPORTANT]\n\u003e 🎉 Splink 4 has been released! Examples of new syntax are [here](https://moj-analytical-services.github.io/splink/demos/examples/examples_index.html) and a release announcement is [here](https://moj-analytical-services.github.io/splink/blog/2024/07/24/splink-400-released.html).\n\n\n# Fast, accurate and scalable data linkage and deduplication\n\nSplink is a Python package for probabilistic record linkage (entity resolution) that allows you to deduplicate and link records from datasets that lack unique identifiers.\n\nIt is used widely by within government, academia and the private sector - see [use cases](https://moj-analytical-services.github.io/splink/#use-cases).\n\n## Key Features\n\n⚡ **Speed:** Capable of linking a million records on a laptop in around a minute.\u003cbr\u003e\n🎯 **Accuracy:** Support for term frequency adjustments and user-defined fuzzy matching logic.\u003cbr\u003e\n🌐 **Scalability:** Execute linkage in Python (using DuckDB) or big-data backends like AWS Athena or Spark for 100+ million records.\u003cbr\u003e\n🎓 **Unsupervised Learning:** No training data is required for model training.\u003cbr\u003e\n📊 **Interactive Outputs:** A suite of interactive visualisations help users understand their model and diagnose problems.\u003cbr\u003e\n\nSplink's linkage algorithm is based on Fellegi-Sunter's model of record linkage, with various customisations to improve accuracy.\n\n## What does Splink do?\n\nConsider the following records that lack a unique person identifier:\n\n![tables showing what splink does](https://raw.githubusercontent.com/moj-analytical-services/splink/master/docs/img/README/what_does_splink_do_1.drawio.png)\n\nSplink predicts which rows link together:\n\n![tables showing what splink does](https://raw.githubusercontent.com/moj-analytical-services/splink/master/docs/img/README/what_does_splink_do_2.drawio.png)\n\nand clusters these links to produce an estimated person ID:\n\n![tables showing what splink does](https://raw.githubusercontent.com/moj-analytical-services/splink/master/docs/img/README/what_does_splink_do_3.drawio.png)\n\n## What data does Splink work best with?\n\nSplink performs best with input data containing **multiple** columns that are **not highly correlated**. For instance, if the entity type is persons, you may have columns for full name, date of birth, and city. If the entity type is companies, you could have columns for name, turnover, sector, and telephone number.\n\nHigh correlation occurs when one column is highly predictable from another - for instance, city can be predicted from postcode.  Correlation is particularly problematic if **all** of your input columns are highly correlated.\n\nSplink is not designed for linking a single column containing a 'bag of words'. For example, a table with a single 'company name' column, and no other details.\n\n## Documentation\n\nThe homepage for the Splink documentation can be found [here](https://moj-analytical-services.github.io/splink/), including a [tutorial](https://moj-analytical-services.github.io/splink/demos/tutorials/00_Tutorial_Introduction.html) and [examples](https://moj-analytical-services.github.io/splink/demos/examples/examples_index.html) that can be run in the browser.\n\n\nThe specification of the Fellegi Sunter statistical model behind `splink` is similar as that used in the R [fastLink package](https://github.com/kosukeimai/fastLink). Accompanying the fastLink package is an [academic paper](http://imai.fas.harvard.edu/research/files/linkage.pdf) that describes this model. The [Splink documentation site](https://moj-analytical-services.github.io/splink/topic_guides/fellegi_sunter.html) and a [series of interactive articles](https://www.robinlinacre.com/probabilistic_linkage/) also explores the theory behind Splink.\n\nThe Office for National Statistics have written a [case study about using Splink](https://github.com/Data-Linkage/Splink-census-linkage/blob/main/SplinkCaseStudy.pdf) to link 2021 Census data to itself.\n\n## Installation\n\nSplink supports python 3.8+. To obtain the latest released version of splink you can install from PyPI using pip:\n\n```sh\npip install splink\n```\n\nor, if you prefer, you can instead install splink using conda:\n\n```sh\nconda install -c conda-forge splink\n```\n\n### Installing Splink for Specific Backends\n\n\nFor projects requiring specific backends, Splink offers optional installations for **Spark**, **Athena**, and **PostgreSQL**. These can be installed by appending the backend name in brackets to the pip install command:\n```sh\npip install 'splink[{backend}]'\n```\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003ci\u003eClick here for backend-specific installation commands\u003c/i\u003e\u003c/summary\u003e\n\n#### Spark\n```sh\npip install 'splink[spark]'\n```\n\n#### Athena\n```sh\npip install 'splink[athena]'\n```\n\n#### PostgreSQL\n```sh\npip install 'splink[postgres]'\n```\n\u003c/details\u003e\n\n## Quickstart\n\nThe following code demonstrates how to estimate the parameters of a deduplication model, use it to identify duplicate records, and then use clustering to generate an estimated unique person ID.\n\nFor more detailed tutorial, please see [here](https://moj-analytical-services.github.io/splink/demos/tutorials/00_Tutorial_Introduction.html).\n\n```py\nimport splink.comparison_library as cl\nimport splink.comparison_template_library as ctl\nfrom splink import DuckDBAPI, Linker, SettingsCreator, block_on, splink_datasets\n\ndb_api = DuckDBAPI()\n\ndf = splink_datasets.fake_1000\n\nsettings = SettingsCreator(\n    link_type=\"dedupe_only\",\n    comparisons=[\n        cl.JaroWinklerAtThresholds(\"first_name\", [0.9, 0.7]),\n        cl.JaroAtThresholds(\"surname\", [0.9, 0.7]),\n        ctl.DateComparison(\n            \"dob\",\n            input_is_string=True,\n            datetime_metrics=[\"year\", \"month\"],\n            datetime_thresholds=[1, 1],\n        ),\n        cl.ExactMatch(\"city\").configure(term_frequency_adjustments=True),\n        ctl.EmailComparison(\"email\"),\n    ],\n    blocking_rules_to_generate_predictions=[\n        block_on(\"first_name\"),\n        block_on(\"surname\"),\n    ]\n)\n\nlinker = Linker(df, settings, db_api)\n\nlinker.training.estimate_probability_two_random_records_match(\n    [block_on(\"first_name\", \"surname\")],\n    recall=0.7,\n)\n\nlinker.training.estimate_u_using_random_sampling(max_pairs=1e6)\n\nlinker.training.estimate_parameters_using_expectation_maximisation(\n    block_on(\"first_name\", \"surname\")\n)\n\nlinker.training.estimate_parameters_using_expectation_maximisation(block_on(\"dob\"))\n\npairwise_predictions = linker.inference.predict(threshold_match_weight=-10)\n\nclusters = linker.clustering.cluster_pairwise_predictions_at_threshold(\n    pairwise_predictions, 0.95\n)\n\ndf_clusters = clusters.as_pandas_dataframe(limit=5)\n```\n\n## Videos\n\n- [Pydata Global 2024 talk](https://www.youtube.com/watch?v=eQtFkI8f02U)\n- [A introductory presentation on Splink](https://www.youtube.com/watch?v=msz3T741KQI)\n- [An introduction to the Splink Comparison Viewer dashboard](https://www.youtube.com/watch?v=DNvCMqjipis)\n\n\n## Support\n\nTo find the best place to ask a question, report a bug or get general advice, please refer to our [Guide](./CONTRIBUTING.md).\n\n## Awards\n\n🥈 Civil Service Awards 2023: Best Use of Data, Science, and Technology - [Runner up](https://www.civilserviceawards.com/best-use-of-data-science-and-technology-award-2/)\n\n🥇 Analysis in Government Awards 2022: People's Choice Award - [Winner](https://analysisfunction.civilservice.gov.uk/news/announcing-the-winner-of-the-first-analysis-in-government-peoples-choice-award/)\n\n🥈 Analysis in Government Awards 2022: Innovative Methods - [Runner up](https://twitter.com/gov_analysis/status/1616073633692274689?s=20\u0026t=6TQyNLJRjnhsfJy28Zd6UQ)\n\n🥇 Analysis in Government Awards 2020: Innovative Methods - [Winner](https://www.gov.uk/government/news/launch-of-the-analysis-in-government-awards)\n\n🥇 MoJ Data and Analytical Services Directorate (DASD) Awards 2020: Innovation and Impact - Winner\n\n\n## Citation\n\nIf you use Splink in your research, please cite as follows:\n\n```BibTeX\n@article{Linacre_Lindsay_Manassis_Slade_Hepworth_2022,\n\ttitle        = {Splink: Free software for probabilistic record linkage at scale.},\n\tauthor       = {Linacre, Robin and Lindsay, Sam and Manassis, Theodore and Slade, Zoe and Hepworth, Tom and Kennedy, Ross and Bond, Andrew},\n\tyear         = 2022,\n\tmonth        = {Aug.},\n\tjournal      = {International Journal of Population Data Science},\n\tvolume       = 7,\n\tnumber       = 3,\n\tdoi          = {10.23889/ijpds.v7i3.1794},\n\turl          = {https://ijpds.org/article/view/1794},\n}\n```\n\n## Acknowledgements\n\nWe are very grateful to [ADR UK](https://www.adruk.org/) (Administrative Data Research UK) for providing the initial funding for this work as part of the [Data First](https://www.adruk.org/our-work/browse-all-projects/data-first-harnessing-the-potential-of-linked-administrative-data-for-the-justice-system-169/) project.\n\nWe are extremely grateful to professors Katie Harron, James Doidge and Peter Christen for their expert advice and guidance in the development of Splink. We are also very grateful to colleagues at the UK's Office for National Statistics for their expert advice and peer review of this work. Any errors remain our own.\n\n## Related Repositories\n\nWhile Splink is a standalone package, there are a number of repositories in the Splink ecosystem:\n\n\n- [splink_scalaudfs](https://github.com/moj-analytical-services/splink_scalaudfs) contains the code to generate [User Defined Functions](https://moj-analytical-services.github.io/splink/dev_guides/udfs.html#spark) in scala which are then callable in Spark.\n- [splink_datasets](https://github.com/moj-analytical-services/splink_datasets) contains datasets that can be installed automatically as a part of Splink through the [In-build datasets](https://moj-analytical-services.github.io/splink/datasets.html) functionality.\n- [splink_synthetic_data](https://github.com/moj-analytical-services/splink_synthetic_data) contains code to generate synthetic data.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmoj-analytical-services%2Fsplink","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmoj-analytical-services%2Fsplink","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmoj-analytical-services%2Fsplink/lists"}