{"id":940,"url":"https://github.com/Evnsn/awsome-entity-resolution","name":"awsome-entity-resolution","description":"A collection of awesome resources regarding Record Linkage.","projects_count":59,"last_synced_at":"2026-08-05T12:00:24.575Z","repository":{"id":167217122,"uuid":"605067078","full_name":"Evnsn/awsome-entity-resolution","owner":"Evnsn","description":"A collection of awesome resources regarding Record Linkage.","archived":false,"fork":false,"pushed_at":"2024-08-16T08:01:39.000Z","size":14,"stargazers_count":8,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-06-28T20:04:09.540Z","etag":null,"topics":["awesome","awesome-list","data-linkage","data-matching","datamatching","dm","em","entity-matching","entity-resolution","entityresolution","er","machine-learning","python","record-linkage","recordlinkage"],"latest_commit_sha":null,"homepage":"","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Evnsn.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2023-02-22T11:24:43.000Z","updated_at":"2026-06-09T05:51:14.000Z","dependencies_parsed_at":"2024-01-11T19:17:50.439Z","dependency_job_id":"5ada4367-7f21-4443-9073-75f24546687f","html_url":"https://github.com/Evnsn/awsome-entity-resolution","commit_stats":null,"previous_names":["evnsn/awsome-entity-resolution"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/Evnsn/awsome-entity-resolution","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Evnsn%2Fawsome-entity-resolution","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Evnsn%2Fawsome-entity-resolution/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Evnsn%2Fawsome-entity-resolution/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Evnsn%2Fawsome-entity-resolution/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Evnsn","download_url":"https://codeload.github.com/Evnsn/awsome-entity-resolution/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Evnsn%2Fawsome-entity-resolution/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":36311008,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-07-20T02:08:10.276Z","status":"online","status_checked_at":"2026-08-05T02:00:06.619Z","response_time":104,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"created_at":"2024-01-04T17:41:51.459Z","updated_at":"2026-08-05T12:00:24.575Z","primary_language":null,"list_of_lists":false,"displayable":true,"categories":["Datasets",":books: Books",":hammer: Frameworks",":page_with_curl: Papers",":pushpin: Miscellaneous"],"sub_categories":["Clustering /","Surveys","Entity Matching Management Systems","Generic Entity Resolution Techniques","Indexing","Miscellaneous (impactful papers?)","Classification"],"readme":"# (_deprecated_) Awesome Entity Resolution [![Awesome](https://cdn.rawgit.com/sindresorhus/awesome/d7305f38d29fed78fa85652e3a63e154dd8e8829/media/badge.svg)](https://github.com/sindresorhus/awesome)\n\n\u003e A collection of awesome resources regarding Entity Resolution.\n\n# Table of Contents\n\n\u003c!-- * What is Record Linkage? --\u003e\n\n- [Books](#books)\n- [Papers](#papers)\n- Frameworks\n- Datasets\n- Projects\n- Miscellaneous\n\n## :point_right: What's Record Linkage?\n*Entity Resolution* (ER) aims to identify different descriptions that refer to the same real-world object. Detecting entities stored in the same database is refeerd to as *deduplication*, while  *record linkage* refeers to detectation in two different databases. \n\n## :books: Books\n\n1. [Data Matching: Concepts and Techniques for Record Linkage, Entity Resolution, and Duplicate Detection](https://link.springer.com/book/10.1007/978-3-642-31164-2) by Peter Christen (2012)\n2. [Data Quality and Record Linkage Techniques](https://link.springer.com/book/10.1007/0-387-69505-2) by Thomas N. Herzog, Fritz J. Scheuren \u0026 William E. Winkler (2007)\n\n## :page_with_curl: Papers\n\n### Surveys\n\n- 2020 | An Overview of End-to-End Entity Resolution for Big Data | Vassilis Christophides, et al. | [`pdf`](https://arxiv.org/pdf/1905.06397.pdf)\n- 2012 | A Survey of Indexing Techniques for Scalable Record Linkage and Deduplication | Peter Christen | [`pdf`](https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=\u0026arnumber=5887335)\n- 2007 | Duplicate Record Detection: A Survey | Ahmed K. Elmagarmid, et a.l | [`pdf`](https://ieeexplore.ieee.org/document/4016511)\n\n### Entity Matching Management Systems\n\n- 2016 | Magellan: Toward Building Entity Matching Management Systems | Pradap Konda, et al. | [`pdf`](http://www.vldb.org/pvldb/vol9/p1197-pkonda.pdf) | [`git`](https://github.com/anhaidgroup/py_entitymatching)\n\n### Generic Entity Resolution Techniques\n- [Swoosh: a generic approach to entity resolution](http://infolab.stanford.edu/serf/swoosh_vldbj.pdf)\n\u003c!-- - [HARRA: Fast Iterative Hashed Record Linkage for Large-Scale Data Collections](https://openproceedings.org/2010/conf/edbt/KimL10.pdf)\n- [(stringMap) Supporting Efficient Record Linkage for Large Data Sets Using Mapping Techniques](https://www.ics.uci.edu/~chenli/pub/2006-ljm.pdf) --\u003e\n\n### Indexing\n\n\u003c!-- - [Corleone: Hands-Off Crowdsourcing for Entity Matching](https://pages.cs.wisc.edu/~anhai/papers/corleone-sigmod14.pdf), 2009 --\u003e\n\n#### Debugging of blocking\n\n\u003c!-- - [MatchCatcher: A Debugger for Blocking in Entity Matching](https://pages.cs.wisc.edu/~anhai/papers1/matchcatcher-edbt18.pdf), 2018 --\u003e\n\n### Pair compairinson\n\n...\n\n### Miscellaneous (impactful papers?)\n\n- 1969 | A Theory for Record Linkage | Fellegi, I.P., Sunter, A.B. | [`pdf`](https://courses.cs.washington.edu/courses/cse590q/04au/papers/Felligi69.pdf)\n\n### Classification\n\n#### Supervised\n\n...\n\n#### Unsupervised\n\n- 2019 | Using a Probabilistic Model to Assist Merging of Large-Scale Administrative Records | T. Enamorado, et al. | [`pdf`](https://imai.fas.harvard.edu/research/files/linkage.pdf) | [`GiT`](https://github.com/kosukeimai/fastLink)\n\n### Clustering /\n\n- 2020 | Entity Matching in the Wild: A Consistent and Versatile Framework to Unify Data in Industrial Applications | Yan Yan, et al. | [`pdf`](https://dl.acm.org/doi/pdf/10.1145/3318464.3386143)\n\n## :hammer: Frameworks\nTable 1 is a composition of tools presented in [(2020, V. Christophides)](https://arxiv.org/pdf/1905.06397.pdf), [(2015, P. Konda)](http://www.vldb.org/pvldb/vol9/p1197-pkonda.pdf) and [J535D165/data-matching-software](https://github.com/J535D165/data-matching-software).\n\n\u003c!-- TODO:\n* Move \n  * 'scaling column', right of 'language' \n* Move Paper?\n --\u003e\n\n**\u003cins\u003eTable 1:\u003c/ins\u003e** \n***Blocking:** Attribute equivalence (AE), Blocking index (BI), Canopy clustering (CC), Canopy index (CI), Clustering (C), Expectation maximization (EM), Full index (FI), Hash-based (HB), Hybrid (H), Induction (I), Predicate-based (PB), Probabilistic (P), Relational clustering (RC), Rule-based (RB), Sorted neighborhood (SN) Sorting index (SoI), Stringmap index (StI), Suffixarray index (SuI). \n**Matching:** Agglomerative hierarchical clustering-based (AHC), Decision trees (DT), Farthest First (FF), Fellegi-Sunter (FS), k-Nearest-neighbour (KNN), Logistic regression (LR), Optimal threshold (OT), Support vector machine (SVM) TwoStep (TS).*\n\n| Tools |Blocking|Matching|Clustering|  | UI | Scaling | Language | OSS | GiT/Inst | Paper |\n|:--|:--|:--|:--|:--| :--|:--|:--|:--|:--|:--\n| Active Atlas  |HB|DT|---|  | GUI, CMD |:x:|Java|:x:|---|---|\n|Atyimo|---|---|---||---|---|Python|---|[`git`](https://github.com/pierrepita/atyimo)|---|\n| BigMatch      |AE, RB|:x:|---|  | CMD |:heavy_check_mark:|C|:x:|---| [(2002, W. E. Yancey)](https://www.census.gov/content/dam/Census/library/working-papers/2002/adrm/rrc2002-01.pdf)     \n| D-Dupe        |AE|RC|---|  | GUI, CMD |:x:|C#|:x:|---|[(2006, M. Bilgic)](https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=\u0026arnumber=4035746)|    \n| Dedoop |AE, SN| DT, LR, SVM, etc| --- |  | GUI | Hadoop | Java|:x:|[`install`](https://dbs.uni-leipzig.de/howto_dedoop)| [(2012, L Kolb)](https://dbs.uni-leipzig.de/file/Dedoop.pdf)|\n| Dedupe |CC, PB| AHC | :x: |  | API, CMD | :heavy_check_mark:| Python | :heavy_check_mark:| [`git`](https://github.com/dedupeio/dedupe)|[(2003, M. Bilenko)](https://www.cs.utexas.edu/~ml/papers/marlin-kdd-03.pdf), [(2006, M. Bilenko)](https://www.cs.utexas.edu/~ml/papers/marlin-dissertation-06.pdf)|\n| DuDe    | SN | RB |:x:|  |CMD|:x:|Java|:heavy_check_mark:|[`install`](https://hpi.de/en/naumann/projects/data-quality-and-cleansing/dude-duplicate-detection.html)|[(2010, U. Draisbach)](https://www.comp.nus.edu.sg/~vldb2010/proceedings/files/vldb_2010_workshop/QDB_2010/Paper5_Draisbach_Naumann.pdf)\n|Duke|:heavy_check_mark:|:heavy_check_mark:|---||CMD|:x:|Java|---|[`git`](https://github.com/larsga/Duke)|Blog: [(2011, L. Marius)](https://www.garshol.priv.no/blog/217.html)|\n| FAMER |:x:|:x:|:heavy_check_mark:|  |---|Apache Flink|---|---|[`gitlab`](https://git.informatik.uni-leipzig.de/dbs/FAMER/) |[(2018, A Saeedi)](https://dbs.uni-leipzig.de/file/FAMER-2407-8454-1-SM.pdf)|\n| fastLink|:x:|---|---|  |API|:heavy_check_mark:|R|:heavy_check_mark:|[`git`](https://github.com/kosukeimai/fastLink)|[(2017, T. Enamorado)](https://imai.fas.harvard.edu/research/files/linkage.pdf)|\n| Febrl |BI, CI, FI, SoI, StI, SuI, Q-gram|FS, OT, K-means, FF, SVM, TS|:x:|  |GUI|:grey_question:|Python|:heavy_check_mark:|[`install`](https://sourceforge.net/projects/febrl/)|[(2013, P. Christen)](https://unstats.un.org/unsd/demographic/meetings/wshops/Ethiopia_14_Sept_09/Manuals/Peter.christen-febrl-demo.pdf)|\n| FRIL |AE, SN|EM|:x:|  |GUI|:grey_question:|Java|:heavy_check_mark:|[`install`](https://fril.sourceforge.net/download.html)|[(2008, P Jurczyk)](https://onlinelibrary.wiley.com/doi/full/10.1002/bdra.20521)|\n| JedAI|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:||GUI|Apache Spark|Java|:heavy_check_mark:|[`git`](https://github.com/scify/JedAIToolkit)|[(2020, G. Papadakis)](https://openproceedings.org/2020/conf/edbt/paper_273.pdf)|\n| KnoFuss|:heavy_check_mark:|:heavy_check_mark:|---||---|:x:|Java|---|---|[(2008, A. Nikolov)](http://oro.open.ac.uk/28010/5/27994.pdf)|\n| LIMES|---|---|---||GUI|:x:|Java|:heavy_check_mark:|[`git`](https://github.com/dice-group/LIMES)|[(2011, A. C. N. Ngomo)](https://www.ijcai.org/Proceedings/11/Papers/385.pdf)|\n| Magellan|:heavy_check_mark:|:heavy_check_mark:|:x:| |API, GUI|Apache Spark|Python|:heavy_check_mark:|[`git`](https://github.com/anhaidgroup/py_entitymatching)|[(2016, P. Konda)](https://arxiv.org/pdf/1905.06397.pdf)\n| MARLIN|CC|DT, SVM|---|  |:x:|---|---|---|---|[(2004, M. Bilenko)](https://www.cs.utexas.edu/~ml/papers/marlin-aaaidc-04.pdf)|\n| Merge Toolbox |AE, CC|P, EM|---|  |GUI|:x:|Java|:x:|[`install`](https://www.record-linkage.de/software/index.html)|[(2004, R. Schnell)](https://d-nb.info/1191659240/34)|\n| MinoanER|:heavy_check_mark:|:heavy_check_mark:|:x:||GUI|Apache Spark|Java|:heavy_check_mark:|---| [(2019, V. Efthymiou)](https://arxiv.org/pdf/1905.06170.pdf)|\n| NADEEF |---|RB|---|  |GUI|:x:|Java|:x:|---|[(2013, M. Dallachiesa)](https://cs.uwaterloo.ca/~ilyas/papers/NADEEFSigmod2013.pdf)|\n| OYSTER|AE|RB|---|  |CMD|:x:|Java|:heavy_check_mark:|[`install`](https://sourceforge.net/projects/oysterer/)|[(2011, E. D. Nelson)](http://worldcomp-proceedings.com/proc/p2011/IKE5074.pdf)|\n| PRIL|---|---|---||GUI|---|C#|---|[`git`](https://github.com/LSHTM-ALPHAnetwork/PIRL_RecordLinkageSoftware)|[(2018, C. T. Rentsch)](https://gatesopenresearch.org/articles/1-8/v1)|\n| pydedupe|AE|KNN, K-means, RB|---||CMD|:x:|Python|:heavy_check_mark:|[`git`](https://github.com/gpoulter/pydedupe)|---|\n| Reclin2|---|---|---||API|---|R|---|[`git`](https://github.com/djvanderlaan/reclin2)|---|\n| RELAIS|---|---|---||GUI|---|R/Java|---|[`install`](https://www.istat.it/en/methods-and-tools/methods-and-it-tools/process/processing-tools/relais)| [(2006, M. Fortini)](https://www.istat.it/it/files/2011/03/FSTT_IQIS06_CR.pdf)|\n|RLTK|---|---|---| |API|---|Python|:heavy_check_mark:|[`git`](https://github.com/usc-isi-i2/rltk)|---|\n| Record Linkage (R)|AE|ML-based|---|  |CMD|:x:|R|:heavy_check_mark:|[`cran`](https://cran.r-project.org/web/packages/RecordLinkage/index.html)|[(2011, M Sariyar)](https://www.sciencedirect.com/science/article/pii/S1532046411000372)|\n| Record Linkage (Python) |FI, BI, SN|DC, LR, SVM, K-means, EM|---| |API|:x:|Python|:heavy_check_mark:|[`git`](https://github.com/J535D165/recordlinkage)|2015, inspired by *FEBRL*|\n| SERIMI|:heavy_check_mark:|:heavy_check_mark:|---||---|---|Ruby|---|[`git`](https://github.com/samuraraujo/SERIMI-RDF-Interlinking)|[(2015, S Araujo)](https://ieeexplore.ieee.org/document/6940278)|\n| SERF|---|R-swoosh|---||CMD|:x:|Java|:x:|[`git`](https://github.com/trevorprater/serf)|[(2009, O. Benjelloun)](http://infolab.stanford.edu/serf/swoosh_vldbj.pdf)|\n| Splink |:heavy_check_mark:|EM, etc?|:heavy_check_mark:| |API, GUI|Apache Spark|Python|:heavy_check_mark:|[`git`](https://github.com/moj-analytical-services/splink)|2019, same as *fastLink*|\n| Silk|---|RB|---||GUI|Hadoop|Scala|:heavy_check_mark:|[`git`](https://github.com/silk-framework/silk), [`install`](http://silkframework.org/download)|[(2009, J. Volz)](http://events.linkeddata.org/ldow2009/papers/ldow2009_paper13.pdf)|\n| TAILOR|AE, SN|P, C, H, I|---||GUI|:x:|Java|:x:|---|[(2002, M. G. Elfeky)](https://www.cs.purdue.edu/homes/ake/pub/TAILOR_ICDE2002.pdf)|\n|WHIRL|---|---|---||CMD|:x:|C++|:x:|[`install`](https://www.cs.cmu.edu/~wcohen/whirl/)|[(2000, W.W Cohen)](https://www.sciencedirect.com/science/article/pii/S0004370299001022)|\n\n## Datasets\n- University of Leipzig: [Benchmark datasets for entity resolution](https://dbs.uni-leipzig.de/research/projects/object_matching/benchmark_datasets_for_entity_resolution)\n- [Restaurant](http://oaei.ontologymatching.org/2010/im/)\n- [Rexa-DBLP](http://oaei.ontologymatching.org/2009/instances/)\n- [BBCmusic-DBpedia](https://old.datahub.io/dataset/bbc-music)\n- [YAGO-IMDb](https://yago-knowledge.org/)\n\n## :pushpin: Miscellaneous\n- List of [blog posts](https://www.robinlinacre.com/probabilistic_linkage/) on \"Probabilistic Record Linkage\" by Robin Linacre (Lead developer of Splink).\n- TWD series\n- GiT: [Data Matching software](https://github.com/J535D165/data-matching-software)\n- Documentation: [FAst Multi-source Entity Resolution system (FAMER)](https://dbs.uni-leipzig.de/research/projects/object_matching/famer)\n- *SERF* - Standford Entity Resolution Framework: [Homepage](http://infolab.stanford.edu/serf/)\n- *Silk* - related [publications](http://silkframework.org/publications).\n- *Magellan* - rlated [material](https://sites.google.com/site/anhaidgroup/current-projects/magellan).\n","projects_url":"https://awesome.ecosyste.ms/api/v1/lists/evnsn%2Fawsome-entity-resolution/projects"}