{"id":15009295,"url":"https://github.com/gaglia88/sparker","last_synced_at":"2025-10-20T07:31:44.839Z","repository":{"id":17526080,"uuid":"81602148","full_name":"Gaglia88/sparker","owner":"Gaglia88","description":"SparkER: an Entity Resolution framework for Apache Spark","archived":false,"fork":false,"pushed_at":"2024-03-29T13:56:52.000Z","size":50766,"stargazers_count":63,"open_issues_count":2,"forks_count":18,"subscribers_count":6,"default_branch":"master","last_synced_at":"2025-03-23T19:38:23.093Z","etag":null,"topics":["apache","apache-spark","entity","entity-resolution","meta-blocking","python","python-library","python27","python3","resolution","scala","spark"],"latest_commit_sha":null,"homepage":"","language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Gaglia88.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2017-02-10T20:03:50.000Z","updated_at":"2024-12-17T14:13:17.000Z","dependencies_parsed_at":"2024-03-29T14:56:29.727Z","dependency_job_id":null,"html_url":"https://github.com/Gaglia88/sparker","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Gaglia88%2Fsparker","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Gaglia88%2Fsparker/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Gaglia88%2Fsparker/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Gaglia88%2Fsparker/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Gaglia88","download_url":"https://codeload.github.com/Gaglia88/sparker/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248075752,"owners_count":21043642,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apache","apache-spark","entity","entity-resolution","meta-blocking","python","python-library","python27","python3","resolution","scala","spark"],"created_at":"2024-09-24T19:24:19.034Z","updated_at":"2025-10-20T07:31:39.805Z","avatar_url":"https://github.com/Gaglia88.png","language":"Scala","funding_links":[],"categories":[],"sub_categories":[],"readme":"# SparkER\n_An Entity Resolution framework developed in Scala and Python for Apache Spark._\n\n**Please note that the Scala version is not maintained anymore, only the Python one is kept updated.**\n\n---\n\nIf use this library, please cite:\n\n```\n@inproceedings{sparker,\n  author    = {Luca Gagliardelli and\n               Giovanni Simonini and\n               Domenico Beneventano and\n               Sonia Bergamaschi},\n  title     = {SparkER: Scaling Entity Resolution in Spark},\n  booktitle = {Advances in Database Technology - 22nd International Conference on\n               Extending Database Technology, {EDBT} 2019, Lisbon, Portugal, March\n               26-29, 2019},\n  pages     = {602--605},\n  publisher = {OpenProceedings.org},\n  year      = {2019},\n  doi       = {10.5441/002/edbt.2019.66}\n}\n```\n\n---\n\n### News\n* **2022-05-18**: we added the Generalized Supervised meta-blocking described in our new paper [6]. [Here](https://github.com/Gaglia88/sparker/blob/master/python/examples/Generalized%20Supervised%20Meta-blocking.ipynb) there is an example of usage.\n\n### Entity Resolution\nEntity Resolution (*ER*) is the task of identifying different records (a.k.a. entity *profiles*) that pertain to the same real-world entity. Comparing all the possible pairs of records in a data set may be very inefficient (quadratic complexity), in particular in the context of Big Data, e.g., when the records to compare are hundreds of millions. To reduce this complexity, usually ER uses different blocking techniques (e.g. token blocking, n-grams, etc.) to create clusters of profiles (called blocks). The goal of this process is to reduce the global number of comparisons, because will be compared only the records that are in the same blocks.\n\nUnfortunately, in the Big Data context the blocking techniques still produces too many comparisons to be managed in a reasonable time, to reduce more the number of comparison the meta-blocking techniques was introduced [2]. The idea is to create a graph using the information learned from the blocks: the profiles in the blocks represents the nodes of the graph, and the comparisons between them represents the edges. Then is possible to calculate some metrics on the graph and use them to pruning the less significant edges.\n\n### Meta-Blocking for Spark\nSparkER implements for Spark the Meta-Blocking techniques described in Simonini et al. [1], Papadakis et al. [2], Gagliardelli et al. [6].\n\n[![stages](https://github.com/Gaglia88/sparker/raw/master/img/stages.png)](#stages)\n\nThe process is composed by different stages\n1. ***Profile loading***: loads the data (supports csv, json and serialized formats) into entity profiles;\n2. ***Blocking***: performs the blocking, token blocking or Loose Schema Blocking [1];\n4. ***Block purging***: removes the biggest blocks that are, usually, stopwords or very common tokens that do not provide significant relations [4];\n5. ***Block filtering***: for each entity profile, filters out the biggest blocks [3];\n6. ***Meta-blocking***: performs the meta-blocking, producing as results the list of candidates pairs that could be matches.\n\n### Datasets\nTo test SparkER we provide a set of datasets that can be downloaded [here](https://sourceforge.net/projects/sparker/files/). It is also possible to use the [datasets](https://sourceforge.net/projects/erframework/files/) proposed in [2].\n\n### Contacts\nFor any questions about SparkER write us at name.surname@unimore.it\n* Luca Gagliardelli\n* Giovanni Simonini\n\n### References\n[1] Simonini, G., Bergamaschi, S., \u0026 Jagadish, H. V. (2016). BLAST: a Loosely Schema-aware Meta-blocking Approach for Entity Resolution. PVLDB, 9(12), 1173–1184. [link](http://www.vldb.org/pvldb/vol9/p1173-simonini.pdf)\n\n[2] Papadakis, G., Koutrika, G., Palpanas, T., \u0026 Nejdl, W. (2014). Meta-blocking: Taking entity resolution to the next level. IEEE TKDE.\n\n[3] Papadakis, G., Papastefanatos, G., Palpanas, T., Koubarakis, M., \u0026 Green, E. L. (2016). Scaling Entity Resolution to Large , Heterogeneous Data with Enhanced Meta-blocking, 221–232. IEEE TKDE.\n\n[4] Papadakis, G., Ioannou, E., Niederée, C., \u0026 Fankhauser, P. (2011). Efficient entity resolution for large heterogeneous information spaces. Proceedings of the Fourth ACM International Conference on Web Search and Data Mining - WSDM ’11, 535.\n\n[5] Gagliardelli, L., Zhu, S., Simonini, G., \u0026 Bergamaschi, S. (2018). Bigdedup: a Big Data integration toolkit for duplicate detection in industrial scenarios. In 25th International Conference on Transdisciplinary Engineering (TE2018) (Vol. 7, pp. 1015-1023). [link](https://iris.unimore.it/retrieve/handle/11380/1165040/201434/ATDE7-1015.pdf)\n\n[6] Gagliardelli, L., Papadakis, G., Simonini, G., Bergamaschi, S., \u0026 Palpanas, T. (2022). Generalized Supervised Meta-Blocking. In PVLDB. [link](https://arxiv.org/abs/2204.08801)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgaglia88%2Fsparker","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgaglia88%2Fsparker","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgaglia88%2Fsparker/lists"}