{"id":18866821,"url":"https://github.com/ing-bank/spark-matcher","last_synced_at":"2025-06-23T00:02:41.934Z","repository":{"id":38402902,"uuid":"480380475","full_name":"ing-bank/spark-matcher","owner":"ing-bank","description":"Record matching and entity resolution at scale in Spark","archived":false,"fork":false,"pushed_at":"2023-10-31T09:57:38.000Z","size":593,"stargazers_count":34,"open_issues_count":5,"forks_count":8,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-04-14T14:58:55.936Z","etag":null,"topics":["deduplication","entity-resolution","record-linkage","spark"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ing-bank.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2022-04-11T12:57:19.000Z","updated_at":"2025-02-12T18:41:35.000Z","dependencies_parsed_at":"2023-02-14T07:40:51.084Z","dependency_job_id":"e838286c-ae98-4f5c-a231-154f6d4d4a4f","html_url":"https://github.com/ing-bank/spark-matcher","commit_stats":null,"previous_names":[],"tags_count":11,"template":false,"template_full_name":null,"purl":"pkg:github/ing-bank/spark-matcher","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ing-bank%2Fspark-matcher","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ing-bank%2Fspark-matcher/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ing-bank%2Fspark-matcher/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ing-bank%2Fspark-matcher/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ing-bank","download_url":"https://codeload.github.com/ing-bank/spark-matcher/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ing-bank%2Fspark-matcher/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":261386701,"owners_count":23150865,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deduplication","entity-resolution","record-linkage","spark"],"created_at":"2024-11-08T05:07:40.190Z","updated_at":"2025-06-23T00:02:36.834Z","avatar_url":"https://github.com/ing-bank.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"\u003c!--- BADGES: START ---\u003e\n[![Version](https://img.shields.io/pypi/v/spark-matcher)](https://pypi.org/project/spark-matcher/)\n[![Downloads](https://pepy.tech/badge/spark-matcher)](https://pepy.tech/project/spark-matcher)\n![](https://img.shields.io/github/license/ing-bank/spark-matcher)\n[![Docs - GitHub.io](https://img.shields.io/static/v1?logo=readthdocs\u0026style=flat\u0026color=pink\u0026label=docs\u0026message=spark-matcher)][#docs-package]\n\n[#docs-package]: https://spark-matcher.readthedocs.io/en/latest/\n\u003c!--- BADGES: END ---\u003e\n\n![spark_matcher_logo](https://spark-matcher.readthedocs.io/en/latest/_images/spark_matcher_logo.png)\n\n# Spark-Matcher\n\nSpark-Matcher is a scalable entity matching algorithm implemented in PySpark. With Spark-Matcher the user can easily\ntrain an algorithm to solve a custom matching problem. Spark Matcher uses active learning (modAL) to train a\nclassifier (Scikit-learn) to match entities. In order to deal with the N^2 complexity of matching large tables, blocking is\nimplemented to reduce the number of pairs. Since the implementation is done in PySpark, Spark-Matcher can deal with\nextremely large tables.\n\nDocumentation with examples can be found [here](https://spark-matcher.readthedocs.io/en/latest/).\n\nDeveloped by data scientists at ING Analytics, www.ing.com.\n\n## Installation\n\n### Normal installation\n\nAs Spark-Matcher is intended to be used with large datasets on a Spark cluster, it is assumed that Spark is already \ninstalled. If that is not the case, first install Spark and PyArrow (`pip install pyspark pyarrow`).\n\nInstall Spark-Matcher using PyPi:\n\n```\npip install spark-matcher\n```\n\n### Install with possibility to create documentation\n\nPandoc, the general markup converter needs to be available. You may follow the official [Pandoc installations instructions](https://pandoc.org/installing.html) or use conda:\n\n```\nconda install -c conda-forge pandoc\n```\n\nThen clone the Spark-Matcher repository and add `[doc]` like this:\n\n```\npip install \".[doc]\"\n```\n\n### Install to contribute\n\nClone this repo and install in editable mode. This also installs PySpark and Jupyterlab:\n\n```\npython -m pip install -e \".[dev]\"\npython setup.py develop\n```\n\n## Documentation\n\nDocumentation can be created using the following command:\n\n```\nmake create_documentation\n```\n\n## Dependencies\n\nThe usage examples in the `examples` directory contain notebooks that run in local mode. \nUsing the SparkMatcher in cluster mode, requires sending the SparkMatcher package and several other python packages (see spark_requirements.txt) to the executors.\nHow to send these dependencies, depends on the cluster. \nPlease read the instructions and examples of Apache Spark on how to do this: https://spark.apache.org/docs/latest/api/python/user_guide/python_packaging.html.\n\nSparkMatcher uses `graphframes` under to hood. \nTherefore, depending on the spark version, the correct version of `graphframes` needs to be added to the `external_dependencies` directory and to the configuration of the spark session.  \nAs a default, `graphframes` for spark 3.0 is used in the spark sessions in the notebooks in the `examples` directory. \nFor a different version, see: https://spark-packages.org/package/graphframes/graphframes.\n\n## Usage\n\nExample notebooks are provided in the `examples` directory.\nUsing the SparkMatcher to find matches between Spark\ndataframes `a` and `b` goes as follows:\n\n```python\nfrom spark_matcher.matcher import Matching\n\nmyMatcher = Matcher(spark_session, col_names=['name', 'suburb', 'postcode'])\n```\n\nNow we are ready for fitting the Matcher object using 'active learning'; this means that the user has to enter whether a\npair is a match or not. You enter 'y' if a pair is a match or 'n' when a pair is not a match. You will be notified when\nthe model has converged and you can stop training by pressing 'f'.\n\n```python\nmyMatcher.fit(a, b)\n```\n\nThe Matcher is now trained and can be used to predict on all data. This can be the data used for training or new data\nthat was not seen by the model yet.\n\n```python\nresult = myMatcher.predict(a, b)\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fing-bank%2Fspark-matcher","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fing-bank%2Fspark-matcher","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fing-bank%2Fspark-matcher/lists"}