{"id":15287742,"url":"https://github.com/itspawanbhardwaj/spark-fuzzy-matching","last_synced_at":"2025-07-29T03:42:37.059Z","repository":{"id":57720290,"uuid":"118232302","full_name":"itspawanbhardwaj/spark-fuzzy-matching","owner":"itspawanbhardwaj","description":"Fuzzy matching function in spark (https://spark-packages.org/package/itspawanbhardwaj/spark-fuzzy-matching)","archived":false,"fork":false,"pushed_at":"2019-12-30T06:35:31.000Z","size":95,"stargazers_count":24,"open_issues_count":0,"forks_count":11,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-05-07T03:47:19.218Z","etag":null,"topics":["algorithm","apache-spark","fuzzy-matching","levenshtein","scala","similarity-metric","soundex"],"latest_commit_sha":null,"homepage":"","language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/itspawanbhardwaj.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-01-20T10:23:41.000Z","updated_at":"2024-12-21T21:33:57.000Z","dependencies_parsed_at":"2022-08-25T15:10:15.357Z","dependency_job_id":null,"html_url":"https://github.com/itspawanbhardwaj/spark-fuzzy-matching","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/itspawanbhardwaj%2Fspark-fuzzy-matching","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/itspawanbhardwaj%2Fspark-fuzzy-matching/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/itspawanbhardwaj%2Fspark-fuzzy-matching/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/itspawanbhardwaj%2Fspark-fuzzy-matching/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/itspawanbhardwaj","download_url":"https://codeload.github.com/itspawanbhardwaj/spark-fuzzy-matching/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252810275,"owners_count":21807759,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["algorithm","apache-spark","fuzzy-matching","levenshtein","scala","similarity-metric","soundex"],"created_at":"2024-09-30T15:36:12.388Z","updated_at":"2025-05-07T03:47:25.179Z","avatar_url":"https://github.com/itspawanbhardwaj.png","language":"Scala","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n## Maven Central\n\n### For Scala 2.10\n```xml\n\u003cdependency\u003e\n  \u003cgroupId\u003ecom.github.itspawanbhardwaj\u003c/groupId\u003e\n  \u003cartifactId\u003espark-fuzzy-matching_2.10\u003c/artifactId\u003e\n  \u003cversion\u003e1.0.0\u003c/version\u003e\n\u003c/dependency\u003e\n```\n ### For Scala 2.11\n```xml\n\u003cdependency\u003e\n  \u003cgroupId\u003ecom.github.itspawanbhardwaj\u003c/groupId\u003e\n  \u003cartifactId\u003espark-fuzzy-matching_2.11\u003c/artifactId\u003e\n  \u003cversion\u003e1.0.1\u003c/version\u003e\n\u003c/dependency\u003e\n```\n\n## Metrics and algorithms\n\n* __[Dice / Sorensen](http://en.wikipedia.org/wiki/Dice%27s_coefficient)__ (Similarity metric)\n* __[Double Metaphone](http://en.wikipedia.org/wiki/Metaphone)__  phonetic metric and algorithm)\n* __[Hamming](http://en.wikipedia.org/wiki/Hamming_distance)__ (Similarity metric)\n* __[Jaccard](http://en.wikipedia.org/wiki/Jaccard_index)__ (Similarity metric)\n* __[Jaro](http://en.wikipedia.org/wiki/Jaro-Winkler_distance)__ (Similarity metric)\n* __[Jaro-Winkler](http://en.wikipedia.org/wiki/Jaro-Winkler_distance)__ (Similarity metric)\n* __[Levenshtein](http://en.wikipedia.org/wiki/Levenshtein_distance)__ (Similarity metric)\n* __[Metaphone](http://en.wikipedia.org/wiki/Metaphone)__ (Phonetic metric and algorithm)\n* __[Monge-Elkan](http://www.cs.cmu.edu/~pradeepr/papers/ijcai03.pdf)__  similarity metric)\n* __[Match Rating Approach](http://en.wikipedia.org/wiki/Match_rating_approach)__  phonetic metric and algorithm)\n* __[Needleman-Wunch](http://en.wikipedia.org/wiki/Needleman%E2%80%93Wunsch_algorithm)__  similarity metric)\n* __[N-Gram](http://en.wikipedia.org/wiki/N-gram)__ (Similarity metric)\n* __[NYSIIS](http://en.wikipedia.org/wiki/New_York_State_Identification_and_Intelligence_System)__ (Phonetic metric and algorithm)\n* __[Overlap](http://en.wikipedia.org/wiki/Overlap_coefficient)__ (Similarity metric)\n* __[Ratcliff-Obershelp](http://xlinux.nist.gov/dads/HTML/ratcliffObershelp.html)__ (Similarity metric)\n* __[Refined NYSIIS](http://www.markcrocker.com/rexxtipsntricks/rxtt28.2.0482.html)__ (Phonetic metric and algorithm)\n* __[Refined Soundex](http://ntz-develop.blogspot.com/2011/03/phonetic-algorithms.html)__ (Phonetic metric and algorithm)\n* __[Tanimoto](http://en.wikipedia.org/wiki/Tanimoto_coefficient)__  similarity metric)\n* __[Tversky](http://en.wikipedia.org/wiki/Tversky_index)__  similarity metric)\n* __[Smith-Waterman](http://en.wikipedia.org/wiki/Smith%E2%80%93Waterman_algorithm)__  similarity metric)\n* __[Soundex](http://en.wikipedia.org/wiki/Soundex)__ (Phonetic metric and algorithm)\n* __Weighted Levenshtein__ (Similarity metric)\n\n\n## Functions\n\n*  All functions are defined under `com.pb.fuzzy.matching.functions`.\n\n  import com.pb.fuzzy.matching.functions._ // import to use fuzzy matching functions\n~~~\n  \n  levenshteinFn(document, document1)\n  diceSorensenFn(document, document1, nGramSize)\n  hammingFn(document, document1)\n  jaccardFn(document, document1, nGramSize)\n  jaroFn(document, document1)\n  jaroWinklerFn(document, document1)\n  nGramFn(document, document1, nGramSize)\n  overlapFn(document, document1, nGramSize)\n  ratcliffObershelpFn(document, document1)\n  weightedLevenshteinFn(document, document1, deleteWeight, insertWeight, substituteWeight)\n  metaphoneFn(document, document1)\n  computeMetaphoneFn(document)\n  nysiisFn(document, document1)\n  computeNysiisFn(document)\n  refinedNysiisFn(document, document1)\n  computeRefinedNysiisFn(document)\n  refinedSoundexFn(document, document1)\n  computeRefinedSoundexFn(document)\n  soundexFn(document, document1)\n  computeSoundexFn(document)\n~~~\n\n\n## Example\nThe project contains a [FuzzyMatchingJoinExample](https://github.com/itspawanbhardwaj/spark-fuzzy-matching/blob/master/src/test/scala/com/pb/fuzzy/matching/FuzzyMatchingJoinExample.scala \"FuzzyMatchingJoinExample\") which works as follows:\n\n~~~\nDataset with proper names\n+--------------------+--------------------+-------+\n|               title|               gener|ratings|\n+--------------------+--------------------+-------+\n|The Shawshank Red...|        Crime. Drama|    9.3|\n|       The Godfather|        Crime. Drama|    9.2|\n|     The Dark Knight|Action. Crime. Drama|    9.0|\n|The Godfather: Pa...|        Crime. Drama|    9.0|\n|        Pulp Fiction|        Crime. Drama|    8.9|\n+--------------------+--------------------+-------+\nonly showing top 5 rows\n\nDataset with misspelled names\n+--------------------+----+--------+\n|               title|year|duration|\n+--------------------+----+--------+\n|dhe Shwshnk Redem...|1994|     142|\n|        dhe Godfdher|1972|     175|\n|      dhe Drk Knighd|2008|     152|\n|dhe Godfdher: Prd II|1974|     202|\n|        Pulp Ficdion|1994|     154|\n+--------------------+----+--------+\nonly showing top 5 rows\n\nDataset after fuzzy join\n+--------------------+--------------------+-------+--------------------+----+--------+\n|               title|               gener|ratings|               title|year|duration|\n+--------------------+--------------------+-------+--------------------+----+--------+\n|The Shawshank Red...|        Crime. Drama|    9.3|dhe Shwshnk Redem...|1994|     142|\n|       The Godfather|        Crime. Drama|    9.2|        dhe Godfdher|1972|     175|\n|     The Dark Knight|Action. Crime. Drama|    9.0|      dhe Drk Knighd|2008|     152|\n|        Pulp Fiction|        Crime. Drama|    8.9|        Pulp Ficdion|1994|     154|\n|    Schindler's List|Biography. Drama....|    8.9|    Schindler's Lisd|1993|     195|\n+--------------------+--------------------+-------+--------------------+----+--------+\nonly showing top 5 rows\n~~~\n\n## Library used\n\n__[stringmetric](https://github.com/rockymadden/stringmetric)__ ( :dart: String metrics and phonetic algorithms for Scala (e.g. Dice/Sorensen, Hamming, Jaccard, Jaro, Jaro-Winkler, Levenshtein, Metaphone, N-Gram, NYSIIS, Overlap, Ratcliff/Obershelp, Refined NYSIIS, Refined Soundex, Soundex, Weighted Levenshtein). )\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fitspawanbhardwaj%2Fspark-fuzzy-matching","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fitspawanbhardwaj%2Fspark-fuzzy-matching","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fitspawanbhardwaj%2Fspark-fuzzy-matching/lists"}