{"id":20619319,"url":"https://github.com/dedupeio/dedupe-examples","last_synced_at":"2025-12-18T13:58:36.520Z","repository":{"id":15629747,"uuid":"18366570","full_name":"dedupeio/dedupe-examples","owner":"dedupeio","description":":id: Examples for using the dedupe library","archived":false,"fork":false,"pushed_at":"2024-08-10T00:15:53.000Z","size":5365,"stargazers_count":410,"open_issues_count":16,"forks_count":215,"subscribers_count":24,"default_branch":"main","last_synced_at":"2025-04-08T20:17:26.909Z","etag":null,"topics":["dedupe","entity-resolution","python","record-linkage"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dedupeio.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE.md","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2014-04-02T13:04:28.000Z","updated_at":"2025-03-11T06:04:26.000Z","dependencies_parsed_at":"2022-07-12T10:22:56.036Z","dependency_job_id":"72346aa7-3f66-4d11-8ba2-fba312f2ddb7","html_url":"https://github.com/dedupeio/dedupe-examples","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dedupeio%2Fdedupe-examples","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dedupeio%2Fdedupe-examples/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dedupeio%2Fdedupe-examples/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dedupeio%2Fdedupe-examples/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dedupeio","download_url":"https://codeload.github.com/dedupeio/dedupe-examples/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254493379,"owners_count":22080126,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dedupe","entity-resolution","python","record-linkage"],"created_at":"2024-11-16T12:11:21.358Z","updated_at":"2025-12-18T13:58:36.449Z","avatar_url":"https://github.com/dedupeio.png","language":"Python","readme":"# Dedupe Examples\n\nExample scripts for the [dedupe](https://github.com/dedupeio/dedupe), a library that uses machine learning to perform de-duplication and entity resolution quickly on structured data.\n\nPart of the [Dedupe.io](https://dedupe.io/) cloud service and open source toolset for de-duplicating and finding fuzzy matches in your data. For more details, see the [differences between Dedupe.io and the dedupe library](https://dedupe.io/documentation/should-i-use-dedupeio-or-the-dedupe-python-library.html).\n\nTo get these examples:\n```bash\ngit clone https://github.com/dedupeio/dedupe-examples.git\ncd dedupe-examples\n```\n\nor [download this repository](https://github.com/dedupeio/dedupe-examples/archive/master.zip)\n\n```bash\ncd /path/to/downloaded/file\nunzip master.zip\ncd dedupe-examples\n```\n\n### Setup\nWe recommend using [virtualenv](http://virtualenv.readthedocs.io/en/stable/) and [virtualenvwrapper](http://virtualenvwrapper.readthedocs.org/en/latest/install.html) for working in a virtualized development environment. [Read how to set up virtualenv](http://docs.python-guide.org/en/latest/dev/virtualenvs/).\n\nOnce you have virtualenvwrapper set up,\n\n```bash\nmkvirtualenv dedupe-examples\npip install -r requirements.txt\n```\n\nAfterwards, whenever you want to work on dedupe-examples,\n\n```bash\nworkon dedupe-examples\n```\n\n### [CSV example](https://dedupeio.github.io/dedupe-examples/docs/csv_example.html) - early childhood locations\n\nThis example works with a list of early childhood education sites in Chicago from 10 different sources.\n\n```bash\ncd csv_example\npip install unidecode\npython csv_example.py\n```\n  (use 'y', 'n' and 'u' keys to flag duplicates for active learning, 'f' when you are finished)\n\n**To see how you might use dedupe with smallish data, see the [annotated source code for csv_example.py](https://dedupeio.github.io/dedupe-examples/docs/csv_example.html).**\n\n### [Patent example](https://dedupeio.github.io/dedupe-examples/docs/patent_example.html) -  patent holders\n\nThis example works with Dutch inventors from the PATSTAT international patent data file\n\n```bash\ncd patent_example\npip install unidecode\npython patent_example.py\n```\n  (use 'y', 'n' and 'u' keys to flag duplicates for active learning, 'f' when you are finished)\n\n### [Record Linkage example](https://dedupeio.github.io/dedupe-examples/docs/record_linkage_example.html) -  electronics products\nThis example links two spreadsheets of electronics products and links up the matching entries. Each dataset individually has no duplicates.\n\n```bash\ncd record_linkage_example\npython record_linkage_example.py\n```\n\n**To see how you might use dedupe for linking datasets, see the [annotated source code for record_linkage_example.py](https://dedupeio.github.io/dedupe-examples/docs/record_linkage_example.html).**\n\n### [Gazetteer example](https://dedupeio.github.io/dedupe-examples/docs/gazetteer_example.html) -  electronics products\nThis example links two spreadsheets of electronics products and links up the matching entries using the Gazetteer class\n\n```bash\ncd gazetteer_example.py\npython gazetteer_example.py\n```\n\n\n### [MySQL example](https://dedupeio.github.io/dedupe-examples/docs/mysql_example.html) - IL campaign contributions\n\nSee `mysql_example/README.md` for details\n\n**To see how you might use dedupe with bigish data, see the [annotated source code for mysql_example](https://dedupeio.github.io/dedupe-examples/docs/mysql_example.html).**\n\n\n### [PostgreSQL big dedupe example](https://dedupeio.github.io/dedupe-examples/docs/pgsql_big_dedupe_example.html) - PostgreSQL example on large dataset\n\nSee `pgsql_big_dedupe_example/README.md` for details\n\nThis is the same example as the MySQL IL campaign contributions dataset above, but ported to run on PostgreSQL.\n\n\n## Training\n\nThe _secret sauce_ of dedupe is human input. In order to figure out the best rules to deduplicate a set of data, you must give it a set of labeled examples to learn from.\n\nThe more labeled examples you give it, the better the deduplication results will be. At minimum, you should try to provide __10 positive matches__ and __10 negative matches__.\n\nThe results of your training will be saved in a JSON file for future runs of dedupe.\n\nHere's an example labeling operation:\n\n```bash\nPhone :  2850617\nAddress :  3801 s. wabash\nZip :\nSite name :  ada s. mckinley st. thomas cdc\n\nPhone :  2850617\nAddress :  3801 s wabash ave\nZip :\nSite name :  ada s. mckinley community services - mckinley - st. thomas\n\nDo these records refer to the same thing?\n(y)es / (n)o / (u)nsure / (f)inished\n```\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdedupeio%2Fdedupe-examples","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdedupeio%2Fdedupe-examples","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdedupeio%2Fdedupe-examples/lists"}