https://github.com/nickcrews/mismo

The SQL/Ibis powered sklearn of record linkage
https://github.com/nickcrews/mismo

deduplication duckdb entity-resolution ibis python record-linkage sql

Last synced: about 5 hours ago
JSON representation

The SQL/Ibis powered sklearn of record linkage

• Host: GitHub
• URL: https://github.com/nickcrews/mismo
• Owner: NickCrews
• License: lgpl-3.0
• Created: 2022-06-10T07:12:41.000Z (over 2 years ago)
• Default Branch: main
• Last Pushed: 2024-10-29T22:52:54.000Z (19 days ago)
• Last Synced: 2024-10-30T00:47:36.006Z (19 days ago)
• Topics: deduplication, duckdb, entity-resolution, ibis, python, record-linkage, sql
• Language: Python
• Homepage: https://nickcrews.github.io/mismo/
• Size: 3.67 MB
• Stars: 14
• Watchers: 1
• Forks: 3
• Open Issues: 29
• Metadata Files:
  • Readme: README.md
  • Contributing: docs/contributing.md
  • License: LICENSE.txt
  • Code of conduct: CODE_OF_CONDUCT.md

Awesome Lists containing this project

README

        
# Mismo

[![PyPI - Version](https://img.shields.io/pypi/v/mismo.svg)](https://pypi.org/project/mismo)

[![PyPI - Python Version](https://img.shields.io/pypi/pyversions/mismo.svg)](https://pypi.org/project/mismo)

The SQL/Ibis powered sklearn of record linkage.

Still in alpha stage. Breaking changes will happen frequently

and with no warning. Once things are more stabilized I

will come up with a stability policy. Any suggestions as

to how you want the API to look like would be greatly appreciated.

-----

## Installation

I have claimed `mismo` on PyPI, but I won't update it often

until this is more stable. Until then, install from source:

```console

python -m pip install "mismo[viz] @ git+https://github.com/NickCrews/mismo@"

```

## Goals

Mismo tries to be the sklearn of record linkage, backed by the scalability

and power of SQL and Ibis. It is made of many small

data structures and functions, each with a well-defined and standard API

that allows them to be composed together and extended easily.

None of the other record linkage packages I have seen, such as

[Splink](https://github.com/moj-analytical-services/splink),

[Dedupe](https://www.github.com/dedupeio/dedupe), or

[Record Linkage Toolkit](https://github.com/J535D165/recordlinkage),

had all of these properties, so I decided to make my own.

See [Goals and Alternatives](https://nickcrews.github.io/mismo/concepts/goals_and_alternatives)

for a more detailed discussion of the goals of Mismo and how it compares to other

record linkage packages.

## Features

- Supports larger-than-memory datasets, executed on powerful SQL engines.

  Use DuckDB for prototyping and for jobs up to maybe ~10M records,

  or Spark or other distributed backends for larger tasks, without

  needing to change your code!

- Use the clean, strong-typed, pythonic, and Dataframe API of Ibis.

- Small, modular functions and data structures that are easy to plug together

  and extend.

- Layered API: Use top-level APIs if your task is common enough that it is

  supported out of the box.

## Examples

See the [example notebook](https://nickcrews.github.io/mismo/examples/patent_deduplication).

## Documentation

See the [documentation](https://nickcrews.github.io/mismo).

## Contributing

See the [contributing guide](https://nickcrews.github.io/mismo/contributing/).

## License

`mismo` is distributed under the terms of the

[LGPL-3.0-or-later](https://spdx.org/licenses/LGPL-3.0-or-later.html) license.