{"id":13552771,"url":"https://github.com/opensanctions/nomenklatura","last_synced_at":"2026-03-17T16:03:55.791Z","repository":{"id":3961277,"uuid":"5055080","full_name":"opensanctions/nomenklatura","owner":"opensanctions","description":"Framework and command-line tools for integrating FollowTheMoney data streams from multiple sources","archived":false,"fork":false,"pushed_at":"2026-03-12T11:35:21.000Z","size":6511,"stargazers_count":234,"open_issues_count":11,"forks_count":42,"subscribers_count":18,"default_branch":"main","last_synced_at":"2026-03-12T17:51:57.270Z","etag":null,"topics":["data-integration","deduplication","record-link"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/opensanctions.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null},"funding":{"custom":"https://www.opensanctions.org/licensing/"}},"created_at":"2012-07-15T09:53:28.000Z","updated_at":"2026-03-12T11:35:24.000Z","dependencies_parsed_at":"2023-09-22T19:40:48.048Z","dependency_job_id":"4e91e9b7-3212-4f7e-aa5a-e1621017a59d","html_url":"https://github.com/opensanctions/nomenklatura","commit_stats":{"total_commits":603,"total_committers":8,"mean_commits":75.375,"dds":0.03814262023217252,"last_synced_commit":"c651754e2fab3aca4f5b49a124da977a64f1e9a4"},"previous_names":["pudo-attic/nomenklatura","pudo/nomenklatura"],"tags_count":210,"template":false,"template_full_name":null,"purl":"pkg:github/opensanctions/nomenklatura","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/opensanctions%2Fnomenklatura","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/opensanctions%2Fnomenklatura/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/opensanctions%2Fnomenklatura/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/opensanctions%2Fnomenklatura/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/opensanctions","download_url":"https://codeload.github.com/opensanctions/nomenklatura/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/opensanctions%2Fnomenklatura/sbom","scorecard":{"id":710025,"data":{"date":"2025-08-11","repo":{"name":"github.com/opensanctions/nomenklatura","commit":"45c1dccc8bfdfa9845aa0ce49543992cb3af6a72"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":6.2,"checks":[{"name":"Code-Review","score":0,"reason":"Found 0/13 approved changesets -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Maintained","score":10,"reason":"30 commit(s) and 3 issue activity found in the last 90 days -- score normalized to 10","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Dangerous-Workflow","score":10,"reason":"no dangerous workflow patterns detected","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Token-Permissions","score":10,"reason":"GitHub workflow tokens follow principle of least privilege","details":["Info: no jobLevel write permissions found"],"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Pinned-Dependencies","score":0,"reason":"dependency not pinned by hash detected -- score normalized to 0","details":["Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/build.yml:38: update your workflow using https://app.stepsecurity.io/secureworkflow/opensanctions/nomenklatura/build.yml/main?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/build.yml:40: update your workflow using https://app.stepsecurity.io/secureworkflow/opensanctions/nomenklatura/build.yml/main?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/build.yml:63: update your workflow using https://app.stepsecurity.io/secureworkflow/opensanctions/nomenklatura/build.yml/main?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/build.yml:65: update your workflow using https://app.stepsecurity.io/secureworkflow/opensanctions/nomenklatura/build.yml/main?enable=pin","Warn: third-party GitHubAction not pinned by hash: .github/workflows/build.yml:80: update your workflow using https://app.stepsecurity.io/secureworkflow/opensanctions/nomenklatura/build.yml/main?enable=pin","Warn: pipCommand not pinned by hash: .github/workflows/build.yml:48","Warn: pipCommand not pinned by hash: .github/workflows/build.yml:49","Warn: pipCommand not pinned by hash: .github/workflows/build.yml:73","Warn: pipCommand not pinned by hash: .github/workflows/build.yml:74","Info:   0 out of   4 GitHub-owned GitHubAction dependencies pinned","Info:   0 out of   1 third-party GitHubAction dependencies pinned","Info:   0 out of   4 pipCommand dependencies pinned"],"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"Vulnerabilities","score":10,"reason":"0 existing vulnerabilities detected","details":null,"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE:0","Info: FSF or OSI recognized license: MIT License: LICENSE:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Branch-Protection","score":0,"reason":"branch protection not enabled on development/release branches","details":["Warn: branch protection not enabled for branch 'main'"],"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}},{"name":"Packaging","score":10,"reason":"packaging workflow detected","details":["Info: Project packages its releases by way of GitHub Actions.: .github/workflows/build.yml:59"],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"SAST","score":10,"reason":"SAST tool is run on all commits","details":["Info: all commits (19) are checked with a SAST tool"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}}]},"last_synced_at":"2025-08-22T07:52:06.950Z","repository_id":3961277,"created_at":"2025-08-22T07:52:06.950Z","updated_at":"2025-08-22T07:52:06.950Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30626906,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-17T14:16:03.965Z","status":"ssl_error","status_checked_at":"2026-03-17T14:16:03.380Z","response_time":56,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-integration","deduplication","record-link"],"created_at":"2024-08-01T12:02:09.712Z","updated_at":"2026-03-17T16:03:55.738Z","avatar_url":"https://github.com/opensanctions.png","language":"Python","funding_links":["https://www.opensanctions.org/licensing/"],"categories":["HTML","others","Anti-Money Laundering and Compliance"],"sub_categories":[],"readme":"# nomenklatura\n\nNomenklatura de-duplicates and integrates different [Follow the Money](https://followthemoney.tech/) entities. It serves to clean up messy data and to find links between different datasets.\n\n![screenshot](./docs/screenshot.png)\n\n## Usage\n\nYou can install `nomenklatura` via PyPI:\n\n```bash\n$ pip install nomenklatura\n```\n\n### Command-line usage\n\nMuch of the functionality of `nomenklatura` can be used as a command-line tool. In the following example, we'll assume that you have a file containing [Follow the Money](https://followthemoney.tech/) entities in your local directory, named `entities.ijson`. If you just want try it out, you can use the file `tests/fixtures/donations.ijson` in this repository for testing (it contains German campaign finance data).\n\nWith the file in place, you will cross-reference the entities to generate de-duplication candidates, then run the interactive de-duplication UI in your console, and eventually apply the judgements to generate a new file with merged entities:\n\n```bash\n# generate merge candidates using an in-memory index:\n$ nomenklatura xref entities.ijson\n# note there is now a sqlite database, `nomenklatura.db` that contains de-duplication info.\n$ nomenklatura dedupe entities.ijson\n# will pop up a user interface.\n$ nomenklatura apply entities.ijson -o merged.ijson\n# de-duplicated data goes into `merged.ijson`:\n$ cat entities.ijson | wc -l \n474\n$ cat merged.ijson | wc -l \n468 \n```\n\nThe resolver graph database location can be customised by setting the environment variable `NOMENKLATURA_DB_URL`\n\n### Programmatic usage\n\nThe command-line use of `nomenklatura` is targeted at small datasets which need to be de-duplicated. For more involved scenarios, the package also offers a Python API which can be used to control the semantics of de-duplication.\n\n* `nomenklatura.Dataset` - implements a basic dataset for describing a set of entities.\n* `nomenklatura.Store` - a general purpose access mechanism for entities. By default, a store is used to access entity data stored in files as an in-memory cache, but the store can be subclassed to work with entities from a database system.\n* `nomenklatura.blocker.Index` - a cross-reference blocker for correlating entities inside of a dataset, or across different datasets.\n* `nomenklatura.Resolver` - the core of the de-duplication process, the resolver is essentially a graph with edges made out of entity judgements. The resolver can be used to store judgements or get the canonical ID for a given entity.\n\nAll of the API classes have extensive type annotations, which should make their integration in any modern Python API simpler.\n\n## Design\n\nThis package offers an implementation of a data deduplication framework centered around the FtM data model. The idea is the following workflow:\n\n* Accept FtM-shaped entities from a given source (e.g. a JSON file, or a database)\n* Build an inverted index of the entities for dedupe blocking\n* Generate merge candidates using the blocking index and FtM compare\n* Provide a SQL persistence abstraction for merge challenges and decisions\n* Provide a text-based user interface to let users make merge decisions\n* Export consolidated entities that cluster source entity data\n\nThe [Enrichment](nomenklatura/enrich/README.md) framework enables linking entities to records in other data sources, and enriching them with information from those records.\n\nLater on, the following might be added:\n\n* A web application to let users make merge decisions on the web\n\n### Resolver graph\n\nThe key implementation detail of nomenklatura is the `Resolver`, a graph structure that manages user decisions regarding entity identity. Edges are `Judgements` of whether two entity IDs are the same, not the same, or undecided. The resolver implements an algorithm for computing connected components, which can the be used to find the best available ID for a cluster of entities. It can also be used to evaluate transitive judgements, e.g. if A \u003c\u003e B, and B = C, then we don't need to ask if A = C.\n\n## Reading\n\n* https://dedupe.readthedocs.org/en/latest/\n* https://github.com/OpenRefine/OpenRefine/wiki/Reconcilable-Data-Sources\n* https://github.com/OpenRefine/OpenRefine/wiki/Clustering-In-Depth\n* https://github.com/OpenRefine/OpenRefine/wiki/Reconciliation-Service-API\n\n\n## Contact, contributions etc.\n\nThis codebase is licensed under the terms of an MIT license (see LICENSE).\n\nWe're keen for any contributions, bug fixes and feature suggestions, please use the GitHub issue tracker for this repository. \n\nNomenklatura is currently developed thanks to a Prototypefund grant for [OpenSanctions](https://opensanctions.org). Previous iterations of the package were developed with support from [Knight-Mozilla OpenNews](http://opennews.org) and the [Open Knowledge Foundation Labs](http://okfnlabs.org).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopensanctions%2Fnomenklatura","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fopensanctions%2Fnomenklatura","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopensanctions%2Fnomenklatura/lists"}