{"id":13582790,"url":"https://github.com/codeforkjeff/conciliator","last_synced_at":"2026-01-31T19:37:24.907Z","repository":{"id":44465832,"uuid":"71844060","full_name":"codeforkjeff/conciliator","owner":"codeforkjeff","description":"OpenRefine reconciliation services for VIAF, ORCID, and Open Library + framework for creating more.","archived":false,"fork":false,"pushed_at":"2025-06-18T02:32:15.000Z","size":958,"stargazers_count":120,"open_issues_count":8,"forks_count":24,"subscribers_count":6,"default_branch":"main","last_synced_at":"2025-08-01T01:07:21.251Z","etag":null,"topics":["entity-resolution","openlibrary","openrefine","orcid","reconciliation-service","solr","viaf"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/codeforkjeff.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-10-25T00:29:56.000Z","updated_at":"2025-07-03T10:59:49.000Z","dependencies_parsed_at":"2024-11-05T23:32:35.958Z","dependency_job_id":"f288f77a-9055-4ef6-9c68-04fcac1aad1f","html_url":"https://github.com/codeforkjeff/conciliator","commit_stats":null,"previous_names":[],"tags_count":23,"template":false,"template_full_name":null,"purl":"pkg:github/codeforkjeff/conciliator","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/codeforkjeff%2Fconciliator","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/codeforkjeff%2Fconciliator/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/codeforkjeff%2Fconciliator/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/codeforkjeff%2Fconciliator/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/codeforkjeff","download_url":"https://codeload.github.com/codeforkjeff/conciliator/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/codeforkjeff%2Fconciliator/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28951709,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-31T18:30:42.805Z","status":"ssl_error","status_checked_at":"2026-01-31T18:30:19.593Z","response_time":128,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["entity-resolution","openlibrary","openrefine","orcid","reconciliation-service","solr","viaf"],"created_at":"2024-08-01T15:03:01.764Z","updated_at":"2026-01-31T19:37:24.883Z","avatar_url":"https://github.com/codeforkjeff.png","language":"Java","funding_links":[],"categories":["Java","数据科学"],"sub_categories":["语音合成"],"readme":"\n# conciliator\n\nconciliator is a growing collection of\n[OpenRefine](http://openrefine.org) reconciliation services, as well\nas a Java framework for creating them. A reconciliation service tries\nto match variant text (usually names of things) to standard IDs for\nthe entity represented by that text.\n\nThis project supercedes [refine_viaf](https://github.com/codeforkjeff/refine_viaf).\n\n\u003c!-- markdown-toc start - Don't edit this section. Run M-x markdown-toc-refresh-toc --\u003e\n\n**Table of Contents**\n\n- [Public Server](#public-server)\n- [General Features](#general-features)\n- [Data Source Features](#data-source-features)\n    - [VIAF](#viaf)\n    - [ORCID](#orcid)\n    - [Open Library](#open-library)\n    - [Solr](#solr)\n- [Running Conciliator on Your Own Computer](#running-conciliator-on-your-own-computer)\n- [Configuring OpenRefine](#configuring-openrefine)\n- [Creating Your Own Data Source](#creating-your-own-data-source)\n- [Advanced Usage](#advanced-usage)\n- [TODO](#todo)\n- [Resources](#resources)\n- [Do you use this thing??](#do-you-use-this-thing)\n- [License](#license)\n\n\u003c!-- markdown-toc end --\u003e\n\n## Public Server\n\nIf your needs are low and you can't or don't want to run this software\nyourself, you can use the public server at\n\u003chttp://refine.codefork.com/\u003e. Visit that address for more\ninstructions.\n\n## General Features\n\n* Out of the box support for the following data sources:\n\n  - [VIAF](http://viaf.org) - Virtual International Authority File\n  - [ORCID](http://orcid.org) - digital identifiers for researchers\n  - [Open Library](http://openlibrary.org) - an open, editable library catalog\n  - Any [Apache Solr](https://lucene.apache.org/solr/) collection\n  - more to come (if you can contribute, please submit pull requests!)\n\n* Good performance (uses threads; stable memory usage; caches results)\n\n* Super easy to run (works on Linux, Mac, Windows)\n\n## Data Source Features\n\n### VIAF\n\n* Support for the following types of names provided by VIAF: Corporate\n  Names, Geographic Names, Personal Names, Works, Expressions\n\n* \"Proxy mode\" to retrieve IDs used by source institutions, instead of\n  VIAF IDs. (NOTE: hyperlinks to source record pages in OpenRefine are\n  supported for BNE, DNB, ICCU, JPG, LC, NDL, SELIBR, SUDOC, and\n  WKP. Links are BROKEN for BNC, BNF, DBC, and NUKAT. For all other\n  sources, the links will take you to the VIAF page.)\n\n### ORCID\n\n* Uses the ORCID v2.1 API. The detailed search results of the v1.2 API\n  are no longer supported, so n+1 requests are made to fetch name\n  details, which is gross, but it's the best we can do. Heavy use may\n  cause the ORCID API to start returning rate-limiting errors.\n\n* Properties are supported as a way to do fielded searches using Solr syntax.\n  For lists of valid field names to use in the \"As Property\" box, see the section titled\n  \"Search for specific elements by field\" on [this page](https://members.orcid.org/api/tutorial/search-orcid-registry),\n  and the list of identifier fields on the\n  [Supported Work Identifiers](https://members.orcid.org/api/resources/supported-work-identifiers) page.\n\n  For example, if you have a column containing Scopus EIDs, you can select the \"Include?\" checkbox\n  for it and enter \"eid\" in the \"As Property\" box on the reconciliation screen.\n\n* By default, queries are keyword searches on the entire ORCID bios,\n  which can return odd results sometimes. The \"smartnames\" mode (see\n  the instructions below) splits up names and searches on the\n  given-names and family-name fields specifically; if there are no\n  results, it falls back to a keyword search.\n\n### Open Library\n\n* Open Library has rate limits on its API, so requests are not run in a\n  threadpool. Expect it to be slow.\n\n* Support for including additional columns (useful for specifying\n  author(s), for example, to help narrow down searches for common book\n  titles). If no results are found, the code tries again with only the\n  original column.\n\n### Solr\n\n* Any Apache Solr collection can be used as a data source. See the\n  sample commented-out lines in the `conciliator.properties` file for\n  more details.\n\n## Running Conciliator on Your Own Computer\n\nUsing Docker is the easiest and preferred way to build and run the application:\n\n```\ndocker build -t conciliator .\n./run_docker.sh\n```\n\nAn alternative way to run conciliator using docker is available\n[here](https://hub.docker.com/r/tobinski/docker-codefork-conciliator/).\n\nIf you don't have Docker, you can run the application as follows:\n\nInstall Java 11 if you don't already have it.\n\nDownload the .jar file for the\n[latest release](https://github.com/codeforkjeff/conciliator/releases). Alternatively,\nyou can download the source code tarball or clone this repository, and\nbuild the .jar file using maven.\n\nRun this command:\n\n```\n# replace VERSION with the release you downloaded\njava -jar conciliator-VERSION.jar\n```\n\nThat's it! You should see some messages as the application starts\nup. Now you're ready to configure OpenRefine to use the service. When\nyou're done with it, hit Ctrl-C to quit the application.\n\nIf a file named `conciliator.properties` exists in the current\ndirectory, conciliator will use the options found in it. See the\nsample file in this repository.\n\nBy default, conciliator will run on port 8080, which is used in the\nexample URLs below. To use a different port, set the `server.port`\nproperty as follows when running the program:\n\n```\njava -Dserver.port=7000 -jar conciliator-VERSION.jar\n```\n\n## Configuring OpenRefine\n\n1. In OpenRefine, chose a column of names you want to reconcile, and\n   select \"Reconcile\" and \"Start Reconciling...\" in the column\n   pull-down menu.\n\n2. Click \"Add Standard Service...\"\n\n3. Enter a URL based on the data source you wish to use. \n\n    To reconcile against names from any VIAF source, type in:\n\n    ```\n    http://localhost:8080/reconcile/viaf\n    ```\n\n    To reconcile against a specific VIAF source, append its code to\n    the end of the path. For example, to search only names from the\n    Bibliothèque nationale de France, type in:\n    \n    ```\n    http://localhost:8080/reconcile/viaf/BNF\n    ```\n\n    To retrieve the IDs used by source institutions, rather than VIAF\n    IDs, use \"proxy mode.\" For example, to search only names from the\n    Library of Congress and retrieve their IDs, type in:\n    \n    ```\n    http://localhost:8080/reconcile/viafproxy/LC\n    ```\n\n    To use ORCID:\n\n    ```\n    http://localhost:8080/reconcile/orcid\n    ```\n\n    To use ORCID with \"smartnames\" mode when reconciliing names:\n\n    ```\n    http://localhost:8080/reconcile/orcid/smartnames\n    ```\n\n    To use Open Library: (On the reconciliation screen, under the\n    \"Also use relevant details from other columns\" panel, you can\n    check the \"Include?\" box for columns to include in the query. Give\n    them any name in the \"As Property\" box. If no results are found\n    with these column values added to the query, the service will try\n    again with only the original selected column.)\n\n    ```\n    http://localhost:8080/reconcile/openlibrary\n    ```\n\n4. Follow the instructions on the dialog box to start reconciling\n   names.\n\n## Creating Your Own Data Source\n\n1. Clone this repository to get the source code. The code you create\n   in the next steps should live under a new\n   `com.codefork.refine.NEW_SOURCE` package so that Spring's\n   auto-scanning picks it up.\n\n2. Create a class for your data source that extends `DataSource` for\n   very bare-bones functionality, or `WebServiceDataSource` if you are\n   making requests to another web service. See the other data sources\n   for some template code. Implement the abstract methods as required.\n   \n3. Create a controller that autowires your new DataSource and hooks up\n   a unique path, e.g. `/reconcile/new_source`. See VIAFController for\n   an example.\n\n4. Write a test or two if you like.\n\n5. Set some default properties in `Config` if your data source has any\n   settings you want to be configurable.\n\n6. Build a new .jar by running `mvn clean package`. Run the .jar file\n   as in the instructions above, and you should be able to access the\n   service for your new data source at:\n\n   ```\n   http://localhost:8080/reconcile/new_source\n   ```\n\n## Advanced Usage\n\nTo build from the source code, install maven and type:\n\n```\nmvn package\n```\n\nIf you want to host this software on a server for long-term usage or\nif you want to enable logging for debugging purposes, take a look at\n`run.sh` for some helpful options.\n\nYou can change run-time options by editing the\n`conciliator.properties` file.\n\nTo see usage/error statistics for the service, go to `http://localhost:8080/stats`  \n\n## TODO\n\n- A few aspects of the Reconciliation Service API aren't implemented by\n  this framework yet.\n\n## Resources\n\nSpecification for the Reconciliation Service API:\n\nhttps://reconciliation-api.github.io/specs/latest/\n\nThis code drew inspiration from these other projects:\n\n* https://github.com/rdmpage/phyloinformatics\n* https://github.com/mikejs/reconcile-demo/\n\n## Do you use this thing??\n\nApparently, you do. Here's a bibliography of things that reference conciliator:\n\nhttps://github.com/codeforkjeff/conciliator/wiki\n\nIf you use conciliator, please take a few seconds to leave a comment\non\n[this page](http://codefork.com/blog/index.php/2016/10/24/announcing-conciliator/). Hearing\nfrom users really motivates me to continue improving this project.\n\n## License\n\nThis code is distributed under a GNU General Public License. See the\nfile LICENSE for details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcodeforkjeff%2Fconciliator","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcodeforkjeff%2Fconciliator","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcodeforkjeff%2Fconciliator/lists"}