{"id":43173610,"url":"https://github.com/eliarizzetto/omid-openalex","last_synced_at":"2026-02-01T02:38:15.177Z","repository":{"id":181840799,"uuid":"645019885","full_name":"eliarizzetto/omid-openalex","owner":"eliarizzetto","description":"Map OMIDs of bibliographic resources already existing in OpenCitations Meta to their corresponding OpenAlex IDs.","archived":false,"fork":false,"pushed_at":"2024-03-14T12:11:58.000Z","size":6786,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2024-05-13T00:46:01.215Z","etag":null,"topics":["bibliographic-metadata","mapping","openalex","opencitations","opencitations-meta"],"latest_commit_sha":null,"homepage":"https://sleepy-geese-5d2.notion.site/Index-6544baacb4e44b14ac4562926070ae6f?pvs=4","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/eliarizzetto.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2023-05-24T18:40:32.000Z","updated_at":"2024-01-26T15:15:09.000Z","dependencies_parsed_at":"2023-10-11T18:26:09.293Z","dependency_job_id":"cb71a3b5-c05b-4d6f-8301-d2a9747fa710","html_url":"https://github.com/eliarizzetto/omid-openalex","commit_stats":null,"previous_names":["eliarizzetto/omid-openalex"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/eliarizzetto/omid-openalex","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eliarizzetto%2Fomid-openalex","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eliarizzetto%2Fomid-openalex/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eliarizzetto%2Fomid-openalex/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eliarizzetto%2Fomid-openalex/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/eliarizzetto","download_url":"https://codeload.github.com/eliarizzetto/omid-openalex/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eliarizzetto%2Fomid-openalex/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28965430,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-01T02:14:24.993Z","status":"ssl_error","status_checked_at":"2026-02-01T02:13:55.706Z","response_time":56,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bibliographic-metadata","mapping","openalex","opencitations","opencitations-meta"],"created_at":"2026-02-01T02:38:14.581Z","updated_at":"2026-02-01T02:38:15.170Z","avatar_url":"https://github.com/eliarizzetto.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# OMID to OpenAlex ID mapping\nThe tasks performed by this software are:\n1. to create a mapping between the bibliographic resources (BRs) already existing in OpenCitations Meta (OC Meta) and their \nequivalents in OpenAlex\n2. to analyse the output of the mapping process with *ad hoc* processes and functions, with a focus on the provenance \nof OC Meta BRs that are not mapped to any BR in OpenAlex (unmapped BRs) and on OC Meta BRs that are mapped to more than \none BR in OpenAlex (multi-mapped BRs).\n\nThe primary goal of the mapping is the addition of OpenAlex IDs to the metadata of the BRs in OC Meta; analysing the \nmapping's output, moreover, provides the chance to detect potential inconsistencies in the input collections (i.e. OC \nMeta and OpenAlex) and better understand the causes of these issues.\n\nThe present guide concerns the mapping process. For the mapping results analysis see the [README.md documentation file](./oc_alignoa/analytics/README.md) inside \"oc_alignoa/analytics\".\n\n### Launch the mapping process\nThe process can be launched from CLI with the following command, executed from inside the `omid-openalex` directory:\n```\npython -m oc_alignoa.main -c \u003cPATH\u003e\n```\nWhere:\n* `-c` `--config`: path to the configuration file.\n\nA guide on how to write the configuration file is provided [here](#configuration).\n\n## The mapping process\nThe mapping between the BRs in the two collections is performed by linking them according to the presence, in both \ncollections, of a common external persistent identifier (PID). For example, if a journal article in OC Meta has, in its \nmetadata the same DOI as a BR in OpenAlex, these entities get aligned ([Fig 1](#fig1)).\n\n\u003cfigure\u003e\n  \u003cimg src=\"./imgs/mapping.png?raw=true\" alt=\"my alt text\"/\u003e\n    \u003cfigcaption style=\"font-style: italic; font-size: 85%\"\u003e\n    \u003ci\u003e\u003ca id=\"fig1\"\u003e\u003cb\u003eFigure 1.\u003c/b\u003e\u003c/a\u003e Visual representation of the mapping between a BR in OC Meta and the corresponding BR in OpenAlex.\u003c/i\u003e\n    \u003c/figcaption\u003e\n\u003c/figure\u003e\n\n\\\nThe output of this mapping process is divided into three separate groups of BRs:\n1. BRs mapped in a 1:1 ratio, i.e. OC Meta BRs that have been mapped to exactly one BR in OpenAlex each and \ntheir corresponding BRs in OpenAlex.\n2. Multi-mapped BRs, i.e. OC Meta BRs that have been mapped to more than one BR in OpenAlex each and their corresponding\nBRs in OpenAlex.\n3. Unmapped BRs, i.e. OC Meta BRs that have not been mapped to any BR in OpenAlex.\n\n### Input \nThe mapping process takes as input the zipped CSV files of an OC Meta dump and the compressed JSON-Lines files of a \nsnapshot of the OpenAlex catalog. The OC Meta CSV dump can be downloaded from [Figshare](https://figshare.com/articles/dataset/OpenCitations_Meta_CSV_dataset_of_all_bibliographic_metadata/21747461/6).\nThe OpenAlex snapshot can be downloaded from Amazon S3 following the dedicated [guide](https://docs.openalex.org/download-all-data/download-to-your-machine) by the OpenAlex team.\n\nEach row of the OC Meta CSV tables store basic metadata about a BR: for the purpose of the mapping and the following \nanalysis, we are interested in the `id` field (storing the BR's OMID and, if present, external PIDs, such as DOIs or ISSNs)\nand the `type` field, specifying the BR type (e.g. journal article, book, journal, dataset, etc.).\n\n``` csv\n# a sample CSV row of OC Meta dump\n\"id\",\"title\",\"author\",\"issue\",\"volume\",\"venue\",\"page\",\"pub_date\",\"type\",\"publisher\",\"editor\"\n\"omid:br/060924 doi:10.4230/lipics.fun.2021.11\",\"Efficient Algorithms For Battleship\",\"Crombez, Loïc [omid:ra/062503694315 orcid:0000-0002-9542-5276]; Fonseca, Guilherme D. Da [omid:ra/0612046462 orcid:0000-0002-9807-028X]; Gerard, Yan [omid:ra/0612046189 orcid:0000-0002-2664-0650]\",\"\",\"\",\"LIPIcs : Leibniz International Proceedings In Informatics [omid:br/060182 issn:1868-8969]\",\"\",\"2020\",\"report\",\"Schloss Dagstuhl - Leibniz-Zentrum Für Informatik [omid:ra/0607497]\",\"Farach, Martin [omid:ra/0614045842 orcid:0000-0003-3616-7788]; Prencipe, Giuseppe [omid:ra/0625025102 orcid:0000-0001-5646-7388]; Uehara, Ryuhei [omid:ra/0617014675 orcid:0000-0003-0895-3765]\"\n```\n\nIn the snapshot of the OpenAlex collection, BRs and their metadata are distinguished into two entity types: Works (e.g. \njournal articles, books, reports) and Sources (i.e. where Works are contained, e.g. journals, conferences, repositories).\nEach Work or Source is represented inside the JSON-Lines files as a JSON object, of which, mapping-wise, we need to \nconsider only the `ids` field; this is itself a JSON object where keys are PID schemes (e.g. `doi`, `issn`) and values \nare the values for those schemes.\n\n``` json\n# a sample from the JSON object representing a Source entity in the OpenAlex snaphot\n{\n   \"id\":\"https://openalex.org/S168707975\",\n   \"issn_l\":\"1941-7012\",\n   \"issn\":[\n      \"1941-7012\"\n   ],\n   \"display_name\":\"Journal of Renewable and Sustainable Energy\",\n    ...\n   \"ids\":{\n      \"openalex\":\"https://openalex.org/S168707975\",\n      \"issn_l\":\"1941-7012\",\n      \"issn\":[\n         \"1941-7012\"\n      ],\n      \"mag\":\"168707975\",\n      \"wikidata\":\"https://www.wikidata.org/entity/Q6295857\",\n      \"fatcat\":\"https://fatcat.wiki/container/xgsd2zrqkzhgpmbd3ofxlhuxci\"\n   },\n    ...\n   \"type\":\"journal\",\n    ...\n}\n```\n### Process\nFirst, the input data (CSV files of the OC Meta dump and JSON-Lines files of the OpenAlex snapshot) is pre-processed to \ncreate new tables:\n- a table storing the OMID and the external PIDs of each BR in the OC Meta CSV dump, and optionally \nits type (`omid`, `ids` and `type` fields, respectively). In this step, OC Meta BRs without external PIDs are discarded, \nmeaning that they will not be included in the produced table and therefore will not be considered for the mapping step.\n- two tables, one for OpenAlex Works and the other for OpenAlex Sources, storing the OpenAlex IDs and the external PIDs \nof each Work/Source in the OpenAlex snapshot, among the ID schemes that are also supported by OC Meta. In both tables, \neach row stores an association between a single, mutually-supported external PID (`supported_id` field) and the OpenAlex ID of the \nresource it is attributed to (`openalex_id` field). In this step, OpenAlex BRs without external PIDs\nthat are also supported by OC Meta are discarded, meaning that they will not be included in the produced tables and therefore\nwill not be considered for the mapping step.\n\nThen, the two tables storing the IDs of the BRs in OpenAlex are converted into an SQLite database\nto enable faster queries on this data. Each external PID is a primary key, to which a single OpenAlex ID is linked ([Fig 2](#fig2)).\n\n\u003cfigure\u003e\n  \u003cimg src=\"./imgs/oa_ids_tables_db.png?raw=true\" alt=\"Entity relationship diagram for the database storing OpenAlex IDs and the external PIDs associated to them\"/\u003e\n    \u003cfigcaption style=\"font-style: italic; font-size: 85%\"\u003e\n    \u003ci\u003e\u003ca id=\"fig2\"\u003e\u003cb\u003eFigure 2.\u003c/b\u003e \u003c/a\u003eDiagram representing the SQLite database storing OpenAlex IDs and the external PIDs associated to them.\u003c/i\u003e\n    \u003c/figcaption\u003e\n\u003c/figure\u003e\n\n\\\nFinally, the actual mapping step takes place: for each row in the table storing OC Meta IDs (i.e. for each OC Meta\nBR with external PIDs), each external PID is looked for in the database: if it is present, the OMID \nlinked to that PID from the CSV table and the OpenAlex ID linked to it in the database are aligned. For every OC Meta BR \n_**all**_ external PIDs are looked for in the database: if all of them point to the same OpenAlex ID, \nthen the resource in OC Meta is mapped to a single resource in OpenAlex; if they point to different OpenAlex IDs, this results \nin a multi-mapped case; if none of the external PIDs is found in the database, no mapping is found for that BR.\n\n### Output\n\nThe three groups of BRs that make up the output of the mapping process are stored in three separate CSV tables. In the tables for \nBRs mapped in a 1:1 ratio and tables for multi-mapped BRs each row constitutes an alignment between the BR in OC Meta\n(represented by its OMID, in the `omid` field) and the BR(s) in OpenAlex (represented by one or multiple OpenAlex IDs, \nin the `openalex_id` field); the optional `type` field stores the type of BRs as registered in OC Meta.\nIn the tables storing unmapped BRs each rows contain only the OMID and, \noptionally, the BR type, in the `omid` and `type` fields respectively.\n\n## Software structure\nThe `utils` module contains the `read_csv_tables` function and the `MultiFileWriter` class, which are used across the \nwhole software to read and write tables.\nThe source code for the mapping process can be found inside the `mapping` module. There are three classes ([Fig 3](#fig3)):\n1. The `MetaProcessor` class deals with creating the tables storing OC Meta BRs that have external IDs (`process_meta_tables` method).\n2. The `OpenAlexProcessor` class deals with creating the tables storing OpenAlex BRs that have external IDs supported also by OC Meta (`create_openalex_ids_tables` method)\nand with converting these tables into SQLite database tables (`create_id_db_table` method).\n3. The `Mapping` class has only one method, `map_omid_openalex_ids`, which is the implementation of the mapping step.\n\nAs of now, some software features concern the pre-processing of responsible agents (authors, publishers, editors) and the separate pre-processing of venues (container publications such as journals, conferences, etc.). For example:\n* the `MetaProcessor.process_meta_tables` function saves separate tables for all bibliographic resources (in a directory named `primary_ents`), for venues only (in a directory named `venues`) and for responsible agents (in a directory named `resp_ags`)\n* the `OpenAlexProcessor` class includes methods for extracting PIDs of other OpenAlex entity types besides Works and Sources (namely Institutions, Publishers, Funders  and Authors)\nThese features are to be considered experimental and are not used for the mapping process.\n\nThe `main` module is the executable to run the whole process (except the result analysis process, which is run separately). The following section illustrates how to write the configuration file storing the arguments passed to the functions called inside `main.py`.\n\n\u003cfigure\u003e\n  \u003cimg src=\"./imgs/mapping_class_diagram.png?raw=true\" alt=\"UML Class diagram\"/\u003e\n    \u003cfigcaption style=\"font-style: italic; font-size: 85%\"\u003e\n    \u003ci\u003e\u003ca id=\"fig3\"\u003e\u003cb\u003eFigure 3.\u003c/b\u003e \u003c/a\u003eUML Class diagram of the software for the mapping process.\u003c/i\u003e\n    \u003c/figcaption\u003e\n\u003c/figure\u003e\n\n## Configuration\nFunction calls in the `main` module (which stores the code to execute the mapping process) take their parameters from a YAML configuration file, \nwhose path is specified in the `--config` argument of the launching command.\nThe YAML file is read as a nested dictionary, and arguments in it are grouped under different keys, according to the purpose they serve. An example of the configuration file can be found in the [config.yaml](config.yaml) file. The \nfollowing illustrates how to compile the configuration file.\n\n#### `meta_tables`\nGroups the parameters to pass to `MetaProcessor.process_meta_tables()` for creating CSV tables of OC Meta BRs with external PIDs.\n- `meta_dump_zip` (str): path to the ZIP file of the OC Meta dump\n- `meta_ids_out` (str): path to the directory where to save the CSV tables. **Here, the tables will be stored in a subdirectory named `primary_ents`**\n- `all_rows` (bool): if True, processes all the BRs in the OC Meta CSV dump, regardless of whether a BR already has an OpenAlex ID. If False, only BRs for which the OpenAlex ID is missing are processed.\n\n#### `openalex_works`\nGroups the parameters to pass to `OpenAlexProcessor.create_openalex_ids_tables()` for creating CSV tables of OpenAlex Works with external PIDs supported also in OC Meta.\n- `inp_dir` (str): the path to the directory storing data of OpenAlex Works\n- `out_dir` (str): the path to the directory where to store the CSV tables with OpenAlex Works\n- `entity_type` (str): the OpenAlex entity type. Since the directory to process contains Works, it must be set to \"work\".\n\n#### `openalex_sources`\nGroups the parameters to pass to `OpenAlexProcessor.create_openalex_ids_tables()` for creating CSV tables of OpenAlex Works with external PIDs supported also in OC Meta.\nThe parameters are named the same as `openalex_works`, as they are used by the same function, but they must be set to deal with OpenAlex Sources\ninstead of Works. The `entity_type` parameter must be set to `source`.\n\n#### `db_works_doi`\nGroups the parameters to pass to `OpenAlexProcessor.create_id_db_table()` for creating a database table storing the DOIs of Work entities in OpenAlex.\n- `inp_dir` (str): path to the directory containing CSV tables storing OpenAlex Works with external PIDs (i.e. the same as `openalex_works.out_dir`)\n- `db_path` (str): the path to the SQLite database where to store the data\n- `id_type` (str): the ID scheme of the IDs to store in the table. Since we want to store Works' _DOIs_, it must be set to \"doi\".\n- `entity_type` (str): the OpenAlex entity type for the database table to produce. Since we want to store DOIs for _Works_, it must be set to \"work\".\n\n#### `db_works_pmid`, `db_works_pmcid`, `db_sources_issn`, `db_sources_wikidata`\nThese group the parameters to pass to `OpenAlexProcessor.create_id_db_table()` for creating database tables for PMIDs and PMCIDs of Works, and\nfor ISSNs and Wikidata IDs of Sources. This follows the same logic as the parameters in `db_works_doi`, but the argument values must be adapted (except `db_path`).\nWhen processing Works, `entity_type` must be set to \"work and `id_type` must be set to \"pmid\" and \"pmcid\"; when processing Sources, `entity_type` must be set to \"source\" and `id_type` must be set to \"issn\" and \"wikidata\".\n\n#### `mapping`\nGroups the parameters to pass to `Mapping.map_omid_openalex_ids()` for creating the mapping.\n- `inp_dir` (str): the directory where the table storing OC Meta BRs with external PIDs are saved, i.e. the directory \"primary_ents\" inside `meta_tables.meta_ids_out`\n- `db_path` (str): The path to the database storing OpenAlex BRs\n- `out_dir` (str): The directory where to save the tables storing BRs mapped in a 1:1 ratio\n- `multi_mapped_dir` (str): The directory where to save the table storing multi-mapped BRs\n- `non_mapped_dir` (str): The directory where to save the table storing unmapped BRs\n- `type_field` (bool): If True, always write the `type` field in the tables.\n- `all_rows` (bool): If True, processes all the BRs in the input table, regardless of whether a BR already has an OpenAlex ID. If False, only BRs for which the OpenAlex ID is missing are processed.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Feliarizzetto%2Fomid-openalex","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Feliarizzetto%2Fomid-openalex","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Feliarizzetto%2Fomid-openalex/lists"}