{"id":43171986,"url":"https://github.com/opencitations/oc_ds_converter","last_synced_at":"2026-02-01T02:35:42.034Z","repository":{"id":141234002,"uuid":"611340510","full_name":"opencitations/oc_ds_converter","owner":"opencitations","description":null,"archived":false,"fork":false,"pushed_at":"2025-10-17T21:26:54.000Z","size":4961,"stargazers_count":1,"open_issues_count":1,"forks_count":3,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-10-18T17:04:28.034Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/opencitations.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2023-03-08T16:14:48.000Z","updated_at":"2025-10-17T21:26:57.000Z","dependencies_parsed_at":"2023-10-13T07:55:25.942Z","dependency_job_id":"8403dab4-89c2-4cfd-9b28-fb680613ce7d","html_url":"https://github.com/opencitations/oc_ds_converter","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/opencitations/oc_ds_converter","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/opencitations%2Foc_ds_converter","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/opencitations%2Foc_ds_converter/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/opencitations%2Foc_ds_converter/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/opencitations%2Foc_ds_converter/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/opencitations","download_url":"https://codeload.github.com/opencitations/oc_ds_converter/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/opencitations%2Foc_ds_converter/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28965430,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-01T02:14:24.993Z","status":"ssl_error","status_checked_at":"2026-02-01T02:13:55.706Z","response_time":56,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-02-01T02:35:41.923Z","updated_at":"2026-02-01T02:35:42.018Z","avatar_url":"https://github.com/opencitations.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"[\u003cimg src=\"https://img.shields.io/badge/powered%20by-OpenCitations-%239931FC?labelColor=2D22DE\" /\u003e](http://opencitations.net)\n[![Run tests](https://github.com/opencitations/ra_processor/actions/workflows/run_tests.yml/badge.svg)](https://github.com/opencitations/oc_meta/actions/workflows/run_tests.yml)\n![Coverage](https://raw.githubusercontent.com/opencitations/ra_processor/main/test/coverage/coverage.svg)\n\u003c!-- ![PyPI](https://img.shields.io/pypi/pyversions/oc_meta) --\u003e\n![GitHub code size in bytes](https://img.shields.io/github/languages/code-size/opencitations/ra_processor)\n\n# OpenCitations Data Sources Converter\n\nThis repository contains scripts for converting scholarly bibliographic metadata from various data sources into the format accepted by OpenCitations Meta. The data sources currently supported are:\n- [Crossref](https://github.com/opencitations/oc_ds_converter/blob/658342c1921126515bc4330b8e6cfce0ead2ba48/oc_ds_converter/crossref/crossref_processing.py)\n- [DataCite](https://github.com/opencitations/oc_ds_converter/blob/658342c1921126515bc4330b8e6cfce0ead2ba48/oc_ds_converter/datacite/datacite_processing.py)\n- [PubMed](https://github.com/opencitations/oc_ds_converter/blob/658342c1921126515bc4330b8e6cfce0ead2ba48/oc_ds_converter/pubmed/pubmed_processing.py)\n- [OpenAIRE](https://github.com/opencitations/oc_ds_converter/blob/658342c1921126515bc4330b8e6cfce0ead2ba48/oc_ds_converter/openaire/openaire_processing.py)\n- [JaLC](https://github.com/opencitations/oc_ds_converter/blob/658342c1921126515bc4330b8e6cfce0ead2ba48/oc_ds_converter/jalc/jalc_processing.py)\n- [mEDRA](https://github.com/opencitations/oc_ds_converter/blob/658342c1921126515bc4330b8e6cfce0ead2ba48/oc_ds_converter/medra/medra_processing.py)\n\n\n\u003c!-- TABLE OF CONTENTS --\u003e\n  \u003csummary\u003e\u003ch2 style=\"display: inline-block\"\u003eTable of Contents\u003c/h2\u003e\u003c/summary\u003e\n  \u003col\u003e\n    \u003cli\u003e\u003ca href=\"#about-the-project\"\u003eAbout The Project\u003c/a\u003e\u003c/li\u003e\n    \u003cli\u003e\u003ca href=\"#components\"\u003eSoftware Components\u003c/a\u003e\u003c/li\u003e\n    \u003cli\u003e\u003ca href=\"#validation\"\u003eID Validation Process\u003c/a\u003e\u003c/li\u003e\n    \u003cli\u003e\u003ca href=\"#run\"\u003eHow to Run the Software\u003c/a\u003e\u003c/li\u003e\n    \u003cli\u003e\u003ca href=\"#extend\"\u003eHow to Extend the Software\u003c/a\u003e\u003c/li\u003e\n    \u003cli\u003e\u003ca href=\"#license\"\u003eLicense\u003c/a\u003e\u003c/li\u003e\n    \u003cli\u003e\u003ca href=\"#contacts\"\u003eContacts\u003c/a\u003e\u003c/li\u003e\n    \u003cli\u003e\u003ca href=\"#acknowledgements\"\u003eAcknowledgements\u003c/a\u003e\u003c/li\u003e\n    \u003cli\u003e\u003ca href=\"#references\"\u003eReferences\u003c/a\u003e\u003c/li\u003e\n  \u003c/ol\u003e\n\n\n\u003c!-- ABOUT THE PROJECT --\u003e\n\u003ch2 id=\"about-the-project\"\u003e About The Project \u003c/h2\u003e\n\nThe main function of this software is to perform a metadata crosswalk between the data sources providing bibliographic and citation data and the [OpenCitations Data Model (OCDM)](https://opencitations.net/model). At the same time, the software also handles the normalization and validation of the identifiers provided by the data source in all cases where it is not also the registration agency for those identifiers. The software generates **two main output datasets**, based on the data provided by a specific source:\n\n- **Bibliographic entities CSV tables**;\n  which are meant to be used as input for the [META software](https://github.com/opencitations/oc_meta.git), an OpenCitations tool and database for managing bibliographic entities' metadata. Example:\n\n| id                             | title                                                | author                                               | pub_date  | venue                                       | volume | issue | page   | type           | publisher                                      | editor |\n|--------------------------------|------------------------------------------------------|------------------------------------------------------|-----------|---------------------------------------------|--------|-------|--------|----------------|------------------------------------------------|--------|\n| doi:10.9799/ksfan.2012.25.1.069 | Nonthermal Sterilization and Shelf-life Extension of Seafood Products by Intense Pulsed Light Treatment | Cheigh, Chan-Ick [orcid:0000-0002-6227-4053]; Mun, Ji-Hye [orcid:0000-0002-6227-4053]; Chung, Myong-Soo | 2012-3-31 | The Korean Journal of Food And Nutrition [issn:1225-4339] | 25     | 1     | 69-76  | journal article | The Korean Society of Food and Nutrition [crossref:4768] |        |\n\n- **AnyID-to-Any-ID citations CSV tables**;\n  which will be used as input for the [INDEX software](https://github.com/opencitations/index), an OpenCitations tool and database for producing and managing citations between bibliographic entities identified by OMIDs (internal and unique identifiers assigned by OpenCitations). Example:\n\n| citing                          | cited                        |\n|---------------------------------|------------------------------|\n| doi:10.11426/nagare1970.2.4_1   | doi:10.1295/kobunshi.16.921  |\n\n\nIn practice, the outputs generated by `oc_ds_converter` are used in subsequent steps of the data ingestion process within the OpenCitations infrastructure. Specifically, the metadata tables are used as input for META. The software assigns an OMID identifier to new entities and propagates an existing OMID to the entities already present in the OpenCitations databases, thereby deduplicating identical entities ingested from different data sources.\n\nSubsequently, the INDEX software, responsible for producing citation data compliant with the OpenCitations data model, takes as input the anyID-to-anyID citation tables produced by the `oc_ds_converter` software. It queries the META database to retrieve the OMID associated with each entity's identifier and produces OMID-to-OMID citations in various formats (RDF, SCHOLIX, CSV) as output.\n\nHere, a diagram of the OpenCitations ingestion workflow:\n![OpenCitations Ingestion Workflow](https://github.com/ariannamorettj/OC_documents/blob/5115cf039b4baa2319c6c22cc270647861ae2f5a/ingestion_workflow_overview.jpg)\n\n\u003c!-- SOFTWARE COMPONENTS --\u003e\n\u003ch2 id=\"components\"\u003e Software Components \u003c/h2\u003e\n\nThe software is built upon three fundamental components, each addressing specific needs: \n1. \u003ca href=\"#mdc\"\u003e Metadata Crosswalk\u003c/a\u003e ([oc_ds_converter](https://github.com/opencitations/oc_ds_converter/tree/297c71577232459a69ca1662011419d452971095/oc_ds_converter)) \n2. \u003ca href=\"#idv\"\u003eIdentifier Validation\u003c/a\u003e ([oc_ds_converter/oc_idmanager](https://github.com/opencitations/oc_ds_converter/tree/297c71577232459a69ca1662011419d452971095/oc_ds_converter/oc_idmanager)) \n3. \u003ca href=\"#dsm\"\u003eData Storage Management\u003c/a\u003e ([oc_ds_converter/oc_idmanager/oc_data_storage](https://github.com/opencitations/oc_ds_converter/tree/297c71577232459a69ca1662011419d452971095/oc_ds_converter/oc_idmanager/oc_data_storage))\n\n\u003ch3 id=\"mdc\"\u003eMetadata Crosswalk\u003c/h3\u003e\nWithin this layer, there is a specific plugin for each data source, which, in turn, contains a Python file (usually named after the data source + \"_ processing.py,\" e.g., oc_ds_converter/datacite/datacite_processing.py). In this file, a class is defined to convert metadata provided by the specific source into metadata compliant with OCDM. For example, in the file datacite_processing.py, the class DataciteProcessing(RaProcessor) is defined, which contains the method `csv_creator(self, item: dict) -\u003e dict`. This method is responsible for producing a dictionary of metadata representing a bibliographic entity extracted from the dump provided by DataCite. \n\u003ch3 id=\"idv\"\u003e Identifier Validation\u003c/h3\u003e\nThis software validates all identifiers not provided by the identifier registration agency itself. Currently, the identifiers handled by OpenCitations are: \u003ca href=\"https://github.com/opencitations/oc_ds_converter/blob/3bf5b93edf88c3f6cb1fe1860490533b5b9d1186/oc_ds_converter/oc_idmanager/doi.py\"\u003eDOI\u003c/a\u003e; \u003ca href=\"https://github.com/opencitations/oc_ds_converter/blob/3bf5b93edf88c3f6cb1fe1860490533b5b9d1186/oc_ds_converter/oc_idmanager/pmid.py\"\u003ePMID\u003c/a\u003e; \u003ca href=\"https://github.com/opencitations/oc_ds_converter/blob/3bf5b93edf88c3f6cb1fe1860490533b5b9d1186/oc_ds_converter/oc_idmanager/pmcid.py\"\u003ePMC\u003c/a\u003e; \u003ca href=\"https://github.com/opencitations/oc_ds_converter/blob/3bf5b93edf88c3f6cb1fe1860490533b5b9d1186/oc_ds_converter/oc_idmanager/viaf.py\"\u003eVIAF\u003c/a\u003e; \u003ca href=\"https://github.com/opencitations/oc_ds_converter/blob/3bf5b93edf88c3f6cb1fe1860490533b5b9d1186/oc_ds_converter/oc_idmanager/wikidata.py\"\u003eWIKIDATA\u003c/a\u003e; \u003ca href=\"https://github.com/opencitations/oc_ds_converter/blob/3bf5b93edf88c3f6cb1fe1860490533b5b9d1186/oc_ds_converter/oc_idmanager/wikipedia.py\"\u003eWIKIPEDIA\u003c/a\u003e; \u003ca href=\"https://github.com/opencitations/oc_ds_converter/blob/3bf5b93edf88c3f6cb1fe1860490533b5b9d1186/oc_ds_converter/oc_idmanager/ror.py\"\u003eROR\u003c/a\u003e; \u003ca href=\"https://github.com/opencitations/oc_ds_converter/blob/3bf5b93edf88c3f6cb1fe1860490533b5b9d1186/oc_ds_converter/oc_idmanager/orcid.py\"\u003eORCID\u003c/a\u003e; \u003ca href=\"https://github.com/opencitations/oc_ds_converter/blob/3bf5b93edf88c3f6cb1fe1860490533b5b9d1186/oc_ds_converter/oc_idmanager/arxiv.py\"\u003eARXIV\u003c/a\u003e; \u003ca href=\"https://github.com/opencitations/oc_ds_converter/blob/3bf5b93edf88c3f6cb1fe1860490533b5b9d1186/oc_ds_converter/oc_idmanager/jid.py\"\u003eJID\u003c/a\u003e; \u003ca href=\"https://github.com/opencitations/oc_ds_converter/blob/3bf5b93edf88c3f6cb1fe1860490533b5b9d1186/oc_ds_converter/oc_idmanager/issn.py\"\u003eISSN\u003c/a\u003e; \u003ca href=\"https://github.com/opencitations/oc_ds_converter/blob/3bf5b93edf88c3f6cb1fe1860490533b5b9d1186/oc_ds_converter/oc_idmanager/isbn.py\"\u003eISBN\u003c/a\u003e; \u003ca href=\"https://github.com/opencitations/oc_ds_converter/blob/3bf5b93edf88c3f6cb1fe1860490533b5b9d1186/oc_ds_converter/oc_idmanager/url.py\"\u003eURL\u003c/a\u003e. \n\nEach identifier schema has its own class (e.g.: `PMIDManager(IdentifierManager)`, defined in `oc_ds_converter/oc_idmanager/pmid.py`), instantiated according to the model provided by the abstract class `IdentifierManager(metaclass=ABCMeta)`, defined in oc_ds_converter/oc_idmanager/base.py. \nEach class provides methods for:\n\u003col\u003e\n  \u003cli\u003enormalising the id string\u003c/li\u003e\n  \u003cli\u003echecking the correctness of the id syntax\u003c/li\u003e\n  \u003cli\u003everifying its existence using specific API services (if available)\u003c/li\u003e\n\u003c/ol\u003e\n\n\u003ch3 id=\"dsm\"\u003e Data Storage Management \u003c/h3\u003e \nOpenCitations ds_converter currently offers three storage systems, which can be alternatively used: \n\n* In Memory (class `InMemoryStorageManager(StorageManager)`, defined in `oc_ds_converter/oc_idmanager/oc_data_storage/in_memory_manager.py)`\n* Redis (class `RedisStorageManager(StorageManager)`, defined in `oc_ds_converter/oc_idmanager/oc_data_storage/redis_manager.py`)\n* Sqlite (class `SqliteStorageManager(StorageManager)`, defined in `oc_ds_converter/oc_idmanager/oc_data_storage/sqlite_manager.py`). \n\nEach of these classes is defined as an instance of the abstract class `StorageManager(metaclass=ABCMeta)`, defined in `oc_ds_converter/oc_idmanager/oc_data_storage/storage_manager.py`. \n\nThe type of storage manager used for a specific data source process can be chosen by the user (however, we suggest using the Redis storage manager). \nAn instance of the chosen storage manager will be used by all the ID Managers instantiated in the process to store validation data at the end of each data chunk management. \nThe temporary storage manager used while processing a data chunk is instead always an instance of the In-Memory storage manager (which is based on the use of a python dictionary). The reason for this choice lies in the fact that, in case of a run stop, the execution would restart processing from the beginning of the chunk that was being managed at the time of the interruption, and thus the data already memorized by a redis or sqlite storage manager would be duplicated, while the data memorized in an instance of an in-memory storage manager are just lost and reprocessed. \n\n\u003c!-- ID VALIDATION PROCESS --\u003e\n\u003ch2 id=\"validation\"\u003eID Validation Process\u003c/h2\u003e\nIn order to avoid redundant API checks, we rely on an ad-hoc data storage system. More in detail, in case the data source is also the id registration agency of at least a part of the identifiers provided in a data dump, we perform a full preliminary iteration of the data to store these identifiers as valid, without any further check. \n\n![Perliminary data dump iteration](https://github.com/ariannamorettj/OC_documents/blob/5115cf039b4baa2319c6c22cc270647861ae2f5a/id_validation_pre_process_dump_iteration_diagram.png) \n\nSubsequently, we perform another full iteration, validating all identifiers not registered by the data source itself. \n\n![Data dump iteration for data validation](https://github.com/ariannamorettj/OC_documents/blob/5115cf039b4baa2319c6c22cc270647861ae2f5a/id_validation_process_dump_iteration_diagram.png) \n\nNote that, to manage the large amount of data provided by each data source, the input dataset is generally divided into data chunks. As mentioned above, in order to avoid data duplication in case of a process interruption and restart, data concerning each chunk are temporarily stored in an instance of the in-memory storage manager (see InMemoryStorageManager(StorageManager) in oc_ds_converter/oc_idmanager/oc_data_storage/in_memory_manager.py). The data stored in the temporary storage manager is transferred to the main storage manager (containing the ID validation data of the full input dataset) at the end of the chunk's process, when both the CSV tables concerning bibliographic metadata and citations are produced.\nFor each encountered identifier to be validated, an ordered list of checks should be performed, stopping as soon as the validity value can be assessed:\n\n1. Search for the identifier in the in-memory storage manager, containing data concerning the current data chunk;\n2. Search for the identifier in the main storage manager, containing data concerning the whole dataset; \n3. Search for the identifier in the OpenCitations databases, containing data of all the datasets ever ingested in OpenCitations.\n4. Use ID-schema specific API services to retrieve the validity information of the ID. \n\n\u003c!-- HOW TO RUN THE SOFTWARE --\u003e\n\u003ch2 id=\"run\"\u003e How to Run the Software \u003c/h2\u003e\n\nTo produce the citations and metadata CSV output from a data source, it is possible to execute its specific process by selecting the correct source from `oc_ds_converter/run/` directory. For example, the oc_ds_converter process for **JaLC** data source can be launched as follows:\n\n```\npython oc_ds_converter/run/jalc_process.py -ja /Volumes/my_disk/JALC_INPUT -out /Volumes/my_disk/JALC_OUTPUT -ca /Volumes/my_disk/JOCI_CACHE.json -r -m 3\n```\n\nThis command launches a process of data conversion from the input data dump (located at `/Volumes/my_disk/JALC_INPUT`) into metadata CSV tables (stored at `/Volumes/my_disk/JALC_OUTPUT`) and citation CSV tables (stored in a directory automatically generated at `/Volumes/my_disk/JALC_OUTPUT_citations`), using up to 3 workers for the process parallelization (`-m 3`) and Redis as storage system (`-r`) . While the process is being executed, a cache file at `/Volumes/my_disk/JOCI_CACHE.json` is created and updated. \n\nMore in detail, each data source run script has a set of arguments that can be adapted to meet the peculiarities of the dataset. However, all the sources should accept a similar list of arguments: \n\n- **'--config'**: The path to a configuration file, where the other arguments can be declared;\n- **'--input_location'**: The path to the input data;\n- **'--output_location'**: The path to the output directory where the metadata CSV files will be stored. From the name of this directory, the name of the directory where to store the citation CSV files will be derived automatically.\n- **'--publishers'**: The path to an optional support CSV file containing additional information about publishers, their crossref members and the DOI prefix they are associated with (id, name, prefix), used to enrich the metadata.\n- **'--orcid'**: The path to an optional support table mapping DOIs to ORCIDs of the publications' authors, used to enrich the metadata.\n- **'--wanted'**: The path to an optional CSV filepath containing a list of DOIs to process.\n- **'--cache'**: The cache file path, that will be automatically deleted at the end of the process.\n- **'--verbose'**: Argument which allows to declare whether a verbose description of the process execution is required. \n- **'--storage_path'**: An argument to optionally choose the path of the file where to store data concerning validated IDs information, in case the process is executed using either an In-Memory or a Sqlite storage manager. Pay attention to specify a \".db\" file in case a SqliteStorageManager is chosen and a \".json\" file otherwise.\n- **'--testing'**: The parameter to define whether or not the script is to be run in testing mode.\n- **'--redis_storage_manager'**: A parameter to define whether or not to use redis as storage manager. In case Redis is not used, the storage manager type is derived by the storage path type (i.e. : In Memory storage in case the file is a JSON file, Sqlite in case of a .db file)\n- **'--max_workers'**: The integer number of workers used to run the process in parallel executions. \n\n\n\u003c!-- HOW TO EXTEND THE SOFTWARE --\u003e\n\u003ch2 id=\"extend\"\u003e How to Extend the Software \u003c/h2\u003e\n\n### Manage a new Data Source\nIn order to manage a new data source, two main software components need to be developed: \n1. a script for reading the data source, extract the bibliographic entities' metadata, and produce the output tables;\n2. a script for reshaping the metadata of each bibliographic entity according to the OpenCitation data model.\n\nIn addition to that, if the data source uses persistent identifiers not managed by OpenCitations yet, a new identifier manager should be developed too. \n\n#### Data Source Reader Script \n\nFor each new data source, a python file should be added to the directory `oc_ds_converter/run/`. The file should be named after the data source, and perform the following tasks: \n\n1. decopress and read the source dataset;\n2. manage the identifiers' validation process;\n3. extract from the source data a data structure representing each bibliographic resource;\n4. call a source-specific metadata crosswalk method to convert this data structure into an OCDM-compliant dictionary representing the bibliographic resource, to be stored as a CSV row;\n5. produce the output tables (citations and metadata)\n\n#### Metadata Crosswalk Script\n\nAll source represents bibliographic records according to a specific data model, which has to be mapped into OCDM. To do so, we implement a source-specific child class of the class `RaProcessor` (defined in `oc_ds_converter/ra_processor.py`) for each new data source. The main method of all `RaProcessor` children classes is `csv_creator`, which is aimed at producing a row for an OpenCitations metadata table from a data structure representing a bibliographic entry according the source data model. As an example, see `OpenaireProcessing(RaProcessor)` class (in `oc_ds_converter/openaire/openaire_processing.py`). \n\n\n### Add a new ID Manager\n\nFor adding a new ID Manager:\n1. create a python file at `oc_ds_converter/oc_idmanager`, named after the id schema, e.g. `oc_ds_converter/oc_idmanager/viaf.py`.\n2. create a new class as an instance of the abstract class `IdentifierManager` (defined in `oc_ds_converter/oc_idmanager/base.py`), e.g.: `ViafManager(IdentifierManager)`, thus following the provided template. In particular:\n3. define all the id-schema specific required methods, i.e.: `syntax_ok`, to check whether the ID is compliant to its own schema syntax, `exists`, to check the ID's existence using the ID-specific API, `normalise`, to normalise the identifier string (for example by removing unexpected character and turing the uppercase into lowercase characters), and `is_valid`, for assessing the overall validity of the identifier.\n4. if possible, add additional ID-schema specific methods. For example, some ID schemas (such as ORCID and ISSN) are formed by following a specific check-digit mechanism, which provides a further control system to verify the ID validity: in these cases, it is possible to add also a `check_digit` method. \n\n### Add a new Storage Manager\n\nFor adding a new type of Storage Manager, i.e. relying on another storage system:\n1. create a python file at `oc_ds_converter/oc_idmanager/oc_data_storage` named after the storage system, e.g.: `oc_ds_converter/oc_idmanager/oc_data_storage/redis_manager.py`.\n2. create a new class as an instance of the abstract class `StorageManager` (defined in `oc_ds_converter/oc_idmanager/oc_data_storage/storage_manager.py`), e.g.: `RedisStorageManager(StorageManager)`, thus following the provided template. In particular:\n3. define all the storage-type-specific required methods, i.e.: `set_value`, to add a single key-value pair to the storage, `set_multi_value`, to store a list of key-value tuple pairs all at once,  `get_value`, to retrieve the value associated to a specific key, `del_value`, to delete a key-value pair, `delete_storage`, to delete all the data previously saved in the storage system, and `get_all_keys`, to retrieve the list of all the keys in the storage.\n\n### Test\nThe repository is managed with [poetry](https://python-poetry.org/docs/). To activate the virtual environment:\n```\npoetry shell\n```\nTo add a package as a dependency to the project:\n```\npoetry add \u003cpackage\u003e\n```\nTo run all tests with poetry: \n```\npoetry run test\n```\nTo run specific tests:\n```\npython -m unittest discover -s test -p \"*.py\" \n```\n\n\n\u003c!-- LICENSE --\u003e\n\u003ch2 id=\"license\"\u003eLicense\u003c/h2\u003e\n\nDistributed under the ISC License. See `LICENSE` for more information.\n\n\u003c!-- CONTACTS --\u003e\n\u003ch2 id=\"contacts\"\u003eContacts\u003c/h2\u003e\n\n#### Authors and Current maintainers of the repository\n\n- Arianna Moretti - [@ariannamorettj](https://github.com/ariannamorettj) - arianna.moretti4@unibo.it\n- Arcangelo Massari - [@arcangelo7](https://github.com/arcangelo7) - arcangelo.massari@unibo.it\n- Elia Rizzetto - [@eliarizzetto](https://github.com/eliarizzetto) - elia.rizzetto@studio.unibo.it\n- Marta Soricetti - [@martasoricetti](https://github.com/martasoricetti) - marta.soricetti@studio.unibo.it\n\nProject Link: [https://github.com/opencitations/oc_ds_converter](https://github.com/opencitations/oc_ds_converter)\n\n\n\u003c!-- ACKNOWLEDGEMENTS --\u003e\n\u003ch2 id=\"acknowledgements\"\u003eAcknowledgements\u003c/h2\u003e\n\nThis project has been developed under the supervision of Prof. Silvio Peroni.\n\n- Silvio Peroni - [@essepuntato](https://github.com/essepuntato) - silvio.peroni@unibo.it\n\n\u003c!-- REFERENCES --\u003e\n\u003ch2 id=\"references\"\u003eReferences\u003c/h2\u003e\n\n- [The OpenCitations Data Model](https://link.springer.com/chapter/10.1007/978-3-030-62466-8_28)\n- [OpenCitations Meta](https://doi.org/10.48550/arXiv.2306.16191)\n- [Metadata crosswalk for citation data production in OpenCitations](https://marketplace.sshopencloud.eu/workflow/MHwO4l)\n\n\u003c!-- MARKDOWN LINKS \u0026 IMAGES --\u003e\n\u003c!-- https://www.markdownguide.org/basic-syntax/#reference-style-links --\u003e\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopencitations%2Foc_ds_converter","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fopencitations%2Foc_ds_converter","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopencitations%2Foc_ds_converter/lists"}