{"id":21198640,"url":"https://github.com/natlibfi/melinda-rest-api-importer","last_synced_at":"2026-07-05T17:31:26.544Z","repository":{"id":42308752,"uuid":"225306797","full_name":"NatLibFi/melinda-rest-api-importer","owner":"NatLibFi","description":"Importer microservice of Melinda REST API","archived":false,"fork":false,"pushed_at":"2025-02-17T11:58:57.000Z","size":2354,"stargazers_count":0,"open_issues_count":1,"forks_count":0,"subscribers_count":7,"default_branch":"main","last_synced_at":"2025-02-17T12:24:23.587Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/NatLibFi.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-12-02T06:58:58.000Z","updated_at":"2024-09-26T14:23:19.000Z","dependencies_parsed_at":"2023-02-16T18:15:49.129Z","dependency_job_id":"ac73c85d-b465-4692-b9d7-28c4f3c6e5ce","html_url":"https://github.com/NatLibFi/melinda-rest-api-importer","commit_stats":null,"previous_names":[],"tags_count":90,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NatLibFi%2Fmelinda-rest-api-importer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NatLibFi%2Fmelinda-rest-api-importer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NatLibFi%2Fmelinda-rest-api-importer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NatLibFi%2Fmelinda-rest-api-importer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/NatLibFi","download_url":"https://codeload.github.com/NatLibFi/melinda-rest-api-importer/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243652707,"owners_count":20325611,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-20T19:52:46.314Z","updated_at":"2025-10-28T09:36:56.629Z","avatar_url":"https://github.com/NatLibFi.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Service for queuing import records from melinda-rest-api\n![Version](https://img.shields.io/github/package-json/v/NatLibFi/melinda-rest-api-importer.svg)\n![Node Version](https://img.shields.io/badge/dynamic/json.svg?url=https%3A%2F%2Fraw.githubusercontent.com%2FNatLibFi%2Fmelinda-rest-api-importer%2Fmaster%2Fpackage.json\u0026label=node\u0026query=$.engines.node)\n\n\n## Usage\n\nWhile service is in use it polls Mongo for jobs is state `'IMPORTER.IN_QUEUE'` for `OPERATION`, prioritizing prio jobs. When a job is found, the record(s) belonging to it are fetched from it's AMQP queue (`OPERATION.correlationId`) in suitable chunks of records and the record file is forwarded to record-load-api for actual importing to Melinda. When all loadprocesses for a job are done, the job is transitioned to state `DONE` in Mongo. \n\n### Environment variables\n| Name                | Mandatory | Description                                                                                                        |\n|---------------------|-----------|--------------------------------------------------------------------------------------------------------------------|\n| AMQP_URL            | Yes       | A serialized object of AMQP connection config                                                                      |\n| OPERATION           | Yes       | A string state of passing operations. Enum: `'CREATE'`, `'UPDATE'` or `'FIX'`                                               |\n| RECORD_LOAD_API_KEY | Yes       | A string key authorized to use the API                                                                             |\n| RECORD_LOAD_LIBRARY | Yes       | A string                                                                                                           |\n| RECORD_LOAD_URL     | Yes       | A serialized URL address of Melinda-record-load-api                                                                |\n| MONGO_URI           | No        | A serialized URL address of Melinda-rest-api's import queue database. Defaults to `'mongodb://localhost:27017/db'` |\n| OFFLINE_PERIOD      | No        | Starting hour and length of offline period. e.g `'11,1'`                                                           |\n| POLL_WAIT_TIME      | No        | A number value presenting time in ms between polling                                                               |\n| LOG_LEVEL           | No        | Log information level                                                                                              |\n| ERROR_503_WAIT_TIME | No        | A number value presenting time in ms for waiting before trying again when receiving 503 error from aleph-record-load-api |\n| KEEP_LOAD_PROCESS_RESULTS | No  | A string telling in which cases load process details are saved in Mongo. Defaults to `NON_HANDLED`. Options: `ALL`, `NONE`, `NON_PROCESSED`, `NON_HANDLED`. |\n| FIX_PRIO            | No | Defaults to `API`|\n| FIX_BULK            | No | Defaults to `INSB`|\n\n\n### Mongo\nDb: `'rest-api'`\nTables: `'prio'` for prio jobs, `'bulk'` for bulk jobs\n\nQueue-item schema example for a prio job queueItem:\n```json\n{\n\"correlationId\":\"FOO\",\n\"cataloger\":\"xxx0000\",\n\"oCatalogerIn\": \"xxx0000\",\n\"operation\":\"UPDATE\",\n\"operationSettings\": {\n  \"noop\": true,\n  \"unique\": false,\n  \"prio\": true,\n  },\n\"recordLoadParams\": {\n  \"pActiveLibrary\": \"XXX00\",\n  \"pInputFile\": \"filename.seq\",\n  \"pRejectFile\": \"filename.rej\",\n  \"pLogFile\": \"filename.syslog\",\n  \"pOldNew\": \"NEW\"\n  },\n\"queueItemState\":\"DONE\",\n\"creationTime\":\"2020-01-01T00:00:00.000Z\",\n\"modificationTime\":\"2020-01-01T00:00:01.000Z\",\n\"handledIds\": [ \"000000001\"],\n\"rejectedIds\": [],\n\"errorStatus\": \"\",\n\"errorMessage\": \"\",\n\"noopValidationMessages\": [],\n\"loadProcessReports\": []\n}\n```\n\nQueue-item schema examle for a bulk job queueItem:\n```json\n{\n\"correlationId\":\"FOO\",\n\"cataloger\":\"xxx0000\",\n\"oCatalogerIn\": \"xxx0000\",\n\"operation\":\"UPDATE\",\n\"operationSettings\": {\n  \"prio\": false,\n },\n\"contentType\": \"application/json\",\n\"recordLoadParams\": {\n  \"pActiveLibrary\": \"XXX00\",\n  \"pInputFile\": \"filename.seq\",\n  \"pRejectFile\": \"filename.rej\",\n  \"pLogFile\": \"filename.syslog\",\n  \"pOldNew\": \"NEW\"\n},\n\"queueItemState\":\"DONE\",\n\"creationTime\":\"2020-01-01T00:00:00.000Z\",\n\"modificationTime\":\"2020-01-01T00:00:01.000Z\",\n\"handledIds\": [ \"000000001\",\"000000002\"],\n\"rejectedIds\": [\"000999999\"],\n\"errorStatus\": \"\",\n\"errorMessage\": \"\",\n\"loadProcessReports\": [{\n  \"status\": 200,\n  \"processId\": 9999,\n  \"processedAll\": false,\n  \"recordAmount\": 3,\n  \"processedAmount\": 2,\n  \"handledAmount\": 1,\n  \"rejectedAmount\": 1,\n  \"rejectMessages\": [\"Cannot overwrite a deleted record. Record 000999999 is written to rej file\"]\n  }],\n\"splitterReport\": [{\n \"recordNumber\": 0,\n \"sequenceNumber\": 2,\n \"readerErrors\": [{\n   \"sequenceNumber\": 1,\n   \"error\": \"Record is invalid\"\n   }]\n }]\n}\n```\n\n## License and copyright\n\nCopyright (c) 2020-2025 **University Of Helsinki (The National Library Of Finland)**\n\nThis project's source code is licensed under the terms of **MIT** or any later version.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnatlibfi%2Fmelinda-rest-api-importer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnatlibfi%2Fmelinda-rest-api-importer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnatlibfi%2Fmelinda-rest-api-importer/lists"}