{"id":28099150,"url":"https://github.com/stadgent/node_service_adlib-backend","last_synced_at":"2025-07-11T06:39:48.148Z","repository":{"id":37803272,"uuid":"362749459","full_name":"StadGent/node_service_adlib-backend","owner":"StadGent","description":null,"archived":false,"fork":false,"pushed_at":"2025-06-12T10:48:09.000Z","size":986,"stargazers_count":5,"open_issues_count":8,"forks_count":2,"subscribers_count":6,"default_branch":"main","last_synced_at":"2025-06-12T11:51:19.314Z","etag":null,"topics":["coghent","d09check22","javascript","lod","nodejs","npm","web"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/StadGent.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2021-04-29T08:44:09.000Z","updated_at":"2025-03-18T10:50:11.000Z","dependencies_parsed_at":"2024-07-16T14:57:11.177Z","dependency_job_id":"c9b4fbb9-de5b-4c8f-8050-875318b77d00","html_url":"https://github.com/StadGent/node_service_adlib-backend","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/StadGent/node_service_adlib-backend","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/StadGent%2Fnode_service_adlib-backend","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/StadGent%2Fnode_service_adlib-backend/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/StadGent%2Fnode_service_adlib-backend/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/StadGent%2Fnode_service_adlib-backend/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/StadGent","download_url":"https://codeload.github.com/StadGent/node_service_adlib-backend/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/StadGent%2Fnode_service_adlib-backend/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":264749875,"owners_count":23658289,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["coghent","d09check22","javascript","lod","nodejs","npm","web"],"created_at":"2025-05-13T17:59:38.628Z","updated_at":"2025-07-11T06:39:48.112Z","avatar_url":"https://github.com/StadGent.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Adlib backend\n\nThe adlib-backend extracts, transforms and loads from an Adlib API into a database.\nThis database is then used by [eventstream-api](https://github.com/StadGent/node_service_eventstream-api) that exposes [event streams*](https://github.com/TREEcg/specification/tree/master/examples/eventstreams).\n\n\\* an event stream is a collection of versioned objects (a version is like an event) and can be updated anytime at their own pace (slow and fast data). This way, consumers can easily discover and harvest the latest changes.\n\n## Prerequisites\n\nThis project requires the [installation of a redis memory store](https://redis.io/docs/getting-started/installation/).\n\n```\ndocker pull redis\ndocker run --rm --name local-redis -p 6379:6379 -d redis\n```\n\n## Install\n\n```\ngit clone https://github.com/StadGent/node_service_adlib-backend.git\ncd node_service_adlib-backend\nnpm install\n```\n\n## Configuration\n\nFill in `config.tml` with the password of the Adlib API you want to use.\n\nadlib-backend uses an ORM (Sequelize) to support following databases: Postgres, MySQL, MariaDB, SQLite and Microsoft SQL Server.\nFill in the connection URI in `config.tml` of the database you want to use. For example: `'sqlite://./eventstream.db'` or `'postgresql://postgres:yourPassword@127.0.0.1:5432'` or `'postgresql://postgres:yourPassword@yourDockerContainer:5432'`\n\n## How to add a mapping\n\n1) Add a new block to `config.tml` with the abbrevation of the institution/dataset, for example ['dmg'] with dmg = Design Museum Ghent.\nThis abbreviation will be used for the creation of URIs and as namespace for the event stream in the Web API. Fill in:\n* `institutionName` with the exact spelling (`institution.name` field) used in Adlib\n* `institutionURI` with a URI of the organization\n\n2) Create a new `start...` function inside `app.js` for your dataset/institution. Copy one of the start functions (e.g. `startDmg`) and rename it with your abbreviation.\n3) Add the new `start...` function inside the `start` function of `app.js` \n4) Adapt the option object so that the value of `institution` matches with abbreviation used in the configuration file.\n5) Adapt the option object so that the value of `adlibDatabase` matches with the database name in Adlib that needs to be harvested\n6) Adapt the option object so that the value of `checkEuropeanaFlag` whether the `EUROPEANA` field of a record needs to be \"checked\" in order to be published.\n7) Create a specific mapper (for example: `TestMapper.js`) in the folder `mappers` for your dataset. Copy (e.g. `dmgMapper.js`) and rename an existing mapper to do this.\n8) Import your mapper in `app.js` by adding an import statement, for example `import TestMapper from './lib/mappers/TestMapper';` Also, rename the mapper in the `start...` function\n9) To add new mapping functions for your mapper, create a new function in `mappers/utils.js` in a similar way as the existing functions.\n\n## Run\n\n### CLI\n\nRun following command to start harvesting the Adlib API:\n\n```\nnpm start\n```\n\nBy running with the NPM command above, versioning is enabled by default using environment variable `npm_package_version` created from `package.json`.\nIf you want to run with `node app.js`,then you need to manually set environment variable `npm_package_version` to enable versioning:\n```\nnpm_package_version=0.0.1 node app.js\n```\n\nWhen `npm_package_version` is not defined, then the `Members` table will be emptied and all objects from Adlib will be harvested again.\nWhen `npm_package_version` is updated, all objects from Adlib will be harvested BUT the already harvested objects will still be maintained (append to the event stream).\n\n## Cronjob\n\nYou can configure in `config.tml` when to periodically run `app.js`.\nFill in `schedule` following the cron syntax, for example every day at midnight (`0 0 * * *`)\n\n\n## Clean data\n\nIf you want to clean up your database (e.g. you updated the mapping of the event stream), then you need to clean the Members table of the database manually.\n\nThis can be configured in the future if preferred.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstadgent%2Fnode_service_adlib-backend","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fstadgent%2Fnode_service_adlib-backend","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstadgent%2Fnode_service_adlib-backend/lists"}