{"id":21425431,"url":"https://github.com/utrechtuniversity/ilab-catalog","last_synced_at":"2026-04-15T18:32:20.386Z","repository":{"id":81546779,"uuid":"132764889","full_name":"UtrechtUniversity/ilab-catalog","owner":"UtrechtUniversity","description":"Deployments scripts for the UU Catalog: https://catalog.data.uu.nl/","archived":false,"fork":false,"pushed_at":"2024-11-20T15:32:40.000Z","size":230,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":8,"default_branch":"master","last_synced_at":"2025-03-16T21:11:50.151Z","etag":null,"topics":["ansible","apache2","ckan","postgresql","redis","solr"],"latest_commit_sha":null,"homepage":"https://catalog.data.uu.nl/","language":"Jinja","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/UtrechtUniversity.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-05-09T13:58:22.000Z","updated_at":"2024-11-20T15:32:44.000Z","dependencies_parsed_at":"2024-11-22T21:41:22.115Z","dependency_job_id":null,"html_url":"https://github.com/UtrechtUniversity/ilab-catalog","commit_stats":null,"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"purl":"pkg:github/UtrechtUniversity/ilab-catalog","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/UtrechtUniversity%2Filab-catalog","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/UtrechtUniversity%2Filab-catalog/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/UtrechtUniversity%2Filab-catalog/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/UtrechtUniversity%2Filab-catalog/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/UtrechtUniversity","download_url":"https://codeload.github.com/UtrechtUniversity/ilab-catalog/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/UtrechtUniversity%2Filab-catalog/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31854753,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-15T15:24:51.572Z","status":"ssl_error","status_checked_at":"2026-04-15T15:24:39.138Z","response_time":63,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ansible","apache2","ckan","postgresql","redis","solr"],"created_at":"2024-11-22T21:28:28.868Z","updated_at":"2026-04-15T18:32:20.374Z","avatar_url":"https://github.com/UtrechtUniversity.png","language":"Jinja","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ilab-catalog\n[Ansible](https://docs.ansible.com) scripts for automatic deployment of Yoda Catalog.\n\n**This repository has been archived. The code is no longer maintained.**\n\n## Requirements\n### Control machine requirements\n* [Ansible](https://docs.ansible.com/ansible/intro_installation.html) (\u003e= 2.8)\n* [VirtualBox](https://www.virtualbox.org/manual/ch02.html) (\u003e= 5.1)\n* [Vagrant](https://www.vagrantup.com/docs/installation/) (\u003e= 1.9)\n\n### Managed node requirements\n\nThe catalog can be deployed on either [CentOS 7](https://www.centos.org/) (\u003e= 7.3),\nor on [RHEL 8](https://www.redhat.com/en/technologies/linux-platforms/enterprise-linux) /\n[Alma Linux 8](https://almalinux.org/).\n\n## Deploying Yoda Catalog development instance\n\nConfigure the virtual machine for development:\n```bash\nvagrant up\n```\n\nOn a Windows host first SSH into the Ansible controller virtual machine (skip this step on GNU/Linux or macOS):\n```bash\nvagrant ssh ilab-catalog-controller\ncd ~/ilab-catalog\n```\n\nDeploy Yoda Catalog to development virtual machine:\n```bash\nansible-playbook playbook.yml\n```\n\nAdd following host to /etc/hosts (GNU/Linux or macOS) or %SystemRoot%\\System32\\drivers\\etc\\hosts (Windows):\n```\n192.168.70.10 ilab-catalog.ckan.test\n```\n\n## Upgrading Yoda Catalog instance\nUpgrading the Yoda Catalog development instance to the latest version can be done by running the Ansible playbooks again.\n\nOn a Windows host first SSH into the Ansible controller virtual machine (skip this step on GNU/Linux or macOS):\n```bash\nvagrant ssh controller\ncd ~/ilab-catalog\n```\n\nUpgrade Ansible scripts:\n```bash\ngit pull\n```\n\nUpgrade Yoda Catalog instance:\n```bash\nansible-playbook playbook.yml\n```\n\n#  CKAN catalog\nYoda Catalog uses CKAN as basis for catalog functionality.  \nCKAN is generic software that can be tailored.\nIt uses several modules / extensions to be able to form CKAN standard product into an application that can handle Yoda specific data.\n\n## Harvest sources   \nA harvest source is a configuration, added within CKAN, to instruct where to harvest for data and how to deal with the harvested data from the source (in this case Yoda).  \nThis by passing information to the ckan module CKANEXT-OAIPMH (see further)\n\nHarvest sources, which is functionality added to CKAN by 3rd party module ‘ckan-harvester’, is mainly a configuration holding:  \n-endpoint  \n-params to tell configure the source freely as an application programmer  \n-harvest type (either CKAN/OAIPMH when OAI-PMH extension added)\n\nCKANEXT HARVESTER uses a POSTGRESQL database to be able to harvest.  \n\n## Harvesting stages\nHarvesting is realised in following three stages:\n\n- Gather stage  \nThrough OAI PMH gather all present datapackage and create a list of them in the postgresql database.\n\n- Import stage  \nOne by one handle the list of gather Ids and add them tot he database as a JSON-object separately.\n\n- Fetch stage  \nThe data that is fetched can now be processed and, in accordance with module CKANEXT-CUSTOM-THEME,  be saved into CKAN itself.\nThis has to be in alignment with the theme as the theme contains the ‘application’ and this needs to be ‘fed’ correctly from the database.  \nThis goes especially for the extra – fields.\n\nThe processing of the fetched data is dealt with within another extension. CKANEXT-OAIPMH.  \nThus, any party can tailor CKAN to its own wishes.  \nThis extension is under development by the Yoda-team: CKANEXT-OAIPMH\n\n\n## CKAN Extension CKANEXT-OAIPMH\nA generic component that currently is used for catalog applications iLAB and EPOS.  \nIt allows for harvesting through the OAI-PMH standard.  \nCKANEXT-OAIPMH processes the harvested data that was collected at the endpoint within the harvest source.\n\nDriven by the parameters as stated in the CKAN **harvest source** that initiated the harvest process, CKANEXT-OAIPMH module can be configured to treat data differently corresponding to the active configuration. Thus, supporting the possibility for totally different applications.  \n\nIn our case this component handles two entirey different sources for entirely different applications:  \n1) EPOS, and  \n2) ILAB\n\nThe differentiation can be made by passing the correct parameters into the Harvest source:\n\nConfiguration:   \n*Voor ILAB datacite:*  \n{\"metadata_prefix\": \"datacite\",  \n\"application\": \"ILAB\"  \n}\n\nThe CKAN-OAIPMH extension is programmed as such that it will provision CKAN based upon the above parameters.\nMain differentation is the application. Currently EPOS and iLAB.\n\n\n# Application specific handling\n\n## iLAB\nmetadataPrefix = **datacite**  \nThe organisation linked to the harvest source is added to each datapackage as being the owning organisation (owner_org: see further CKAN fields).\n\nExample ckan harvest source configuration:  \n{  \n\u0026nbsp;\u0026nbsp;\"metadata_prefix\": \"datacite\",  \n\u0026nbsp;\u0026nbsp;\"application\": \"ILAB\"  \n}\n\n\n## EPOS\nmetadataPrefix = **ISO**\n\n*-Owner organisation is determined from within harvested data*  \nNO new organisations are added programmatically when processing the harvested data.  \nOtto Lange, CKAN admin, controls this proces in the sense that he controls which organisations are known within CKAN.  \nIf a passed organisation is present in CKAN already, that organisation is used as owner organisatin.  \nIf not known within CKAN -\u003e ‘Other lab’ is default.\n\nThus, organization ‘Other lab’ is an indicator of datapackages from unkown origins (from the point of view of CKAN).\n\n\n\n*-Maintainer is derived from harvest source organisation*  \nSimilarly like in iLAB the harvest source organisation is used.\nFor iLAB the harvest source organisation is used as owner organisation of a datapackage.\nFOr EPOS however, the harvest source organisation is used maintainer of the datapackage.\nConsequence of this is that maintainer organisations need to be added to CKAN as organisations. Otherwise, it is not possible to add the organisation to a harvest source. And consequently use it as maintainer regarding the harvested datapackage.\n\n        self.package_dict['maintainer'] = harvest_source.organisation\n        self.package_dict['maintainer_email'] = harvest_source.organisation.email\n\n\n\n*-Specific EPOS 'package reference' handling *  \nWithin the EPOS application there are 3 types of references regarding a package:\n\n-supplement to  \n-cites  \n-references  \n\nWithin the harvested XML that minimal information is passed regarding these references.  \nTherefore, lookup functionality is added to get more details via GFZ.\n\n\nhttp://dataservices.gfz-potsdam.de/getcitationinfo.php?doi=http://dx.doi.org/' op basis van DOI\n\nExample:  \nhttp://dataservices.gfz-potsdam.de/getcitationinfo.php?doi=http://dx.doi.org/10.1007/s11214-010-9693-4\n\nBased upon the received citation info, the CKAN package can be filled with more elaborate information.\n\nThis extra provisioning was deliberately created for EPOS handling - GFZ specifically.\nIn order run this functionality, it must be added to the corresponding harvest source configuration.\n\n\nexample configuration for harvest source where extra information will be recovered for citations.\n\n{  \n\u0026nbsp;\u0026nbsp;\u0026nbsp;\"metadata_prefix\": \"iso19139\",  \n\u0026nbsp;\u0026nbsp;\u0026nbsp;\"application\": \"EPOS\",  \n\u0026nbsp;\u0026nbsp;\u0026nbsp;**\"collect_extra_info_from_gfz\"**:\"http://dataservices.gfz-potsdam.de/getcitationinfo.php?doi=http://dx.doi.org/\",  \n\u0026nbsp;\u0026nbsp;\u0026nbsp;\"set\":\"~P3E9c3ViamVjdCUzQSUyMm11bHRpLXNjYWxlK2xhYm9yYXRvcmllcyUyMg\"  \n}\n\nwhen 'collect extra info' is left out, there will be not retrieval for this information and a simple list of doi's will be presented.\n\n\n\n*Email addresses for maintainers*   \nOrganisations (labs in CKAN) that will be used as maintainers, will have to have an email address as well.   \nWithin CKAN this can only be done by adding extra fields. It is not a standard field on organization level.  \nTherefore use extra fields on organization level:\n\nextras: [  \n\u0026nbsp;{  \n  \u0026nbsp;\u0026nbsp;\u0026nbsp;**value: \"hdr@hdr.nl\"**,  \n  \u0026nbsp;\u0026nbsp;\u0026nbsp;state: \"active\",  \n  \u0026nbsp;\u0026nbsp;\u0026nbsp;**key: \"email\"**,  \n  \u0026nbsp;\u0026nbsp;\u0026nbsp;revision_id: \"b2356692-7f95-475e-88d3-1390451ccec7\",  \n  \u0026nbsp;\u0026nbsp;\u0026nbsp;group_id: \"7ad73491-aec5-4379-ae2c-44e03d05eaf4\",  \n  \u0026nbsp;\u0026nbsp;\u0026nbsp;id: \"2426a9b8-bfa3-4790-b5f9-f78d0d3eb2ee\"  \n\u0026nbsp;}  \n]\n\n\n\n## Standard CKAN fields\n**Standard CKAN fields**\n\n-author  \nAuthor of the package.  \n-owner_org    \nOrganisation that owns the package and as such shown on the datapackage view  \n-title  \nTitle of a package  \n-notes  \nDescription of a package  \n-license_id  \nThe license under which the datapackage can be used.  \n-url  \nURL referring to more detailed information regarding the datapackage  \n-groups  \nA datapackage can be linked to N groups.  \n-tags  \nA datapackage can be tagged to…  \n-formats  \n-maintainer  \nReference to where the CKAN package is maintained.  \n-maintainer_email  \n-extras\nKey value pairs enabling a CKAN programmer to customise datapackage data to\n\n**Extra fields for iLAB**:  \nLocations covered  \nDataset contact  \nCreated at repository  \nYear of publicatoin  \nPublisher  \nCollection period  \n\n## CKANEXT-CUSTOM-THEMES\nMainly frontend items that transform standard CKAN into a tailored application. Eg. the iLAB catalog.  \nThis by overruling view files,  add static content like images and javascript.\n\n## License\nThis project is licensed under the GPL-v3 license.\nThe full license can be found in [LICENSE](LICENSE).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Futrechtuniversity%2Filab-catalog","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Futrechtuniversity%2Filab-catalog","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Futrechtuniversity%2Filab-catalog/lists"}