{"id":20570058,"url":"https://github.com/digibib/marc2rdf","last_synced_at":"2026-01-10T19:52:03.122Z","repository":{"id":140037204,"uuid":"2101455","full_name":"digibib/marc2rdf","owner":"digibib","description":"MARC to RDF toolkit - converter and harvester through json mapping","archived":true,"fork":false,"pushed_at":"2015-07-10T10:49:54.000Z","size":2968,"stargazers_count":36,"open_issues_count":0,"forks_count":7,"subscribers_count":13,"default_branch":"master","last_synced_at":"2025-03-21T18:51:58.377Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/digibib.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2011-07-25T14:29:40.000Z","updated_at":"2024-10-30T08:42:25.000Z","dependencies_parsed_at":"2023-03-13T02:16:53.803Z","dependency_job_id":null,"html_url":"https://github.com/digibib/marc2rdf","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/digibib%2Fmarc2rdf","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/digibib%2Fmarc2rdf/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/digibib%2Fmarc2rdf/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/digibib%2Fmarc2rdf/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/digibib","download_url":"https://codeload.github.com/digibib/marc2rdf/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253312026,"owners_count":21888559,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-16T05:11:14.564Z","updated_at":"2025-05-09T19:30:34.751Z","avatar_url":"https://github.com/digibib.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# MARC bibliographic record to RDF converter\n\n    MARC2RDF - a ruby toolkit to convert bibliographic MARC to RDF by YAML mapping\n    Copyright (C) 2014 Benjamin Rokseth\n    Purpose: Convert binary/xml marc to semantic markup using yaml mapping file\n             Import into RDF triplestore and maintain via OAI-PMH harvesting\n\n## GPLv3 LICENSE\n    \n    This program is free software: you can redistribute it and/or modify\n    it under the terms of the GNU General Public License as published by\n    the Free Software Foundation, either version 3 of the License, or\n    (at your option) any later version.\n\n    This program is distributed in the hope that it will be useful,\n    but WITHOUT ANY WARRANTY; without even the implied warranty of\n    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the\n    GNU General Public License for more details.\n\n    You should have received a copy of the GNU General Public License\n    along with this program.  If not, see \u003chttp://www.gnu.org/licenses/\u003e.\"\n\n### Architecture\n![API architecture](https://github.com/digibib/marc2rdf/raw/master/doc/schematics.png)\n\n## HOW IT WORKS\n\nThe webapp is built upon the notion of syncing one or more bibliographic repositories, `Libraries`, to an updated \nRDF triplestore. Each `Library` connects to an OAI-PMH repository (bibliographic metadata) service and harvests \nmetadata once or regularly, maps it to RDF in the `mapping` module, and converts it to RDF and inserts it to\na triplestore via the `conversion` module.\n\nIn addition there is a `rules` module, in which SPARQL rules can be made for updating/modifying RDFstore on the fly, \neither once or regularly. And a `harvester` module, which can be used to harvest metadata from other web APIs, \nbased on OAI harvested content or existing RDF.\n\n### HOW TO RUN\n\nA typical setup would be:\n\n* Setup a RDF store and edit settings.json to reflect admin rights\n* start app: `foreman start` (will default to 'development' mode, reading 'Procfile')\n* Create a `Library` and fill in settings such as OAI endpoint, tag and resource and default graph\n* Validate OAI, choose specific set if wanted\n* Create a `Mapping` either from scratch or clone existing\n* Test mapping against library in the `Conversion` module, by inputting a known id from the OAI repo\n* Modify `Mapping` and test until expected results\n* Alternatively, generate `Rules` to massage data after harvest and activate on `Library`\n* Alternatively, generate `Harvest` to harvest further metadata from other sources and activate on `Library`\n* Set OAI harvest schedule to automate process daily \n\n\n## FEATURES\n\nThe marc2rdf toolkit consists of a web frontend and three parts:\n\n* app.rb       - Main Sinatra webapp\n* api.rb       - Grape API for RESTful interaction between app and browser \n* scheduler.rb - Rufus Scheduler to manage job/cron queues\n\nScheduler takes these job types:\n\n* Single oai harvest (job)\n* Recurring oai harvest (cronjob)\n* Single SPARQL job (isql job) \n* Recurring SPARQL job (isql cronjob)\n\nFull harvest is generally recommended done in two steps and not enabled in web frontend.\nTo do this you need to fire two RESTful requests\n\n* Full OAI harvest. This will harvest entire repo from beginning of time to XML dumps\n(all saved OAI responses will end up in './db/converted/full')\n\n    http PUT http://localhost:3000/api/oai/harvest_full \\\n      SECRET_SESSION_KEY:'secretsessionkey' id=\u003cid of library\u003e\n\n* Full OAI conversion. This will use full XML dumps harvested in above step.\nIt will also run any rules and harvester rules activated on library\n\n    http PUT http://localhost:3000/api/oai/convert_full \\\n      SECRET_SESSION_KEY:'secretsessionkey' id=\u003cid of library\u003e\n      sparql_update=bool write_records=bool\n\n### REQUIREMENTS\n\n* ruby \u003e= 1.9.3, recommended installed via Ruby Version Manager (rvm)\n* ruby bundler \u003e= 1.3.5, install by `gem install bundler`\n\nRequirements for tools/marcfix.pl\n\n * Perl\n * MARC::Record perl module from [CPAN](http://search.cpan.org/~gmcharlt/MARC-Record-2.0.3/lib/MARC/Record.pm)\n\n### INSTALL\n\nHere is a short walk-trough on how to install the needed tools and libraries.\n\n1. Clone this repository form github  \n\t* ```git://github.com/digibib/marc2rdf.git```  \n\tThis creates a subdirectory called marc2rdf\n\n1. Either install ruby via rvm (Ruby Version Manager, instructions at https://rvm.io/rvm/install) or install ruby-dev  \n\t*  Ubuntu install (for ruby and rdf-xml support)  \n\t```sudo apt-get install ruby-dev libxml2-dev libxslt1-dev libyaml-ruby libzlib-ruby rubygems```  \n\t* Debian note  \n\tDebian adds a version postfix to the ruby executables. Thus all references to `ruby` becomes \n\t`ruby1.8` and references to `gem` becomes `gem1.8`.\n\n1. Install/Update RubyGems bundler  \n\t```sudo gem install bundler```  \n\tIf you can not or do not want to install RubyGems into system folder locations, please have a look at \n\thttp://docs.rubygems.org/read/chapter/3/\n\n1. Install needed gems given in Gemfile:  \n\t```\n\tcd marc2rdf\n\tbundle install\n\t```\n\n### Configuration\n\n  Most configurations are made within app, but for startup repository connection and login settings, \n  copy configuration file  :\n\t```\n\tcp ./config/example.settings.json ./config/settings.json\n\t```  \n\t* Make changes to the new files as needed to fit your system\n  * Set login username and password \n  * Port and host settings are in Procfile (development) and Procfile.production (production)\n\n  setup of libraries, mappings, rules, harvests \u0026 vocabularies are made within app, \n  but for convenience we have added our current setup in the following gist:\n  https://gist.github.com/anonymous/6206748\n\n### Production\n\n  When ready for production mode, the included Procfile.production can be used as an example:\n  \n  `foreman start -f Procfile.production`\n  \n  Webapp should then run blazingly fast, as pages are no longer reloaded each load. (Development\n  mode reloads app on each page load).\n  \n  Foreman can also be used to create upstart jobs easily (managed by linux system and respawned if down):\n  \n  `rvmsudo foreman export upstart /etc/init -p Procfile.production -a marc2rdf`\n  \n  Read more about foreman at: http://ddollar.github.io/foreman/\n  \n### Binary MARC\n  \nUsage of binary MARC is generally not recommended, and only supported in single batch conversion. But you'll need to:\n\n1. Install Perl and MARC::Record  \n\tOn Debian and Ubuntu these are installed with apt:  \n\t```\n\tapt-get install perl libmarc-record-perl\n\t```  \n\tMARC::Record can also be installed with \n\t[CPAN](http://search.cpan.org/~gmcharlt/MARC-Record-2.0.3/lib/MARC/Record.pm)  \n\n### RDF STORE\n\n * Recommended triplestore is OpenLink Virtuoso, minimum version v6.1.4, recommended v7.0.0\n * Rules Engine is based on isql Sparql scripts, so rules will only work on virtuoso for now \n\n#### Ubuntu\n\n```\nsudo apt-get install virtuoso-opensource\n```\n\n#### Debian\n\nDebian squeeze does not have a recent version of Virtuoso. Please compile from source as described \nover at [their site](http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/VOSDebianNotes). \n\nDebian wheezy comes with Virtuoso version 6.1.4, as does Ubuntu 12.04 Precise.\n\n#### Mapping extras\n\n* tag numbers can be regex (e.g. \"^5(?!71)\" for 500-599 minus 571)\n* predicates are given in format PREFIX.suffix and non-standard prefixes must exist in ./lib/vocabularies.rb\n* objects prefixes must be exploded \n* predicates can be conditionally mapped from subfields or indicators\n* objects can have language tags given as symbols (:se, :en_UK etc)\n* objects can be mapped key =\u003e values\n* relations can have subfields\n* string replace non-ascii characters to create uris\n* oai harvester uses same mapping\n* erroneous 000 marc field in normarc can be removed with perl script ./tools/marcfix.pl\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdigibib%2Fmarc2rdf","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdigibib%2Fmarc2rdf","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdigibib%2Fmarc2rdf/lists"}