{"id":41894768,"url":"https://github.com/wimmuskee/shell-oaiharvester","last_synced_at":"2026-01-25T14:19:07.332Z","repository":{"id":2287178,"uuid":"3244797","full_name":"wimmuskee/shell-oaiharvester","owner":"wimmuskee","description":"OAI-PMH harvester in shell.","archived":false,"fork":false,"pushed_at":"2025-12-23T07:45:35.000Z","size":149,"stargazers_count":17,"open_issues_count":2,"forks_count":4,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-12-24T22:14:34.707Z","etag":null,"topics":["bash","command-line-tool","oai-harvester","oai-pmh"],"latest_commit_sha":null,"homepage":"","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/wimmuskee.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"COPYING","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2012-01-23T06:55:05.000Z","updated_at":"2025-12-23T07:45:13.000Z","dependencies_parsed_at":"2022-07-19T15:04:13.188Z","dependency_job_id":null,"html_url":"https://github.com/wimmuskee/shell-oaiharvester","commit_stats":null,"previous_names":[],"tags_count":17,"template":false,"template_full_name":null,"purl":"pkg:github/wimmuskee/shell-oaiharvester","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wimmuskee%2Fshell-oaiharvester","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wimmuskee%2Fshell-oaiharvester/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wimmuskee%2Fshell-oaiharvester/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wimmuskee%2Fshell-oaiharvester/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/wimmuskee","download_url":"https://codeload.github.com/wimmuskee/shell-oaiharvester/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wimmuskee%2Fshell-oaiharvester/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28754159,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-25T13:59:49.818Z","status":"ssl_error","status_checked_at":"2026-01-25T13:59:33.728Z","response_time":113,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bash","command-line-tool","oai-harvester","oai-pmh"],"created_at":"2026-01-25T14:19:06.733Z","updated_at":"2026-01-25T14:19:07.324Z","avatar_url":"https://github.com/wimmuskee.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Shell OAI Harvester\nThe OAI-PMH Shell Harvester is able to harvest OAI-PMH targets. It supports multiple configurable targets which can be updated individually. Furthermore, it is able to execute a preset command for each record it updates or deletes.\nView the [CHANGELOG](CHANGELOG.md) for important changes.\n\n## Installation\nUse sudo/root where needed.\n\n### Dependencies\n - bash\n - curl\n - coreutils (cat, cut, head, date, printf, test)\n - grep\n - sed\n - xmllint (optional for validating responses)\n - xsltproc (libxslt)\n\n### Manual\n - git clone https://github.com/wimmuskee/shell-oaiharvester.git\n - cd shell-oaiharvester\n - cp oaiharvester /usr/bin/oaiharvester\n - mkdir /usr/share/shell-oaiharvester\n - cp libs/* /usr/share/shell-oaiharvester/.\n - mkdir /etc/shell-oaiharvester\n - cp config.example.xml /etc/shell-oaiharvester/config.xml\n\n### Make\n - git clone https://github.com/wimmuskee/shell-oaiharvester.git\n - cd shell-oaiharvester\n - autoreconf -i\n - ./configure\n - make\n - make install\n\n### Gentoo\nPackage available in the flavour overlay.\n\n## Usage\n\n### Configuration\nThe harvester has several configuration options that apply to the general behaviour while other options are repository specific.\nDifferent config files can be used with different levels of precedence (without fallback).\n1. commandline option provided location: `oaiharvester -c \u003cconfig file\u003e`\n2. *$HOME/.config/shell-oaiharvester/config.xml*\n3. */etc/shell-oaiharvester/config.xml*\n\nSome notes on the options:\n - deletecmd: executed before a record is deleted, optional\n - updatecmd: executed after a record is updated, optional\n   - ${identifier} holds the identifier\n   - ${filename) is the identifier with optional xz extension\n   - ${path} holds the absolute path of the record\n   - ${responsedatetime} the full oai page responsedate\n   - ${responsedate} the YYYY-MM-DD format of responsedatetime\n - repository update and delete cmd's are executed before the generic ones if available\n - set, from, until, conditional and recordpath are optional\n\n### Operation\nView all available options:\n```oaiharvester -h```\n\nTo harvest records from a specified repository, run:\n```oaiharvester -r \u003crepository_id\u003e```\n\nTo only retrieve identifiers:\n```oaiharvester -r \u003crepository_id\u003e -n```\n\nNot harvest, but validate the repository\n```oaiharvester -r \u003crepository_id\u003e -t```\n\nTo perform strict validation use:\n```oaiharvester -r \u003crepository_id\u003e --test-strict```\n\nTo list configured repositories:\n```oaiharvester -l```\n\n### Logs\nUnless customized, the log file of the harvest process is stored at */tmp/oaiharvester-log.csv*. The logger uses a simple csv format with a line for each downloaded page:\n```YYYY-MM-DD HH:MM:SS,PID,repository,record count,download time,process time```.\n\nA recordlog can also be set, by default at */dev/null* because depending on repositories this file can increase in size a lot. This has the following format:\n```YYYY-MM-DD HH:MM:SS,repository,record datestamp,record identifier```.\n\n### Status\nOnce harvested, each repository directory will hold a status file. In version 1, this is *lasttimestamp.txt* containing just the timestamp where the harvester should continue.\nStarting from version 2, a hidden *.oaiharvester* file is used to contain not only the timestamp, but also harvester version information which should help with future migrations.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwimmuskee%2Fshell-oaiharvester","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwimmuskee%2Fshell-oaiharvester","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwimmuskee%2Fshell-oaiharvester/lists"}