{"id":21620780,"url":"https://github.com/miku/oaicrawl","last_synced_at":"2026-03-11T18:02:40.502Z","repository":{"id":64318913,"uuid":"103181308","full_name":"miku/oaicrawl","owner":"miku","description":"OAI crawler for strange endpoints.","archived":false,"fork":false,"pushed_at":"2019-05-08T16:15:02.000Z","size":76,"stargazers_count":5,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-03-25T06:33:15.865Z","etag":null,"topics":["code4lib","oai"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/miku.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-09-11T19:53:40.000Z","updated_at":"2020-07-17T14:53:57.000Z","dependencies_parsed_at":"2023-01-15T12:00:47.224Z","dependency_job_id":null,"html_url":"https://github.com/miku/oaicrawl","commit_stats":null,"previous_names":["miku/haprot"],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/miku%2Foaicrawl","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/miku%2Foaicrawl/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/miku%2Foaicrawl/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/miku%2Foaicrawl/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/miku","download_url":"https://codeload.github.com/miku/oaicrawl/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248365351,"owners_count":21091773,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["code4lib","oai"],"created_at":"2024-11-24T23:12:48.422Z","updated_at":"2025-12-15T18:31:57.924Z","avatar_url":"https://github.com/miku.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"oaicrawl\n========\n\nHarvest an OAI endpoint by fetching records one by one. This is a different\nstrategy from the windowed and cached approach used in\n[metha](https://github.com/miku/metha).\n\nInstall via *go get* or [releases](https://github.com/miku/oaicrawl/releases).\n\noaicrawl does not cache anything and will write the raw responses directly to\nstandard output.\n\n```shell\n$ oaicrawl -f oai_dc -verbose http://www.academicpub.org/wapoai/OAI.aspx \u003e harvest.data\n...\nDEBU[0025] fetched 1395 identifiers with 1 requests in 25.140568087s\n```\n\nUse the `-b` flag to crawl in a best effort way (continue in the presence of\nerrors):\n\n```shell\n$ oaicrawl -b -f oai_dc -verbose http://www.academicpub.org/wapoai/OAI.aspx \u003e harvest.data\n...\n... worker-11 backoff [3]: ..ertasacademica.com/5318\u0026metadataPrefix=oai_dc\n\n```\n\nThis crawler was written for working with endpoints that are slightly\noff-standard and cannot be harvested easily in chunks.\n\nTest it yourself (might take a day to harvest completely):\n\n```shell\n$ oaicrawl -f mets -b -verbose http://zvdd.de/oai2/\n...\n```\n\nUsage\n-----\n\n```shell\n$ oaicrawl -h\nUsage of oaicrawl:\n  -b    create best effort data set\n  -e duration\n        max elapsed time (default 10s)\n  -f string\n        format (default \"oai_dc\")\n  -retry int\n        max number of retries (default 3)\n  -verbose\n        more logging\n  -version\n        show version\n  -w int\n        number of parallel connections (default 16)\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmiku%2Foaicrawl","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmiku%2Foaicrawl","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmiku%2Foaicrawl/lists"}