{"id":21620883,"url":"https://github.com/miku/metha","last_synced_at":"2026-01-04T11:46:47.547Z","repository":{"id":37819530,"uuid":"56384577","full_name":"miku/metha","owner":"miku","description":"Command line OAI-PMH harvester and client with built-in cache.","archived":false,"fork":false,"pushed_at":"2024-04-08T10:40:01.000Z","size":79110,"stargazers_count":113,"open_issues_count":14,"forks_count":13,"subscribers_count":8,"default_branch":"master","last_synced_at":"2024-04-09T10:50:45.070Z","etag":null,"topics":["code4lib","hacktoberfest","harvest","oai"],"latest_commit_sha":null,"homepage":"https://lab.ub.uni-leipzig.de/metha/","language":"HTML","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/miku.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2016-04-16T13:18:58.000Z","updated_at":"2024-04-23T14:30:03.184Z","dependencies_parsed_at":"2023-02-16T18:15:49.550Z","dependency_job_id":"c40f9e6a-84d5-427f-a088-bdbc618419de","html_url":"https://github.com/miku/metha","commit_stats":null,"previous_names":[],"tags_count":115,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/miku%2Fmetha","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/miku%2Fmetha/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/miku%2Fmetha/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/miku%2Fmetha/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/miku","download_url":"https://codeload.github.com/miku/metha/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244279407,"owners_count":20427679,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["code4lib","hacktoberfest","harvest","oai"],"created_at":"2024-11-24T23:13:10.476Z","updated_at":"2026-01-04T11:46:47.525Z","avatar_url":"https://github.com/miku.png","language":"HTML","funding_links":[],"categories":[],"sub_categories":[],"readme":"# metha\n\n\u003e The Open Archives Initiative Protocol for Metadata Harvesting (OAI-PMH) is a\n\u003e low-barrier mechanism for repository interoperability. Data Providers are\n\u003e repositories that expose structured metadata via OAI-PMH. Service Providers\n\u003e then make OAI-PMH service requests to harvest that metadata. -- https://www.openarchives.org/pmh/\n\nThe metha command line tools can gather information on OAI-PMH endpoints and\nharvest data incrementally. The goal of metha is to make it simple to get\naccess to data, its focus is not to manage it.\n\n[![DOI](https://zenodo.org/badge/56384577.svg)](https://zenodo.org/badge/latestdoi/56384577) [![Project Status: Active – The project has reached a stable, usable state and is being actively developed.](https://www.repostatus.org/badges/latest/active.svg)](https://www.repostatus.org/#active)\n\nThe metha tool has been developed for [project finc](https://finc.info) at\n[Leipzig University Library](https://ub.uni-Leipzig.de) ([lab](https://lab.ub.uni-leipzig.de/metha/)).\n\n## Why yet another OAI harvester?\n\n* I wanted to crawl [Arxiv](http://export.arxiv.org/oai2) but found that existing tools would timeout.\n* Some harvesters would start to download all records anew, if I interrupted a running harvest.\n* There are many OAI\n  [endpoints](https://raw.githubusercontent.com/miku/metha/master/contrib/sites.tsv) out\n  there. It is a widely used\n  [protocol](http://www.openarchives.org/OAI/openarchivesprotocol.html) and\n  somewhat worth knowing.\n* I wanted something simple for the command line; also fast and robust - metha\n  as it is implemented now, is relatively robust and more efficient than\n  requesting all record one-by-one (there is one\n  [annoyance](https://github.com/miku/metha/issues/6) which will hopefully be\n  fixed soon).\n\n## How it works\n\nThe functionality is spread accross a few different executables:\n\n* metha-sync for harvesting\n* metha-cat for viewing\n* metha-id for gathering data about endpoints\n* metha-ls for inspecting the local cache\n* metha-files for listing the associated files for a harvest\n\nTo harvest and endpoint in the default *oai_dc* format:\n\n```sh\n$ metha-sync http://export.arxiv.org/oai2\n...\n```\n\nAll downloaded files are written to a directory below a base directory. The base\ndirectory is `~/.cache/metha` by default and can be adjusted with the `METHA_DIR`\nenvironment variable.\n\nWhen the `-dir` flag is set, only the directory corresponding to a harvest is printed.\n\n```\n$ metha-sync -dir http://export.arxiv.org/oai2\n/home/miku/.metha/I29haV9kYyNodHRwOi8vZXhwb3J0LmFyeGl2Lm9yZy9vYWky\n```\n\n```sh\n$ METHA_DIR=/tmp/harvest metha-sync -dir http://export.arxiv.org/oai2\n/tmp/harvest/I29haV9kYyNodHRwOi8vZXhwb3J0LmFyeGl2Lm9yZy9vYWky\n```\n\nThe harvesting can be interrupted at any time and the HTTP client will\nautomatically retry failed requests a few times before giving up.\n\nCurrently, there is a limitation which only allows to harvest data up to the\nlast day. Example: If the current date would be *Thu Apr 21 14:28:10 CEST\n2016*, the harvester would request all data since the repositories earliest\ndate and *2016-04-20 23:59:59*.\n\nTo stream the harvested XML data to stdout run:\n\n```sh\n$ metha-cat http://export.arxiv.org/oai2\n```\n\nYou can emit records based on datestamp as well:\n\n```sh\n$ metha-cat -from 2016-01-01 http://export.arxiv.org/oai2\n```\n\nThis will only stream records with a datestamp equal or after 2016-01-01.\n\nTo just stream all data really fast, use `find` and `zcat` over the harvesting\ndirectory.\n\n```sh\n$ find $(metha-sync -dir http://export.arxiv.org/oai2) -name \"*gz\" | xargs unpigz -c\n```\n\nTo display basic repository information:\n\n```sh\n$ metha-id http://export.arxiv.org/oai2\n```\n\nTo list all harvested endpoints:\n\n```sh\n$ metha-ls\n```\n\nFurther examples can be found in the metha [man page](https://github.com/miku/metha/blob/master/docs/metha.md):\n\n```\n$ man metha\n```\n\n## Installation\n\nUse a deb, rpm [release](https://github.com/miku/metha/releases), or the go tool:\n\n```sh\n$ go install -v github.com/miku/metha/cmd/...@latest\n```\n\n## Limitations\n\nCurrently the endpoint URL, the format and the set are concatenated and base64\nencoded to form the target directory, e.g:\n\n```\n$ echo \"U291bmRzI29haV9kYyNodHRwOi8vY29wYWMuamlzYy5hYy51ay9vYWktcG1o\" | base64 -d\nSounds#oai_dc#http://copac.jisc.ac.uk/oai-pmh\n```\n\nIf you have very long set names or a very long URL and the target directory\nexceeds e.g. 255 chars (on ext4), the harvest won't work.\n\n## Harvesting Roulette\n\n```sh\n$ URL=$(shuf -n 1 \u003c(curl -Lsf https://git.io/vKXFv)); metha-sync $URL \u0026\u0026 metha-cat $URL\n```\n\nIn 0.1.27 a `metha-fortune` command was added, which fetches a random article\ndescription and displays it.\n\n```shell\n$ metha-fortune\nActive Networking is concerned with the rapid definition and deployment of\ninnovative, but reliable and robust, networking services. Towards this end we\nhave developed a composite protocol and networking services architecture that\nencourages re-use of protocol functions, is well defined, and facilitates\nautomatic checking of interfaces and protocol component properties. The\narchitecture has been used to implement common Internet protocols and services.\nWe will report on this work at the workshop.\n\n    -- http://drops.dagstuhl.de/opus/phpoai/oai2.php\n\n$ metha-fortune\nIn this paper we show that the Lempert property (i.e., the equality between the\nLempert function and the Carathéodory distance) holds in the tetrablock, a\nbounded hyperconvex domain which is not biholomorphic to a convex domain. The\nquestion whether such an equality holds was posed by Abouhajar et al. in J.\nGeom. Anal. 17(4), 717–750 (2007).\n\n    -- http://ruj.uj.edu.pl/oai/request\n\n$ metha-fortune\nI argue that Gödel's incompleteness theorem is much easier to understand when\nthought of in terms of computers, and describe the writing of a computer\nprogram which generates the undecidable Gödel sentence.\n\n    -- http://quantropy.org/cgi/oai2\n\n$ metha-fortune\nNigeria, a country in West Africa, sits on the Atlantic coast with a land area\nof approximately 90 million hectares and a population of more than 140 million\npeople. The southern part of the country falls within the tropical rainforest\nwhich has now been largely depleted and is in dire need of reforestation. About\n10 percent of the land area was constituted into forest reserves for purposes\nof conservation but this has suffered perturbations over the years to the\nextent that what remains of the constituted forest reserves currently is less\nthan 4 percent of the country land area. As at today about 382,000 ha have been\nreforested with indigenous and exotic species representing about 4 percent of\nthe remaining forest estate. Regrettably, funding of the Forestry sector in\nNigeria has been critically low, rendering reforestation programme near\nimpossible, especially in the last two decades. To revive the forestry sector\ngovernment at all levels must re-strategize and involve the local communities\nas co-managers of the forest estates in order to create mutual dependence and\ninteraction in resource conservation.\n\n    -- http://journal.reforestationchallenges.org/index.php/REFOR/oai\n```\n\n## Scrape all metadata in a best-effort way\n\nUse an endless loop with a timeout to get out of any hanging connection (which\nhappen). Example scrape, converted to JSON (326M records, 60+ GB:\n[2023-11-01-metha-oai.ndjson.zst](https://archive.org/download/oai_harvest_2023-11-01/2023-11-01-metha-oai.ndjson.zst)).\n\n```shell\n$ while true; do \\\n    timeout 120 metha-sync -list | \\\n    shuf | \\\n    parallel -j 64 -I {} \"metha-sync -base-dir ~/.cache/metha {}\"; \\\ndone\n```\n\nAlternatively, use a\n[metha.service](https://raw.githubusercontent.com/miku/metha/master/extra/linux/metha.service)\nfile to run harvests continuously.\n\nmetha stores harvested data in one file per interval; to combine all XML files\ninto a single JSON file you can utilize the\n[xmlstream.go](https://github.com/miku/metha/blob/master/extra/largecrawl/xmlstream.go) (adjust the harvest directory):\n\n```shell\n$ fd . '/data/.cache/metha' -e xml.gz | parallel unpigz -c | xmlstream -D\n```\n\nFor notes on parallel processing of XML see: [Faster XML processing in Go](https://golangleipzig.space/posts/faster-go-xml-processing/).\n\n![](docs/metha-net-zenith.png)\n\n## Errors this harvester can somewhat handle\n\n* responses with resumption tokens that lead to empty responses\n* gzipped responses, that are not advertised as such\n* funny (illegal) control characters in XML responses\n* repositories, that won't respond unless the dates are given with the exact granualarity\n* repositories with endless token loops\n* repositories that do not support selective harvesting, use `-no-intervals` flag\n* limited repositories, metha will try a few times with an exponential backoff\n* repositories, which throw occasional HTTP errors, although most of the responses look good, use `-ignore-http-errors` flag\n\n## Authors\n\n* Martin Czygan \u003cmartin.czygan@uni-leipzig.de\u003e\n* Natanael Arndt, [https://github.com/white-gecko](https://github.com/white-gecko)\n* Gunnar Þór Magnússon, [https://github.com/gunnihinn](https://github.com/gunnihinn)\n* Thomas Gersch, [https://github.com/titabo2k](https://github.com/titabo2k)\n* [Andreas Czerniak](https://github.com/ACz-UniBi)\n* [David Glück](https://github.com/dvglc)\n* [Justin Kelly](https://github.com/justinkelly)\n\n## Misc\n\nShow formats of random repository:\n\n```shell\n$ shuf -n 1 \u003c(curl -Lsf https://git.io/vKXFv) | xargs -I {} metha-id {} | jq .formats\n```\n\nA snippet from a 2010 publication:\n\n\u003e The Open Archives Protocol for Metadata Harvesting\n(OAI-PMH) (Lagoze and van de Sompel, 2002) is currently implemented by more\nthan 1,700 digital library reposi- tories world-wide and enables the exchange\nof metadata via HTTP. -- [Interweaving OAI-PMH Data Sources with the Linked Data Cloud](http://eprints.cs.univie.ac.at/73/1/ijmso2010_haslhofer_schandl.pdf)\n\n## Metha elsewhere\n\n* [The finc project](https://finc.info/de/datenquellen)\n* [UB LEIPZIG LAB](https://lab.ub.uni-leipzig.de/metha/)\n* [Getting a dump of arXiv metadata](https://academia.stackexchange.com/questions/38969/getting-a-dump-of-arxiv-metadata) at [academia.stackexchange.com](https://academia.stackexchange.com/)\n* [Keyword Extraction from arXiv - Part 1](http://web.archive.org/web/20191111162743/http://akumano.site/posts/arxiv-keyword-extraction-part1/)\n* [Openrefine use case: Automated workflow for harvesting, transforming and indexing of bibliographic metadata](https://groups.google.com/forum/#!topic/openrefine/RqQwlF-ll1c)\n* [Sammeln und Finden. Über das Sichtbarmachen von Open Science in Hamburg](https://opus4.kobv.de/opus4-bib-info/files/3645/HOS+Bibliothekartag.pdf) (PDF)\n* [acohan/arxiv-tools](https://github.com/acohan/arxiv-tools)\n* [Arxiv on Archive](https://archive.org/details/arxiv-bulk-metadata)\n* [Metadata analysis of 80,000 arxiv:physics/astro-ph articles](https://quantumdynamics.wordpress.com/2016/06/12/metadata-analysis-of-80000-arxivastro-ph-articles-reveals-biased-moderation/)\n* [Arxiv Harvesting](https://trislee.com/arxiv/)\n* [Orcid](https://trello.com/c/3OrWa2ZY/5771-load-issn-metadata-into-registry-db-8)\n* [Tutorial on Topological Data Analysis](https://github.com/shizuo-kaji/TutorialTopologicalDataAnalysis/blob/master/NLP_example.md)\n* [Connectome](https://cms.www.switch.ch/fr/connectome/) (linked open research data, [#28](https://github.com/miku/metha/issues/28#issuecomment-1144526453)), [From MARCXML to Records in Contexts](https://zenodo.org/record/7400442/files/SWITCH%20Patrinum%20RiC_20221205_Wildi.pdf)\n* [Comparison w/ other OAI tools](https://github.com/Deutsche-Digitale-Bibliothek/ddblabs-ometha#geschwindigkeit) (de)\n* [biblio.ai](https://github.com/biblio-ai/extract)\n\n## Asciicast\n\n[![asciicast](https://asciinema.org/a/271660.svg)](https://asciinema.org/a/271660)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmiku%2Fmetha","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmiku%2Fmetha","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmiku%2Fmetha/lists"}