{"id":46824259,"url":"https://github.com/wellcomecollection/oai-pmh","last_synced_at":"2026-03-10T09:38:47.023Z","repository":{"id":311754570,"uuid":"1044817774","full_name":"wellcomecollection/oai-pmh","owner":"wellcomecollection","description":"Lightweight Python OAI-PMH client for harvesting records (arXiv-ready), with pluggable datestamp granularity.","archived":false,"fork":false,"pushed_at":"2025-12-01T15:07:41.000Z","size":226,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-12-04T03:38:51.125Z","etag":null,"topics":["glam","oai-pmh","python"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/wellcomecollection.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-08-26T08:55:50.000Z","updated_at":"2025-12-01T15:07:33.000Z","dependencies_parsed_at":null,"dependency_job_id":"6cd5b4c2-dcc0-4769-bba5-1eacd754f233","html_url":"https://github.com/wellcomecollection/oai-pmh","commit_stats":null,"previous_names":["kenoir/oai-pmh","wellcomecollection/oai-pmh"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/wellcomecollection/oai-pmh","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wellcomecollection%2Foai-pmh","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wellcomecollection%2Foai-pmh/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wellcomecollection%2Foai-pmh/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wellcomecollection%2Foai-pmh/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/wellcomecollection","download_url":"https://codeload.github.com/wellcomecollection/oai-pmh/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wellcomecollection%2Foai-pmh/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30328862,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-10T05:25:20.737Z","status":"ssl_error","status_checked_at":"2026-03-10T05:25:17.430Z","response_time":106,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["glam","oai-pmh","python"],"created_at":"2026-03-10T09:38:45.953Z","updated_at":"2026-03-10T09:38:46.885Z","avatar_url":"https://github.com/wellcomecollection.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# OAI-PMH Client\n\nA modern Python client for OAI-PMH (Open Archives Initiative Protocol for Metadata Harvesting).\n\n## Installation\n\nThis project uses `uv` for package management. To install the client and its dependencies, you can use the following commands:\n\n```bash\nuv venv\nsource .venv/bin/activate\nuv pip install .\n```\n\n## Documentation\n\nFull documentation is available at [https://wellcomecollection.github.io/oai-pmh/](https://wellcomecollection.github.io/oai-pmh/).\n\n### Updating the documentation\n\nThe static HTML docs under `docs/pdoc/` are generated with [pdoc](https://pdoc.dev/). To refresh them after making code changes:\n\n1. Install the project (and dev extras if desired):\n\n    ```bash\n    uv pip install -e \".[dev]\"\n    ```\n\n2. Regenerate the documentation:\n\n    ```bash\n    uv run python -m pdoc oai_pmh_client --output-dir docs/pdoc --docformat google\n    ```\n\nThis overwrites the HTML (and supporting assets) inside `docs/pdoc/`. Commit the updated files if you want the published site to reflect the latest API.\n\n## Usage\n\nHere is a simple example of how to use the client:\n\n```python\nfrom oai_pmh_client.client import OAIClient\n\n# Create a client for the arXiv OAI-PMH endpoint.\nclient = OAIClient(\"https://oaipmh.arxiv.org/oai\")\n\n# Get the repository's identity.\nidentity = client.identify()\nprint(identity)\n\n# List the available metadata formats.\nformats = client.list_metadata_formats()\nprint(formats)\n\n# List the sets in the repository.\nsets = client.list_sets()\nprint(sets)\n```\n\n### More Examples\n\n#### Listing Records\n\nYou can list records with optional `from_date`, `until_date`, and `set_spec` filters.\n\n```python\nfrom datetime import datetime\n\n# List all records updated since the start of 2024 in the \"cs\" (Computer Science) set\nrecords = client.list_records(\n    metadata_prefix=\"oai_dc\",\n    from_date=datetime(2024, 1, 1),\n    set_spec=\"cs\"\n)\nfor record in records:\n    print(record.header.identifier, record.header.datestamp)\n```\n\n##### Datestamp granularity\n\nDifferent OAI-PMH repositories declare (via the `Identify` response) which datestamp granularity they accept for the `from` and `until` parameters:\n\n* `YYYY-MM-DD` (day-level)\n* `YYYY-MM-DDThh:mm:ssZ` (second-level, UTC)\n\nThe client automatically chooses the right format based on the `datetime` you provide: midnight values are sent with day-level precision, and any timestamp with a time component uses second-level precision. This makes it easy to harvest narrow windows (e.g. a few seconds) without additional configuration.\n\nIf you need to override the automatic behaviour—for example to force day-level timestamps for repositories that reject second-level granularity—you can set the `datestamp_granularity` argument when instantiating the client:\n\n```python\nclient = OAIClient(\"https://oaipmh.arxiv.org/oai\", datestamp_granularity=\"YYYY-MM-DD\")\n\nfrom datetime import datetime\nrecords = client.list_records(\n    metadata_prefix=\"oai_dc\",\n    from_date=datetime(2024, 1, 1, 12, 0, 0),  # still sent as 2024-01-01\n)\n```\n\nYou may also supply a pre-formatted string to override formatting entirely:\n\n```python\nrecords = client.list_records(\n    metadata_prefix=\"oai_dc\",\n    from_date=\"2024-01-01\",  # already correctly formatted\n)\n```\n\n#### Getting a Single Record\n\nRetrieve a single record by its identifier and a metadata prefix.\n\n```python\nrecord = client.get_record(\"oai:arXiv.org:2401.00001\", \"oai_dc\")\nprint(record.metadata)\n```\n\n#### Handling Deleted Records\n\nOAI-PMH repositories may return records that have been deleted. These records will have a header with `status=\"deleted\"` and no metadata. The client exposes this via the `is_deleted` property on the record header.\n\n```python\nrecords = client.list_records(metadata_prefix=\"oai_dc\")\n\nfor record in records:\n    if record.header.is_deleted:\n        print(f\"Record {record.header.identifier} has been deleted.\")\n        continue\n        \n    # Process active records\n    print(record.metadata)\n```\n\n#### Error Handling\n\nThe client will raise an `OAIError` subclass for errors returned by the OAI-PMH server.\n\n```python\nfrom oai_pmh_client.exceptions import IdDoesNotExistError\n\ntry:\n    record = client.get_record(\"oai:arXiv.org:this-id-does-not-exist\", \"oai_dc\")\nexcept IdDoesNotExistError as e:\n    print(f\"Caught expected error: {e}\")\n```\n\n#### Notebook example\n\nSee the [`notebooks/arxiv_recent_changes.ipynb`](notebooks/arxiv_recent_changes.ipynb) notebook for an example of using the client to fetch recent changes from the arXiv OAI-PMH endpoint.\n\n## Testing\n\nTo run the tests, you will need to install the development dependencies:\n\n```bash\nuv pip install -e \".[dev]\"\n```\n\nThen, you can run the tests using `pytest`:\n\n```bash\npytest\n```\n\n## Releases\n\nThis repository uses [Release Please](https://github.com/googleapis/release-please) to automate releases.\n\nWhen you merge a pull request to the `main` branch, Release Please will:\n\n1.  Create or update a \"Release PR\" with the changelog and version bump.\n2.  When you merge that Release PR, it will create a GitHub Release and tag the commit.\n\nTo trigger a release:\n\n1.  Ensure your PR titles follow the [Conventional Commits](https://www.conventionalcommits.org/) specification (e.g., `feat: add new feature`, `fix: bug fix`).\n2.  Merge your PRs into `main`.\n3.  Review and merge the Release PR created by the bot.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwellcomecollection%2Foai-pmh","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwellcomecollection%2Foai-pmh","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwellcomecollection%2Foai-pmh/lists"}