{"id":23440864,"url":"https://github.com/living-with-machines/hmd_newspaper_dl","last_synced_at":"2025-10-14T20:11:52.570Z","repository":{"id":37819886,"uuid":"341499854","full_name":"Living-with-machines/hmd_newspaper_dl","owner":"Living-with-machines","description":"Bulk download British Library Heritage Made Digital Newspapers 📰","archived":false,"fork":false,"pushed_at":"2022-06-16T14:54:12.000Z","size":225,"stargazers_count":3,"open_issues_count":1,"forks_count":2,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-04-13T09:13:23.299Z","etag":null,"topics":["dataset","glam","nbdev","newspapers"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Living-with-machines.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-02-23T09:30:49.000Z","updated_at":"2022-09-15T06:10:49.000Z","dependencies_parsed_at":"2022-06-23T10:14:41.343Z","dependency_job_id":null,"html_url":"https://github.com/Living-with-machines/hmd_newspaper_dl","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":"fastai/nbdev_template","purl":"pkg:github/Living-with-machines/hmd_newspaper_dl","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Living-with-machines%2Fhmd_newspaper_dl","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Living-with-machines%2Fhmd_newspaper_dl/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Living-with-machines%2Fhmd_newspaper_dl/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Living-with-machines%2Fhmd_newspaper_dl/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Living-with-machines","download_url":"https://codeload.github.com/Living-with-machines/hmd_newspaper_dl/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Living-with-machines%2Fhmd_newspaper_dl/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":262704458,"owners_count":23351077,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dataset","glam","nbdev","newspapers"],"created_at":"2024-12-23T16:19:25.317Z","updated_at":"2025-10-14T20:11:47.539Z","avatar_url":"https://github.com/Living-with-machines.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# hmd_download\n\u003e Bulk download Heritage Made Digital digitised newspapers from the British Library Research Repository \n\n\nThis [command line tool](https://en.wikipedia.org/wiki/Command-line_interface) is intended to make it easy to bulk download [Heritage Made Digital Newspapers](https://bl.iro.bl.uk/collections/353c908d-b495-4413-b047-87236d2573e3?utf8=%E2%9C%93\u0026sort=score+desc%2C+system_create_dtsi+desc\u0026per_page=100\u0026locale=en) from the [British Library](https://www.bl.uk/) [Research Repository](https://bl.iro.bl.uk/). \n\nThe tool has been used by Living with Machines but may be of use to other people. Since the tool is intended to download the collection in 'bulk' it is likely to be useful if you either want:\n- all HMD newspapers \n- a random sample i.e. 10 newspaper\n\nThis tool was developed for internal use so it might not be suitable for your needs. If you have problems or suggestions with the tool please [open an issue](https://github.com/Living-with-machines/hmd_newspaper_dl/issues/new/choose). \n\n## Install\n\nThe tool was developed using `nbdev` so although all of the code for this tool lives inside a single Jupyter notebook you can still install it as a Python package. At the moment this is done via GitHub:\n\n```bash\npython -m pip install git+https://github.com/Living-with-machines/hmd_newspaper_dl\n```\n\nIt is recommened to install the package insdide a virtual environment. Since this is a command line tool one simple option for installing is [pipx](https://pypa.github.io/pipx/) which will install the tool inside a new virtual environment for you:\n\n```bash\npipx install git+https://github.com/Living-with-machines/hmd_newspaper_dl\n```\n\n## How to use\n\nOnce you have installed the packaghe you will also have made available a console script `hmd_download`:\n\n    usage: hmd_download [-h] [--n_threads N_THREADS] [--subset SUBSET] save_dir\n    \n    Download HMD newspaper from iro to `save_dir` using `n_threads`\n    \n    positional arguments:\n      save_dir               Output Directory\n    \n    optional arguments:\n      -h, --help             show this help message and exit\n      --n_threads N_THREADS  Number threads to use (default: 8)\n      --subset SUBSET        Download subset of HMD\n\n\nThis will by default download all available newspaper titles. If you just want a subset you can pass in a subset parameter to specify how many titles you want. At the moment this is just a random selection. \n\n## Feedback \n\nThis tool was put together for internal Living with Machines but is shared in case it is helpful for other people. If you have feedback, problems or want to suggest changes please open a [new issue](https://github.com/Living-with-machines/hmd_newspaper_dl/issues/new/choose). \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fliving-with-machines%2Fhmd_newspaper_dl","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fliving-with-machines%2Fhmd_newspaper_dl","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fliving-with-machines%2Fhmd_newspaper_dl/lists"}