{"id":15555415,"url":"https://github.com/sckott/pyminer","last_synced_at":"2025-04-23T20:02:46.770Z","repository":{"id":52243894,"uuid":"50943199","full_name":"sckott/pyminer","owner":"sckott","description":"Text-mining toolset for Crossref data","archived":false,"fork":false,"pushed_at":"2021-05-03T21:48:19.000Z","size":8316,"stargazers_count":8,"open_issues_count":4,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-04-18T18:29:24.880Z","etag":null,"topics":["crossref","python","text-mining"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sckott.png","metadata":{"files":{"readme":"README.rst","changelog":"Changelog.rst","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2016-02-02T18:45:50.000Z","updated_at":"2023-09-08T18:25:09.000Z","dependencies_parsed_at":"2022-08-30T19:51:48.481Z","dependency_job_id":null,"html_url":"https://github.com/sckott/pyminer","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sckott%2Fpyminer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sckott%2Fpyminer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sckott%2Fpyminer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sckott%2Fpyminer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sckott","download_url":"https://codeload.github.com/sckott/pyminer/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250506146,"owners_count":21441722,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["crossref","python","text-mining"],"created_at":"2024-10-02T15:09:14.754Z","updated_at":"2025-04-23T20:02:46.678Z","avatar_url":"https://github.com/sckott.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"pyminer\n=======\n\n|pypi| |docs| |travis| |coverage|\n\nPython client for text mining levaraging `Crossrefs Text and Data Mining service\n\u003chttp://tdmsupport.crossref.org/researchers\u003e`__.\n\n`Source on GitHub at sckott/pyminer \u003chttps://github.com/sckott/pyminer\u003e`__\n\nOther Crossref text mining (and related) clients:\n\n* R: `rcrossref`, `ropensci/rcrossref \u003chttps://github.com/ropensci/rcrossref\u003e`__\n* R: `crminer`, `ropensci/crminer \u003chttps://github.com/ropenscilabs/crminer\u003e`__\n* R: `fulltext`, `ropensci/fulltext \u003chttps://github.com/ropensci/fulltext\u003e`__\n* Ruby: `textminer`, `sckott/textminer \u003chttps://github.com/sckott/textminer\u003e`__\n* Python: `habanero`, `sckott/habanero \u003chttps://github.com/sckott/habanero\u003e`__\n\nInstallation\n============\n\nStable from pypi\n\n.. code-block:: console\n\n    pip install pyminer\n\nDevelopment version\n\n.. code-block:: console\n\n    [sudo] pip install git+git://github.com/sckott/pyminer.git#egg=pyminer\n\n\nSearch\n======\n\nStrongly recommend for search using your email in the mailto parameter in the \nMiner() call to get in the \"fast lane\".\n\n.. code-block:: python\n\n    from pyminer import Miner\n    import os\n    m = Miner(mailto = os.environ['crossref_email'])\n    m.search(filter = {'has_full_text': True}, limit = 5)\n\n\nFetch\n=====\n\nIf you have a Crossref Text and Data Mining key/token, you can give it in the \ntdmkey parameter in the Miner() call\n\n.. code-block:: python\n\n    # a Pensoft article\n    from pyminer import Miner\n    import os\n    m = Miner(mailto = os.environ['crossref_email'])\n    x = m.search(ids = '10.3897/rio.2.e10445')\n    x\n    out = x.fetch(type = \"pdf\")\n    out\n    out[0].url\n    out[0].path\n    out[0].type\n    out[0].parse()\n\n    # an Elsevier article - BEWARE, they check IP addresses, so your IP address \n    # must be at a member institution or similar\n    from pyminer import Miner\n    import os\n    m = Miner(mailto = os.environ['crossref_email'], tdmkey = os.environ['CROSSREF_TDM'])\n    x = m.search(ids = \"10.1016/j.funeco.2010.11.003\")\n    out = x.fetch(type = \"xml\")\n    out\n    out[0].path\n    out[0].parse()\n\n\nExtract\n=======\n\n.. code-block:: python\n\n    from pyminer import fetch, extract\n    url = 'http://www.nepjol.info/index.php/JSAN/article/viewFile/13527/10928'\n    x = fetch(url)\n    extract(x.path)\n\nMeta\n====\n\n* License: MIT, see `LICENSE file \u003cLICENSE\u003e`__\n* Please note that this project is released with a `Contributor Code of Conduct \u003cCODE_OF_CONDUCT.md\u003e`__. By participating in this project you agree to abide by its terms.\n\n.. |pypi| image:: https://img.shields.io/pypi/v/pyminer.svg\n   :target: https://pypi.python.org/pypi/pyminer\n\n.. |docs| image:: https://readthedocs.org/projects/pyminer/badge/?version=latest\n   :target: http://pyminer.readthedocs.io/en/latest/?badge=latest\n\n.. |travis| image:: https://travis-ci.org/sckott/pyminer.svg\n   :target: https://travis-ci.org/sckott/pyminer\n\n.. |coverage| image:: https://coveralls.io/repos/sckott/pyminer/badge.svg?branch=master\u0026service=github\n   :target: https://coveralls.io/github/sckott/pyminer?branch=master\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsckott%2Fpyminer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsckott%2Fpyminer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsckott%2Fpyminer/lists"}