{"id":20824568,"url":"https://github.com/hadro/hathi_analysis","last_synced_at":"2026-02-08T23:03:01.474Z","repository":{"id":88705963,"uuid":"91818615","full_name":"hadro/hathi_analysis","owner":"hadro","description":"Usage analysis of HathiTrust","archived":false,"fork":false,"pushed_at":"2018-07-11T16:01:12.000Z","size":27108,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-08-01T16:56:12.051Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://hadro.github.io/hathi_analysis/","language":"HTML","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/hadro.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-05-19T15:15:21.000Z","updated_at":"2018-07-11T16:01:14.000Z","dependencies_parsed_at":null,"dependency_job_id":"0c56e986-919b-4c87-a7e1-f58f9b29c727","html_url":"https://github.com/hadro/hathi_analysis","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/hadro/hathi_analysis","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hadro%2Fhathi_analysis","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hadro%2Fhathi_analysis/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hadro%2Fhathi_analysis/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hadro%2Fhathi_analysis/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/hadro","download_url":"https://codeload.github.com/hadro/hathi_analysis/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hadro%2Fhathi_analysis/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29248493,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-08T22:49:53.206Z","status":"ssl_error","status_checked_at":"2026-02-08T22:49:51.384Z","response_time":57,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-17T22:22:54.719Z","updated_at":"2026-02-08T23:03:01.224Z","avatar_url":"https://github.com/hadro.png","language":"HTML","funding_links":[],"categories":[],"sub_categories":[],"readme":"## N.B. Text below being revised, new charts added, etc.\nSince writing this originally, the analysis has evolved somewhat, and this needs\nto be revised before being shared broadly.\n\n# HathiTrust Usage Analysis\n\nAs part of some work to expand the [SimplyE\nproject](http://www.librarysimplified.org/) to include more materials\nuseful to the research community, I've done some basic analysis of\n[HathiTrust](https://www.hathitrust.org/about)\nusage.\n\nHathiTrust describes itself as: \n\n\u003e    a partnership of major research institutions and libraries working to ensure that the cultural record is preserved and accessible long into the future. The mission of HathiTrust is to contribute to research, scholarship, and the common good by collaboratively collecting, organizing, preserving, communicating, and sharing the record of human knowledge. There are more than 120 partners in HathiTrust, and membership is open to institutions worldwide.\n\nIt contains more than 15 million volumes, including 5.8 million open volumes in the\npublic domain. NYPL, through its Google Library Project partnership, has\ncontributed more than 300,000 volumes to HathiTrust. For more info see the\n[HathiTrust about page](https://www.hathitrust.org/about).\n\nThe data below covers the period May 8, 2014 - May 7, 2017.\n## Usage\n\n### All Items\n\nThis graph displays all volumes, open volumes accessed, and closed volumes with\nattempted access by year. (See [full graph for larger view](https://hadro.github.io/hathi_analysis/percentage_years.html).)\n\n\u003ciframe width=\"900\" height=\"800\" frameborder=\"0\" scrolling=\"no\" src=\"//plot.ly/~hadro/15.embed\"\u003e\u003c/iframe\u003e\n\n## Open items\n\nThis graph displays all open volumes and all open volumes accessed by year\n(toggle \"Total volumes\" in the legend to see all HathiTrust volumes for a given\nyear as well). (See [full graph for larger view](https://hadro.github.io/hathi_analysis/access.html).)\n\n\u003ciframe width=\"900\" height=\"800\" frameborder=\"0\" scrolling=\"no\" src=\"//plot.ly/~hadro/19.embed\"\u003e\u003c/iframe\u003e\n\n## Headline numbers\n\n**15,095,384** retrievable records from the Hathifiles\n(Pandas choked on a small subset of records which are not included here, and are\nalmost certainly not among the most-viewed items)\n\n**3,247,601** distinct HathiTrust volume IDs\n(that I was able to scrape out of three years of analytics data)\n\n### Most viewed titles\n\n**See [top_open_items.txt](txt/top_open_items.txt) for the full list**\n\n| Title |  Uses | Publication Date|\n|-------|-------|-----------------|\n| \u003ca href=\"https://babel.hathitrust.org/cgi/pt?id=mdp.39015054061430\"\u003eQuicksand,\u003c/a\u003e | 823739 views | 1928 |\n| \u003ca href=\"https://babel.hathitrust.org/cgi/pt?id=mdp.39015011274175\"\u003eThe surnames of Scotland, their origin meaning and history /\u003c/a\u003e | 619250 views | 1962 |\n| \u003ca href=\"https://babel.hathitrust.org/cgi/pt?id=mdp.39015064340733\"\u003eSolid mensuration,\u003c/a\u003e | 452809 views | 1934 |\n| \u003ca href=\"https://babel.hathitrust.org/cgi/pt?id=pst.000057937434\"\u003eThe human figure /\u003c/a\u003e | 348795 views | 1907 |\n| \u003ca href=\"https://babel.hathitrust.org/cgi/pt?id=mdp.39015000566789\"\u003eAmerica is in the heart, a personal history,\u003c/a\u003e | 326990 views | 1946 |\n| \u003ca href=\"https://babel.hathitrust.org/cgi/pt?id=mdp.39015004111095\"\u003eGodey's magazine.\u003c/a\u003e | 311573 views | 1850 |\n| \u003ca href=\"https://babel.hathitrust.org/cgi/pt?id=wu.89059402255\"\u003eRoster of the Confederate soldiers of Georgia, 1861-1865 /\u003c/a\u003e | 285162 views | 9999 |\n\n\n\n### Most viewed NYPL titles\n\n**See [NYPL_top_open_items.txt](txt/NYPL_top_open_items.txt) for the full list**\n\n|Title |  Uses | Publication Date|\n|------|-------|-----------------|\n| \u003ca href=\"https://babel.hathitrust.org/cgi/pt?id=nyp.33433076064025\"\u003eMiranda /\u003c/a\u003e | 48465 views | 1915|\n| \u003ca href=\"https://babel.hathitrust.org/cgi/pt?id=nyp.33433082137914\"\u003eWife no. 19, or the story of a life in bondage : being a complete exposé of Mormonism, and revealing the sorrows, sacrifices and sufferings of women in polygamy /\u003c/a\u003e | 36158 views | 1875|\n|\u003ca href=\"https://babel.hathitrust.org/cgi/pt?id=nyp.33433067286868\"\u003eMen of West Virginia ...\u003c/a\u003e | 30308 views | 1903|\n|\u003ca href=\"https://babel.hathitrust.org/cgi/pt?id=nyp.33433069455859\"\u003eIllustrated trade catalogue and price list : manufacturers, importers and jobbers of watchmakers', jewelers' and engravers' supplies of every description : optical goods, chains, charms, etc. : originators of the box matetial [sic] and makers of Swartchild's celebrated watchmakers' benches : 1897-1898 /\u003c/a\u003e | 28594 views | 1897|\n|\u003ca href=\"https://babel.hathitrust.org/cgi/pt?id=nyp.33433081844692\"\u003eA standard history of Stark County, Ohio : an authentic narrative of the past, with particular attention to the modern era in the commercial, industrial, civic and social development : a chronicle of the people, with family lineage and memoirs /\u003c/a\u003e | 26722 views | 1916|\n\n\n### Closed (\"Limited View\") titles with most attempted views\n\n**See [top_closed_items.txt](txt/top_closed_items.txt) for the full list**\n\n|Title |  Attempted Uses | Publication Date|\n|------|-----------------|-----------------|\n|\u003ca href=\"https://babel.hathitrust.org/cgi/pt?id=uc1.b4906221\"\u003e The competent manager : a model for effective performance /\u003c/a\u003e | 12226 views | 1982|\n|\u003ca href=\"https://babel.hathitrust.org/cgi/pt?id=mdp.39015010576356\"\u003e Objects of daily use, with over 1800 figures from University college, London,\u003c/a\u003e | 12150 views | 1927|\n|\u003ca href=\"https://babel.hathitrust.org/cgi/pt?id=mdp.39015014559135\"\u003e My experiences in the world war,\u003c/a\u003e | 10976 views | 1931|\n|\u003ca href=\"https://babel.hathitrust.org/cgi/pt?id=mdp.39015010574575\"\u003e Catalogue of Alexandrian coins,\u003c/a\u003e | 9132 views | 1933|\n|\u003ca href=\"https://babel.hathitrust.org/cgi/pt?id=uc1.b4522148\"\u003e The regimental history of the 3rd Queen Alexandra's own Gurkha rifles from April 1815 to December 1927,\u003c/a\u003e | 9088 views | 1929|\n\n\n## Publication Date analysis\n\n**See [full chart](histogram.html) for better view**\n\nThis chart describes the frequency of publication year for the top 500,000 requested items in HathiTrust from\nMay 2014 - May 2017.\n\n\u003ciframe width=\"900\" height=\"800\" frameborder=\"0\" scrolling=\"no\" src=\"https://plot.ly/~hadro/12.embed\"\u003e\u003c/iframe\u003e\n\nNote: the embedded histogram on this page only includes the top 40,000 data points\nbecause the free version of plotly has a 40K data point limit; for a full-screen\ninteractive version of this chart with 500,000 data points, see the **full\n[Histogram](histogram.html)**.\n\nMeanwhile, you can toggle the data series on this chart, for example if you want to\nview just the items among the top 500K that were requested but could not be\nviewed because they are \"limited view\" items (i.e. closed for copyright reasons).\n\nNotes:\n- There are 18,455 volumes not displayed on the [complete chart](histogram.html),\n  because they did not fall within the 1600-2020 publication date range\n    - 10,349 volumes with a publication date either before 1600 or after 2020\n        - 7,770 of those have publication dates of \"9999\", which almost always\n          means they are part of an ongoing publication or serial\n    - 8,106 volumes with no valid publication date value in the HathiFiles\n\n\n## Usage analysis\n\n**See [full chart](usage_log.html) for better view**\n\nThis chart describes the usage curve for the top 10,000 items in HathiTrust from\nMay 2014 - May 2017.\n\n\u003ciframe width=\"900\" height=\"800\" frameborder=\"0\" scrolling=\"no\" src=\"https://plot.ly/~hadro/11.embed\"\u003e\u003c/iframe\u003e\n\n(For full-screen interactive version of this chart, see [Line Chart](usage_log.html) (or the\n[linear scale](usage.html) version as well).\n\n## Tools and method\n\nMy steps and the tools I used were very roughly as follows:\n- Scraped three years of daily complete Google Analytics urls, using Corey Harper's\n  helpful [PygAnalytics tool](https://github.com/chrpr/pyganalytics)\n- Downloaded the complete [HathiFiles](https://www.hathitrust.org/hathifiles)\n  for May 2017, which includes basic bibliographic and rights metadata for every\n      volume in HathiTrust\n- Various slicing, dicing, matching, joining, and other manipulations using the\n  invaluable [Pandas Python Data Analysis\n  Library](http://pandas.pydata.org/)\n    - Unholy amounts of regular expressions, via the Pandas `.extract()` and\n      `.extractall()` methods\n- Ingest of ~15 million rows of HathiFiles into Postgres database, using\n  the Pandas `.to_sql()` method\n- Data visualization using the [Plotly Python Library](https://plot.ly/python/)\n  (including the handy ability to [run Plotly in 'offline\n  mode'](https://www.reddit.com/r/IPython/comments/3tibc8/tip_on_how_to_run_plotly_examples_in_offline_mode/),\n  so you don't have to constantly upload each iteration of a revised chart).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhadro%2Fhathi_analysis","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhadro%2Fhathi_analysis","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhadro%2Fhathi_analysis/lists"}