{"id":15913106,"url":"https://github.com/34j/cached-historical-data-fetcher","last_synced_at":"2025-03-22T19:34:36.804Z","repository":{"id":197749652,"uuid":"699145493","full_name":"34j/cached-historical-data-fetcher","owner":"34j","description":"Python utility for fetching any historical data using caching. Suitable for news, posts, weather, etc.","archived":false,"fork":false,"pushed_at":"2025-03-11T19:27:07.000Z","size":505,"stargazers_count":3,"open_issues_count":13,"forks_count":1,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-03-11T20:29:40.944Z","etag":null,"topics":["aiohttp","asyncio","cache","fetch","hacktoberfest","historical-data","http","joblib","lz4","pagenation","pandas","python","realtime","scraper","scraping","tqdm","update","web-scraping"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/34j.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG","contributing":"CONTRIBUTING.md","funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":".github/CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":".github/SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":["34j"]}},"created_at":"2023-10-02T02:43:01.000Z","updated_at":"2025-01-30T18:21:19.000Z","dependencies_parsed_at":null,"dependency_job_id":"6ca97c9a-e45a-46f4-939c-5b03aa14bbc9","html_url":"https://github.com/34j/cached-historical-data-fetcher","commit_stats":{"total_commits":85,"total_committers":4,"mean_commits":21.25,"dds":"0.19999999999999996","last_synced_commit":"1c6c166aef0d0b38be8f4d6cfcb2bda0e1940a28"},"previous_names":["34j/cached-historical-data-fetcher"],"tags_count":28,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/34j%2Fcached-historical-data-fetcher","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/34j%2Fcached-historical-data-fetcher/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/34j%2Fcached-historical-data-fetcher/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/34j%2Fcached-historical-data-fetcher/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/34j","download_url":"https://codeload.github.com/34j/cached-historical-data-fetcher/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245013743,"owners_count":20547175,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aiohttp","asyncio","cache","fetch","hacktoberfest","historical-data","http","joblib","lz4","pagenation","pandas","python","realtime","scraper","scraping","tqdm","update","web-scraping"],"created_at":"2024-10-06T16:23:07.994Z","updated_at":"2025-03-22T19:34:36.490Z","avatar_url":"https://github.com/34j.png","language":"Python","funding_links":["https://github.com/sponsors/34j"],"categories":[],"sub_categories":[],"readme":"# Cached Historical Data Fetcher\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://github.com/34j/cached-historical-data-fetcher/actions/workflows/ci.yml?query=branch%3Amain\"\u003e\n    \u003cimg src=\"https://img.shields.io/github/actions/workflow/status/34j/cached-historical-data-fetcher/ci.yml?branch=main\u0026label=CI\u0026logo=github\u0026style=flat-square\" alt=\"CI Status\" \u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://cached-historical-data-fetcher.readthedocs.io\"\u003e\n    \u003cimg src=\"https://img.shields.io/readthedocs/cached-historical-data-fetcher.svg?logo=read-the-docs\u0026logoColor=fff\u0026style=flat-square\" alt=\"Documentation Status\"\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://codecov.io/gh/34j/cached-historical-data-fetcher\"\u003e\n    \u003cimg src=\"https://img.shields.io/codecov/c/github/34j/cached-historical-data-fetcher.svg?logo=codecov\u0026logoColor=fff\u0026style=flat-square\" alt=\"Test coverage percentage\"\u003e\n  \u003c/a\u003e\n\u003c/p\u003e\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://python-poetry.org/\"\u003e\n    \u003cimg src=\"https://img.shields.io/badge/packaging-poetry-299bd7?style=flat-square\u0026logo=data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAA4AAAASCAYAAABrXO8xAAAACXBIWXMAAAsTAAALEwEAmpwYAAAAAXNSR0IArs4c6QAAAARnQU1BAACxjwv8YQUAAAJJSURBVHgBfZLPa1NBEMe/s7tNXoxW1KJQKaUHkXhQvHgW6UHQQ09CBS/6V3hKc/AP8CqCrUcpmop3Cx48eDB4yEECjVQrlZb80CRN8t6OM/teagVxYZi38+Yz853dJbzoMV3MM8cJUcLMSUKIE8AzQ2PieZzFxEJOHMOgMQQ+dUgSAckNXhapU/NMhDSWLs1B24A8sO1xrN4NECkcAC9ASkiIJc6k5TRiUDPhnyMMdhKc+Zx19l6SgyeW76BEONY9exVQMzKExGKwwPsCzza7KGSSWRWEQhyEaDXp6ZHEr416ygbiKYOd7TEWvvcQIeusHYMJGhTwF9y7sGnSwaWyFAiyoxzqW0PM/RjghPxF2pWReAowTEXnDh0xgcLs8l2YQmOrj3N7ByiqEoH0cARs4u78WgAVkoEDIDoOi3AkcLOHU60RIg5wC4ZuTC7FaHKQm8Hq1fQuSOBvX/sodmNJSB5geaF5CPIkUeecdMxieoRO5jz9bheL6/tXjrwCyX/UYBUcjCaWHljx1xiX6z9xEjkYAzbGVnB8pvLmyXm9ep+W8CmsSHQQY77Zx1zboxAV0w7ybMhQmfqdmmw3nEp1I0Z+FGO6M8LZdoyZnuzzBdjISicKRnpxzI9fPb+0oYXsNdyi+d3h9bm9MWYHFtPeIZfLwzmFDKy1ai3p+PDls1Llz4yyFpferxjnyjJDSEy9CaCx5m2cJPerq6Xm34eTrZt3PqxYO1XOwDYZrFlH1fWnpU38Y9HRze3lj0vOujZcXKuuXm3jP+s3KbZVra7y2EAAAAAASUVORK5CYII=\" alt=\"Poetry\"\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://github.com/ambv/black\"\u003e\n    \u003cimg src=\"https://img.shields.io/badge/code%20style-black-000000.svg?style=flat-square\" alt=\"black\"\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://github.com/pre-commit/pre-commit\"\u003e\n    \u003cimg src=\"https://img.shields.io/badge/pre--commit-enabled-brightgreen?logo=pre-commit\u0026logoColor=white\u0026style=flat-square\" alt=\"pre-commit\"\u003e\n  \u003c/a\u003e\n\u003c/p\u003e\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://pypi.org/project/cached-historical-data-fetcher/\"\u003e\n    \u003cimg src=\"https://img.shields.io/pypi/v/cached-historical-data-fetcher.svg?logo=python\u0026logoColor=fff\u0026style=flat-square\" alt=\"PyPI Version\"\u003e\n  \u003c/a\u003e\n  \u003cimg src=\"https://img.shields.io/pypi/pyversions/cached-historical-data-fetcher.svg?style=flat-square\u0026logo=python\u0026amp;logoColor=fff\" alt=\"Supported Python versions\"\u003e\n  \u003cimg src=\"https://img.shields.io/pypi/l/cached-historical-data-fetcher.svg?style=flat-square\" alt=\"License\"\u003e\n\u003c/p\u003e\n\nPython utility for fetching any historical data using caching.\nSuitable for acquiring data that is added frequently and incrementally, e.g. news, posts, weather, etc.\n\n## Installation\n\nInstall this via pip (or your favourite package manager):\n\n```shell\npip install cached-historical-data-fetcher\n```\n\n## Features\n\n- Uses cache built on top of [`joblib`](https://github.com/joblib/joblib), [`lz4`](https://github.com/lz4/lz4) and [`aiofiles`](https://github.com/Tinche/aiofiles).\n- Ready to use with [`asyncio`](https://docs.python.org/3/library/asyncio.html), [`aiohttp`](https://github.com/aio-libs/aiohttp), [`aiohttp-client-cache`](https://github.com/requests-cache/aiohttp-client-cache). Uses `asyncio.gather` for fetching chunks in parallel. (For performance reasons, only using `aiohttp-client-cache` is probably not a good idea when fetching large number of chunks (web requests).)\n- Based on [`pandas`](https://github.com/pandas-dev/pandas) and supports `MultiIndex`.\n\n## Usage\n\n### `HistoricalDataCache`, `HistoricalDataCacheWithChunk` and `HistoricalDataCacheWithFixedChunk`\n\nOverride `get_one()` method to fetch data for one chunk. `update()` method will call `get_one()` for each unfetched chunk and concatenate results, then save to cache.\n\n```python\nfrom cached_historical_data_fetcher import HistoricalDataCacheWithFixedChunk\nfrom pandas import DataFrame, Timedelta, Timestamp\nfrom typing import Any\n\n# define cache class\nclass MyCacheWithFixedChunk(HistoricalDataCacheWithFixedChunk[Timestamp, Timedelta, Any]):\n    delay_seconds = 0.0 # delay between chunks (requests) in seconds\n    interval = Timedelta(days=1) # interval between chunks, can be any type\n    start_index = Timestamp.utcnow().floor(\"10D\") # start index, can be any type\n\n    async def get_one(self, start: Timestamp, *args: Any, **kwargs: Any) -\u003e DataFrame:\n        \"\"\"Fetch data for one chunk.\"\"\"\n        return DataFrame({\"day\": [start.day]}, index=[start])\n\n# get complete data\nprint(await MyCacheWithFixedChunk().update())\n```\n\n```shell\n                           day\n2023-09-30 00:00:00+00:00   30\n2023-10-01 00:00:00+00:00    1\n2023-10-02 00:00:00+00:00    2\n```\n\n**See [example.ipynb](example.ipynb) for real-world example.**\n\n### `IdCacheWithFixedChunk`\n\nOverride `get_one` method to fetch data for one chunk in the same way as in `HistoricalDataCacheWithFixedChunk`.\nAfter updating `ids` by calling `set_ids()`, `update()` method will call `get_one()` for every unfetched id and concatenate results, then save to cache.\n\n```python\nfrom cached_historical_data_fetcher import IdCacheWithFixedChunk\nfrom pandas import DataFrame\nfrom typing import Any\n\nclass MyIdCache(IdCacheWithFixedChunk[str, Any]):\n    delay_seconds = 0.0 # delay between chunks (requests) in seconds\n\n    async def get_one(self, start: str, *args: Any, **kwargs: Any) -\u003e DataFrame:\n        \"\"\"Fetch data for one chunk.\"\"\"\n        return DataFrame({\"id+hello\": [start + \"+hello\"]}, index=[start])\n\ncache = MyIdCache() # create cache\ncache.set_ids([\"a\"]) # set ids\ncache.set_ids([\"b\"]) # set ids again, now `cache.ids` is [\"a\", \"b\"]\nprint(await cache.update(reload=True)) # discard previous cache and fetch again\ncache.set_ids([\"b\", \"c\"]) # set ids again, now `cache.ids` is [\"a\", \"b\", \"c\"]\nprint(await cache.update()) # fetch only new data\n```\n\n```shell\n       id+hello\n    a   a+hello\n    b   b+hello\n       id+hello\n    a   a+hello\n    b   b+hello\n    c   c+hello\n```\n\n## Contributors ✨\n\nThanks goes to these wonderful people ([emoji key](https://allcontributors.org/docs/en/emoji-key)):\n\n\u003c!-- prettier-ignore-start --\u003e\n\u003c!-- ALL-CONTRIBUTORS-LIST:START - Do not remove or modify this section --\u003e\n\u003c!-- markdownlint-disable --\u003e\n\u003c!-- markdownlint-enable --\u003e\n\u003c!-- ALL-CONTRIBUTORS-LIST:END --\u003e\n\u003c!-- prettier-ignore-end --\u003e\n\nThis project follows the [all-contributors](https://github.com/all-contributors/all-contributors) specification. Contributions of any kind welcome!\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2F34j%2Fcached-historical-data-fetcher","html_url":"https://awesome.ecosyste.ms/projects/github.com%2F34j%2Fcached-historical-data-fetcher","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2F34j%2Fcached-historical-data-fetcher/lists"}