{"id":15678141,"url":"https://github.com/rams3sh/botocache","last_synced_at":"2025-05-07T02:23:47.206Z","repository":{"id":59390713,"uuid":"340289938","full_name":"rams3sh/botocache","owner":"rams3sh","description":"Caching layer for botocore / boto3 SDK.","archived":false,"fork":false,"pushed_at":"2024-03-13T18:21:34.000Z","size":49,"stargazers_count":12,"open_issues_count":0,"forks_count":5,"subscribers_count":5,"default_branch":"main","last_synced_at":"2024-03-14T18:33:11.443Z","etag":null,"topics":["aws","boto","boto3","botocore","cache","cachetools","caching","caching-library","sdk"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rams3sh.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-02-19T07:23:22.000Z","updated_at":"2023-07-20T14:27:23.000Z","dependencies_parsed_at":"2024-03-09T03:27:32.674Z","dependency_job_id":"fbf4ddca-6cc3-43f6-a4d6-8c3e97daac77","html_url":"https://github.com/rams3sh/botocache","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rams3sh%2Fbotocache","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rams3sh%2Fbotocache/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rams3sh%2Fbotocache/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rams3sh%2Fbotocache/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rams3sh","download_url":"https://codeload.github.com/rams3sh/botocache/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":242884322,"owners_count":20201108,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","boto","boto3","botocore","cache","cachetools","caching","caching-library","sdk"],"created_at":"2024-10-03T16:17:36.523Z","updated_at":"2025-03-10T16:31:38.747Z","avatar_url":"https://github.com/rams3sh.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Botocache\n\nCaching layer for Boto / Botocore libraries.\n\n\n## Background\n\n---\n\nThis project was started to solve the issue raised [here](https://github.com/boto/boto3/issues/2723).\n\nMy day job requires me to write and use multiple tools and standalone scripts to audit AWS environments. \nMost of these tools are hacky and written in python which uses boto3 / botocore in the backend to interact with AWS API.\n\nSometimes, I would have to write a wrapper to combine these scripts to get a custom consolidated report. \nSince these scripts are standalone, they repeat the same API calls that a previous script would have already \ncalled before, leading to redundant API calls, throttling and unnecessary IO wait. \n\nThese wrapped tools are sometimes even used as Lambda for automation of certain things. \nThe IO wait times becomes a bottleneck when used in environment such as Lambda where execution is a \ntime-bound activity. \n\nHence I was looking for a caching layer over boto that can solve reducing these direct redundant calls and \nthe wait times. Thus the birth of botocache.\n\n\n## About\n\n---\n\nBotocache caches the response of API calls initiated through boto3 / botocore.\nAny subsequent redundant call will end up getting the previously cached response from Botocache as long as the call is\nwithin the expiry timeout of the cached response. \n\nBotocache can work with any cache library that is based on [cachetools](https://github.com/tkem/cachetools/)\n\nIt uses the unittest module's patch as the magic component to achieve this. :wink:\n\nThis project is little hacky given the nature how it achieves caching, but it gets the job done. \n\n\n## Installation\n\n---\nStable release installation (PyPI) :-\n```bash\npip3 install botocache\n```\n\nTest release installation (Test PyPI)\n```bash\npip install -i https://test.pypi.org/pypi/ --extra-index-url https://pypi.org/simple botocache\n```\n\nInstallation directly from this Repository:-\n```bash\npip3 install git+https://github.com/rams3sh/botocache.git\n```\n\n\n## Usage\n\n---\n\nBelow snippet demonstrates usage of botocache.\n\n```python\nfrom boto3.session import Session\nfrom cachetools_ext.fs import FSLRUCache\n\nfrom botocache.botocache import botocache_context\n\ncache = FSLRUCache(ttl=900, path=\".cache\", maxsize=1000)\n\n# action_regex_to_cache parameter consists list of regex to be matched against a given action for considering the call to be cached\nwith botocache_context(cache=cache,\n                       action_regex_to_cache=[\"List.*\", \"Get.*\", \"Describe.*\"],  \n                       call_log=True, # This helps in logging all calls made to AWS. Useful while debugging. Default value is False.\n                       supress_warning_message=False # This supresses warning messages encountered while caching. Default value is False. \n                       ):\n\n  cached_session = Session()\n  # Initialising client with botocache context\n  cached_client = cached_session.client('iam')\n\n\n\"\"\"\nUsing cached client for paginating list users\n\"\"\"\npaginator = cached_client.get_paginator('list_users')\nfor page in paginator.paginate():\n  print(page)\n\n\"\"\"\nAlways use a fresh initialised session client and subsequent new objects outside the context of botocache to use \nboto3 without caching layer.\n\"\"\"\n\nnon_cached_session = Session()\nnon_cached_client = non_cached_session.client('iam')\npaginator = non_cached_client.get_paginator('list_users')\nfor page in paginator.paginate():\n  print(page)\n\n```\n**Note:** \n\nBotocache has been tested with cachetools_ext's FSLRU cache, but it should logically work with any cache that is \ncompatible with cachetools.\n\n\n## Disclaimer(s)\n\n---\n\n* This project was created mainly to support my specific internal use cases. \nHence, there is a good scope of it having bugs and functional issues. Feel free to raise a PR / Issue in those cases.\n\n\n* Botocache does not understand [HTTP related caching specification](https://tools.ietf.org/html/rfc7234).\nIt works based on function caching.Its a very simple dumb caching layer that checks for specific attributes in an API call, converts into a key \nand stores the response against the key. \nAny subsequent call having matching attributes will be returned with the value stored against the same key.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frams3sh%2Fbotocache","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frams3sh%2Fbotocache","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frams3sh%2Fbotocache/lists"}