Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/karlicoss/cachew
Transparent and persistent cache/serialization powered by type hints
https://github.com/karlicoss/cachew
cache dataclass decorator mypy namedtuple python-decorator serialization sqlite sqlite-database
Last synced: about 2 months ago
JSON representation
Transparent and persistent cache/serialization powered by type hints
- Host: GitHub
- URL: https://github.com/karlicoss/cachew
- Owner: karlicoss
- License: mit
- Created: 2019-07-27T07:11:05.000Z (about 5 years ago)
- Default Branch: master
- Last Pushed: 2024-03-31T05:30:46.000Z (6 months ago)
- Last Synced: 2024-07-19T16:57:53.676Z (2 months ago)
- Topics: cache, dataclass, decorator, mypy, namedtuple, python-decorator, serialization, sqlite, sqlite-database
- Language: Python
- Homepage: https://beepb00p.xyz/unnecessary-db.html#cachew
- Size: 346 KB
- Stars: 208
- Watchers: 4
- Forks: 6
- Open Issues: 6
-
Metadata Files:
- Readme: README.ipynb
- License: LICENSE.txt
Awesome Lists containing this project
- awesome-starred - karlicoss/cachew - Transparent and persistent cache/serialization powered by type hints (others)
README
{
"cells": [
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"autoscroll": false,
"ein.hycell": false,
"ein.tags": "worksheet-0",
"slideshow": {
"slide_type": "-"
},
"tags": [
"noexport"
]
},
"outputs": [],
"source": [
"from pathlib import Path\n",
"import sys; sys.path.insert(0, str(Path('src').absolute()))\n",
"import os\n",
"cwd =os.getcwd()\n",
"\n",
"import ast\n",
"import inspect\n",
"\n",
"from IPython.display import Markdown as md\n",
"\n",
"def flink(title: str, name: str=None):\n",
" # name is method name\n",
" if name is None:\n",
" name = title.replace('`', '') # meh\n",
" split = name.rsplit('.', maxsplit=1)\n",
" if len(split) == 1:\n",
" modname = split[0]\n",
" fname = None\n",
" else:\n",
" [modname, fname] = split\n",
" module = sys.modules[modname]\n",
"\n",
" file = Path(module.__file__).relative_to(cwd)\n",
"\n",
" if fname is not None:\n",
" func = module\n",
" for p in fname.split('.'):\n",
" func = getattr(func, p)\n",
" _, number = inspect.getsourcelines(func)\n",
" numbers = f'#L{number}'\n",
" else:\n",
" numbers = ''\n",
" return f'[{title}]({file}{numbers})'\n",
"\n",
"dmd = lambda x: display(md(x.strip()))\n",
"\n",
"import cachew\n",
"import cachew.extra\n",
"import cachew.marshall.cachew\n",
"import cachew.tests.test_cachew as tests\n",
"sys.modules['tests'] = tests # meh"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"autoscroll": false,
"ein.hycell": false,
"ein.tags": "worksheet-0",
"slideshow": {
"slide_type": "-"
}
},
"outputs": [],
"source": [
"dmd(f'''\n",
"\n",
"''')"
]
},
{
"cell_type": "markdown",
"metadata": {
"ein.tags": "worksheet-0",
"slideshow": {
"slide_type": "-"
}
},
"source": [
"# What is Cachew?\n",
"TLDR: cachew lets you **cache function calls** into an sqlite database on your disk in a matter of **single decorator** (similar to [functools.lru_cache](https://docs.python.org/3/library/functools.html#functools.lru_cache)). The difference from `functools.lru_cache` is that cached data is persisted between program runs, so next time you call your function, it will only be a matter of reading from the cache.\n",
"Cache is **invalidated automatically** if your function's arguments change, so you don't have to think about maintaining it.\n",
"\n",
"In order to be cacheable, your function needs to return a simple data type, or an [Iterator](https://docs.python.org/3/library/typing.html#typing.Iterator) over such types.\n",
"\n",
"A simple type is defined as:\n",
"\n",
"- primitive: `str`/`int`/`float`/`bool`\n",
"- JSON-like types (`dict`/`list`/`tuple`)\n",
"- `datetime`\n",
"- `Exception` (useful for [error handling](https://beepb00p.xyz/mypy-error-handling.html#kiss) )\n",
"- [NamedTuples](https://docs.python.org/3/library/typing.html#typing.NamedTuple)\n",
"- [dataclasses](https://docs.python.org/3/library/dataclasses.html)\n",
"\n",
"\n",
"That allows to **automatically infer schema from type hints** ([PEP 526](https://www.python.org/dev/peps/pep-0526)) and not think about serializing/deserializing.\n",
"Thanks to type hints, you don't need to annotate your classes with any special decorators, inherit from some special base classes, etc., as it's often the case for serialization libraries.\n",
"\n",
"## Motivation\n",
"\n",
"I often find myself processing big chunks of data, merging data together, computing some aggregates on it or extracting few bits I'm interested at. While I'm trying to utilize REPL as much as I can, some things are still fragile and often you just have to rerun the whole thing in the process of development. This can be frustrating if data parsing and processing takes seconds, let alone minutes in some cases.\n",
"\n",
"Conventional way of dealing with it is serializing results along with some sort of hash (e.g. md5) of input files,\n",
"comparing on the next run and returning cached data if nothing changed.\n",
"\n",
"Simple as it sounds, it is pretty tedious to do every time you need to memorize some data, contaminates your code with routine and distracts you from your main task.\n",
"\n",
"\n",
"# Examples\n",
"## Processing Wikipedia\n",
"Imagine you're working on a data analysis pipeline for some huge dataset, say, extracting urls and their titles from Wikipedia archive.\n",
"Parsing it (`extract_links` function) takes hours, however, as long as the archive is same you will always get same results. So it would be nice to be able to cache the results somehow.\n",
"\n",
"\n",
"With this library your can achieve it through single `@cachew` decorator."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"autoscroll": false,
"ein.hycell": false,
"ein.tags": "worksheet-0",
"slideshow": {
"slide_type": "-"
}
},
"outputs": [],
"source": [
"doc = inspect.getdoc(cachew.cachew)\n",
"doc = doc.split('Usage example:')[-1].lstrip()\n",
"dmd(f\"\"\"```python\n",
"{doc}\n",
"```\"\"\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"When you call `extract_links` with the same archive, you start getting results in a matter of milliseconds, as fast as sqlite reads it.\n",
"\n",
"When you use newer archive, `archive_path` changes, which will make cachew invalidate old cache and recompute it, so you don't need to think about maintaining it separately."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Incremental data exports\n",
"This is my most common usecase of cachew, which I'll illustrate with example.\n",
"\n",
"I'm using an [environment sensor](https://bluemaestro.com/products/product-details/bluetooth-environmental-monitor-and-logger) to log stats about temperature and humidity.\n",
"Data is synchronized via bluetooth in the sqlite database, which is easy to access. However sensor has limited memory (e.g. 1000 latest measurements).\n",
"That means that I end up with a new database every few days, each of them containing only a slice of data I need, e.g.:\n",
"\n",
" ...\n",
" 20190715100026.db\n",
" 20190716100138.db\n",
" 20190717101651.db\n",
" 20190718100118.db\n",
" 20190719100701.db\n",
" ...\n",
"\n",
"To access **all** of historic temperature data, I have two options:\n",
"\n",
"- Go through all the data chunks every time I wan to access them and 'merge' into a unified stream of measurements, e.g. something like:\n",
" \n",
" def measurements(chunks: List[Path]) -> Iterator[Measurement]:\n",
" for chunk in chunks:\n",
" # read measurements from 'chunk' and yield unseen ones\n",
"\n",
" This is very **easy, but slow** and you waste CPU for no reason every time you need data.\n",
"\n",
"- Keep a 'master' database and write code to merge chunks in it.\n",
"\n",
" This is very **efficient, but tedious**:\n",
" \n",
" - requires serializing/deserializing data -- boilerplate\n",
" - requires manually managing sqlite database -- error prone, hard to get right every time\n",
" - requires careful scheduling, ideally you want to access new data without having to refresh cache\n",
"\n",
" \n",
"Cachew gives the best of two worlds and makes it both **easy and efficient**. The only thing you have to do is to decorate your function:\n",
"\n",
" @cachew \n",
" def measurements(chunks: List[Path]) -> Iterator[Measurement]:\n",
" # ...\n",
" \n",
"- as long as `chunks` stay same, data stays same so you always read from sqlite cache which is very fast\n",
"- you don't need to maintain the database, cache is automatically refreshed when `chunks` change (i.e. you got new data)\n",
"\n",
" All the complexity of handling database is hidden in `cachew` implementation."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"autoscroll": false,
"ein.hycell": false,
"ein.tags": "worksheet-0",
"slideshow": {
"slide_type": "-"
}
},
"outputs": [],
"source": [
"[composite] = [x\n",
" for x in ast.walk(ast.parse(inspect.getsource(cachew))) \n",
" if isinstance(x, ast.FunctionDef) and x.name == 'composite_hash'\n",
"]\n",
"\n",
"link = f'{Path(cachew.__file__).relative_to(cwd)}:#L{composite.lineno}'\n",
"\n",
"dmd(f'''\n",
"# How it works\n",
"\n",
"- first your objects get {flink('converted', 'cachew.marshall.cachew.CachewMarshall')} into a simpler JSON-like representation \n",
"- after that, they are mapped into byte blobs via [`orjson`](https://github.com/ijl/orjson).\n",
"\n",
"When the function is called, cachew [computes the hash of your function's arguments ]({link})\n",
"and compares it against the previously stored hash value.\n",
" \n",
"- If they match, it would deserialize and yield whatever is stored in the cache database\n",
"- If the hash mismatches, the original function is called and new data is stored along with the new hash\n",
"''')"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"autoscroll": false,
"ein.hycell": false,
"ein.tags": "worksheet-0",
"slideshow": {
"slide_type": "-"
}
},
"outputs": [],
"source": [
"dmd('# Features')\n",
"types = [f'`{t}`' for t in ['str', 'int', 'float', 'bool', 'datetime', 'date', 'Exception']]\n",
"dmd(f\"\"\"\n",
"* automatic schema inference: {flink('1', 'tests.test_return_type_inference')}, {flink('2', 'tests.test_return_type_mismatch')}\n",
"* supported types: \n",
"\n",
" * primitive: {', '.join(types)}\n",
" \n",
" See {flink('tests.test_types')}, {flink('tests.test_primitive')}, {flink('tests.test_dates')}, {flink('tests.test_exceptions')}\n",
" * {flink('@dataclass and NamedTuple', 'tests.test_dataclass')}\n",
" * {flink('Optional', 'tests.test_optional')} types\n",
" * {flink('Union', 'tests.test_union')} types\n",
" * {flink('nested datatypes', 'tests.test_nested')}\n",
" \n",
"* detects {flink('datatype schema changes', 'tests.test_schema_change')} and discards old data automatically \n",
"\"\"\")\n",
"# * custom hash function TODO example with mtime?"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Performance\n",
"Updating cache takes certain overhead, but that would depend on how complicated your datatype in the first place, so I'd suggest measuring if you're not sure.\n",
"\n",
"During reading cache all that happens is reading blobls from sqlite/decoding as JSON, and mapping them onto your target datatype, so the overhead depends on each of these steps.\n",
"\n",
"It would almost certainly make your program faster if your computations take more than several seconds.\n",
"\n",
"You can find some of my performance tests in [benchmarks/](benchmarks) dir, and the tests themselves in [src/cachew/tests/marshall.py](src/cachew/tests/marshall.py)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"autoscroll": false,
"ein.hycell": false,
"ein.tags": "worksheet-0",
"slideshow": {
"slide_type": "-"
}
},
"outputs": [],
"source": [
"dmd(f\"\"\"\n",
"# Using\n",
"See {flink('docstring', 'cachew.cachew')} for up-to-date documentation on parameters and return types. \n",
"You can also use {flink('extensive unit tests', 'tests')} as a reference.\n",
" \n",
"Some useful (but optional) arguments of `@cachew` decorator:\n",
" \n",
"* `cache_path` can be a directory, or a callable that {flink('returns a path', 'tests.test_callable_cache_path')} and depends on function's arguments.\n",
" \n",
" By default, `settings.DEFAULT_CACHEW_DIR` is used.\n",
" \n",
"* `depends_on` is a function which determines whether your inputs have changed, and the cache needs to be invalidated.\n",
" \n",
" By default it just uses string representation of the arguments, you can also specify a custom callable.\n",
" \n",
" For instance, it can be used to {flink('discard cache', 'tests.test_custom_hash')} if the input file was modified.\n",
" \n",
"* `cls` is the type that would be serialized.\n",
"\n",
" By default, it is inferred from return type annotations, but can be specified explicitly if you don't control the code you want to cache. \n",
"\"\"\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"ein.tags": "worksheet-0",
"slideshow": {
"slide_type": "-"
}
},
"source": [
"# Installing\n",
"Package is available on [pypi](https://pypi.org/project/cachew/).\n",
"\n",
" pip3 install --user cachew\n",
" \n",
"## Developing\n",
"I'm using [tox](tox.ini) to run tests, and [Github Actions](.github/workflows/main.yml) for CI."
]
},
{
"cell_type": "markdown",
"metadata": {
"ein.tags": "worksheet-0",
"slideshow": {
"slide_type": "-"
}
},
"source": [
"# Implementation\n",
"\n",
"* why NamedTuples and dataclasses?\n",
" \n",
" `NamedTuple` and `dataclass` provide a very straightforward and self documenting way to represent data in Python.\n",
" Very compact syntax makes it extremely convenient even for one-off means of communicating between couple of functions.\n",
" \n",
" If you want to find out more why you should use more dataclasses in your code I suggest these links:\n",
" \n",
" - [What are data classes?](https://stackoverflow.com/questions/47955263/what-are-data-classes-and-how-are-they-different-from-common-classes)\n",
" - [basic data classes](https://realpython.com/python-data-classes/#basic-data-classes)\n",
" \n",
"* why not `pandas.DataFrame`?\n",
"\n",
" DataFrames are great and can be serialised to csv or pickled.\n",
" They are good to have as one of the ways you can interface with your data, however hardly convenient to think about it abstractly due to their dynamic nature.\n",
" They also can't be nested.\n",
"\n",
"* why not [ORM](https://en.wikipedia.org/wiki/Object-relational_mapping)?\n",
" \n",
" ORMs tend to be pretty invasive, which might complicate your scripts or even ruin performance. It's also somewhat an overkill for such a specific purpose.\n",
"\n",
" * E.g. [SQLAlchemy](https://docs.sqlalchemy.org/en/13/orm/tutorial.html#declare-a-mapping) requires you using custom sqlalchemy specific types and inheriting a base class.\n",
" Also it doesn't support nested types.\n",
" \n",
"* why not [pickle](https://docs.python.org/3/library/pickle.html) or [`marshmallow`](https://marshmallow.readthedocs.io/en/3.0/nesting.html) or `pydantic`?\n",
"\n",
" Pickling is kinda heavyweigh for plain data class, it's slower just using JSON. Lastly, it can only be loaded via Python, whereas JSON + sqlite has numerous bindings and tools to explore and interface.\n",
"\n",
" Marshmallow is a common way to map data into db-friendly format, but it requires explicit schema which is an overhead when you have it already in the form of type annotations. I've looked at existing projects to utilize type annotations, but didn't find them covering all I wanted:\n",
" \n",
" * https://marshmallow-annotations.readthedocs.io/en/latest/ext/namedtuple.html#namedtuple-type-api\n",
" * https://pypi.org/project/marshmallow-dataclass\n",
" \n",
" I wrote up an extensive review of alternatives I considered: see [doc/serialization.org](doc/serialization.org).\n",
" So far looks like only `cattrs` comes somewhere close to the feature set I need, but still not quite.\n",
"\n",
"* why `sqlite` database for storage?\n",
"\n",
" It's pretty efficient and iterables (i.e. sequences) map onto database rows in a very straightforward manner, plus we get some concurrency guarantees.\n",
"\n",
" There is also a somewhat experimental backend which uses a simple file (jsonl-like) for storage, you can use it via `@cache(backend='file')`, or via `settings.DEFAULT_BACKEND`.\n",
" It's slightly faster than sqlite judging by benchmarks, but unless you're caching millions of items this shouldn't really be noticeable.\n",
" \n",
" It would also be interesting to experiment with in-RAM storages.\n",
"\n",
" I had [a go](https://github.com/karlicoss/cachew/issues/9) at Redis as well, but performance for writing to cache was pretty bad. That said it could still be interesting for distributed caching if you don't care too much about performance.\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Tips and tricks\n",
"## Optional dependency\n",
"You can benefit from `cachew` even if you don't want to bloat your app's dependencies. Just use the following snippet:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import cachew.extra\n",
"dmd(f\"\"\"```python\n",
"{inspect.getsource(cachew.extra.mcachew)}\n",
"```\"\"\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now you can use `@mcachew` in place of `@cachew`, and be certain things don't break if `cachew` is missing.\n",
"\n",
"## Settings"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"dmd(f'''\n",
"{flink('cachew.settings')} exposes some parameters that allow you to control `cachew` behaviour:\n",
"- `ENABLE`: set to `False` if you want to disable caching for without removing the decorators (useful for testing and debugging).\n",
" You can also use {flink('cachew.extra.disabled_cachew')} context manager to do it temporarily.\n",
"- `DEFAULT_CACHEW_DIR`: override to set a different base directory. The default is the \"user cache directory\" (see [appdirs docs](https://github.com/ActiveState/appdirs#some-example-output)).\n",
"- `THROW_ON_ERROR`: by default, cachew is defensive and simply attemps to cause the original function on caching issues.\n",
" Set to `True` to catch errors earlier.\n",
"- `DEFAULT_BACKEND`: currently supported are `sqlite` and `file` (file is somewhat experimental, although should work too). \n",
"\n",
"''')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Updating this readme\n",
"This is a literate readme, implemented as a Jupiter notebook: [README.ipynb](README.ipynb). To update the (autogenerated) [README.md](README.md), use [generate-readme](generate-readme) script."
]
}
],
"metadata": {
"celltoolbar": "Tags",
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.12"
},
"name": "README.ipynb"
},
"nbformat": 4,
"nbformat_minor": 4
}