{"id":25659480,"url":"https://github.com/josuakrause/redipy","last_synced_at":"2025-04-19T17:08:34.082Z","repository":{"id":200243474,"uuid":"697938779","full_name":"JosuaKrause/redipy","owner":"JosuaKrause","description":"redipy is a uniform interface to Redis-like storage systems. It allows you to use the same Redis API with different backends.","archived":false,"fork":false,"pushed_at":"2024-07-08T01:11:58.000Z","size":507,"stargazers_count":6,"open_issues_count":7,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-29T10:42:42.392Z","etag":null,"topics":["backends","in-memory","redis","uniform-api"],"latest_commit_sha":null,"homepage":"https://pypi.org/project/redipy/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/JosuaKrause.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-09-28T19:36:20.000Z","updated_at":"2024-06-27T18:39:43.000Z","dependencies_parsed_at":null,"dependency_job_id":"598432b8-fc8a-47fd-a178-e588e54f9be0","html_url":"https://github.com/JosuaKrause/redipy","commit_stats":null,"previous_names":["josuakrause/pyredis","josuakrause/redist"],"tags_count":10,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JosuaKrause%2Fredipy","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JosuaKrause%2Fredipy/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JosuaKrause%2Fredipy/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JosuaKrause%2Fredipy/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/JosuaKrause","download_url":"https://codeload.github.com/JosuaKrause/redipy/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249745977,"owners_count":21319581,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["backends","in-memory","redis","uniform-api"],"created_at":"2025-02-24T01:17:21.150Z","updated_at":"2025-04-19T17:08:34.062Z","avatar_url":"https://github.com/JosuaKrause.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# RediPy\n\n`redipy` is a Python library that provides a uniform interface to Redis-like\nstorage systems. It allows you to use the same Redis API with different backends\nthat implement the same functionality, such as:\n\n- `redipy.memory`: A backend that runs inside the current process and stores\n  data in memory using Python data structures.\n- `redipy.redis`: A backend that connects to an actual Redis instance and\n  delegates all operations to it.\n\n[![redipy logo][logo-small]][logo]\n\n## Overview\u003ca id=\"toc\"\u003e\u003c/a\u003e\n\n\u003cdetails\u003e\u003csummary\u003eTable of Contents\u003c/summary\u003e\n\n1. [Installation](#installation)\n2. [Usage](#usage)\n3. [Features](#features)\n4. [Custom Scripts](#custom-scripts)\n   * [Simple Example](#simple-example)\n   * [Advanced Example](#advanced-example)\n5. [Limitations](#limitations)\n6. [Contributing](#contributing)\n   * [If You Find a Bug](#if-you-find-a-bug)\n   * [Missing Redis or Lua Functions](#missing-redis-or-lua-functions)\n   * [Implementing New Redis Functions](#implementing-new-redis-functions)\n7. [Changelog](#changelog)\n8. [License](#license)\n9. [Feedback](#feedback)\n\n\u003c/details\u003e\n\nThis [medium article][medium] explores some of the rationale behind the library.\n\nIf you need certain functionality or found a bug, have a look at the\n[contributing](#contributing) section.\n\n## Installation\u003ca id=\"installation\"\u003e\u003c/a\u003e\nYou can install `redipy` using pip:\n\n```sh\npip install redipy\n```\n\n[🔝](#toc)\n\n## Usage\u003ca id=\"usage\"\u003e\u003c/a\u003e\nTo use `redipy`, you need to import the library and create a `redipy` client\nobject with the desired backend. For example:\n\n```python\n# Import the redipy library\nimport redipy\n\n# Create a redipy client using the memory backend\nr = redipy.Redis()\n\n# Create a redipy client using the redis backend\nr = redipy.Redis(host=\"localhost\", port=6379)\n\n# Or preferred\nr = redipy.Redis(\n    cfg={\n        \"host\": \"localhost\",\n        \"port\": 6379,\n        \"passwd\": \"\",\n        # A prefix that gets added to every key.\n        # Can be used to implement namespaces.\n        \"prefix\": \"\",\n    })\n\n# You can specify the backend explicitly to ensure that the correct parameters\n# are passed to the constructor\nr = redipy.Redis(\n    backend=\"redis\",\n    cfg={\n        \"host\": \"localhost\",\n        \"port\": 6379,\n        \"passwd\": \"\",\n        \"prefix\": \"\",\n    })\n```\n\nThe `redipy` client object supports similar methods and attributes to the\nofficial [Redis][redis] Python client library.\nYou can use them as you would normally do with `redis`. For example:\n\n```python\n# Set some values\nr.set_value(\"foo\", \"bar\")\nr.set_value(\"baz\", \"qux\")\n\n# Get some values\nr.get_value(\"foo\")  # \"bar\"\nr.get_value(\"baz\")  # \"qux\"\n\n# Push some values\nr.lpush(\"mylist\", \"a\", \"b\", \"c\")\nr.rpush(\"mylist\", \"d\")\n\n# Pop values\nr.lpop(\"mylist\")  # \"c\"\nr.rpop(\"mylist\", 3)  # [\"d\", \"a\", \"b\"]\n```\n\nMore examples can be found in the [examples folder][examples].\n\n[🔝](#toc)\n\n## Features\u003ca id=\"features\"\u003e\u003c/a\u003e\nThe main features of `redipy` are:\n\n- Flexibility: You can choose from different backends that suit your needs and\n  preferences, without changing your code or learning new APIs.\n\n- Adaptability: You can start your project small with the memory backend and\n  only switch to a full Redis server once the application grows.\n\n- Scripting: You can create backend independent Redis scripts without using Lua.\n  Scripts are written using a symbolic API in python.\n\n- Compatibility: You can use any Redis client or tool with any backend.\n\n- Mockability: You can use redipy in tests that require Redis with the memory\n  backend to easily mock the functionality without actually having to run a\n  Redis server in the background. Also, this avoids issues that might occur\n  when running tests in parallel with an actual Redis server.\n\n- Performance: You can leverage the high performance of Redis or other backends\n  that offer fast and scalable data storage and retrieval.\n\n- Migration: You can easily migrate data between different backends, or use\n  multiple backends simultaneously.\n\n[🔝](#toc)\n\n## Custom Scripts\u003ca id=\"custom-scripts\"\u003e\u003c/a\u003e\n\nRedis scripts can be defined via a symbolic API in python and can be executed\nby any backend.\n\n[🔝](#toc)\n\n### Simple Example\u003ca id=\"simple-example\"\u003e\u003c/a\u003e\n\nHere, we are writing a filter function that drains a Redis list\nand puts items into a \"left\" and a \"right\" list by comparing each items\nnumerical value with a given `cmp` value:\n\n```python\nimport redipy\n\n# set up script\nctx = redipy.script.FnContext()\n# add argument\ncmp = ctx.add_arg(\"cmp\")\n# add key arguments\ninp = redipy.script.RedisList(ctx.add_key(\"inp\"))\nleft = redipy.script.RedisList(ctx.add_key(\"left\"))\nright = redipy.script.RedisList(ctx.add_key(\"right\"))\n\n# add local variable which contains the current value pop'ed from the list\ncur = ctx.add_local(inp.lpop())\n# we consume \"inp\" until it is empty\nloop = ctx.while_(cur.ne_(None))\n# push the value to the list depending on whether it is smaller than `cmp`\nb_then, b_else = loop.if_(redipy.script.ToNum(cur).lt_(cmp))\nb_then.add(left.rpush(cur))\nb_else.add(right.rpush(cur))\n# pop next value and store in local variable\nloop.add(cur.assign(inp.lpop()))\n# the script doesn't return a value\nctx.set_return_value(None)\n\n# make sure to build the script only once and reuse the filter_list function\nfilter_list = r.register_script(ctx)\n\nr.rpush(\"mylist\", \"1\", \"3\", \"2\", \"4\")\nfilter_list(\n    keys={\n        \"inp\": \"mylist\",\n        \"left\": \"small\",\n        \"right\": \"big\",\n    },\n    args={\n        \"cmp\": 3,\n    })\n\nr.lpop(\"mylist\", 4)  # []\nr.lpop(\"small\", 4)  # [\"1\", \"2\"]\nr.lpop(\"big\", 4)  # [\"3\", \"4\"]\n```\n\n[🔝](#toc)\n\n### Advanced Example\u003ca id=\"advanced-example\"\u003e\u003c/a\u003e\n\nHere, we are implementing and object stack with fall-through lookup. Each frame\nin the stack has its own fields. If the user tries to access a field that\ndoesn't exist in the current stack frame (and they are using `get_cascading`)\nthe accessor will recursively go down the stack until a value for the given\nfield is found (or the end of the stack is reached).\n\n```python\nfrom typing import cast\nfrom redipy import RedisClientAPI\nfrom redipy.script import (\n    ExecFunction,\n    FnContext,\n    JSONType,\n    RedisHash,\n    RedisVar,\n    Strs,\n    ToIntStr,\n    ToNum,\n)\n\n\nclass RStack:\n    \"\"\"An example class that simulates a key value stack.\"\"\"\n    def __init__(self, rt: RedisClientAPI) -\u003e None:\n        self._rt = rt\n\n        self._set_value = self._set_value_script()\n        self._get_value = self._get_value_script()\n        self._pop_frame = self._pop_frame_script()\n        self._get_cascading = self._get_cascading_script()\n\n    def key(self, base: str, name: str) -\u003e str:\n        \"\"\"\n        Compute the key.\n\n        Args:\n            base (str): The base key.\n\n            name (str): The name.\n\n        Returns:\n            str: The key associated with the name.\n        \"\"\"\n        return f\"{base}:{name}\"\n\n    def push_frame(self, base: str) -\u003e None:\n        \"\"\"\n        Pushes a new stack frame.\n\n        Args:\n            base (str): The base key.\n        \"\"\"\n        self._rt.incrby(self.key(base, \"size\"), 1)\n\n    def pop_frame(self, base: str) -\u003e dict[str, str]:\n        \"\"\"\n        Pops the current stack frame and returns its values.\n\n        Args:\n            base (str): The base key.\n\n        Returns:\n            dict[str, str] | None: The content of the stack frame.\n        \"\"\"\n        res = self._pop_frame(\n            keys={\n                \"size\": self.key(base, \"size\"),\n                \"frame\": self.key(base, \"frame\"),\n            },\n            args={})\n        if res is None:\n            return {}\n        return cast(dict, res)\n\n    def set_value(self, base: str, field: str, value: str) -\u003e None:\n        \"\"\"\n        Set a value in the current stack frame.\n\n        Args:\n            base (str): The base key.\n\n            field (str): The field.\n\n            value (str): The value.\n        \"\"\"\n        self._set_value(\n            keys={\n                \"size\": self.key(base, \"size\"),\n                \"frame\": self.key(base, \"frame\"),\n            },\n            args={\"field\": field, \"value\": value})\n\n    def get_value(self, base: str, field: str) -\u003e JSONType:\n        \"\"\"\n        Returns a value from the current stack frame.\n\n        Args:\n            base (str): The base key.\n\n            field (str): The field.\n\n        Returns:\n            JSONType: The value.\n        \"\"\"\n        return self._get_value(\n            keys={\n                \"size\": self.key(base, \"size\"),\n                \"frame\": self.key(base, \"frame\"),\n            },\n            args={\"field\": field})\n\n    def get_cascading(self, base: str, field: str) -\u003e JSONType:\n        \"\"\"\n        Returns a value from the stack. If the value is not in the current\n        stack frame the value is recursively retrieved from the previous\n        stack frames.\n\n        Args:\n            base (str): The base key.\n\n            field (str): The field.\n\n        Returns:\n            JSONType: The value.\n        \"\"\"\n        return self._get_cascading(\n            keys={\n                \"size\": self.key(base, \"size\"),\n                \"frame\": self.key(base, \"frame\"),\n            },\n            args={\"field\": field})\n\n    def _set_value_script(self) -\u003e ExecFunction:\n        ctx = FnContext()\n        rsize = RedisVar(ctx.add_key(\"size\"))\n        rframe = RedisHash(Strs(\n            ctx.add_key(\"frame\"),\n            \":\",\n            ToIntStr(rsize.get_value(default=0))))\n        field = ctx.add_arg(\"field\")\n        value = ctx.add_arg(\"value\")\n        ctx.add(rframe.hset({\n            field: value,\n        }))\n        ctx.set_return_value(None)\n        return self._rt.register_script(ctx)\n\n    def _get_value_script(self) -\u003e ExecFunction:\n        ctx = FnContext()\n        rsize = RedisVar(ctx.add_key(\"size\"))\n        rframe = RedisHash(Strs(\n            ctx.add_key(\"frame\"),\n            \":\",\n            ToIntStr(rsize.get_value(default=0))))\n        field = ctx.add_arg(\"field\")\n        ctx.set_return_value(rframe.hget(field))\n        return self._rt.register_script(ctx)\n\n    def _pop_frame_script(self) -\u003e ExecFunction:\n        ctx = FnContext()\n        rsize = RedisVar(ctx.add_key(\"size\"))\n        rframe = RedisHash(Strs(\n            ctx.add_key(\"frame\"),\n            \":\",\n            ToIntStr(rsize.get_value(default=0))))\n        lcl = ctx.add_local(rframe.hgetall())\n        ctx.add(rframe.delete())\n\n        b_then, b_else = ctx.if_(ToNum(rsize.get_value(default=0)).gt_(0))\n        b_then.add(rsize.incrby(-1))\n        b_else.add(rsize.delete())\n\n        ctx.set_return_value(lcl)\n        return self._rt.register_script(ctx)\n\n    def _get_cascading_script(self) -\u003e ExecFunction:\n        ctx = FnContext()\n        rsize = RedisVar(ctx.add_key(\"size\"))\n        base = ctx.add_local(ctx.add_key(\"frame\"))\n        field = ctx.add_arg(\"field\")\n        pos = ctx.add_local(ToNum(rsize.get_value(default=0)))\n        res = ctx.add_local(None)\n        cur = ctx.add_local(None)\n        rframe = RedisHash(cur)\n\n        loop = ctx.while_(res.eq_(None).and_(pos.ge_(0)))\n        loop.add(cur.assign(Strs(base, \":\", ToIntStr(pos))))\n        loop.add(res.assign(rframe.hget(field)))\n        loop.add(pos.assign(pos - 1))\n\n        ctx.set_return_value(res)\n        return self._rt.register_script(ctx)\n```\n\n[🔝](#toc)\n\n## Limitations\u003ca id=\"limitations\"\u003e\u003c/a\u003e\nThe current limitations of `redipy` are:\n\n- Some Redis commands are not supported yet: This is likely due to redundant\n  functionality. For all other cases it will eventually be resolved.\n  Check [this issue to see the status of redis functions][implemented].\n- The API differs slightly: Most notably stored values are always strings\n  (i.e., the bytes returned by Redis are decoded as utf-8).\n- The semantic of Redis functions inside scripts has been altered to feel more\n  natural coming from python: Redis functions inside Lua scripts often differ\n  greatly from the documented behavior. For example, `LPOP` returns `false` for\n  an empty list inside Lua (instead of `nil` or `cjson.null`). While `LPOP`\n  returns `None` in the python API. The script API of `redipy` has been altered\n  to match the python API more closely. As the user doesn't code in Lua directly\n  the benefit of having a more consistent API outweighs the more complicated Lua\n  code that needs to be generated in the backend.\n- Scripts aim to use python semantics as best as possible: In Lua array indices\n  start at 1. The script API uses a 0 based indexing system and transparently\n  adjusts indices in the Lua backend. Other, similar changes are performed\n  as well.\n- Scripts use JSON to pass arguments and return values: The arguments to the\n  script are passed as JSON bytes for the Lua backend. Keys are passed as is.\n  The return value of the script is also converted into JSON when moving from\n  Lua to python. Note, that the empty dictionary (`{}`) and the empty list\n  (`[]`) are indistinguishable in Lua so `None` is returned instead of setting\n  the return value to either of these.\n\n[🔝](#toc)\n\n## Contributing\u003ca id=\"contributing\"\u003e\u003c/a\u003e\n\nAny contribution, even if it is just creating an issue for a bug,\nis much appreciated.\n\n[🔝](#toc)\n\n### If You Find a Bug\u003ca id=\"if-you-find-a-bug\"\u003e\u003c/a\u003e\n\nIf you encounter a bug, please open an issue to draw attention to it or give\na thumbsup if the issue already exists. This helps with prioritizing\nimplementation efforts. Even if you cannot solve the bug yourself,\ninvestigating why it happens or creating a PR to add test cases helps a lot.\nIf you have a fix for a bug don't hesistate to open a PR.\n\n[🔝](#toc)\n\n### Missing Redis or Lua Functions\u003ca id=\"missing-redis-or-lua-functions\"\u003e\u003c/a\u003e\nIf you encounter a missing Redis or Lua function please consider adding it\nyourself (see the [implementing](#implementing-new-redis-functions) section).\nHere also opening an issue or giving a thumbsup to existing issues helps\nwith prioritization.\n\nHowever, if you need it only in your local setup\nwithout API support or support for multiple backends, pipelines, etc. you can\nuse the raw underlying Redis connection via\n`redipy.main.Redis.get_redis_runtime` and\n`redipy.redis.conn.RedisConnection.get_connection` or make use of\nthe plug-in mechanism.\n\nFor the memory backend you can use\n`redipy.memory.rt.LocalRuntime.add_redis_function_plugin` or\n`redipy.memory.rt.LocalRuntime.add_general_function_plugin`. The methods need\na module that contains subclasses of `redipy.plugin.LocalRedisFunction` and\n`redipy.plugin.LocalGeneralFunction` respectively. Once the new functions are\ndefined via loading the plugin they can be used in a `redipy.script.FnContext`\nvia `redipy.script.RedisFn` or `redipy.script.CallFn` respectively.\n\nNote, that `redipy.script.RedisFn` and `redipy.script.CallFn` can always be\nused in Redis backend scripts. However, calling functions this way will have\nthe native Lua behavior which can lead to surprising results. To patch those\nup as well you can use `redipy.redis.lua.LuaBackend.add_redis_patch_plugin`,\n`redipy.redis.lua.LuaBackend.add_general_patch_plugin`, and\n`redipy.redis.lua.LuaBackend.add_helper_function_plugin` to add the subclasses\nof `redipy.plugin.LuaRedisPatch`, `redipy.plugin.LuaRedisPatch`, and\n`redipy.plugin.HelperFunction` respectively. Those functions then can also be\nused with the `redipy.script.RedisFn` and `redipy.script.CallFn` commands.\n\nAdding functions as described above is discouraged as it may lead to\ninconsistent support of different backends and inconsistent behavior across\ndifferent backends.\n\n[🔝](#toc)\n\n### Implementing New Redis Functions\u003ca id=\"implementing-new-redis-functions\"\u003e\u003c/a\u003e\n\nThe easiest way to contribute to `redipy` is to pick some Redis API functions\nthat have not (or not completely) been [implemented][implemented] in `redipy`\nyet. It is also much appreciated if you just add test cases or the stubs in a\nPR. For a full implementation follow these steps:\n\n1. Add the signature of the function to `redipy.api.RedisAPI`. Adjust as\n  necessary from the Redis spec to get a pythonic feel. Also, add the signature\n  to `redipy.api.PipelineAPI` but with `None` as return value. Additionally,\n  add the redirect to the backend in `redipy.main.Redis`.\n2. Implement the function in `redipy.redis.conn.RedisConnection` and\n  `redipy.redis.conn.PipelineConnection`. This should\n  be straightforward as there are not too many changes expected. Don't forget\n  to convert bytes into strings via `...decode(\"utf-8\")` (there are various\n  helper functions for this in `redipy.util`).\n3. Add tests to `test/test_sanity.py` to determine the function's behavior in\n  Lua (especially its edge cases).\n4. If the Lua behavior needs to be changed to provide a better feel you can add\n  a monkeypatch for the function call by either creating a class in\n  `redipy.redis.rpatch` to directly change the returned expr for the execution\n  graph or using a Lua helper function via adding a class to\n  `redipy.redis.helpers` (you need to use a patch to use the helper in the\n  right location).\n5. Next, add and implement the functionality in\n  `redipy.memory.state.Machine` and add the appropriate redirects in\n  `redipy.memory.rt.LocalRuntime` and `redipy.memory.rt.LocalPipeline`.\n6. To make the new function accessible in scripts from the memory backend add\n  a class in `redipy.memory.rfun`.\n7. Add the approriate class or method in the right `redipy.symbolic.r...py`\n  file. If it is a new class / file add an import to `redipy.script`.\n8. Add a new test in `test/test_api.py` to verify the new function works inside\n  a script for all backends. You can run `make pytest FILE=test/test_api.py`\n  to execute the test and `make coverage-report` to verify that the new code\n  is executed.\n9. Make sure `make lint-all` passes, as well as, all tests (`make pytest`)\n  run without issue.\n\nYou can submit your patch as pull request [here][pulls].\n\n[🔝](#toc)\n\n## Changelog\u003ca id=\"changelog\"\u003e\u003c/a\u003e\nThe changelog can be found [here][changelog].\n\n[🔝](#toc)\n\n## License\u003ca id=\"license\"\u003e\u003c/a\u003e\n`redipy` is licensed under the [Apache License (Version 2.0)][license].\n\n[🔝](#toc)\n\n## Feedback\u003ca id=\"feedback\"\u003e\u003c/a\u003e\nIf you have any questions, suggestions, or issues with `redipy`, please feel\nfree to [open an issue][issues] on GitHub. I would love to hear your feedback\nand improve `redipy`. Thank you!\n\n[🔝](#toc)\n\n[changelog]: https://github.com/JosuaKrause/redipy/blob/main/CHANGELOG.md\n[examples]: https://github.com/JosuaKrause/redipy/tree/main/examples\n[implemented]: https://github.com/JosuaKrause/redipy/issues/8\n[issues]: https://github.com/JosuaKrause/redipy/issues\n[license]: https://github.com/JosuaKrause/redipy/blob/v0.4.2/LICENSE\n[logo-small]: https://raw.githubusercontent.com/JosuaKrause/redipy/v0.4.2/img/redipy_logo_small.png\n[logo]: https://raw.githubusercontent.com/JosuaKrause/redipy/v0.4.2/img/redipy_logo.png\n[medium]: https://medium.josuakrause.com/a-backend-agnostic-redis-interface-9fdeb8641bc5?source=friends_link\u0026sk=565113f29f4cfadf174aa3ab23ab9a16\n[pulls]: https://github.com/JosuaKrause/redipy/pulls\n[redis]: https://pypi.org/project/redis/\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjosuakrause%2Fredipy","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjosuakrause%2Fredipy","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjosuakrause%2Fredipy/lists"}