{"id":24345943,"url":"https://github.com/chordify/redis-schema","last_synced_at":"2025-04-09T18:08:52.542Z","repository":{"id":58299861,"uuid":"505506270","full_name":"chordify/redis-schema","owner":"chordify","description":"Typed, schema-based, composable Redis library for Haskell","archived":false,"fork":false,"pushed_at":"2024-07-16T15:22:39.000Z","size":63,"stargazers_count":8,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-04-03T05:55:49.578Z","etag":null,"topics":["haskell","library","redis","schema","typed"],"latest_commit_sha":null,"homepage":"","language":"Haskell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/chordify.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-06-20T15:54:50.000Z","updated_at":"2024-11-20T21:12:50.000Z","dependencies_parsed_at":"2024-04-15T09:43:15.825Z","dependency_job_id":"aa16af7c-6f8a-47c1-8c7b-27351e477dff","html_url":"https://github.com/chordify/redis-schema","commit_stats":{"total_commits":83,"total_committers":3,"mean_commits":"27.666666666666668","dds":0.03614457831325302,"last_synced_commit":"017048f67d5c13ee1b27da4fa41faf7669c4aba4"},"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chordify%2Fredis-schema","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chordify%2Fredis-schema/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chordify%2Fredis-schema/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chordify%2Fredis-schema/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/chordify","download_url":"https://codeload.github.com/chordify/redis-schema/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248085155,"owners_count":21045135,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["haskell","library","redis","schema","typed"],"created_at":"2025-01-18T10:21:16.857Z","updated_at":"2025-04-09T18:08:52.522Z","avatar_url":"https://github.com/chordify.png","language":"Haskell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# redis-schema\n\nA typed, schema-based, composable Redis library.\nIt strives to provide a solid layer on top of which you can\ncorrectly build your application or another library.\n\n## Table of contents\n* [Table of contents](#table-of-contents)\n* [Why `redis-schema`](#why-redis-schema)\n  * [Statically typed schema](#statically-typed-schema)\n    * [Hedis](#hedis)\n    * [`redis-schema`](#redis-schema)\n  * [Composability](#composability)\n* [Tutorial by example](#tutorial-by-example)\n  * [Simple variables](#simple-variables)\n  * [Parameterised references](#parameterised-references)\n  * [Lists, Sets, Hashes, etc.](#lists-sets-hashes-etc)\n  * [Hashes](#hashes)\n    * [Aside: Hashes vs. composite keys](#aside-hashes-vs-composite-keys)\n  * [Records](#records)\n    * [Aside: non-fixed record fields](#aside-non-fixed-record-fields)\n  * [Transactions](#transactions)\n    * [The `Tx` functor](#the-tx-functor)\n    * [Working with transactions](#working-with-transactions)\n    * [What Redis transactions cannot do](#what-redis-transactions-cannot-do)\n    * [Errors in transactions](#errors-in-transactions)\n    * [Monads vs applicative functors](#monads-vs-applicative-functors)\n  * [Exceptions](#exceptions)\n  * [Custom data types](#custom-data-types)\n    * [Simple values](#simple-values)\n    * [Non-simple values](#non-simple-values)\n    * [Redis instances](#redis-instances)\n  * [Meta-records](#meta-records)\n    * [Aside: references](#aside-references)\n    * [Aside: instances](#aside-instances)\n* [Libraries](#libraries)\n  * [Locks](#locks)\n  * [Remote jobs](#remote-jobs)\n* [Future work](#future-work)\n* [License](#license)\n\n## Why `redis-schema`\n\n### Statically typed schema\n\n#### Hedis\n\nThe most common Redis library seems to be\n[Hedis](https://hackage.haskell.org/package/hedis), and `redis-schema` builds\non top of it. However, consider the type of `get` in Hedis:\n\n```haskell\nget\n    :: (RedisCtx m f)\n    =\u003e ByteString -- ^ key\n    -\u003e m (f (Maybe ByteString))\n```\n\nFor most use cases, it would be nice if:\n* the value could be decoded from a `ByteString` automatically\n  * provides convenience but also type safety\n* the key could imply the type of the value\n  * provides type safety\n  * guides programmer, documents structures, etc. -- everything we love about static types\n  * it's also immediately clear which instance to use for decoding\n\n#### `redis-schema`\n\nIn `redis-schema`, the type of `get` is:\n```haskell\nget :: Ref ref =\u003e ref -\u003e RedisM (RefInstance ref) (Maybe (ValueType ref))\n```\nand it makes use of user-supplied declarations:\n```haskell\ndata NumberOfVisitors = NumberOfVisitors Date\n\ninstance Ref NumberOfVisitors where\n  type ValueType NumberOfVisitors = Integer\n  toIdentifier (NumberOfVisitors date) =\n    SviTopLevel $ Redis.colonSep [\"number-of-visitors\", BS.pack (show date)]\n```\n\nThe differences are:\n* Instead of `ByteStrings`, `redis-schema` uses references that are usually\n  bespoke ADTs, such as `NumberOfVisitors`.\n* Bespoke reference types eliminate string operations scattered across the code:\n  you write `get (NumberOfVisitors today)` instead of\n  `get (\"number-of-visitors:\" \u003c\u003e BS.pack (show today))`.\n  `ByteString` concatenation of course needs to be done somewhere\n  but it's implemented only once: in the `toIdentifier` method.\n* References are more abstract than bytestring keys, which improves composability.\n  For example, meta-records [use this abstractness](#aside-references),\n  as a meta-record consists of multiple Redis keys, and thus there's no single bytestring\n  that could reasonably identify it.\n* The `Ref` instance of that data type determines that\n  the reference stores `Integer`s. This can be seen\n  in the associated type family `ValueType`.\n\nMore complex data structures, like records, work similarly.\n\n### Composability\n\nA major goal of `redis-schema` is to provide typed primitives,\non top of which one can safely and conveniently build further typed libraries,\nsuch as [`Database.Redis.Schema.Lock`](#locks)\nor [`Database.Redis.Schema.RemoteJob`](#remote-jobs).\n[Meta-records](#meta-records) are another example of how low-level\nprimitives compose into higher-level \"primitives\" of the same kind.\n\nThe focus at composability is reflected in the design decisions of various typeclasses,\nand in the design and use of Redis transactions to ensure that\ncomposability is not broken by race conditions.\n\n## Tutorial by example\n\nImagine you want to use Redis to count the number of the visitors\non your website. This is how you would do it with `redis-schema`.\n\n### Simple variables\n\n(For demonstration purposes, the following example also includes some\nbasic operations you might *not* do while counting visitors, too. :) )\n\n```haskell\n-- This module is generally intended to be imported qualified.\nimport qualified Database.Redis.Schema as Redis\n\n-- The type of references to the number of visitors.\n-- Since we want only one number of visitors, this type is a singleton.\n-- Later on, we'll see more interesting types of references.\ndata NumberOfVisitors = NumberOfVisitors\n\n-- We define that NumberOfVisitors is indeed a Redis reference.\ninstance Redis.Ref NumberOfVisitors where\n  -- The type of the value that NumberOfVisitors refers to is Int.\n  type ValueType NumberOfVisitors = Int\n\n  -- The location of the value that NumberOfVisitors refers to is \"visitors:number\".\n  toIdentifier NumberOfVisitors = \"visitors:number\"\n\nf :: Redis.Pool -\u003e IO ()\nf pool = Redis.run pool $ do\n  -- write to the reference\n  set NumberOfVisitors 42\n  setTTL NumberOfVisitors (24 * Redis.hour)\n\n  -- atomically increment the number of visitors\n  incrementBy NumberOfVisitors 1\n\n  -- atomically read and clear (zero) the reference\n  -- useful for transactional moves of data\n  n2 \u003c- take NumberOfVisitors\n  liftIO $ print n2\n\n  -- read the value of the reference\n  n \u003c- get NumberOfVisitors\n  liftIO $ print n  -- this prints \"Just 0\", assuming no writes from other threads\n```\n\n### Parameterised references\n\nIf you want a separate counter for every day,\nyou define a slightly more interesting reference type.\n\n```haskell\n-- Note that the type constructor is still nullary (no parameters)\n-- but the data constructor takes the 'Date' in question.\ndata DailyVisitors = DailyVisitors Date\n\ninstance Redis.Ref DailyVisitors where\n  -- Again, the reference points to an 'Int'.\n  -- We're talking about the type of the reference so no date is present here.\n  type ValueType DailyVisitors = Int\n\n  -- The location does depend on the value of the reference,\n  -- so it can depend on the date. We include the date in the Redis path.\n  toIdentifier (DailyVisitors date) =\n    Redis.colonSep [\"visitors\", \"daily\", ByteString.pack (show date)]\n\nf :: Redis.Pool -\u003e Date -\u003e IO ()\nf pool today = Redis.run pool $ do\n  -- atomically bump the number of visitors\n  incrementBy (DailyVisitors today) 1\n\n  -- (other threads may modify the value here)\n\n  -- read and print the reference\n  n \u003c- get (DailyVisitors today)\n  liftIO $ print n\n```\n\nWith composite keys, it's sometimes useful to use `Redis.colonSep`,\nwhich builds a single colon-separated `ByteString` from the provided components.\n\n### Lists, Sets, Hashes, etc.\n\nWhat we've read/written so far were `SimpleValue`s: data items that can be\nencoded as `ByteString`s and used without restrictions.\nHowever, Redis also provides richer data structures, including lists, sets,\nand maps/hashes.\n\nThe advantage is that Redis provides operations to manipulate these data\nstructures directly. You can insert elements, delete elements, etc., without\nreading a `ByteString`-encoded structure and writing its modified version back.\n\nThe disadvantage is that Redis does not support nesting them.\n\nThat does not mean there's absolutely no way to put sets in sets --\nif you encode the inner sets into ByteString, you can nest them however you want.\nHowever, you will not be able to use native Redis functions like `sInsert` or `sDelete`\nto modify the inner sets; you'd have to read, modify, and write back the entire inner value to do it\n-- and that, besides being inconvenient and inefficient,\n[cannot be done atomically in Redis](#transactions).\n\nThis is reflected in `redis-schema` by the fact that\nthe `SimpleValue` instance is not defined for `Set a`, `Map k v` and `[a]`,\nwhich prevents nesting them directly.\n\nOn the other hand, `redis-schema` defines additional functions\nspecific to these data structures, such as the above mentioned\n`sInsert`, which is used to insert elements into a Redis set.\n\n```haskell\n-- The set of visitor IDs for the given date.\ndata DailyVisitorSet = DailyVisitorSet Date\n\ninstance Redis.Ref DailyVisitorSet where\n  -- This reference points to a set of visitor IDs.\n  type ValueType DailyVisitorSet = Set VisitorId\n\n  -- The Redis location of the value.\n  toIdentifier (DailyVisitorSet date) =\n    Redis.colonSep [\"visitor_set\", \"daily\", ByteString.pack (show date)]\n\nf :: Redis.Pool -\u003e Date -\u003e VisitorId -\u003e IO ()\nf pool today vid = Redis.run pool $ do\n  -- insert the visitor ID\n  sInsert (DailyVisitorSet today) vid\n\n  -- get the size of the updated set\n  -- (and print it)\n  liftIO . print =\u003c\u003c sSize (DailyVisitorSet today)\n\n  -- atomically get and clear the visitor set\n  -- (and print it)\n  liftIO . print =\u003c\u003c take (DailyVisitorSet today)\n```\n\nThere is a number of functions available for these structures,\nrefer to the reference documentation / source code for a complete list.\n\nAlso, we add functions when we need them, so it's quite possible that the function\nthat you require has not been added yet. Pull requests are welcome.\n\n### Hashes\n\nThere is a special operator `(:/)` to access the items of a hash,\nas if they were individual Redis `Ref`s.\nHere's our running example with website visitors,\nexcept that now instead of just the count of visits, or just the set of visitors,\nwe will store exactly how many times each visitor has visited us.\n\n```haskell\ndata Visitors = Visitors Date\n\ninstance Redis.Ref Visitors where\n  -- Each daily visitor structure is a map from visitor ID to the number of visits.\n  type ValueType Visitors = Map VisitorId Int\n\n  toIdentifier (Visitors date) =\n    Redis.colonSep [\"visitors\", ByteString.pack (show date)]\n\nf :: Redis.Pool -\u003e Date -\u003e VisitorId -\u003e IO ()\nf pool today visitorId = do\n  -- increment one specific counter inside the hash\n  incrementBy (Visitors today :/ visitorId) 1\n\n  -- print all visitors\n  allVisitors \u003c- get (Visitors today)\n  print allVisitors\n```\n\nUsing operator `(:/)`, we could write `Visitors today :/ visitorId`\nto reference a single field of a hash. However, we can also\nretrieve and print the whole hash if we choose to.\n\n#### Aside: Hashes vs. composite keys\n\nIn the previous example, the reference `Visitors date`\npoints to a `Map VisitorId Int`. This is one realisation of a mapping\n`(Date, VisitorId) -\u003e Int` but not the only possible one.\nAnother way would be including the `VisitorId` in the key like this:\n\n```haskell\ndata VisitCount = VisitCount Date VisitorId\n\ninstance Redis.Ref VisitCount where\n  type ValueType VisitCount = Int\n\n  toIdentifier (VisitCount date visitorId) =\n    Redis.colonSep\n      [ \"visitors\"\n      , ByteString.pack (show date)\n      , ByteString.pack (show visitorId)\n      ]\n```\n\nThis way, every date-visitor combination gets its own full key-value entry\nin Redis. There are advantages and disadvantages to either representation.\n\n* With hashes, you also implicitly get a list of visitor IDs for each day.\n  With composite keys, you have to use the `SCAN` or `KEYS` Redis command.\n\n* It's easy to `get`, `set` or `take` whole hashes (atomically).\n  With separate keys, you have to use an explicit transaction,\n  and code up these operations manually.\n\n* Hashes take less space than the same number of values in separate keys.\n\n* You cannot set the TTL of items in a hash separately: only the whole hash has a TTL.\n  With separate keys, you can set TTL individually.\n\n* You cannot have complex data types (Redis sets, Redis hashes, etc.)\n  nested inside hashes without encoding them as `ByteString`s first.\n  (See [Lists, sets, hashes, etc.](#lists-sets-hashes-etc))\n  There are no such restrictions for separate keys.\n\nHence the encoding depends on your use case. If you're caching\na set of related things for a certain visitor, which you want to read as a whole\nand expire as a whole, it makes sense to put them in a hash.\n\nIf your items are rather separate, you want to expire them separately,\nor you want to store structures like hashes inside,\nyou have to put them in separate keys.\nFields like `date` should probably generally go in the (possibly composite) key\nbecause they will likely affect the required expiration time.\n\n### Records\n\nWe have just seen how to use Redis hashes to store values of type `Map k v`.\nThe number of items in the map is unlimited\nbut all keys and values must have the same type.\n\nThere's another (major) use case for Redis hashes: records.\nRecords are structures which contain a fixed number of named values,\nwhere each value can have a different type.\nIt is therefore a natural way of clustering related data together.\n\nHere's an example showing how records are modelled in `redis-schema`.\n\n```haskell\n-- First, we use GADTs to describe the available fields and their types.\n-- Here, 'Email' has type 'Text', 'DateOfBirth' has type 'Date',\n-- and 'Visits' and 'Clicks' have type 'Int'.\ndata VisitorField :: * -\u003e * where\n  Email :: VisitorField Text\n  DateOfBirth :: VisitorField Date\n  Visits :: VisitorField Int\n  Clicks :: VisitorField Int\n\n-- We define how to translate record keys to strings\n-- that will be used to key the Redis hash.\ninstance Redis.RecordField VisitorField where\n  rfToBS Email = \"email\"\n  rfToBS DateOfBirth = \"date-of-birth\"\n  rfToBS Visits = \"visits\"\n  rfToBS Clicks = \"clicks\"\n\n-- Then we define the type of references pointing to the visitor statistics\n-- for any given visitor ID.\ndata VisitorStats = VisitorStats VisitorId\n\n-- Finally, we declare that the type of references is indeed a Redis reference.\ninstance Redis.Ref VisitorStats where\n  -- The type pointed to is 'Redis.Record VisitorField', which means\n  -- a record with the fields defined by 'VisitorField'.\n  type ValueType VisitorStats = Redis.Record VisitorField\n\n  -- As usual, this defines what key in Redis this reference points to.\n  toIdentifier (VisitorStats visitorId) =\n    Redis.colonSep [\"visitors\", \"statistics\", Redis.toBS visitorId]\n```\n\nThis example is a bit silly because if you know `DateOfBirth` about your unregistered visitors,\nthere's something very wrong. However, for demonstrational purposes, it'll suffice.\n\nNow we can get references to the individual fields with the specialised operator `:.`.\n\n```haskell\nhandleClick :: VisitorId -\u003e Redis ()\nhandleClick visitorId = do\n  -- for demonstration purposes, log the email\n  email \u003c- Redis.get (VisitorStats visitorId :. Email)\n  liftIO $ print email\n\n  -- atomically increase the counter of clicks\n  Redis.incrementBy (VisitorStats visitorId :. Clicks) 1\n```\n\nIn the current implementation, `Record`s cannot be read or written as a whole.\n(However, they *can* be deleted and their TTL can be set.)\nThere is no special reason for that, except that it would be too much type-level code\nthat we currently do not need, so we keep it simple.\n\nHowever, see [Meta-records](#meta-records) for the next best solution.\n\n#### Aside: non-fixed record fields\n\nThe number of fields in a record is not *really* fixed.\nConsider the following declaration.\n\n```haskell\ndata VisitorField :: * -\u003e * where\n  Visits :: Date -\u003e VisitorField Int\n\ninstance Redis.RecordField VisitorField where\n  rfToBS (Visits date) = Redis.colonSep [\"visits\", Redis.toBS date]\n```\n\nThis creates a record with a separate field for every date:\n\n```haskell\nhandleVisit :: VisitorId -\u003e Date -\u003e Redis ()\nhandleVisit visitorId today = do\n  Redis.incrementBy (VisitorStats visitorId :. Visits today) 1\n```\n\n### Transactions\n\nRedis does support transactions and `redis-schema` supports them,\nbut they are not like SQL transactions, which you may be accustomed to.\nA more suggestive name for Redis transactions might be\n\"[mostly](#errors-in-transactions) atomic operation batches\".\n\nThe main difference between SQL-like transactions and batched Redis transactions\nis that in SQL, you can start a transaction, run a query, receive its output,\nand then run another query in the same transaction. Sending queries and receiving their outputs\ncan be interleaved in the same transaction, and later queries can depend on the output\nof previous queries, while the database takes care of the ACIDity of the transaction.\n\nWith Redis-style batched transactions, on the other hand, you can batch up\nmultiple operations but the atomicity of a transaction ends at the moment you\nreceive the output of those operations. Anything you do with the output is not\nenclosed in that transaction anymore, and other clients could have modified the\ndata in the meantime. In other words, later operations in a batched transaction\ncannot depend on the output of the previous operations, as that output is not\navailable yet.\n\nWhile the structure of SQL-like transactions is captured by the `Monad` typeclass,\nRedis-style fixed-effects transactions are described by `Applicative` functors --\nand this is exactly the interface that `redis-schema` provides for Redis transactions.\n\n#### The `Tx` functor\n\n`redis-schema` defines the `Tx` functor for transactional computations.\n\n```haskell\nnewtype Tx inst a\ninstance Functor (Tx inst)\ninstance Applicative (Tx inst)\ninstance Alternative (Tx inst)\n\natomically :: Tx inst a -\u003e RedisM inst a\ntxThrow :: RedisException -\u003e Tx inst a\n```\n\nThe type parameter `inst` is explained in section [Redis instances](#redis-instances),\nbut can be ignored for now.\n\nRedis transactions are run using the combinator called `atomically`.\nA failing operation (or using `txThrow`)\nin a transaction [will not prevent any other side effects from taking place](#errors-in-transactions);\nonly the exception will be re-thrown in the `RedisM` monad\ninstead of returning the output of the transaction. The `Alternative` instance\nof `Tx` can be used to address exceptions.\n\n#### Working with transactions\n\nMost functions, like `get`, `set` or `take`,\nhave a sibling that can be used in a transaction, usually prefixed with `tx`:\n\n```haskell\nget   :: Ref ref =\u003e ref -\u003e RedisM (RefInstance ref) (Maybe (ValueType ref))\ntxGet :: Ref ref =\u003e ref -\u003e Tx     (RefInstance ref) (Maybe (ValueType ref))\n```\n\nWith `ApplicativeDo`, these transactional functions can be used as conveniently\nas their non-transactional counterparts. For example, the function `take`,\nwhich atomically reads and deletes a Redis value, could be (re-)implemented as follows:\n\n```haskell\n{-# LANGUAGE ApplicativeDo #-}\n\ntake :: Ref ref =\u003e ref -\u003e RedisM (RefInstance ref) (Maybe (ValueType ref))\ntake ref = atomically $ do\n  value \u003c- txGet ref\n  txDelete_ ref\n  pure value\n```\n\n#### What Redis transactions cannot do\n\nOne might try to attempt an alternative implementation of `txIncrementBy`:\n\n```haskell\nimport Data.Maybe (fromMaybe)\n\ntxIncrementBy' :: (SimpleRef ref, Num (ValueType ref))\n  =\u003e ref -\u003e Integer -\u003e Tx (RefInstance ref) (ValueType ref)\ntxIncrementBy' ref incr = do\n  oldValue \u003c- fromMaybe 0 \u003c$\u003e txGet ref        -- COMPILER ERROR\n  let newValue = oldValue + fromInteger incr\n  txSet ref newValue\n  pure newValue\n```\n\nThe compiler complains\n```\n• Could not deduce (Monad (Tx (RefInstance ref)))\n    arising from a do statement\n```\nbecause `oldValue` is used non-trivially in the `do` block,\nbut `Tx` implements only `Applicative` and not `Monad`.\n\nThis error is exactly a goal of the design: it indicates at compile time\nthat Redis does not support this usage pattern.\n\n#### Errors in transactions\n\nBeware that Redis won't roll back failed transactions, which means they\nare not atomic in that sense, and may be carried out incompletely.\nA Redis transaction that fails in the middle\nwill keep going and retain all effects except for any failed operations.\nSee [the Redis documentation](https://redis.io/docs/manual/transactions/#errors-inside-a-transaction)\nfor details and rationale.\n\n#### Monads vs applicative functors\n\nThe underlying library of `redis-schema`, Hedis, provides a monad `RedisTx`\nto describe Redis transactions. Since monads would be too powerful, Hedis uses\nan opaque wrapper for `Queued` results to prevent the users from accessing\nvalues that are not available yet. We believe that using an applicative functor\ninstead is a perfect match for this use case: it allows exactly the right\noperations, and all wrapping/unwrapping can be done entirely transparently.\n`Tx` also propagates exceptions from transactions transparently.\n\n### Exceptions\n\nThe type of exceptions in `redis-schema` is `RedisException`,\nand they are thrown using `throwIO` under the hood.\nThese arise mostly from internal error conditions, such as\nconnection errors, decoding errors, etc.,\nbut library users can nevertheless still throw them manually\nusing `throw :: RedisException -\u003e RedisM inst a`.\n\nUnlike `hedis`, `redis-schema` does support throwing exceptions\nin transactions. Exceptions do *not* abort transactions\n-- all effects of a transaction will persist even if an exception has been thrown --\nbut `RedisException`s thrown using `txThrow` are transparently propagated out of the transaction\nand thrown at the `RedisM` level instead of returning the result of the transaction.\n\n### Custom data types\n\nEvery type that can be stored in Redis using `redis-schema`\ncomes with a `Value` instance that describes how to read, write, and perform\nother operations on values of that type in Redis.\n\nThere are two kinds of Redis `Value`s: simple values and non-simple values.\nSimple values are those that encode/decode to/from a `ByteString`, and thus\nhave no restrictions on how they can be used in Redis.\nThey can be stored in top-level keys, as well as in Redis lists,\nRedis sets, Redis hashes, etc. Simple values include\nintegers, floats, text, bytestrings, etc.\n\nNon-simple values are all values that are more complicated than a bytestring,\nand thus will come with restrictions. For example, Redis lists are not simple values.\n\nLet's start by discussing simple values.\n\n#### Simple values\n\nThe easiest case of declaring Redis instances for custom data types\nare newtypes of types that already have Redis instances. For example,\nif your user IDs are textual but you would still like to keep them apart\nfrom other `Text` data, you could use the following declarations.\n\n```haskell\n{-# LANGUAGE DerivingStrategies #-}\n\nnewtype UserId = UserId Text\n  deriving newtype (Redis.Serializable)\n\ninstance Redis.Value inst UserId\ninstance Redis.SimpleValue inst UserId\n```\n\nThanks to `deriving newtype`, we did not have to write\nany wrapping/unwrapping boilerplate, and thanks to\nthe default implementations of `Value` methods,\nwe did not have to write those, either.\n\nThe class `SimpleValue` does not have any methods, and it mostly\nonly stands for the list of constraints in its declaration\n(primarily, for the `Serializable` constraint).\n`SimpleValue` is a typeclass rather than a constraint alias\nbecause you may want to have a `Serializable` instance for\na non-simple `Value`. Thus a `SimpleValue` instance also represents\nthe intentional declaration that the type in question should be regarded\nas a simple value.\n\nFor other types, we need to supply a `Serializable` instance,\nwhich is, however, often not too hard.\n\n```haskell\ndata Color = Red | Green | Blue\n\ninstance Redis.Serializable Color where\n  fromBS = Redis.readBS\n  toBS   = Redis.showBS\n\n-- Convenience functions available:\n-- Redis.readBS :: Read val =\u003e ByteString -\u003e Maybe val\n-- Redis.showBS :: Show val =\u003e val -\u003e ByteString\n\ninstance Redis.Value inst Color\ninstance Redis.SimpleValue inst Color\n```\n\nThe typeclass `Serializable` is separate from `Show`, `Read`, and `Binary` because:\n* `Show` and `Read` quote strings, and we need the ability to avoid doing it\n* `Binary` does not produce human-readable output and would thus affect the usability of tools like `redis-cli`\n\nSince `redis-schema` is intended to be imported qualified as `Redis`,\n`Redis.Serializable` is an accurate name for the typeclass.\n\n#### Non-simple values\n\nNon-simple values have instances only for `Value`.\nThe default implementations of methods of `Value` require a `SimpleValue` instance,\nthus relieving us from defining them whenever a `SimpleValue` instance exists.\nFor non-simple values, we have to implement the methods of `Value` manually.\n\nNot all methods of `Value` may make sense for all data types,\nor not all methods may be practically implementable.\nIn such cases, it's acceptable to fill the definition with an `error` message.\n\nFor example, the `Record` type defined by `redis-schema` does not support\nreading/writing whole records because that would require more type-level\nmachinery than we needed at the time.\n\nAnother example is the fact that `setTTL` does not make (a lot of)\nsense for values represented by `SviHash`,\ni.e. for values that exist inside a Redis hash, as TTL can be set only for the whole hash.\nPragmatically, `redis-schema` resorts to silently changing the TTL for the whole hash.\n\nYet another example are the `PubSub` channels,\nwhere the operations of `get` and `set` do not make sense.\n\nIn all these cases, the \"correct\" solution would be splitting the `Value`\ntypeclass into smaller classes per supported feature so that the availability\nof the individual operations is declared at the type level. We decided to keep\nthings simple (if perhaps a bit crude) and use a single `Value` typeclass. This\nmay be revisited in the future.\n\n#### Redis instances\n\nIn section [Simple Variables](#simple-variables), we have seen that\na `Redis.Ref` determines a \"path to a variable\" in Redis.\nBut what if you run more Redis servers? You might want that to use different\nkey eviction policies and different memory limits for different purposes.\n\nThe definition of `Redis.Ref` includes an extra associated type family\ncalled `RefInstance`, which identifies the server, representing the hitherto\nmissing part of the \"path to the variable\". This type family has a default\nvalue `DefaultInstance`, which is why we have not needed to deal with it so far.\nHere's what it looks like:\n\n```haskell\n-- | The kind of Redis instances. Ideally, this would be a user-defined DataKind,\n--   but since Haskell does not have implicit arguments,\n--   that would require that we index everything with it explicitly,\n--   which would create a lot of syntactic noise.\n--\n--   (Ab)using the * kind for instances is a compromise.\ntype Instance = *\n\n-- | We also define a default instance.\n--   This is convenient for code bases using only one Redis instance,\n--   since 'RefInstance' defaults to this. (See the 'Ref' typeclass below.)\ndata DefaultInstance\n\n-- | The Redis monad related to the default instance.\ntype Redis = RedisM DefaultInstance\n\nclass Value (RefInstance ref) (ValueType ref) =\u003e Ref ref where\n  -- | Type of the value that this ref points to.\n  type ValueType ref :: *\n\n  -- | RedisM instance this ref points into, with a default.\n  type RefInstance ref :: Instance\n  type RefInstance ref = DefaultInstance\n\n  -- | How to convert the ref to an identifier that its value accepts.\n  toIdentifier :: ref -\u003e Identifier (ValueType ref)\n```\n\nA Redis instance can be added by declaring an empty tag type,\nfor example as follows:\n\n```haskell\n-- For data that should not get lost\ntype InstReliable = Redis.DefaultInstance\n\n-- For throwaway data to speed things up\ndata InstCacheLRU\n```\n\nThen a `Redis.Ref` can be placed in the appropriate Redis instance:\n```haskell\ndata VisitorCount = VisitorCount\n\ninstance Redis.Ref VisitorCount where\n  type ValueType VisitorCount = Integer\n  type RefInstance VisitorCount = InstReliable  -- reliable\n  toIdentifier VisitorCount = \"visitor_count\"\n\n\ndata CachedFile = CachedFile FilePath\n\ninstance Redis.Ref CachedFile where\n  type ValueType CachedFile = ByteString\n  type RefInstance CachedFile = InstCacheLRU  -- evicted as necessary\n  toIdentifier (CachedFile path) = Redis.colonSep [\"cached_files\", BS.pack path]\n```\n\nFinally, all connections and the Redis monad are tagged\nby the Redis instance, best illustrated by this type signature:\n\n```haskell\nrun :: MonadIO m =\u003e Pool inst -\u003e RedisM inst a -\u003e m a\n```\n\nThere are two consequences.\nFirst, all operations in a `RedisM` computation must work with the same instance.\nSecond, it is practical to have a wrapper function around `run` that automatically\nselects the right connection `Pool` from the environment, based on the Redis instance\nspecified in the type of the `RedisM` computation.\n\n### Meta-records\n\nIn Haskell, records can be nested arbitrarily. You can have a record\nthat contains some fields alongside another couple of records,\nwhich themselves contain arbitrarily nested maps and lists of further records.\n\nRedis does not support such arbitrary nesting while being able to\naccess and manipulate the inner structures like you would a top-level one\n(e.g. increment a counter deep in the structure).\nHowever, we can often work around this limitation\nby distributing the datastructure over a number of separate Redis keys.\nFor example, consider a case where each visitor should be associated with\nthe number of visits, the number of clicks, and the set of their favourite songs.\nHere we can keep the visits+clicks in one record reference per visitor, and the set of favourites\nin another reference, again per visitor.\nHowever, we still need to read the visits+clicks separately from the favourites.\nThis is not just an impediment to convenience: two separate reads may lead to a race condition,\nunless we run them in a transaction.\n\nSince `redis-schema` encourages compositionality, it is possible to make data structures\nthat gather (or scatter) all their data across Redis automatically, without having\nto manipulate every component separately every time. Here's an example.\n\n```haskell\n-- VisitorFields are visits and clicks.\ndata VisitorField :: * -\u003e * where\n  Visits :: VisitorField Int\n  Clicks :: VisitorField Int\n\n-- VisitorStats is a record with VisitorFields\ndata VisitorStats = VisitorStats VisitorId\ninstance Redis.Ref VisitorStats where\n  type ValueType VisitorStats = Redis.Record VisitorField\n  toIdentifier = {- ...omitted... -}\n\n-- A separate reference to the favourite songs.\ndata FavouriteSongs = FavouriteSongs VisitorId\ninstance Redis.Ref FavouriteSongs where\n  type ValueType FavouriteSongs = Set SongId\n  toIdentifier = {- ...omitted... -}\n\n-- Finally, here's our composite record that we want to read/write atomically.\ndata VisitorInfo = VisitorInfo\n  { viVisits :: Int\n  , viClicks :: Int\n  , viFavouriteSongs :: Set SongId\n  }\n\ninstance Redis.Value Redis.DefaultInstance VisitorInfo where\n  type Identifier VisitorInfo = VisitorId\n\n  txValGet visitorId = do\n    visits \u003c- fromMaybe 0 \u003c$\u003e Redis.txGet (VisitorStats visitorId :. Visits)\n    clicks \u003c- fromMaybe 0 \u003c$\u003e Redis.txGet (VisitorStats visitorId :. Clicks)\n    favourites \u003c- fromMaybe Set.empty \u003c$\u003e Redis.txGet (FavouriteSongs visitorId)\n    return $ Just VisitorInfo\n      { viVisits = visits\n      , viClicks = clicks\n      , viFavourites = favourites\n      }\n\n  txValSet visitorId vi = do\n    Redis.txSet (VisitorStats visitorId :. Visits) (viVisits vi)\n    Redis.txSet (VisitorStats visitorId :. Clicks) (viClicks vi)\n    Redis.txSet (FavouriteSongs visitorId) (viFavourites vi)\n\n  txValDelete visitorId = do\n    Redis.txDelete (VisitorStats visitorId)\n    Redis.txDelete (FavouriteSongs visitorId)\n\n  {- etc. -}\n```\n\nIt's a bit of a boilerplate, but now all the scatter/gather code is packed\nin the `Value` instance, it's safe and it composes. Moreover, using `let`-bound\nshorthand functions for common expressions, the repetition can be greatly minimised.\n\n#### Aside: references\n\nA reference to `VisitorInfo` would look as follows.\n```haskell\ndata VisitorInfoRef = VisitorInfoFor VisitorId\n\ninstance Redis.Ref VisitorInfoRef where\n  type ValueType VisitorInfoRef = VisitorInfo\n  toIdentifier (VisitorInfoFor visitorId) = visitorId\n```\n\nMeta-records demonstrate why reference ADTs are more flexible than bytestring keys.\nSince `VisitorInfo` is identified by `VisitorId`, as determined by the associated\ntype family `Identifier`, it would be impractical to extract `VisitorId`\nfrom a `ByteString` reference.\n\nMore fundamentally, a meta-record is not associated with any single\nkey in Redis so there is no bytestring key to speak of -- and that's why\nwe used `VisitorId` to identify the meta-record above instead.\n\nWe *could* approach the bytestring as the prefix of all keys that constitute the meta-record\nbut that's less flexible than the ADT approach, which lets us extract\nthe components of the key and rearrange them as we see fit.\nThe optimal arrangement of data in Redis may not coincide with a single\nfixed bytestring key prefix.\n\n#### Aside: instances\n\nLooking back at this instance head:\n```haskell\ninstance Redis.Value Redis.DefaultInstance VisitorInfo where\n```\nWe see that unlike in the usual case, this `Value` instance has been declared specifically\nfor `DefaultInstance`. The reason is that the definition of the `Value` instance\nfor `VisitorInfo` accesses Redis refs `VisitorStats` and `FavouriteSongs`,\nand these refs are linked to `DefaultInstance`.\n\nSince every Redis `Ref` must be linked to a specific Redis instance, and cannot be polymorphic\nin the instance (its purpose is to give a path to the variable, as discussed),\nall meta-records that access them under the hood must be declared for that particular instance.\nConsequently, all `Ref`s that make up a meta-record must be linked to the same Redis instance.\n\n## Libraries\n\n### Locks\n\nLocks are implemented in `Database.Redis.Schema.Lock`.\nThe basic type is the exclusive lock; the shared lock is implemented using an exclusive lock.\nHence the shared lock is also slower, and it's sometimes better to use an exclusive lock,\neven though a shared lock would be sufficient.\n\nThe library does not export much API; the main points of interest\nare functions `withExclusiveLock` and `withShareableLock`, which bracket\na synchronised operation.\n```haskell\nwithExclusiveLock ::\n  ( MonadCatch m, MonadThrow m, MonadMask m, MonadIO m\n  , Redis.Ref ref, Redis.ValueType ref ~ ExclusiveLock\n  )\n  =\u003e Redis.Pool (Redis.RefInstance ref)\n  -\u003e LockParams  -- ^ Params of the lock, such as timeouts or TTL.\n  -\u003e ref         -- ^ Lock ref\n  -\u003e m a         -- ^ The action to perform under lock\n  -\u003e m a\n```\n\nAnother purpose of `Database.Redis.Schema.Lock` is to demonstrate\nhow a library can be implemented on top of `Database.Redis.Schema`.\n\n### Remote jobs\n\nIn `Database.Redis.Schema.RemoteJob` a Redis-based worker queue is implemented, to run CPU\nintensive jobs on remote machines. The queue is strongly typed, and can contain multiple\ndifferent jobs to be executed, with priorities, that workers can pick up.\n\nAs an example, we define a queue that can contain three types of jobs:\n```haskell\nnewtype ComputeFactorial = CF Integer deriving ( Binary )\nnewtype ComputeSquare    = CS Integer deriving ( Binary )\n\ndata MyQueue\ninstance JobQueue MyQueue where\n  type RPC MyQueue =\n    '[ ComputeFactorial -\u003e Integer\n     , ComputeSquare -\u003e Integer\n     , String -\u003e String\n     ]\n  keyPrefix = \"myqueue\"\n```\nHere the `MyQueue` type is used only during compile time to let the compiler find the right\ninstances. To distinguish between the two `Integer -\u003e Integer` functions, we wrap them in\nnewtypes. A `Binary` instance must exist for all inputs and outputs, so that they can be put\ninto Redis.\n\nBased on this queue, we can now define a worker that executes the jobs. This worker must\ndefine a function for each the the types in `RPC`, and runs in a monadic context (which\nwe fixed to `IO` for the example).\n\n```haskell\nfac :: ComputeFactorial -\u003e IO Integer\nfac (CF n) = do\n  putStrLn $ \"Computing the factorial of \" ++ show n\n  pure $ product [1..n]\n\nsm :: ComputeSquare -\u003e IO Integer\nsm (CS n) = pure $ n * n\n\nrunWorker :: IO ()\nrunWorker = do\n  pool \u003c- connect \"redis:///\" 10\n  let myId = \"localworker\"\n  let err e = error $ \"Something went wrong: \" ++ show e\n  remoteJobWorker @MyQueue myId pool err fac sm (pure . reverse)\n```\nThe arguments to `remoteJobWorker` are a unique identifier for this worker (for counting\nthe workers, executing jobs will work fine even with overlapping ids), a connection pool,\na logging function for exceptions, and then for each element in `RPC` the right function.\n\nNow if we call `runWorker` it will block until work needs to be done, and it will never\nreturn except when an async exception is thrown. In production cases it is adviced to use\n`withRemoteJobWorker` instead, which forks off a worker thread and provides a `WorkerHandle`\nto it's continuation, which can be passed to `gracefulShutdown` to handle the currently\nrunning job and then gracefully return.\n\nNow from another process or even other machine we can 'execute jobs', e.g. add them to the\nqueue and synchronously wait for their result. For example:\n```haskell\nrunJobs :: IO ()\nrunJobs = do\n  pool \u003c- connect \"redis:///\" 10\n  a \u003c- runRemoteJob @MyQueue @String @String False pool 1 \"test\"\n  print a\n  b \u003c- runRemoteJob @MyQueue @ComputeFactorial @Integer False pool 1 (CF 5)\n  print b\n```\nThis will print:\n```ghci\u003e runJobs\nRight \"tset\"\nRight 120\n```\nThe underlying Redis implementation is based on blocking reads from sorted sets (`BZPOPMIN`),\nwhich is concurrency safe and no polling is needed. An arbitrary amount of workers can be run\nand jobs can be executed from arbitrary machines. Only the `countWorkers` implementation\nis based on a keep-alive loop on the workers, to properly deal with TCP connection losses.\n\n## Future work\n\n* Reading numeric types in Redis never returns `Nothing`; they'll return `Just 0` instead.\n  Perhaps the return types could reflect that somehow.\n\n* Different Redis `Value`s sometimes support different operations, as briefly discussed\n  at [non-simple values](#non-simple-values). We may want to split `Value` into multiple\n  type classes, depending on the supported operations.\n\n* [Records](#records) cannot be read/written as a whole.\n  The only reason is that we did not need it,\n  and thus opted to avoid all the type-level machinery\n  coming with extensible records.\n  However, adopting an established library like `vinyl`\n  as an optional dependency might be worth it.\n\n## License\n\nBSD 3-clause.\n\n\u003c!--\nvim: ts=2 sts=2 sw=2 et\n--\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchordify%2Fredis-schema","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fchordify%2Fredis-schema","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchordify%2Fredis-schema/lists"}