Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/replikativ/konserve

A clojuresque key-value/document store protocol with core.async.
https://github.com/replikativ/konserve

clojure key-value-store konserve

Last synced: 4 days ago
JSON representation

A clojuresque key-value/document store protocol with core.async.

Awesome Lists containing this project

README 

        
* konserve

  :PROPERTIES:

  :CUSTOM_ID: h:6f85a7f4-3694-4703-8c0b-ffcc34f2e5c9

  :END:



[[https://clojurians.slack.com/archives/CB7GJAN0L][https://img.shields.io/badge/slack-join_chat-brightgreen.svg]]

[[https://clojars.org/io.replikativ/konserve][https://img.shields.io/clojars/v/io.replikativ/konserve.svg]]

[[https://circleci.com/gh/replikativ/konserve][https://circleci.com/gh/replikativ/konserve.svg?style=shield]]

[[https://github.com/replikativ/konserve/tree/development][https://img.shields.io/github/last-commit/replikativ/konserve/main.svg]]



[[https://whilo.github.io/old/articles/16/unified-storage-io][Simple durability, made flexible.]]



A simple document store protocol defined with synchronous and [[https://github.com/clojure/core.async][core.async]]

semantics to allow Clojuresque collection operations on associative key-value

stores, both from Clojure and ClojureScript for different backends. Data is

generally serialized with [[https://github.com/edn-format/edn][edn]] semantics or, if supported, as native binary blobs

and can be accessed similarly to =clojure.core= functions =get-in=, =assoc-in=

and =update-in=. =update-in= especially allows to run functions atomically and

returns old and new value. Each operation is run atomically and must be

consistent (in fact ACID), but further consistency is not supported (Riak,

CouchDB and many scalable solutions don't have transactions over keys for that

reason). This is meant to be a building block for more sophisticated storage

solutions (Datomic also builds on kv-stores). A simple append-log for fast

write operations is also implemented.



** Features

   :PROPERTIES:

   :CUSTOM_ID: h:115591f9-90d2-4c25-8499-6f53a8ae4bc6

   :END:



- /cross-platform/ between Clojure and ClojureScript

- /lowest-common denominator interface/ for an associative datastructure

  with =edn= semantics

- /thread-safety with atomicity over key operations/

- /consistent error handling/ for core.async

- /fast serialization/ options (fressian, transit, ...), independent of

  the underlying kv-store

- /very low overhead/ protocol, including direct binary access for high

  throughput

- /no additional dependencies and setup/ required for IndexedDB in the

  browser and the file backend on the JVM

- /avoids blocking io/, the filestore for instance will not block any

  thread on reading. Fully asynchronous support for writing and other

  stores is in the pipeline.



*** Garbage Collector

:PROPERTIES:

:CUSTOM_ID: h:5529aa34-11b1-4499-bf62-7fc7be2b8a12

:END:



Konserve has a garbage collector that can be called manually when the store gets

too crowded. For that, the function =konserve.gc/sweep!= allows you to provide a

cut-off date to evict old keys and a whitelist for keys that should be kept.



*** Error handling

:PROPERTIES:

:CUSTOM_ID: h:10edb2cf-b2fc-4cc5-8854-77e6e8a1d82d

:END:



For synchronous execution normal exceptions will be thrown. For asynchronous

error handling we follow the semantics of =go-try= and == introduced [[https://swannodette.github.io/2013/08/31/asynchronous-error-handling][here]]. We

have the [[https://github.com/replikativ/superv.async/][superv.async]] library around the error handling in core.async, but since

there is no need to push it onto the users of konserve, you just need these two

macros that properly handle the errors. == needs to check for an exception and

rethrow and =go-try= needs to catch and pass it along as a return value such

that it does not get lost.



** Usage

   :PROPERTIES:

   :CUSTOM_ID: h:07b8872b-1b84-412b-8133-4dbb9d2a7430

   :END:



Add to your dependencies: [[http://clojars.org/io.replikativ/konserve][http://clojars.org/io.replikativ/konserve/latest-version.svg]]



*** Synchronous Execution

:PROPERTIES:

:CUSTOM_ID: h:e290028c-78d8-4af6-8742-18b6d46680e3

:END:



Run the following synchronous code if you are not using core.async in your scope:



#+BEGIN_SRC clojure

  (ns test-db

    (:require [konserve.filestore :refer [connect-fs-store]]

              [konserve.core :as k]))



  (def store (connect-fs-store "/tmp/store" :opts {:sync? true}))



  (k/assoc-in store ["foo" :bar] {:foo "baz"} {:sync? true})

  (k/get-in store ["foo"] nil {:sync? true})

  (k/exists? store "foo" {:sync? true})



  (k/assoc-in store [:bar] 42 {:sync? true})

  (k/update-in store [:bar] inc {:sync? true})

  (k/get-in store [:bar] nil {:sync? true})

  (k/dissoc store :bar {:sync? true})



  (k/append store :error-log {:type :horrible} {:sync? true})

  (k/log store :error-log {:sync? true})



  (let [ba (byte-array (* 10 1024 1024) (byte 42))]

    (time (k/bassoc store "banana" ba {:sync? true})))



  (k/bget store "banana"

          (fn [{is :input-stream}]

            (your-read-does-all-work-here is))

          {:sync? true})

#+END_SRC



*** Asynchronous Execution

:PROPERTIES:

:CUSTOM_ID: h:929c501d-2a31-4f05-b231-132f79ee6cb5

:END:



In a ClojureScript REPL you can evaluate the expressions from the REPL

each wrapped in a go-block.



#+BEGIN_SRC clojure

  (ns test-db

    (:require [konserve.memory :refer [new-mem-store]]

              [clojure.core.async :refer [go  1000 values)

- filestore, indexeddb: allow to safely custom deserialize

  file-inputstream in transaction/lock

- filestore, indexeddb, memstore: implement -exists?



*** 0.2.1

    :PROPERTIES:

    :CUSTOM_ID: h:9a3e49e9-9dd0-474d-949e-eb8eb0a15b80

    :END:



- filestore: fix fressian collection types for clojure, expose

  read-handlers/write-handlers

- filestore: fix -update-in behaviour for nested values

- filestore: fix rollback renaming order



*** 0.2.0

    :PROPERTIES:

    :CUSTOM_ID: h:757b5af0-3262-4bb4-82ea-85aee87d77e1

    :END:



- experimental native ACID file-store for Clojure

- native binary blob support for file-store, IndexedDB and mem-store



** Contributors

   :PROPERTIES:

   :CUSTOM_ID: h:dd1ebb1a-2748-4f04-86f1-c2a5347ec9f8

   :END:



- Björn Ebbinghaus

- Daniel Szmulewicz

- Konrad Kühne

- Christian Weilbach



** License

   :PROPERTIES:

   :CUSTOM_ID: h:8153b6f6-d253-4863-86b4-038dd383b6fe

   :END:



Copyright © 2014-2023 Christian Weilbach and contributors



Distributed under the Eclipse Public License either version 1.0 or (at

your option) any later version.