{"id":18672323,"url":"https://github.com/stephenmcd/curiodb","last_synced_at":"2025-04-05T19:13:14.440Z","repository":{"id":31035131,"uuid":"34593732","full_name":"stephenmcd/curiodb","owner":"stephenmcd","description":"Distributed NoSQL Database","archived":false,"fork":false,"pushed_at":"2018-11-04T22:44:22.000Z","size":1216,"stargazers_count":513,"open_issues_count":5,"forks_count":47,"subscribers_count":34,"default_branch":"master","last_synced_at":"2025-03-29T18:07:51.032Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"http://curiodb.jupo.org","language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-2-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/stephenmcd.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-04-26T01:47:43.000Z","updated_at":"2025-01-09T10:00:11.000Z","dependencies_parsed_at":"2022-07-13T07:40:50.183Z","dependency_job_id":null,"html_url":"https://github.com/stephenmcd/curiodb","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stephenmcd%2Fcuriodb","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stephenmcd%2Fcuriodb/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stephenmcd%2Fcuriodb/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stephenmcd%2Fcuriodb/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/stephenmcd","download_url":"https://codeload.github.com/stephenmcd/curiodb/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247386265,"owners_count":20930619,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-07T09:10:39.162Z","updated_at":"2025-04-05T19:13:14.406Z","avatar_url":"https://github.com/stephenmcd.png","language":"Scala","funding_links":[],"categories":["Distributed Systems","数据库"],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\u003cimg src=\"http://blog.jupo.org/static/img/curiodb-logo.png\"\u003e\u003c/p\u003e\n\nCreated by [Stephen McDonald][twitter]\n\nCurioDB is a distributed and persistent [Redis][redis] clone, built\nwith [Scala][scala] and [Akka][akka]. Please note that despite the\nfancy logo, this is a toy project, hence the name \"Curio\", and any\nsuitability as a drop-in replacement for Redis is purely incidental. :-)\n\n## Installation\n\nI've been using [sbt][sbt] to build the project, which you can install\non [OS X][sbt-osx], [Linux][sbt-linux] or [Windows][sbt-windows]. With\nthat done, you just need to clone this repository and run it:\n\n```\n$ git clone git://github.com/stephenmcd/curiodb.git\n$ cd curiodb\n$ sbt \"~re-start --config=path/to/config.file\"\n```\n\nYou can also build a binary (executable JAR file):\n\n```\n$ sbt assembly\n$ ./target/scala-2.11/curiodb-0.0.1 --config=path/to/config.file\n```\n\nNote that the `--config=path/to/config.file` argument is optional, see\nthe *Configuration* section below for more detail.\n\n## Overview\n\nWhy build a Redis clone? Well, I'd been learning Scala and Akka and\nwanted a nice project I could take them further with. I've used Redis\nheavily in the past, and Akka gave me some really cool ideas for\nimplementing a clone, based on each key/value pair (or KV pair) in the\nsystem being implemented as an [actor][actor-model]:\n\n##### Concurrency\n\nSince each KV pair in the system is an actor, CurioDB will happily use\nall your CPU cores, so you can run 1 server using 32 cores instead of\n32 servers each using 1 core (or use all 1,024 cores of your 32 server\ncluster, why not). Each actor operates on its own value atomically,\nso the atomic nature of Redis commands is still present, it just occurs\nat the individual KV level instead of in the context of an entire\nrunning instance of Redis.\n\n##### Distributed by Default\n\nSince each KV pair in the system is an actor, the interaction between\nmultiple KV pairs works the same way when they're located across the\nnetwork as it does when they're located on different processes on a\nsingle machine. This negates the need for features of Redis like \"hash\ntagging\", and allows commands that deal with multiple keys (`SUNION`,\n`SINTER`, `MGET`, `MSET`, etc) to operate seamlessly across a cluster.\n\n##### Virtual Memory\n\nSince each KV pair in the system is an actor, the unit of disk storage\nis the individual KV pair, not a single instance's entire data\nset. This makes Redis' [abandoned virtual memory feature][redis-vm] a\nlot more feasible. With CurioDB, an actor can simply persist its value\nto disk after some criteria occurs, and shut itself down until\nrequested again.\n\n##### Simple Implementation\n\nScala is concise, you get a lot done with very little code, but that's\njust the start - CurioDB leverages Akka very heavily, taking care of\nclustering, concurrency, persistence, and a whole lot more. This means\nthe bulk of CurioDB's code mostly deals with implementing all of the\n[Redis commands][redis-commands], so far weighing in at only a paltry\n1,000 lines of Scala! Currently, the majority of commands have been\nfully implemented, as well as the [Redis wire protocol][redis-protocol]\nitself, so [existing client libraries][redis-clients] can be used. Some\ncommands have been purposely omitted where they don't make sense, such\nas cluster management, and things specific to Redis' storage format.\n\n##### Pluggable Storage\n\nSince Akka Persistence is used for storage, many strange scenarios\nbecome available. Want to use [PostgreSQL][postgresql] or\n[Cassandra][cassandra] for storage, with CurioDB as the front-end\ninterface for Redis commands? [This should be possible!][storage-backends]\nBy default, CurioDB uses Akka's built-in [LevelDB storage][leveldb-storage].\n\n## Design\n\nHere's a bad diagram representing one server in the cluster, and the\nflow of a client sending a command:\n\n\u003cp align=\"center\"\u003e\u003cimg src=\"http://blog.jupo.org/static/img/curiodb.png\"\u003e\u003c/p\u003e\n\n* An outside client sends a command to the server actor\n  ([Server.scala][server-source]). There's at most one per cluster node\n  (which could be used to support load balancing), and at least one per\n  cluster (not all nodes need to listen for outside clients).\n* Upon receiving a new outside client connection, the server actor will\n  create a Client Node actor ([System.scala][system-source]), it's\n  responsible for the life-cycle of a single client connection, as well\n  as parsing the incoming and writing the outgoing protocol, such as the\n  Redis protocol for TCP clients, or JSON for HTTP clients\n  ([Server.scala][server-source]).\n* Key Node actors ([System.scala][system-source]) manage the key space\n  for the entire system, which are distributed across the entire\n  cluster using consistent hashing. A Client Node will forward the\n  command to the matching Key Node for its key.\n* A Key Node is then responsible for creating, removing, and\n  communicating with each KV Node actor, which are the actual actors\n  that store the underlying value for each key, such as a strings,\n  hashes, and sorted sets, and perform the operations on them for each\n  of their respective commands. ([Data.scala][data-source]).\n* The KV Node then sends a response back to the originating Client\n  Node, which returns it to the outside client.\n\nNot diagrammed, but in addition to the above:\n\n* Some commands require coordination with multiple KV Nodes, in which\n  case a temporary Aggregate actor\n  ([Aggregation.scala][aggregation-source]) is created by the Client\n  Node, which coordinates the results for multiple commands via Key\n  Nodes and KV Nodes in the same way a Client Node does.\n* PubSub is implemented by adding behavior to Key Nodes and Client\n  Nodes, which act as PubSub servers and clients respectively\n  ([PubSub.scala][pubsub-source]).\n* Lua scripting is fully supported\n  ([Scripting.scala][scripting-source]) thanks to [LuaJ][luaj], and is\n  implemented similarly to PubSub, where behavior is added to Key Nodes\n  which store and run compiled Lua scripts (via `EVALSHA`), and Client\n  Nodes which can run uncompiled scripts directly (via `EVAL`).\n\n## Transactions\n\nDistributed transactions are fully supported, both by way of the\n`MULTI` and `EXEC` commands, and for Lua scripts with the `EVAL` and\n`EVALSHA` commands. Transactions are implemented using basic\n[two-phase commit (2PC)][2pc] with rollback support,\n[multiversion concurrency control (MVCC)][mvcc], and configurable\n[isolation levels][isolation].\n\n##### 2PC\n\nA Client Node acts as a transaction coordinator in 2PC parlance. It is\nresponsible for coordinating initial agreement with each Node that will\nparticipate in the transaction, aggregating responses for all executed\n(but uncommitted) commands, and then finally coordinating the commit\nphase for each participating Node. Given the use of MVCC, performing\nrollback on errors during a transaction is fully supported, and is the\ndefault behavior. This differs however from the way Redis deals with\nerrors, as it [does not support transaction rollbacks][redis-rollback],\ntherefore in CurioDB the behavior can be configured by changing the\n`curiodb.transactions.on-error` setting from `rollback` to `commit`, if\nthis level of compatibility with Redis is required.\n\n##### MVCC\n\nEach KV Node in the system stores multiple versions of its underlying\nvalue internally, using a map that contains each transaction's version,\nas well as the current committed version of the value, or \"main\" value.\nWhen a transaction begins, the main value is copied into the map,\nstored against its transaction ID, and from that point, all commands\nreceived within the transaction will read and write to the transaction\nversion until the transaction is commited, at which point the\ntransaction version is copied back to the main value.\n\n##### Isolation\n\nThree levels of [transaction isolation][isolation] are available,\nwhich can be configured by the `curiodb.transactions.isolation`\nsetting, to control how a key's value is read during a command:\n\n* `repeatable` (default): Inside a transaction, only the transaction's\n  version will be read, otherwise when outside of a transaction, the\n  current committed version will be read.\n* `committed`: Inside or outside of a transaction, the current\n  committed version will be read.\n* `uncommitted`: Inside or outside of a transaction, the most recently\n  written version will be read, even if uncommitted.\n\nNote there is no `serializable` isolation level typically found in SQL\ndatabases, since neither Redis nor CurioDB have a notion of range\nqueries.\n\n## Configuration\n\nHere are the few configuration settings and their default values that\nCurioDB implements, along with the large range of settings\n[provided by Akka][akka-config] itself, which both use\n[typesafe-config][typesafe-config] - consult those projects for\ndetailed information on configuration implementation.\n\n```\ncuriodb {\n\n  // Addresses listening for clients.\n  listen = [\n    \"tcp://127.0.0.1:6379\"    // TCP server using Redis protocol.\n    \"http://127.0.0.1:2600\"   // HTTP server using JSON.\n    \"ws://127.0.0.1:6200\"     // WebSocket server, also using JSON.\n  ]\n\n  // Duration settings (either time value, or \"off\").\n  persist-after = 1 second    // Like \"save\" in Redis.\n  sleep-after   = 10 seconds  // Virtual memory threshold.\n  expire-after  = off         // Automatic key expiry.\n\n  transactions {\n    timeout   = 3 seconds     // Max time a transaction may take to run.\n    isolation = repeatable    // \"repeatable\", \"committed\", or \"uncommitted\".\n    on-error  = rollback      // \"commit\" or \"rollback\".\n  }\n\n  commands {\n    timeout  = 1 second       // Max time a command may take to run.\n    disabled = [SHUTDOWN]     // List of disabled commands.\n    debug    = off            // Print debug info for every command run.\n  }\n\n  // Cluster nodes.\n  nodes = {\n    node1: \"tcp://127.0.0.1:9001\"\n    // node2: \"tcp://127.0.0.1:9002\"\n    // node3: \"tcp://127.0.0.1:9003\"\n  }\n\n  // Current cluster node (from the \"nodes\" keys above).\n  node = node1\n\n}\n```\n\nYou can also optionally provide your own configuration file, using the\n`--config=path/to/config.file` command-line argument. Your\nconfiguration file need only define the values you wish to override.\nFor example, suppose you wanted to only listen over TCP, and disable\nextra commands:\n\n```\ncuriodb.listen = [\"tcp://127.0.0.1:3333\"]\n\ncuriodb.commands.disabled = [SHUTDOWN, DEL, FLUSHDB, FLUSHALL]\n```\n\nThe `sleep-after` and `expire-after` settings are worth some\nexplanation. Each of these configure a time duration that starts each\ntime a command is run against a key, and elapses if no further commands\nrun against that key within the duration. Once the duration elapses,\nan action is performed. After the `sleep-after` duration elapses for a\nkey, it will persist its value to disk, and shut down, essentially\ngoing to sleep - this is how virtual memory is implemented.\n`expire-after` is similar, but after its duration elapses, the key is\ndeleted entirely, just as if the `EXPIRE` command was used.\n\n## HTTP/WebSocket JSON API\n\nAs alluded to in the configuration example above, CurioDB also supports\na HTTP/WebSocket JSON API, as well as the same wire protocol that Redis\nimplements over TCP. Commands are issued with HTTP POST requests, or\nWebSocket messages, containing a JSON Object with a single `args` key,\ncontaining an Array of arguments. Responses are returned as a JSON\nObject with a single `result` key.\n\nHTTP:\n\n```\n$ curl -X POST -d '{\"args\": [\"SET\", \"foo\", \"bar\"]}' http://127.0.0.1:2600\n{\"result\":\"OK\"}\n\n$ curl -X POST -d '{\"args\": [\"MGET\", \"foo\", \"baz\"]}' http://127.0.0.1:2600\n{\"result\":[\"bar\",null]}\n```\n\nWebSocket:\n\n```\nvar socket = new WebSocket('ws://127.0.0.1:6200');\n\nsocket.onmessage = function(response) {\n  console.log(JSON.parse(response.data));\n};\n\nsocket.send(JSON.stringify({args: [\"DEL\", \"foo\"]}));\n```\n\n`SUBSCRIBE` and `PSUBSCRIBE` commands work as expected over WebSockets,\nand are also fully supported by the HTTP API, by using chunked transfer\nencoding to allow a single HTTP connection to receive a stream of\npublished messages over an extended period of time.\n\nIn the case of errors such as invalid arguments to a command, WebSocket\nconnections will transmit a JSON Object with a single `error` key\ncontaining the error message, while HTTP requests will return a\nresponse with a 400 status, containing the error message in the response\nbody.\n\n## Disadvantages compared to Redis\n\n* I haven't measured it, but it's safe to say memory consumption is\n  much poorer due to the JVM. Somewhat alleviated by the virtual memory\n  feature.\n* It's slower, but not by as much as you'd expect. Without any\n  optimization, it's roughly about half the speed of Redis. See the\n  performance section below.\n* PubSub pattern matching may perform poorly. PubSub channels are\n  distributed throughout the cluster using consistent hashing, which\n  makes pattern matching impossible. To work around this, patterns get\n  stored on every node in the cluster, and the `PSUBSCRIBE` and\n  `PUNSUBSCRIBE` commands get broadcast to all of them. This needs\n  rethinking!\n\nMainly though, Redis is an extremely mature and battle-tested project\nthat's been developed by many over the years, while CurioDB is a\none-man hack project worked on over a few months. As much as this\ndocument attempts to compare them, they're really not comparable in\nthat light. That said, it's been tons of fun building it, and it has\nsome cool ideas thanks to Akka. I hope others can get something out of\nit too.\n\n## Performance\n\nThese are the results of `redis-benchmark -q` on an early 2014\nMacBook Air running OS X 10.9 (the numbers are requests per second):\n\nBenchmark      | Redis    | CurioDB  | %\n---------------|----------|----------|----\n`PING_INLINE`  | 57870.37 | 46296.29 | 79%\n`PING_BULK`    | 55432.37 | 44326.24 | 79%\n`SET`          | 50916.50 | 33233.63 | 65%\n`GET`          | 53078.56 | 38580.25 | 72%\n`INCR`         | 57405.28 | 33875.34 | 59%\n`LPUSH`        | 45977.01 | 28082.00 | 61%\n`LPOP`         | 56369.79 | 23894.86 | 42%\n`SADD`         | 59101.65 | 25733.40 | 43%\n`SPOP`         | 50403.23 | 33886.82 | 67%\n`LRANGE_100`   | 22246.94 | 11228.38 | 50%\n`LRANGE_300`   |  9984.03 |  6144.77 | 61%\n`LRANGE_500`   |  6473.33 |  4442.67 | 68%\n`LRANGE_600`   |  5323.40 |  3511.11 | 65%\n`MSET`         | 34554.25 | 15547.26 | 44%\n\n*Generated with the bundled [benchmark.py][benchmark-script] script.*\n\n## Further Reading\n\nThese are some articles I published on developing CurioDB:\n\n* [CurioDB: A Distributed and Persistent Redis Clone][intro-article]\n* [Embedding Lua in Scala using Java][lua-article]\n* [Distributed Transactions in Actor Systems][transactions-article]\n\n## License\n\nBSD.\n\n[twitter]: http://twitter.com/stephen_mcd\n[scala]: http://www.scala-lang.org/\n[akka]: http://akka.io/\n[sbt]: http://www.scala-sbt.org/\n[sbt-osx]: http://www.scala-sbt.org/0.13/tutorial/Installing-sbt-on-Mac.html\n[sbt-linux]: http://www.scala-sbt.org/0.13/tutorial/Installing-sbt-on-Linux.html\n[sbt-windows]: http://www.scala-sbt.org/0.13/tutorial/Installing-sbt-on-Windows.html\n[redis]: http://redis.io/\n[actor-model]: https://en.wikipedia.org/wiki/Actor_model\n[redis-vm]: http://redis.io/topics/virtual-memory\n[redis-commands]: http://redis.io/commands\n[redis-protocol]: http://redis.io/topics/protocol\n[redis-clients]: http://redis.io/clients\n[postgresql]: http://www.postgresql.org/\n[cassandra]: http://cassandra.apache.org/\n[storage-backends]: http://akka.io/community/#snapshot-plugins\n[leveldb-storage]: http://doc.akka.io/docs/akka/snapshot/scala/persistence.html#Local_snapshot_store\n[server-source]: https://github.com/stephenmcd/curiodb/blob/master/src/main/scala/Server.scala\n[system-source]: https://github.com/stephenmcd/curiodb/blob/master/src/main/scala/System.scala\n[data-source]: https://github.com/stephenmcd/curiodb/blob/master/src/main/scala/Data.scala\n[aggregation-source]: https://github.com/stephenmcd/curiodb/blob/master/src/main/scala/Aggregation.scala\n[pubsub-source]: https://github.com/stephenmcd/curiodb/blob/master/src/main/scala/PubSub.scala\n[scripting-source]: https://github.com/stephenmcd/curiodb/blob/master/src/main/scala/Scripting.scala\n[luaj]: http://www.luaj.org/luaj/3.0/README.html\n[2pc]: https://en.wikipedia.org/wiki/Two-phase_commit_protocol\n[mvcc]: https://en.wikipedia.org/wiki/Multiversion_concurrency_control\n[isolation]: https://en.wikipedia.org/wiki/Isolation_(database_systems)\n[redis-rollback]: http://redis.io/topics/transactions#why-redis-does-not-support-roll-backs\n[akka-config]: http://doc.akka.io/docs/akka/snapshot/general/configuration.html#listing-of-the-reference-configuration\n[typesafe-config]: https://github.com/typesafehub/config#standard-behavior\n[benchmark-script]: https://github.com/stephenmcd/curiodb/blob/master/benchmark.py\n[intro-article]: http://blog.jupo.org/2015/07/08/curiodb-a-distributed-persistent-redis-clone/\n[lua-article]: http://blog.jupo.org/2015/08/08/embedding-lua-in-scala-with-java-oh-my/\n[transactions-article]: http://blog.jupo.org/2016/01/28/distributed-transactions-in-actor-systems/\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstephenmcd%2Fcuriodb","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fstephenmcd%2Fcuriodb","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstephenmcd%2Fcuriodb/lists"}