Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/coleifer/greendb
server frontend for lmdb
https://github.com/coleifer/greendb
gevent lmdb python
Last synced: about 2 months ago
JSON representation
server frontend for lmdb
- Host: GitHub
- URL: https://github.com/coleifer/greendb
- Owner: coleifer
- Created: 2018-11-12T16:42:18.000Z (about 6 years ago)
- Default Branch: master
- Last Pushed: 2022-12-14T02:40:28.000Z (about 2 years ago)
- Last Synced: 2024-03-15T15:08:16.235Z (9 months ago)
- Topics: gevent, lmdb, python
- Language: Python
- Size: 84 KB
- Stars: 24
- Watchers: 5
- Forks: 2
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
### greendb
async server frontend for symas lmdb.
#### description
greendb is a lightweight server (and Python client) for symas lmdb. The server
uses the new Redis RESPv3 protocol.
greendb supports multiple independent databases, much like Redis. Values are
serialized using `msgpack`, so the server is capable of storing all the
data-types supported by msgpack. greendb also provides per-database
configuration of multi-value support, allowing you to efficiently store
multiple values at a given key, in sorted order (e.g. for secondary indexes).
With greendb, database keys are always bytestrings. Values may be:
* dict
* list
* set
* bytestrings
* unicode strings
* integers
* floating-point
* boolean
* `None`
#### installing
```
$ pip install greendb
```
Alternatively, you can install from git:
```
$ git clone https://github.com/coleifer/greendb
$ cd greendb
$ python setup.py install
```
Dependencies:
* [gevent](http://www.gevent.org/)
* [lmdb](https://github.com/dw/py-lmdb)
* [msgpack-python](https://github.com/msgpack/msgpack-python)
#### running
```
$ greendb.py -h
Usage: greendb.py [options]
Options:
-h, --help show this help message and exit
-c CONFIG, --config=CONFIG
Config file (default="config.json")
-D DATA_DIR, --data-dir=DATA_DIR
Directory to store db environment and data.
-d, --debug Log debug messages.
-e, --errors Log error messages only.
-H HOST, --host=HOST Host to listen on.
-l LOG_FILE, --log-file=LOG_FILE
Log file.
-m MAP_SIZE, --map-size=MAP_SIZE
Maximum size of memory-map used for database. The
default value is 256M and should be increased. Accepts
value in bytes or file-size using "M" or "G" suffix.
--max-clients=MAX_CLIENTS
Maximum number of clients.
-n MAX_DBS, --max-dbs=MAX_DBS
Number of databases in environment. Default=16.
-p PORT, --port=PORT Port to listen on.
-r, --reset Reset database and config. All data will be lost.
-s, --sync Flush system buffers to disk when committing a
transaction. Durable but much slower.
-u DUPSORT, --dupsort=DUPSORT
db index(es) to support dupsort
-M, --no-metasync Flush system buffers to disk only once per
transaction, omit the metadata flush.
-W, --writemap Use a writeable memory map.
-A, --map-async When used with "--writemap" (-W), use asynchronous
flushes to disk.
```
Complete config file example with default values:
```javascript
{
"host": "127.0.0.1",
"port": 31337,
"max_clients": 1024,
"path": "data", // Directory for data storage, default is "data" in CWD.
"map_size": "256M", // Default map size is 256MB. INCREASE THIS!
"read_only": false, // Open the database in read-only mode.
"metasync": true, // Sync metadata changes (recommended).
"sync": false, // Sync all changes (durable, but much slower).
"writemap": false, // Use a writable map (probably safe to do).
"map_async": false, // Asynchronous writable map.
"meminit": true, // Initialize new memory pages to zero.
"max_dbs": 16, // Maximum number of DBs.
"max_spare_txns": 64,
"lock": true, // Lock the database when opening environment.
"dupsort": false // Either a boolean or a list of DB indexes.
}
```
Example custom configuration:
* 1GB max database size
* dupsort enabled on databases 13, 14 and 15
* data stored in /var/lib/greendb/data
```javascript
{
"map_size": "1G",
"dupsort": [13, 14, 15],
"path": "/var/lib/greendb/data/"
}
```
Equivalent configuration using command-line arguments:
```
$ greendb.py -m 1G -u 13 -u 14 -u 15 -D /var/lib/greendb/data/
```
### client
A Python client is included in the `greendb` module. All server commands are
implemented as client methods using the lower-case command name, for example:
```python
from greendb import Client
client = Client(host='10.0.0.3')
# Execute the ENVINFO command.
print(client.envinfo())
# Set multiple key/value pairs, read a key, then delete two keys.
client.mset({'k1': 'v1', 'k2': 'v2'}) # MSET
print(client.get('k1')) # GET
client.mdelete(['k1', 'k2']) # MDELETE
```
Additionally, the `Client` implements much of the Python `dict` interface, such
as item get/set/delete, iteration, length, contains, etc.
If an error occurs, either due to a malformed command (e.g., missing required
parameters) or for any other reason (e.g., attempting to write to a read-only
database), then a `CommandError` will be raised by the client with a message
indicating what caused the error.
**A note about connections**: the greendb client will automatically connect the
first time you issue a command to the server. The client maintains its own
thread-safe (and greenlet-safe) connection pool. If you wish to explicitly
connect, the client may be used as a context manager.
In a multi-threaded or multi-greenlet application (e.g. a web app), the client
will maintain a separate connection for each thread/greenlet.
### command reference
Below is the list of supported commands. **Commands are available on the client
using the lower-case command name as the method name**.
command
description
arguments
return value
ENVINFO
metadata and storage configuration settings
(none)
dict
ENVSTAT
metadata related to the global b-tree
(none)
dict
FLUSH
delete all records in the currently-selected database
(none)
boolean indicating success
FLUSHALL
delete all records in all databases
(none)
dict mapping database index to boolean
PING
ping the server
(none)
"pong"
STAT
metadata related to the currently-selected database b-tree
(none)
dict
SYNC
synchronize database to disk (use when sync=False)
(none)
(none)
USE
select the given database
database index, 0 through (max_dbs - 1)
int: active database index
KV commands
COUNT
get the number of key/value pairs in active database
(none)
int
DECR
decrement the value at the given key
amount to decrement by (optional, default is 1)
int or float
INCR
increment the value at the given key
amount to increment by (optional, default is 1)
int or float
CAS
compare-and-set
key, original value, new value
boolean indicating success or failure
DELETE
delete a key and any value(s) associated
key
int: number of keys removed (1 on success, 0 if key not found)
DELETEDUP
delete a particular key/value pair when dupsort is enabled
key, value to delete
int: number of key+value removed (1 on success, 0 if key+value not found)
DELETEDUPRAW
delete a particular key/value pair when dupsort is enabled using an
unserialized bytestring as the value
key, value to delete
int: number of key+value removed (1 on success, 0 if key+value not found)
DUPCOUNT
get number of values stored at the given key (requires dupsort)
key
int: number of values, or None if key does not exist
EXISTS
determine if the given key exists
key
bool
GET
get the value associated with a given key. If dupsort is enabled and
multiple values are present, the one that is sorted first will be returned.
key
value or None
GETDUP
get all values associated with a given key (requires dupsort)
key
list of values or None if key does not exist
LENGTH
get the length of the value stored at a given key, e.g. for a string
this returns the number of characters, for a list the number of items, etc.
key
length of value or None if key does not exist
POP
atomically get and delete the value at a given key. If dupsort is
enabled and multiple values are present, the one that is sorted first will
be removed and returned.
key
value or None
REPLACE
atomically get and set the value at a given key. If dupsort is enabled,
the first value will be returned (if exists) and ALL values will be removed
so that only the new value is stored.
key
previous value or None
SET
store a key/value pair. If dupsort is enabled, duplicate values will be
stored at the given key in sorted-order. Additionally, if dupsort is
enabled and the exact key/value pair already exist, no changes are made.
key, value
int: 1 if new key/value added, 0 if dupsort is enabled and the
key/value already exist
SETDUP
store a key/value pair, treating duplicates as successful writes
(requires dupsort). Unlike SET, if the exact key/value pair already exists,
this command will return 1 indicating success.
key, value
int: 1 on success
SETDUPRAW
store a key/value pair, treating duplicates as successful writes
(requires dupsort). Additionally, the value is not serialized, but is
stored as a raw bytestring.
key, value (bytes)
int: 1 on success
SETNX
store a key/value pair only if the key does not exist
key, value
int: 1 on success, 0 if key already exists
Bulk KV commands
MDELETE
delete multiple keys
list of keys
int: number of keys deleted
MGET
get the value of multiple keys
list of keys
dict of key and value. Keys that were requested, but which do not
exist are not included in the response.
MGETDUP
get all values of multiple keys (requires dupsort)
list of keys
dict of key to list of values. Keys that were requested, but which do
not exist, are not included in the response.
MPOP
atomically get and delete the value of multiple keys. If dupsort is
enabled and multiple values are stored at a given key, only the first value
will be removed.
list of keys
dict of key to value
MREPLACE
atomically get and set the value of multiple keys. If dupsort is
enabled and multiple values are stored at a given key, only the first value
will be returned and all remaining values discarded.
dict of key to value
dict of key to previous value. Keys that did not exist previously will
not be included in the response.
MSET
set the value of multiple keys.
dict of key to value
int: number of key / value pairs set
MSETDUP
store multiple key/value pairs, treating duplicates as successful
writes (requires dupsort). Unlike MSET, if the exact key/value pair already
exists, this command will treat the write as a success.
dict of key to value
int: number of key / value pairs set
MSETNX
store multiple key/value pair only if the key does not exist
dict of key to value
int: number of key / value pairs set
Cursor / range commands
DELETERANGE
delete a range of keys using optional inclusive start/end-points
start key (optional), end key (optional), count (optional)
int: number of keys deleted
GETRANGE
retrieve a range of key/value pairs using optional inclusive
start/end-points
start key (optional), end key (optional), count (optional)
list of [key, value] lists
GETRANGEDUPRAW
retrieve a range of duplicate values stored in a given key, using
optional (inclusive) start/end-points
key, start value (optional), end value (optional), count (optional)
list of values (as bytestrings)
KEYS
retrieve a range of keys using optional inclusive start/end-points
start key (optional), end key (optional), count (optional)
list of keys
PREFIX
retrieve a range of key/value pairs which match the given prefix
prefix, count (optional)
list of [key, value] lists
VALUES
retrieve a range of values using optional inclusive start/end-points
start key (optional), end key (optional), count (optional)
list of values
Client commands
QUIT
disconnect from server
(none)
int: 1
SHUTDOWN
terminate server process from client (be careful!)
(none)
(none)
#### protocol
The protocol is the Redis RESPv3 protocol. I have extended the protocol with
one additional data-type, a dedicated type for representing UTF8-encoded
unicode text (notably absent from RESPv3). This type is denoted by the leading
`^` byte in responses.
Details can be found here: https://github.com/antirez/RESP3/blob/master/spec.md