https://github.com/everythingme/kickass-redis
Loose python framework for kickass redis patterns
https://github.com/everythingme/kickass-redis
Last synced: 11 months ago
JSON representation
Loose python framework for kickass redis patterns
- Host: GitHub
- URL: https://github.com/everythingme/kickass-redis
- Owner: EverythingMe
- Created: 2012-06-18T07:32:04.000Z (almost 14 years ago)
- Default Branch: master
- Last Pushed: 2017-04-18T04:50:14.000Z (about 9 years ago)
- Last Synced: 2025-04-15T19:46:19.500Z (about 1 year ago)
- Language: Python
- Homepage:
- Size: 339 KB
- Stars: 154
- Watchers: 25
- Forks: 23
- Open Issues: 3
-
Metadata Files:
- Readme: readme.md
Awesome Lists containing this project
README
Kickass-Redis - a loose framework of Redis based data solutions
==================================================
This project aims to create a repository of useful python libraries built on top of redis (and using each other),
to automate data modeling with Redis.
##For discussion, help and contributing code - join the google group at https://groups.google.com/forum/#!forum/kickass-redis
Redis is relatively low level, and while it is simple to start using, getting a good knowledge of how to model problems
with it in an efficient way can be tricky. So I've created this project to wrap common use cases, into a loose framework
of redis based solutions for real world problems.
The project has started out as a bunch of code examples for a presentation I recently gave on [IL Tech Talks](http://www.iltechtalks.org.il/), that can be found here: http://www.slideshare.net/dvirsky/kicking-ass-with-redis
#Installation
Option 1:
1. Clone this repo or download the sources
2. `cd kickass-redis`
3. `python setup.py build`
4. `sudo python setup.py install`
Option 2:
`sudo pip install kickass_redis`
#Components
----------
To kick things off, the framework includes the following components:
## object_store
a fast yet simple ORM (well, OM actually) that automates creation, indexing and searching for complex objects using redis.
Indexes include: simple string index, numeric index that supports sorting and ranges, simplistic full text index, and a unique key.
###Example:
```python
from kickass_redis.patterns.object_store.objects import IndexedObject, KeySpec
from kickass_redis.patterns.object_store.indexing import UnorderedKey, OrderedNumericalKey
from kickass_redis.patterns.object_store.condition import Condition
class User(IndexedObject):
#which fields should be saved to redis
_spec = ('id', 'name', 'email', 'pwhash', 'registrationDate', 'score')
#The keys for this object
_keySpec = KeySpec(
UnorderedKey(prefix='users',fields=('name',)),
OrderedNumericalKey(prefix='users', field='score')
)
def __init__(self, **kwargs):
IndexedObject.__init__(self, **kwargs)
self.registrationDate = int(kwargs.get('registrationDate', time.time()))
#Creating a user
user = User(email = 'user@domain.com', name = 'John Doe', pwhash = 'eabc626ec26bc6ae6cb2', score = 100)
user.save()
#loading by name key
users = User.get(Condition({'name': 'John Doe'}))
#loading by id:
users = User.loadObjects((1,))
#See example/users_example for a more detailed exmample and some benchmarks
```
## bitmap_counter
efficient unique value counter (to be used mostly as a unique users counter) with time slots, making use of redis bitmaps.
It makes use of new redis-2.6 commands BITCOUNT and BITOP, so it will not function on redis-2.4.
###Example:
```python
from kickass_redis.patterns.bitmap_counter import BitmapCounter
#Daily unique users counter
counter = BitmapCounter('unique_users', timeResolutions=(BitmapCounter.RES_DAY,))
#sampling current user
counter.add(3)
#Getting the unique user count for today
counter.getCount((time.time(),), counter.RES_DAY)
#Getting cohort analysis on your users for the past week
week = tuple((int(time.time() - i*86400) for i in xrange(7, 0, -1)))
print counter.cohortAnalysis(week, counter.RES_DAY)
#Getting funnel analysis on your users for the past week
print counter.funnelAnalysis(week, counter.RES_DAY)
```
###New:
It now also supports mapping of non sequential or non numeric ids to incemental ids, that makes it memory optimized.
## LuaCall
A convenience wrapper that allows you to edit, precache and call Lua scripts available in redis-2.6, as if they were native python functions.
### Example:
let mult.lua contain the code:
```lua
local val = ARGV[1]*ARGV[2]
redis.call('set', KEYS[1], val)
return redis.call('get', KEYS[1])
```
Running it from python:
```python
import redis
from kickass_redis.patterns.lua import LuaCall, LuaScriptError
conn = redis.Redis()
#Define the call, and make it runn on our connection
mult = LuaCall(open('mult.lua'), conn)
#Call it once:
print "Result: %s" % mult(keys = ('foor',), args = (3,10))
#call it again
print "Result: %s" % mult(keys = ('foor2',), args = (5,20))
```
## idgenerator
Used in the object store, this can also be used standalone, as a centralized unique, incremental id generator using redis.
To optimize performance, it reserves in local memory many ids when accessing redis, which can be tuned.
## redis_unit
A unit-test like set of assertions about redis data to be used to validate the data inside a redis database.
## requirements
* redis-2.6 server(BITCOUNT/BITOP)
* redis-py
* [pyhash package](https://code.google.com/p/pyfasthash/)
###Example:
```python
from kickass_redis.patterns.redis_unit import RedisDataTest
class MyTest(RedisDataTest):
def testSomeStuff(self):
self.assertPrefixCount('users:*', minAmount=100000)
self.assertKeysExists('users:1')
self.assertKeysType(self.test.T_HASH, 'users:1')
self.assertHashValue('users:1', 'name', self.equals('John'))
test = MyTest('localhost', 6379)
test.run()
```
---------------------------
### To follow soon:
* geo search
* full text search
* hierarchical counters
* MySQL data sync
* Generic expiring object cache.
# Feel free to contribute more recipes...
#Project TODO:
1. Add unit tests for all objects
2. Add geo-key