Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/dcrosta/dabble

Simple A/B testing framework for Python
https://github.com/dcrosta/dabble

Last synced: 17 days ago
JSON representation

Simple A/B testing framework for Python

Awesome Lists containing this project

README 

        
# Dabble



Dabble is a simple A/B testing framework for Python. Using dabble, you

configure your tests in code, collect results, and analyze them later to

make informed decisions about design changes, feature implementations, etc.



You define an A/B test in dabble with class `ABTest`, which describes the

test name, the names of each of the alternatives, and the set of steps the

users will progress through during the test (in the simplest case, this is

just two steps). You then define one or more `ABParameter`s, which contain

the values you wish to vary for each alternative in the test. Each test can

have one or more alternatives, though the most common case is to have 2

(hence "A/B testing").



## Example



    import dabble

    dabble.configure(

        CookieIdentityProvider('dabble_id'),

        FSResultStorage('/path/to/results.data')

    )



    class Signup(page):

        path = '/signup'



        signup_button = ABTest('signup button',

                               alternatives=['red', 'green'],

                               steps=['show', 'signup'])

        button_color = ABParameter('signup button', ['#ff0000', '#00ff00'])



        def GET(self):

            self.signup_button.record('show')

            return render('index.html', button_color=self.button_color)



        def POST(self):

            self.signup_button.record('signup')

            return redirect('/account')



Behind the scenes, dabble has used a cookie for each user on your site to

assigne them each an *identity*, so that each user always ever sees the same

*alternative*. Users may visit the homepage many times over many browsing

sessions, but as long as they have the same cookie present in their browser,

they will always see either the red or the green button, depending on which

was chosen the first time the viewed the page.



When a user signs up, the `record()` method of `ABTest` is called, to track

the user's action. Later on, reports can be generated to determine whether

the red or the green button induced more users to sign up.



## Configuring Dabble



In addition to `ABTest` and `ABParameter`, dabble also needs an

`IdentityProvider` and a `ResultsStorage`. Dabble provides several

alternatives for each of these out of the box, and it is also

straightforward to write your own.



`IdentityProvider`s should make their best possible effort to always

identify individuals, rather than browsing sessions (particularly if cookies

are set to expire when the user closes his/her browser). If you are testing

a feature that requires users to be logged in, then their username is a good

choice for identity.



`ResultsStorage` stores configuration and results of A/B tests, and provides

some facilities for generating reports based on the stored results. Dabble

provides several backends, including `MongoResultsStorage`, and

`FSResultsStorage`.



At this time it is not possible to configure different `IdentityProvider`s

or `ResultsStorage`s for different tests within the same application.



## Reporting



Dabble will also produce reports on all users who have taken part in an A/B

test, by way of the `report()` method. The report is a dictionary which

describes, for each alternative, how many users attempted and converted at

each of the defined steps. For the above example, a report might look like:



    >>> storage = FSResultStorage('/path/to/results.data')

    >>> storage.report('signup button')

    {

        'test_name': 'signup button',

        'results': [

            {

                'alternative': 'red',

                'funnel': [{

                    'stage': ('show', 'signup'),

                    'attempted': 187,

                    'converted': 22,

                }],

            },

            {

                'alternative': 'green',

                'funnel': [{

                    'stage': ('show', 'signup'),

                    'attempted': 195

                    'converted': 18,

                }],

            }

        ],

    }



The `funnel` key in each of the `results` entries will have one element

fewer than the number of steps, since each entry describes the progression

of users from one step to the next.