Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/ErlyORM/boss_db

BossDB: a sharded, caching, pooling, evented ORM for Erlang
https://github.com/ErlyORM/boss_db

erlang orm

Last synced: 12 days ago
JSON representation

BossDB: a sharded, caching, pooling, evented ORM for Erlang

Awesome Lists containing this project

README 

        
BossDB: A sharded, caching, pooling, evented ORM for Erlang

===========================================================

[![Build Status](https://travis-ci.org/ErlyORM/boss_db.svg?branch=master)](https://travis-ci.org/ErlyORM/boss_db)



Attention! This is a master branch supporting Erlang 18 and above. For older Erlang versions use legacy branch.



Supported databases

-------------------



* *NEW* DynamoDB (experimental)

* Mnesia

* MongoDB

* MySQL

* PostgreSQL

* Riak

* Tokyo Tyrant



Complete API references

-----------------------



Querying: http://www.chicagoboss.org/doc/api-db.html



Records: http://www.chicagoboss.org/doc/api-record.html



BossNews: http://chicagoboss.org/doc/api-news.html



Write an adapter: https://github.com/ChicagoBoss/ChicagoBoss/wiki/DB-Adapter-Quickstart



Usage

-----

```erlang

boss_db:start(DBOptions),

boss_cache:start(CacheOptions), % If you want cacheing with Memcached

boss_news:start() % Mandatory! Hopefully will be optional one day



DBOptions = [

    {adapter, mock | tyrant | riak | mysql | pgsql | mnesia | mongodb},

    {db_host, HostName::string()},

    {db_port, PortNumber::integer()},

    {db_username, UserName::string()},

    {db_password, Password::string()},

    {db_database, Database::string()},

    {db_configure, DatabaseOptions::list()},

    {db_ssl, UseSSL::boolean() | required}, % for now pgsql only



    {shards, [

        {db_shard_models, [ModelName::atom()]},

        {db_shard_id, ShardId::atom()},

        {db_host, _}, {db_port, _}, ...

    ]},

    {cache_enable, true | false},

    {cache_exp_time, TTLSeconds::integer()},



    {size, PoolSize::non_neg_integer()}, % the size of the connection pool - defaults to 5

    {max_overflow, MaxOverflow::non_neg_integer()} % the maximum number of temporary extra workers that can be created past the `size' just above - defaults to 10

    %% the sum size + max_overflow effectively controls how many concurrent mysql queries can run

]



CacheOptions = [

    {adapter, memcached_bin | redis | ets},

    {cache_servers, MemcachedCacheServerOpts | RedisCacheServerOpts | EtsCacheServerOpts}

]



MemcachedCacheServerOpts = [

    { HostName::string() = "localhost"

    , Port::integer()    = 11211

    , Weight::integer()  = 1

    }, ...

]



RedisCacheServerOpts = [

    {host,      HostName::string()   = "localhost"},

    {port,      Port::integer()      = 6379},

    {pass,      Password::string()   = undefined},

    {db,        Db::integer()        = 0},

    {reconnect, Reconnect::boolean() = true}

]



EtsCacheServerOpts = [

    {ets_maxsize,   MaxSize::integer() = 32 * 1024 * 1024},

    {ets_threshold, Threshold::float() = 0.85},

    {ets_weight,    Weight::integer()  = 30}

]

```



Introduction

------------



BossDB is a compiler chain and run-time library for accessing a database via

Erlang parameterized modules. It solves the age-old problem of retrieving

named fields without resorting to verbosities like proplists:get_value/2 or

dict:find/2. For example, if you want to look up a puppy by ID and print its

name, you would write:



```erlang

Puppy = boss_db:find("puppy-1"),

io:format("Puppy's name: ~p~n", [Puppy:name()]).

```



Functions for accessing field names are generated automatically. All you need

to do is create a model file and compile it with boss_record_compiler. Example:



The model file, call it puppy.erl:



```erlang

-module(puppy, [Id, Name, BreedId]).

```



Then compile it like:



```erlang

{ok, puppy} = boss_record_compiler:compile("puppy.erl")

```



...and you're ready to go.



You can also enable boss_db_rebar plugin in your rebar.config to automatize

compilation:



```erlang

{plugin_dir, ["deps/boss_db/priv/rebar"]}.

{plugins, [boss_db_rebar]}.

{boss_db_opts, [

    {model_dir, "src/model"}

]}.

```



Associations

------------



BossDB supports database associations. Suppose you want to model the dog breed

(golden retriever, poodle, etc). You would create a model file with a special

"-has" attribute, like:



```erlang

-module(breed, [Id, Name]).

-has({puppies, many}).

```

Then back in puppy.erl you'd add a "-belongs_to" attribute:



```erlang

-module(puppy, [Id, Name, BreedId]).

-belongs_to(breed).

```



Once you've compiled breed.erl with boss_record_compiler, you can print a puppy's

associated breed like:



```erlang

Breed = Puppy:breed(),

io:format("Puppy's breed: ~p~n", [Breed:name()]).

```



Similarly, you could iterate over all the puppies of a particular breed:



```erlang

Breed = boss_db:find("breed-47"),

lists:map(fun(Puppy) -> 

        io:format("Puppy: ~p~n", [Puppy:name()]) 

    end, Breed:puppies())

```



Querying

--------



You can search the database with the boss_db:find functions. Example:



```erlang

Puppies = boss_db:find(puppy, [{breed_id, 'equals', "breed-47"}])

```



This is somewhat verbose. If you compile the source file with boss_compiler,

you'll be able to write the more simple expression:



```erlang

Puppies = boss_db:find(puppy, [breed_id = "breed-47"])

```



BossDB supports many query operators, as well as sorting, offsets, and limits;

see the API references at the top.



Validating and saving

---------------------



To create and save a new record, you would write:



```erlang

Breed = breed:new(id, "Golden Retriever"),

{ok, SavedBreed} = Breed:save()

```



You can provide validation logic by adding a validation_tests/0 function

to your model file, e.g.



```erlang

-module(breed, [Id, Name]).

-has({puppies, many}).

-export([validation_tests/0]).



validation_tests() ->

    [{fun() -> length(Name) > 0 end,

        "Name must not be empty!"}].

```



If validation fails, the save/0 function will return a list of error messages

instead of the saved record.



You can also provide spec strings in the parameter declaration if you want to

validate the attribute types before saving, e.g.



```erlang

-module(puppy, [Id, Name::string(), BirthDate::datetime()]).

```



Accepted types are:



* string()

* binary()

* datetime()

* date()

* timestamp() [e.g. returned by erlang:now()]

* integer()

* float()



If the type validation fails, then validation_tests/0 will not be called.



Working with existing SQL databases

-----------------------------------



By default, SQL columns must be underscored versions of the attribute names,

and SQL tables must be plural versions of the model names. (If the model is

"puppy", the database table should be "puppies".)



You may want to override these defaults if you are working with an existing

database. To specify your own column and table names, you can use the

-columns() and -table() attributes in a model file like so:



```erlang

-module(puppy, [Id, Name]).

-columns([{id, "puppy_id"}, {name, "puppy_name"}]).

-table("puppy_table").

```



Events

------



BossDB provides two kinds of model events: synchronous save hooks, and

asynchronous notifications via BossNews. Save hooks are simple; just

define one or more of these functions in your model file:



```erlang

before_create/0 -> ok | {ok, ModifiedRecord} | {error, Reason}

before_update/0 -> ok | {ok, ModifiedRecord} | {error, Reason}

before_update/1 -> ok | {ok, ModifiedRecord} | {error, Reason}

after_create/0

after_update/0

after_update/1

before_delete/0 -> ok | {error, Reason}

```



The before_update/1 and after_update/1 hooks provide access to the old

boss record as an additional parameter. If both hooks like before_update/0

and before_update/1 exists in a boss model module, before_update/0 will

be ignored. The same behaviour applies to after_update/1 and after_update/0.



BossNews is more complicated but also more powerful. It is a notification

system that executes asynchronously, so the code that calls "save" does

not have to wait for callbacks to complete. The central concept in BossNews

is a "watch", which is an event observer. You can create and destroy watches

programmatically:



```

{ok, WatchId} = boss_news:watch(TopicString, CallBack),

boss_news:cancel_watch(WatchId)

```

Four kinds of topic strings are supported:



```erlang

"puppies" => watch for new and deleted Puppy records

"puppy-42.*" => watch all attributes of Puppy #42

"puppy-*.name" => watch the "name" attribute of all Puppy records

"puppy-*.*" => watch all attributes of all Puppy records

```



The callback is passed two or three arguments: the event name

(created/updated/deleted), information about the event (i.e. the new and old

values of the watched record), and optionally user information passed as the

third argument to boss_news:watch/3.



BossNews is suited to providing real-time notifications and alerts. For example,

if you want to log each time a puppy's name is changed,



```erlang

boss_news:watch("puppy-*.name", 

        fun(updated, {Puppy, 'name', OldName, NewName}) ->

            error_logger:info_msg("Puppy's name changed from ~p to ~p", [OldName, NewName])

        end)

```



For more details see the documentation at http://www.chicagoboss.org/doc/api-news.html



Caching

-------



If caching is enabled, queries and records are automatically cached. BossDB

uses BossNews events to automatically invalidate out-of-date cache entries; you do

not need to write any cache logic in your save hooks.



Sharding

--------



Vertical sharding is supported via the db_shards config option. Simply add shard-specific

configuration in a proplist along with an extra config parameter called db_shard_models,

which should be a list of models (atoms) in the shard.



Pooling

-------



BossDB uses Poolboy to create a connection pool to the database. Connection pooling

is supported with all databases.



Primary Keys

------------



The Id field of each model is assumed to be an integer supplied by the

database (e.g., a SERIAL type in Postgres or AUTOINCREMENT in MySQL).

Specifying an Id value other than the atom 'id' for a new record will

result in an error.



When using the mock or pgsql adapters, the Id may have a type of

::uuid().  This will coerce boss_db into generating a v4 UUID for the

Id field before saving the record (in other words, the UUID is

provided by boss_db and not by the application nor by the DB).  UUIDs

are useful PKs when data are being aggregated from multiple sources.



The default Id type ::serial() may be explicitly supplied.  Note that

all Id types, valid or otherwise, pass type validation.