Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

https://github.com/norton-archive/lets

LETS - LevelDB-based Erlang Term Storage
https://github.com/norton-archive/lets

Last synced: 3 months ago
JSON representation

LETS - LevelDB-based Erlang Term Storage

Lists

README 

        



# LETS - LevelDB-based Erlang Term Storage #



Copyright (c) 2011-2016 by Joseph Wayne Norton



__Authors:__ Joseph Wayne Norton ([`[email protected]`](mailto:[email protected])).

LETS is an alternative Erlang Term Storage using LevelDB as the

storage implementation.  LETS tries to address some bad properties of

ETS and DETS.  ETS is limited by physical memory.  DETS is limited by

a 2 GB file size limitation and does not implement ordered sets.  LETS

has neither of these limitations.  Data can also be automatically

compressed using the Snappy compression library.  In addition, the

name of a LETS table can be any Erlang term.


LETS is not intended to be an exact clone of ETS.  The currently

supported ETS APIs are:


    

  • 

    

    all/0

    

    • 

  • 

    

    delete/1

    

    • 

  • 

    

    delete/2

    

    • 

  • 

    

    delete_all_objects/1 only ets implementation

    

    • 

  • 

    

    first/1

    

    • 

  • 

    

    foldl/3

    

    • 

  • 

    

    foldr/3

    

    • 

  • 

    

    info/1 only a subset of items

    

    • 

  • 

    

    info/2 only a subset of items

    

    • 

  • 

    

    insert/2

    

    • 

  • 

    

    insert_new/2 only ets implementation

    

    • 

  • 

    

    last/1

    

    • 

  • 

    

    lookup/2

    

    • 

  • 

    

    lookup_element/3

    

    • 

  • 

    

    match/1

    

    • 

  • 

    

    match/2

    

    • 

  • 

    

    match/3

    

    • 

  • 

    

    match_delete/2

    

    • 

  • 

    

    match_object/1

    

    • 

  • 

    

    match_object/2

    

    • 

  • 

    

    match_object/3

    

    • 

  • 

    

    member/2

    

    • 

  • 

    

    new/2

    

    • 

  • 

    

    next/2

    

    • 

  • 

    

    prev/2

    

    • 

  • 

    

    select/1

    

    • 

  • 

    

    select/2

    

    • 

  • 

    

    select/3

    

    • 

  • 

    

    select_count/2

    

    • 

  • 

    

    select_delete/2

    

    • 

  • 

    

    select_reverse/1

    

    • 

  • 

    

    select_reverse/2

    

    • 

  • 

    

    select_reverse/3

    

    • 

  • 

    

    tab2list/1

    

    • 



For testing and comparison purposes, LETS supports five backend

implementations:


    

  • 

    

    drv CPP Driver with LevelDB backend (default)

    

    • 

  • 

    

    nif CPP NIF with LevelDB backend

    

    • 

  • 

    

    hyper drv CPP Driver with HyperLevelDB backend

    

    • 

  • 

    

    hyper nif CPP NIF with HyperLevelDB backend

    

    • 

  • 

    

    rocks drv CPP Driver with RocksDB backend

    

    • 

  • 

    

    rocks nif CPP NIF with RocksDB backend

    

    • 

  • 

    

    ets Erlang ETS backend

    

    • 



If desired, all of the five backends can be called directly using the

table identifier returned by new/2, tid/1, and/or tid/2.  The

tid/1 and tid/2 approach must be used for named tables.  The five

backends implement the gen_ets_ns behaviour to provide batch

insertion, batch iteration by key, batch iteration by object, and

other basic table operations.  See the following modules for the

corresponding backend:


    

  • 

    

    drv - src/lets_impl_drv.erl

    

    • 

  • 

    

    nif - src/lets_impl_nif.erl

    

    • 

  • 

    

    hyper drv - src/hets_impl_drv.erl

    

    • 

  • 

    

    hyper nif - src/hets_impl_nif.erl

    

    • 

  • 

    

    rocks drv - src/rets_impl_drv.erl

    

    • 

  • 

    

    rocks nif - src/rets_impl_nif.erl

    

    • 

  • 

    

    ets - deps/gen_ets/src/gen_ets_impl_ets.erl

    

    • 



The Driver backends are recommended over the NIF backends for two

reasons:


    

  • 

    

    The NIF backends when deleting the table rely on garbage collection

      of the NIF resource to delete the corresponding CPP database object.

      The calling application must manage this carefully to ensure that a

      call to re-open an existing database is not performed until the CPP

      database object is deleted.  The Driver backends do not have this

      limitation.

    

    • 

  • 

    

    The NIF backends are not built as "dirty" NIFs and can easily block

      the emulator's schedulers.  The Driver backends support the async

      option for such purpose.  It is recommended to use the Driver

      backends until the "dirty" NIF functionality is a non-experimental

      Erlang/OTP feature and the LETS NIF backends are built as "dirty"

      NIFs.

    

    • 



Development and testing of LETS is done solely on Mac OS X.  Feedback

and/or bug reports on other operating systems is welcome.


This repository is experimental in nature - use at your own risk and

please contribute if you find LETS useful.



Quick Start Recipe



To download and build the lets application in one shot, please follow

this recipe:



$ mkdir working-directory-name

$ cd working-directory-name

$ git clone https://github.com/norton/lets.git lets

$ cd lets

$ make deps clean compile



OR if QuickCheck is available then follow this recipe:



$ mkdir working-directory-name

$ cd working-directory-name

$ git clone https://github.com/norton/lets.git lets

$ cd lets

$ make deps clean compile-for-eqc

$ (cd .qc; erl -smp +A 5 -pz -pz ../deps/{sext,gen_ets,qc}/ebin)



1> qc_statem_lets:qc_run(500).

.......

OK, passed 500 tests



100.0% {1,attempts}



3.46% {{undefined,[]},{new,2},ok}

1.21% {{ets,[]},{select_delete,2},ok}

1.19% {{ets,[]},{tid,1},ok}

1.11% {{ets,[]},{match_object,2},ok}

1.11% {{ets,[]},{insert,2},ok}

1.11% {{ets,[]},{all,1},ok}

1.11% {{ets,[]},{select_reverse,2},ok}

1.11% {{ets,[]},{match_delete,2},ok}

1.08% {{ets,[]},{member,2},ok}

1.08% {{ets,[]},{insert_new,2},ok}

1.03% {{ets,[]},{foldl,3},ok}

1.02% {{ets,[]},{select,2},ok}

1.01% {{ets,[]},{foldr,3},ok}

1.01% {{ets,[]},{delete,2},ok}

1.01% {{ets,[]},{select31,3},ok}

1.00% {{ets,[]},{match_object31,3},ok}

0.98% {{ets,[]},{tid,2},ok}

0.97% {{ets,[]},{select_reverse31,3},ok}

0.97% {{ets,[]},{select_count,2},ok}

0.97% {{ets,[]},{delete,1},ok}

0.95% {{ets,[]},{delete_all_objects,1},ok}

0.94% {{ets,[]},{last,1},ok}

0.93% {{ets,[]},{lookup_element,3},{error,badarg}}

0.93% {{ets,[]},{first,1},ok}

0.92% {{ets,[]},{tab2list,1},ok}

0.90% {{ets,[]},{match,2},ok}

0.90% {{ets,[]},{lookup,2},ok}

0.90% {{ets,[]},{match31,3},ok}

0.89% {{ets,[]},{new,3},ok}

0.67% {{drv,[hyper]},{tid,1},ok}

0.66% {{drv,[hyper]},{select,2},ok}

0.66% {{drv,[hyper]},{all,1},ok}

0.65% {{drv,[hyper]},{match_object31,3},ok}

0.64% {{nif,[]},{match_object,2},ok}

0.64% {{drv,[hyper]},{select_reverse31,3},ok}

0.64% {{nif,[]},{select_reverse31,3},ok}

0.64% {{drv,[hyper]},{tid,2},ok}

0.63% {{nif,[]},{tab2list,1},ok}

0.62% {{nif,[]},{select31,3},ok}

0.62% {{drv,[hyper]},{foldr,3},ok}

0.61% {{drv,[hyper]},{next,2},ok}

0.61% {{nif,[]},{select_delete,2},ok}

0.61% {{drv,[hyper]},{first,1},ok}

0.60% {{nif,[]},{match_object31,3},ok}

0.60% {{drv,[hyper]},{match_object,2},ok}

0.60% {{drv,[hyper]},{match_delete,2},ok}

0.60% {{drv,[hyper]},{delete,2},ok}

0.59% {{drv,[hyper]},{foldl,3},ok}

0.58% {{nif,[]},{select,2},ok}

0.58% {{drv,[hyper]},{tab2list,1},ok}

0.58% {{drv,[hyper]},{select_count,2},ok}

0.57% {{drv,[hyper]},{member,2},ok}

0.56% {{drv,[hyper]},{match31,3},ok}

0.56% {{drv,[hyper]},{insert,2},ok}

0.56% {{nif,[]},{prev,2},ok}

0.56% {{nif,[]},{next,2},ok}

0.55% {{nif,[]},{select_reverse,2},ok}

0.55% {{nif,[]},{lookup,2},ok}

0.55% {{drv,[hyper]},{select_delete,2},ok}

0.54% {{nif,[]},{all,1},ok}

0.54% {{ets,[]},{next,2},{error,badarg}}

0.53% {{nif,[]},{match31,3},ok}

0.53% {{nif,[]},{delete,1},ok}

0.53% {{nif,[]},{new,3},ok}

0.51% {{nif,[]},{foldr,3},ok}

0.51% {{drv,[hyper]},{lookup,2},ok}

0.51% {{nif,[]},{last,1},ok}

0.51% {{ets,[]},{next,2},ok}

0.51% {{drv,[hyper]},{select_reverse,2},ok}

0.51% {{drv,[hyper]},{prev,2},ok}

0.50% {{drv,[hyper]},{last,1},ok}

0.48% {{nif,[]},{member,2},ok}

0.48% {{ets,[]},{prev,2},ok}

0.48% {{drv,[hyper]},{select31,3},ok}

0.48% {{nif,[]},{match_delete,2},ok}

0.48% {{nif,[]},{delete,2},ok}

0.47% {{drv,[hyper]},{match,2},ok}

0.47% {{drv,[hyper]},{delete,1},ok}

0.46% {{drv,[]},{delete,2},ok}

0.45% {{nif,[]},{foldl,3},ok}

0.44% {{nif,[hyper]},{match_object31,3},ok}

0.44% {{nif,[]},{tid,2},ok}

0.44% {{nif,[]},{insert,2},ok}

0.43% {{nif,[rocks]},{member,2},ok}

0.43% {{drv,[hyper]},{new,3},ok}

0.43% {{drv,[hyper]},{lookup_element,3},{error,badarg}}

0.43% {{nif,[rocks]},{all,1},ok}

0.43% {{ets,[]},{prev,2},{error,badarg}}

0.42% {{nif,[hyper]},{match_object,2},ok}

0.42% {{drv,[]},{last,1},ok}

0.41% {{nif,[]},{tid,1},ok}

0.40% {{drv,[]},{tid,1},ok}

0.40% {{drv,[]},{select_reverse,2},ok}

0.39% {{nif,[hyper]},{tid,1},ok}

0.39% {{nif,[hyper]},{select31,3},ok}

0.39% {{drv,[]},{insert,2},ok}

0.38% {{nif,[rocks]},{select,2},ok}

0.38% {{nif,[rocks]},{insert,2},ok}

0.38% {{nif,[rocks]},{foldr,3},ok}

0.38% {{nif,[hyper]},{lookup,2},ok}

0.38% {{nif,[hyper]},{last,1},ok}

0.38% {{nif,[hyper]},{first,1},ok}

0.38% {{nif,[]},{match,2},ok}

0.38% {{nif,[]},{first,1},ok}

0.38% {{drv,[]},{match_object31,3},ok}

0.38% {{nif,[rocks]},{select31,3},ok}

0.38% {{nif,[hyper]},{insert,2},ok}

0.38% {{nif,[]},{select_count,2},ok}

0.38% {{drv,[]},{tid,2},ok}

0.37% {{nif,[rocks]},{select_reverse31,3},ok}

0.37% {{nif,[rocks]},{prev,2},ok}

0.37% {{nif,[rocks]},{match,2},ok}

0.37% {{drv,[rocks]},{select_delete,2},ok}

0.37% {{drv,[]},{first,1},ok}

0.36% {{drv,[]},{tab2list,1},ok}

0.35% {{nif,[rocks]},{tab2list,1},ok}

0.35% {{nif,[hyper]},{select_reverse31,3},ok}

0.35% {{nif,[hyper]},{next,2},ok}

0.35% {{drv,[rocks]},{tid,2},ok}

0.35% {{drv,[]},{delete,1},ok}

0.35% {{nif,[rocks]},{match_object,2},ok}

0.35% {{nif,[hyper]},{select,2},ok}

0.35% {{nif,[hyper]},{match_delete,2},ok}

0.35% {{nif,[hyper]},{delete,2},ok}

0.35% {{drv,[]},{select_reverse31,3},ok}

0.35% {{drv,[]},{foldr,3},ok}

0.34% {{nif,[rocks]},{select_delete,2},ok}

0.34% {{nif,[hyper]},{select_reverse,2},ok}

0.34% {{nif,[hyper]},{foldr,3},ok}

0.33% {{nif,[hyper]},{tab2list,1},ok}

0.33% {{nif,[]},{lookup_element,3},{error,badarg}}

0.33% {{drv,[rocks]},{next,2},ok}

0.33% {{drv,[]},{match_object,2},ok}

0.33% {{nif,[rocks]},{select_count,2},ok}

0.33% {{nif,[rocks]},{match_object31,3},ok}

0.33% {{nif,[rocks]},{match_delete,2},ok}

0.33% {{nif,[hyper]},{prev,2},ok}

0.33% {{nif,[hyper]},{match31,3},ok}

0.33% {{drv,[rocks]},{prev,2},ok}

0.32% {{nif,[rocks]},{foldl,3},ok}

0.32% {{nif,[hyper]},{match,2},ok}

0.32% {{nif,[hyper]},{lookup_element,3},{error,badarg}}

0.32% {{drv,[rocks]},{last,1},ok}

0.32% {{drv,[]},{new,3},ok}

0.32% {{drv,[]},{match,2},ok}

0.31% {{nif,[rocks]},{tid,2},ok}

0.31% {{nif,[rocks]},{match31,3},ok}

0.31% {{nif,[rocks]},{delete,2},ok}

0.31% {{nif,[rocks]},{delete,1},ok}

0.31% {{drv,[rocks]},{tid,1},ok}

0.31% {{drv,[]},{lookup,2},ok}

0.31% {{drv,[]},{all,1},ok}

0.30% {{nif,[rocks]},{select_reverse,2},ok}

0.30% {{nif,[rocks]},{new,3},ok}

0.30% {{nif,[hyper]},{all,1},ok}

0.30% {{drv,[rocks]},{first,1},ok}

0.30% {{drv,[]},{prev,2},ok}

0.30% {{drv,[]},{next,2},ok}

0.30% {{drv,[]},{lookup_element,3},{error,badarg}}

0.30% {{nif,[hyper]},{tid,2},ok}

0.30% {{drv,[rocks]},{tab2list,1},ok}

0.30% {{drv,[rocks]},{select_reverse,2},ok}

0.30% {{drv,[]},{match_delete,2},ok}

0.29% {{nif,[rocks]},{next,2},ok}

0.29% {{nif,[hyper]},{select_count,2},ok}

0.29% {{drv,[rocks]},{delete,2},ok}

0.28% {{drv,[rocks]},{insert,2},ok}

0.28% {{drv,[]},{foldl,3},ok}

0.28% {{nif,[rocks]},{tid,1},ok}

0.28% {{nif,[hyper]},{select_delete,2},ok}

0.28% {{nif,[hyper]},{member,2},ok}

0.28% {{nif,[hyper]},{foldl,3},ok}

0.28% {{nif,[hyper]},{delete,1},ok}

0.28% {{drv,[rocks]},{delete,1},ok}

0.28% {{drv,[]},{select,2},ok}

0.27% {{drv,[rocks]},{new,3},ok}

0.27% {{drv,[rocks]},{lookup,2},ok}

0.27% {{drv,[]},{select_delete,2},ok}

0.27% {{drv,[]},{select_count,2},ok}

0.27% {{drv,[]},{match31,3},ok}

0.26% {{nif,[rocks]},{lookup_element,3},{error,badarg}}

0.26% {{nif,[hyper]},{new,3},ok}

0.26% {{drv,[rocks]},{select31,3},ok}

0.26% {{drv,[rocks]},{match_object,2},ok}

0.26% {{drv,[rocks]},{match_delete,2},ok}

0.26% {{drv,[]},{member,2},ok}

0.25% {{nif,[rocks]},{lookup,2},ok}

0.25% {{drv,[rocks]},{match,2},ok}

0.25% {{nif,[rocks]},{last,1},ok}

0.25% {{nif,[rocks]},{first,1},ok}

0.25% {{drv,[rocks]},{match31,3},ok}

0.25% {{drv,[rocks]},{foldr,3},ok}

0.24% {{drv,[rocks]},{select_count,2},ok}

0.23% {{drv,[rocks]},{member,2},ok}

0.23% {{drv,[rocks]},{all,1},ok}

0.22% {{drv,[rocks]},{select,2},ok}

0.22% {{drv,[]},{select31,3},ok}

0.21% {{drv,[rocks]},{match_object31,3},ok}

0.21% {{drv,[rocks]},{foldl,3},ok}

0.20% {{drv,[rocks]},{lookup_element,3},{error,badarg}}

0.17% {{drv,[rocks]},{select_reverse31,3},ok}

0.13% {{ets,[]},{lookup_element,3},ok}

0.11% {{nif,[]},{lookup_element,3},ok}

0.08% {{drv,[hyper]},{lookup_element,3},ok}

0.07% {{nif,[rocks]},{lookup_element,3},ok}

0.04% {{drv,[]},{lookup_element,3},ok}

0.03% {{nif,[hyper]},{lookup_element,3},ok}

0.03% {{drv,[rocks]},{lookup_element,3},ok}

true

.......



Tip



For testing LevelDB directly using the C bindings, tryqc_statemc_lets:qc_run(500).



OR if PropEr is available then follow this recipe:



$ mkdir working-directory-name

$ cd working-directory-name

$ git clone https://github.com/norton/lets.git lets

$ cd lets

$ make deps clean compile-for-proper

$ (cd .qc; erl -smp +A 5 -pz -pz ../deps/{sext,gen_ets,qc}/ebin)



1> qc_statem_lets:qc_run(500).

.......

OK: Passed 500 test(s).



9% {{undefined,[]},{new,2},ok}

1% {{ets,[]},{foldr,3},ok}

1% {{ets,[]},{tid,2},ok}

1% {{ets,[]},{delete,1},ok}

1% {{ets,[]},{tid,1},ok}

1% {{ets,[]},{delete_all_objects,1},ok}

1% {{ets,[]},{new,3},ok}

1% {{ets,[]},{select,2},ok}

1% {{ets,[]},{all,1},ok}

1% {{ets,[]},{last,1},ok}

1% {{ets,[]},{select_delete,2},ok}

1% {{ets,[]},{select_reverse31,3},ok}

1% {{ets,[]},{match_delete,2},ok}

1% {{ets,[]},{select31,3},ok}

1% {{ets,[]},{insert_new,2},ok}

1% {{ets,[]},{match_object,2},ok}

0% {{ets,[]},{delete,2},ok}

0% {{ets,[]},{tab2list,1},ok}

0% {{ets,[]},{insert,2},ok}

0% {{ets,[]},{match_object31,3},ok}

0% {{ets,[]},{foldl,3},ok}

0% {{ets,[]},{lookup,2},ok}

0% {{ets,[]},{lookup_element,3},{error,badarg}}

0% {{ets,[]},{first,1},ok}

0% {{ets,[]},{member,2},ok}

0% {{ets,[]},{select_count,2},ok}

0% {{ets,[]},{match,2},ok}

0% {{ets,[]},{select_reverse,2},ok}

0% {{ets,[]},{match31,3},ok}

0% {{ets,[]},{prev,2},ok}

0% {{nif,[]},{select,2},ok}

0% {{ets,[]},{next,2},{error,badarg}}

0% {{nif,[]},{tid,1},ok}

0% {{drv,[]},{match,2},ok}

0% {{drv,[]},{select_delete,2},ok}

0% {{nif,[]},{delete,2},ok}

0% {{nif,[]},{first,1},ok}

0% {{nif,[]},{insert,2},ok}

0% {{nif,[]},{select31,3},ok}

0% {{drv,[]},{delete,1},ok}

0% {{nif,[]},{match_object,2},ok}

0% {{drv,[hyper]},{foldr,3},ok}

0% {{drv,[rocks]},{prev,2},ok}

0% {{ets,[]},{next,2},ok}

0% {{ets,[]},{prev,2},{error,badarg}}

0% {{nif,[]},{delete,1},ok}

0% {{nif,[]},{match_delete,2},ok}

0% {{nif,[rocks]},{tid,1},ok}

0% {{drv,[hyper]},{match31,3},ok}

0% {{nif,[]},{match,2},ok}

0% {{nif,[]},{new,3},ok}

0% {{nif,[]},{select_reverse31,3},ok}

0% {{drv,[]},{match31,3},ok}

0% {{drv,[]},{tab2list,1},ok}

0% {{nif,[]},{foldr,3},ok}

0% {{nif,[rocks]},{last,1},ok}

0% {{nif,[rocks]},{lookup_element,3},{error,badarg}}

0% {{nif,[rocks]},{match,2},ok}

0% {{drv,[]},{new,3},ok}

0% {{drv,[hyper]},{select31,3},ok}

0% {{drv,[hyper]},{select_reverse,2},ok}

0% {{drv,[hyper]},{select_reverse31,3},ok}

0% {{nif,[]},{next,2},ok}

0% {{nif,[rocks]},{foldl,3},ok}

0% {{nif,[rocks]},{select_delete,2},ok}

0% {{nif,[rocks]},{tab2list,1},ok}

0% {{drv,[]},{all,1},ok}

0% {{drv,[]},{delete,2},ok}

0% {{drv,[]},{first,1},ok}

0% {{drv,[]},{foldr,3},ok}

0% {{drv,[]},{match_object31,3},ok}

0% {{drv,[]},{next,2},ok}

0% {{drv,[hyper]},{match_object31,3},ok}

0% {{drv,[hyper]},{tab2list,1},ok}

0% {{nif,[]},{select_delete,2},ok}

0% {{nif,[rocks]},{prev,2},ok}

0% {{drv,[hyper]},{insert,2},ok}

0% {{drv,[hyper]},{select,2},ok}

0% {{drv,[rocks]},{tid,1},ok}

0% {{nif,[]},{last,1},ok}

0% {{nif,[]},{lookup_element,3},{error,badarg}}

0% {{nif,[]},{match_object31,3},ok}

0% {{nif,[]},{member,2},ok}

0% {{drv,[]},{insert,2},ok}

0% {{drv,[hyper]},{tid,2},ok}

0% {{drv,[rocks]},{match31,3},ok}

0% {{nif,[]},{lookup,2},ok}

0% {{nif,[]},{match31,3},ok}

0% {{nif,[]},{select_count,2},ok}

0% {{drv,[]},{last,1},ok}

0% {{drv,[]},{lookup,2},ok}

0% {{drv,[]},{select_count,2},ok}

0% {{drv,[hyper]},{lookup_element,3},{error,badarg}}

0% {{drv,[hyper]},{match,2},ok}

0% {{drv,[hyper]},{select_delete,2},ok}

0% {{nif,[]},{prev,2},ok}

0% {{nif,[]},{tab2list,1},ok}

0% {{nif,[rocks]},{next,2},ok}

0% {{drv,[]},{foldl,3},ok}

0% {{drv,[]},{match_object,2},ok}

0% {{drv,[]},{member,2},ok}

0% {{drv,[]},{select,2},ok}

0% {{drv,[]},{select31,3},ok}

0% {{drv,[]},{tid,2},ok}

0% {{drv,[hyper]},{foldl,3},ok}

0% {{drv,[rocks]},{all,1},ok}

0% {{drv,[rocks]},{match_object,2},ok}

0% {{nif,[rocks]},{foldr,3},ok}

0% {{nif,[rocks]},{lookup,2},ok}

0% {{nif,[rocks]},{member,2},ok}

0% {{drv,[hyper]},{all,1},ok}

0% {{drv,[rocks]},{tab2list,1},ok}

0% {{nif,[]},{all,1},ok}

0% {{nif,[]},{select_reverse,2},ok}

0% {{nif,[hyper]},{tab2list,1},ok}

0% {{nif,[rocks]},{first,1},ok}

0% {{nif,[rocks]},{insert,2},ok}

0% {{nif,[rocks]},{select31,3},ok}

0% {{nif,[rocks]},{select_reverse31,3},ok}

0% {{drv,[]},{lookup_element,3},{error,badarg}}

0% {{drv,[]},{select_reverse,2},ok}

0% {{drv,[hyper]},{first,1},ok}

0% {{drv,[hyper]},{member,2},ok}

0% {{drv,[rocks]},{delete,1},ok}

0% {{drv,[rocks]},{last,1},ok}

0% {{drv,[rocks]},{lookup,2},ok}

0% {{drv,[rocks]},{match,2},ok}

0% {{drv,[rocks]},{new,3},ok}

0% {{drv,[rocks]},{select_reverse,2},ok}

0% {{nif,[]},{foldl,3},ok}

0% {{nif,[hyper]},{delete,2},ok}

0% {{nif,[hyper]},{last,1},ok}

0% {{nif,[hyper]},{select_count,2},ok}

0% {{nif,[rocks]},{delete,1},ok}

0% {{nif,[rocks]},{match_delete,2},ok}

0% {{nif,[rocks]},{match_object31,3},ok}

0% {{nif,[rocks]},{select_count,2},ok}

0% {{drv,[]},{match_delete,2},ok}

0% {{drv,[]},{prev,2},ok}

0% {{drv,[hyper]},{delete,1},ok}

0% {{drv,[hyper]},{next,2},ok}

0% {{drv,[hyper]},{prev,2},ok}

0% {{drv,[hyper]},{tid,1},ok}

0% {{drv,[rocks]},{foldl,3},ok}

0% {{drv,[rocks]},{match_delete,2},ok}

0% {{nif,[]},{tid,2},ok}

0% {{nif,[hyper]},{foldr,3},ok}

0% {{nif,[hyper]},{lookup,2},ok}

0% {{nif,[rocks]},{delete,2},ok}

0% {{nif,[rocks]},{select,2},ok}

0% {{nif,[rocks]},{tid,2},ok}

0% {{drv,[hyper]},{last,1},ok}

0% {{drv,[hyper]},{lookup,2},ok}

0% {{drv,[hyper]},{match_delete,2},ok}

0% {{drv,[hyper]},{select_count,2},ok}

0% {{drv,[rocks]},{match_object31,3},ok}

0% {{ets,[]},{lookup_element,3},ok}

0% {{nif,[hyper]},{match31,3},ok}

0% {{nif,[hyper]},{prev,2},ok}

0% {{nif,[hyper]},{select,2},ok}

0% {{nif,[rocks]},{new,3},ok}

0% {{drv,[]},{select_reverse31,3},ok}

0% {{drv,[hyper]},{match_object,2},ok}

0% {{drv,[hyper]},{new,3},ok}

0% {{drv,[rocks]},{first,1},ok}

0% {{drv,[rocks]},{foldr,3},ok}

0% {{drv,[rocks]},{insert,2},ok}

0% {{nif,[hyper]},{first,1},ok}

0% {{nif,[hyper]},{insert,2},ok}

0% {{nif,[hyper]},{select_reverse31,3},ok}

0% {{nif,[rocks]},{all,1},ok}

0% {{nif,[rocks]},{match31,3},ok}

0% {{drv,[]},{tid,1},ok}

0% {{drv,[rocks]},{select_delete,2},ok}

0% {{nif,[hyper]},{match_object31,3},ok}

0% {{nif,[hyper]},{select_delete,2},ok}

0% {{nif,[hyper]},{tid,1},ok}

0% {{nif,[rocks]},{select_reverse,2},ok}

0% {{drv,[hyper]},{delete,2},ok}

0% {{nif,[hyper]},{all,1},ok}

0% {{nif,[hyper]},{match,2},ok}

0% {{nif,[hyper]},{match_delete,2},ok}

0% {{nif,[hyper]},{member,2},ok}

0% {{drv,[rocks]},{delete,2},ok}

0% {{drv,[rocks]},{select,2},ok}

0% {{drv,[rocks]},{select31,3},ok}

0% {{nif,[hyper]},{delete,1},ok}

0% {{nif,[hyper]},{lookup_element,3},{error,badarg}}

0% {{nif,[hyper]},{next,2},ok}

0% {{nif,[hyper]},{select_reverse,2},ok}

0% {{nif,[hyper]},{tid,2},ok}

0% {{nif,[rocks]},{match_object,2},ok}

0% {{drv,[rocks]},{member,2},ok}

0% {{drv,[rocks]},{select_reverse31,3},ok}

0% {{drv,[rocks]},{tid,2},ok}

0% {{nif,[hyper]},{foldl,3},ok}

0% {{nif,[hyper]},{new,3},ok}

0% {{drv,[rocks]},{next,2},ok}

0% {{drv,[rocks]},{select_count,2},ok}

0% {{nif,[]},{lookup_element,3},ok}

0% {{nif,[hyper]},{match_object,2},ok}

0% {{drv,[rocks]},{lookup_element,3},{error,badarg}}

0% {{nif,[hyper]},{select31,3},ok}

0% {{drv,[rocks]},{lookup_element,3},ok}

0% {{nif,[hyper]},{lookup_element,3},ok}

0% {{drv,[hyper]},{lookup_element,3},ok}

0% {{nif,[rocks]},{lookup_element,3},ok}

true

.......



Documentation



Where should I start?


This README is the only bit of documentation.


The QC (a.k.a. QuickCheck) tests underneath the "tests/qc" directory

should be helpful for understanding the specification and behavior of

ETS and LETS.  These QC tests also illustrate several strategies for

testing Erlang Driver-based and NIF-based implementations.



What is ETS and DETS?


ETS and DETS are Erlang/OTP's standard library modules for Erlang

term storage.  ETS is a memory-based implementation.  DETS is a

disk-based implementation.


See http://www.erlang.org/doc/man/ets.html and

http://www.erlang.org/doc/man/dets.html for further details.



What is LevelDB?


LevelDB is a fast key-value storage library written at Google that

provides an ordered mapping from string keys to string values.


See https://github.com/google/leveldb for further details.



What is HyperLevelDB?


HyperLevelDB is the data storage engine that powers HyperDex.

HyperLevelDB was forked from Google's LevelDB and adapted to more

closely meet the needs of HyperDex.


See https://github.com/rescrv/HyperLevelDB for further details.



What is RocksDB?


RocksDB is a persistent Key-Value Store for Flash and RAM Storage.

RocksDB is built on earlier work from Google's LevelDB.


See https://github.com/facebook/rocksdb for further details.



What is Snappy?


Snappy is a fast compression/decompression library written at Google.


See https://github.com/google/snappy for further details.



Tools



For further information and help for related tools, please refer to

the following links:





Roadmap



    

  • 

    

    Documentation

    

      

    • 

      

      Explain how to run QuickCheck tests using a new rebar plugin.

      

      • 

    • 

      

      Explain how to build and to run lets with valgrind enabled

          OTP/Erlang virtual machine

      

      • 

    

    • 

  • 

    

    Backends (TBD)

    

      

    • 

      

      Implement the NIF backends as "dirty" NIFs.

      

      • 

    

    • 

  • 

    

    New APIs (TBD)

    

    

    • 

  • 

    

    Existing APIs (TBD)

    

    

    • 

  • 

    

    Testing - Black Box

    

      

    • 

      

      Functional

      

        

      • 

        

        Update test model to include LevelDB's database, read, and

               write options.  These options have not been tested.

        

        • 

      • 

        

        Update test model to include LevelDB's destroy and repair

               operations.  These operations have not been tested.

        

        • 

      

      • 

    • 

      

      Performance (TBD)

      

      • 

    • 

      

      Stability (TBD)

      

      • 

    

    • 

  • 

    

    Testing - White (or more like "Grey") Box

    

      

    • 

      

      Goals

      

        

      • 

        

        Test normal, abnormal, and corner test cases without having to

               actually use "big data" or invoke lots of operations.  Invoke

               operations using small inputs but with varying sizes, ranges,

               and patterns.

        

        • 

      • 

        

        Learn about what special parameters exist, their default values

               and ranges, and the difficulty to control these parameters on a

               request-by-request basis (at best case).

        

        • 

      

      • 

    • 

      

      Functional (TBD)

      

        

      • 

        

        Enable/disable background compaction

        

        • 

      • 

        

        Invoke/suspend manual compaction

        

        • 

      • 

        

        Force new memtable creation

        

        • 

      • 

        

        Force new level creation

        

        • 

      • 

        

        Database Recovery (i.e. closing/reopening the db)

        

        • 

      • 

        

        Large keys (e.g. 1KB)

        

        • 

      • 

        

        Adjacent keys that share a long prefix (e.g ~1KB); useful

               since file format has prefix compression

        

        • 

      • 

        

        Snapshots that are live across compactions and are read from

               after compaction

        

        • 

      • 

        

        Iterators that are live across compactions and are read from

               after compaction

        

        • 

      • 

        

        File system writes return errors (e.g., disk-full)

        

        • 

      

      • 

    

    • 




## Modules ##



hets_impl_drv

hets_impl_nif

lets

lets_impl_drv

lets_impl_nif

rets_impl_drv

rets_impl_nif