Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/goatdb/llama

LLAMA Graph Analytics Engine
https://github.com/goatdb/llama

Last synced: 30 days ago
JSON representation

LLAMA Graph Analytics Engine

Awesome Lists containing this project

README

        

LLAMA
=========

0. Introduction
-----------------

LLAMA is a graph storage and analysis system that supports mutability and
out-of-memory execution built on top of the compressed sparse row (CSR)
representation. Its goal is to perform comparably to immutable main-memory
analysis systems for graphs that fit in memory and to match or outperform
existing out-of-memory analysis systems for graphs that exceed main memory.

Relevant publications:
* Peter Macko, Virendra Marathe, Daniel Margo, and Margo Seltzer. "LLAMA:
Efficient Graph Analytics Using Large Multiversioned Arrays." 31st IEEE
International Conference on Data Engineering (ICDE '15), Seoul, Korea,
April 2015.
* Peter Macko. "LLAMA: A Persistent, Mutable Representation for Graphs."
PhD Dissertation. Harvard University, Cambridge, MA, January 2015.

1. Configuring and Building LLAMA
-----------------------------------

Prerequisites:
- Linux or NetBSD
- GCC 4.4 or newer (4.8 or newer recommended)
- gperftools (recommended)

LLAMA comes in four different flavors, which you can choose at the compile-time
by defining an appropriate preprocessor directive:
* LL_MEMORY_ONLY - an in-memory version of LLAMA
* LL_PERSISTENCE - a persistent version of LLAMA
* LL_SLCSR - traditional (single-snapshot) CSR

You can also define the following directives:
* LL_DELETIONS - enable deletion vector (required to support deletions in the
write-optimized graph store and also required for multiversioned sliding
windows)
* LL_FLAT_VT - a multiversioned flat vertex table (disable COW)
* LL_ONE_VT - a single shared flat vertex table

The benchmark suite bypasses the write-optimized store by default, but you can
change that by defining:
* BENCHMARK_WRITABLE - use the write-optimized store in the benchmark

The Makefile has several defined targets for these configurations; the most
useful targets are:
* benchmark-memory - uses an in-memory version of LLAMA
* benchmark-persistent - uses a persistent version of LLAMA
* benchmark-slcsr - uses the single-snapshot (single-level) version of LLAMA
* benchmark-memory-wd - uses the in-memory version with a deletion vector
* benchmark-persistent-wd - uses the persistent version with a deletion
vector (currently broken)
* benchmark-w-memory - in-memory version of LLAMA, use write-optimized store
Most benchmark targets have corresponding %_debug targets useful for debugging
LLAMA, such as benchmark-memory_debug or benchmark-persistent_debug. Typing
"make all" builds benchmark-memory, benchmark-persistent, benchmark-slcsr,
benchmark-memory-wd, and benchmark-persistent-wd.

You can further customize the build by passing additional definitions to the
Makefile. If you use the global Makefile, you can use the targets above, but if
you use benchmark/Makefile directly, you might need to modify the targets. Here
are examples of a few things that you can do:
* make TASK=pagerank benchmark-memory - compile a PageRank-specific benchmark
so that you do not need to use the -r argument to specify the task when
running it; using this compile-time specialization usually results in a
more optimized (and therefore faster) code
* make ONE_VT=1 benchmark-memory - enable LL_ONE_VT, a single shared flat
vertex table
* make FLAT_VT=1 benchmark-memory - enable LL_FLAT_VT, a multiversioned flat
vertex table
* make NO_CONT=1 benchmark-memory - enable LL_NO_CONTINUATIONS, which
disables explicit adjacency list linking
You can use any other combination of these except combining ONE_VT and FLAT_VT.

2. Using LLAMA
----------------

If you would like to use LLAMA in your project,
1. Include llama.h in your code
2. Compile with -fopenmp -std=gnu++11

Please refer to the following files for examples about how to use LLAMA:
* tools/llama-load.cc - a stand-alone example that loads a LLAMA database
* examples/llama-pagerank.cc - a stand-alone example that opens an existing
LLAMA database and computes a few iterations of PageRank

The code includes Doxygen comments, which documents the functionality,
arguments, and return values of most data functions and describes most data
structures and their fields.

A lot of the streaming functionality is currently hard-coded in benchmark.cc,
and it is in a desperate need for refactoring (see the corresponding issue in
TODO.txt) so that it would be possible to easily build stand-alone streaming
applications. If you would like to do that before that happens, implement your
program as another benchmark in the benchmark suite -- or go ahead and do the
refactoring yourself :) We will be grateful for the contribution!

3. Modifying LLAMA
--------------------

If you would like to contribute to LLAMA, we would certainly appreciate your
help! There is a long list of TODO items and known bugs in docs/TODO.txt, so
feel free to tackle any of those, or if there is anything else that you would
like to do, please feel free!

All source code of LLAMA is in the header files to facilitate deep inlining,
which is crucial for performance, and to enable the use of templates.

Please refer to docs/FILES.txt for short descriptions of files. In short:
* benchmark/ - contains the benchmark suite
* examples/ - contains examples (currently just one PageRank program)
* llama/ - contains the actual source code of LLAMA
* tools/ - contains the database loading tool

The files in llama/include/llama roughly break into the following categories:
* Shared high-level functionality, definitions, and configuration:
ll_common.h - Shared definitions
ll_config.h - Configuration classes
ll_database.h - A high-level database class
* The read-optimized, multi-snapshot (multi-level) graph store:
ll_edge_table.h - The edge table
ll_mem_array.h - In-memory arrays, both COW and flat, used for
the vertex table and properties
ll_mlcsr_graph.h - The (multi-level CSR) read-only graph that
consists of CSR graph stores, such as in- and
out- edges, and node and edge properties
ll_mlcsr_helpers.h - Various multi-level CSR helpers
ll_mlcsr_iterator.h - Definitions of edge and node iterators
ll_mlcsr_properties.h - Multi-level properties
ll_mlcsr_sp.h - The actual multi-level CSR implementation
ll_page_manager.h - The reference-counted page manager used by
in-memory vertex tables and properties
ll_persistent_storage.h - Almost everything related to persistence,
including persistent COW arrays used for
persistent vertex tables and properties
* The write-optimized graph store:
ll_growable_array.h - A growable array
ll_writable_array.h - Arrays used by the write-optimized store
ll_writable_elements.h - Writable node and edge objects
ll_writable_graph.h - The actual write-optimized store
* Miscellaneous functionality:
ll_slcsr.h - Traditional CSR implementation
* Miscellaneous utilities:
ll_lock.h - Locks and atomic functions
ll_bfs_template.h - A BFS template (adapted from GM)
ll_dfs_template.h - A DFS template (adapted from GM)
ll_external_sort.h - An external sort routine
ll_mem_helper.h - Various types of memory pools
ll_seq.h - The sequence container (from GM)
ll_utils.h - Miscellaneous utilities
* Graph loaders and generators, all inside loaders/ :
ll_gen_erdosrenyi.h - Random graph generator
ll_gen_rmat.h - R-MAT graph generator
ll_load_async_writable.h - Helpers for loading the write-store
ll_loaders.h - A collection of all loaders
ll_load_fgf.h - FGF loader
ll_load_net.h - SNAP edge-list loader
ll_load_utils.h - Common loader functionality
ll_load_xstream1.h - X-Stream Type 1 edge-list loader