https://github.com/sfu-dis/ermia
ERMIA: Memory-Optimized OLTP engine for Heterogeneous Workloads (SIGMOD 2016)
https://github.com/sfu-dis/ermia
ermia oltp serial-safety-net serializable serializable-snapshot-isolation ssi ssn transactions
Last synced: about 1 month ago
JSON representation
ERMIA: Memory-Optimized OLTP engine for Heterogeneous Workloads (SIGMOD 2016)
- Host: GitHub
- URL: https://github.com/sfu-dis/ermia
- Owner: sfu-dis
- License: mit
- Created: 2015-12-07T11:31:48.000Z (over 9 years ago)
- Default Branch: master
- Last Pushed: 2022-12-11T05:39:19.000Z (over 2 years ago)
- Last Synced: 2025-03-25T10:04:38.087Z (about 2 months ago)
- Topics: ermia, oltp, serial-safety-net, serializable, serializable-snapshot-isolation, ssi, ssn, transactions
- Language: C++
- Homepage:
- Size: 6.69 MB
- Stars: 88
- Watchers: 9
- Forks: 26
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
## ERMIA
Fast and Robust OLTP using Epoch-based Resource Management and Indirection Array
See our SIGMOD'16 paper [1] for a description of the system, our VLDBJ paper [2] for details in concurrency control, and our VLDB paper for replication.
\[1\] Kangnyeon Kim, Tianzheng Wang, Ryan Johnson and Ippokratis Pandis. [ERMIA: Fast Memory-Optimized Database System for Heterogeneous Workloads](https://www2.cs.sfu.ca/~tzwang/ermia.pdf). SIGMOD 2016.
\[2\] Tianzheng Wang, Ryan Johnson, Alan Fekete and Ippokratis Pandis. [Efficiently making (almost) any concurrency control mechanism serializable](https://link.springer.com/article/10.1007/s00778-017-0463-8). The VLDB Journal, Volume 26, Issue 4. 2017. [preprint](https://arxiv.org/pdf/1605.04292.pdf).
\[3\] Tianzheng Wang, Ryan Johnson and Ippokratis Pandis. [Query Fresh: Log Shipping on Steroids](http://www.vldb.org/pvldb/vol11/p406-wang.pdf). VLDB 2018.
#### Environment configurations
* Software dependencies: `libnuma`. Install from your favorite package manager. ERMIA uses `mmap` with `MAP_HUGETLB` to allocate huge pages. `MAP_HUGETLB` is available after Linux 2.6.32.
* Make sure you have enough huge pages. Almost all memory allocations come from the space carved out here. Assuming 2MB pages, the command below will allocate 40GB of memory:
```
sudo sh -c 'echo [x pages] > /proc/sys/vm/nr_hugepages'
```
This limits the maximum for --node-memory-gb to 10 for a 4-socket machine (see below).
* `mlock` limits. Add the following to `/etc/security/limits.conf` (replace "[user]" with your login):
```
[user] soft memlock unlimited
[user] hard memlock unlimited
```
*Re-login to apply.*
#### Adjust maximum concurrent workers
By default we support up to 256 cores. The limit can be adjusted by setting `MAX_THREADS` defined under `config` in `dbcore/sm-config.h.` `MAX_THREADS` must be a multiple of 64.
#### Build it
--------
Currently the code only compiles with clang. We do not allow building in the source directory. Suppose we build in a separate directory:
```
$ mkdir build
$ cd build
$ CC=clang CXX=clang++ cmake ../ -DCMAKE_BUILD_TYPE=[Debug/Release/RelWithDebInfo]
$ make -jN
```
After `make` there will be three executables under `build`:
`ermia_SI` that runs snapshot isolation (not serializable);
`ermia_SI_SSN` that runs snapshot isolation + Serial Safety Net (serializable)
`ermia_SSI` that runs serializable snapshot isolation *
* Serializable Isolation for Snapshot Databases, M. Cahill, U. Rohm, A. Fekete, SIGMOD 2008.
#### Run it
```
$run.sh \
[executable] \
[benchmark] \
[scale-factor] \
[num-threads] \
[duration (seconds)] \
"[other system-wide runtime options]" \
"[other benchmark-specific runtime options]"`
```
#### System-wide runtime options
`-node_memory_gb`: how many GBs of memory to allocate per socket.
`-null_log_device`: flush log buffer to `/dev/null`. With more than 30 threads, log flush (even to tmpfs) can easily become a bottleneck because of a mutex in the kernel held during the flush. This option does *not* disable logging, but it voids the ability to recover.
`-tmpfs_dir`: location of the log buffer's mmap file. Default: `/tmpfs/`.
`-enable_gc`: turn on garbage collection. Currently there is only one GC thread.
`-enable_chkpt`: enable checkpointing.
`-phantom_prot`: enable phantom protection.
`-warm-up`: strategy to load versions upon recovery. Candidates are:
- `eager`: load all latest versions during recovery, so the database is fully in-memory when it starts to process new transactions;
- `lazy`: start a thread to load versions in the background after recovery, so the database is partially in-memory when it starts to process new transactions.
- `none`: load versions on-demand upon access.
*SSI and SSN specific:*
`--safesnap`: enable safe snapshot for read-only transactions.
*SSN-specific:*
`--ssn-read-opt-threshold`: versions that are read by a read-mostly transaction and older than this value are considered "old" and will not be tracked; setting it to 0 will skip all read tracking for read-mostly transactions (`TXN_FLAG_READ_MOSTLY`).
*SSI-specific:*
`--ssi-read-only-opt`: enable P&G style read-only optimization for SSI.