https://github.com/rohanvdvivedi/bufferpool

A user-space buffer pool manager library in C for building a HDD/SSD based database storage engine.
https://github.com/rohanvdvivedi/bufferpool

bufferpool database database-storage-engine disk-usage disklrucache heap-file

Last synced: 12 months ago
JSON representation

A user-space buffer pool manager library in C for building a HDD/SSD based database storage engine.

• Host: GitHub
• URL: https://github.com/rohanvdvivedi/bufferpool
• Owner: RohanVDvivedi
• Created: 2020-01-24T17:02:14.000Z (about 6 years ago)
• Default Branch: master
• Last Pushed: 2025-03-30T11:33:21.000Z (12 months ago)
• Last Synced: 2025-03-30T12:24:24.085Z (12 months ago)
• Topics: bufferpool, database, database-storage-engine, disk-usage, disklrucache, heap-file
• Language: C
• Homepage:
• Size: 670 KB
• Stars: 4
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

          
# Bufferpool

It is an implementation of a Buffer Pool Manager library in C, used for accessing (with Read/Write lcoks) and caching pages of secondary memory (SSD/HDD).

 * It primarily solves 2 problems

   * Caching disk pages in a maximum number of page frames, and evicting them as necessary using an LRU policy (Actual LRU not clock based LRU) for replacement.

   * Databases need page level latches (mutexes in database terminology) on pages, while modifying individual disk pages, **this library provides you an api to latch pages that have reader-writer lock like api and functionality**.

 * Additionally,

   * It provides plenty of flags with an extensive api to acquire read/write locks on pages, upgrading read lock to write lock on a page, downgrade write lock to read lock on a page and release read/write lock on page, gives wide flexibility to build your storage engine on top of this bufferpool.

   * This library allows you to lock (traditionally called latch in Database terminology) pages with reader and writer lock while accessing them, allowing higher concurrency to your database storage engine.

   * It also allows you to prefetch pages, both synchronously and asynchronously.

   * It supports periodic jobs that can be run in background to flush dirty pages to disk, keeping your bufferpool pages clean.

   * It does not perfrom I/O for you, but instead takes structure of call back functions that get called to perform I/O, allowing you to intercept read and write I/O calls.

   * This also allows you to switch the underlying storage from (conventionally) SSD/HDD to some network storage accessed over TCP/UDP, very easily.

*Information on ARIES support :*

 * This bufferpool implementation provides 2 callback functions can_be_flushed_to_disk and was_flushed_to_disk, both of which are passed page_id and read only frame pointer, on their designated calls.

 * The *can_be_flushed_to_disk* callback can be used to check, if the pageLSN of the dirty page, is greater than or equal to the flushedLSN of the write-ahead-log (WAL).

 * The *was_flushed_to_disk* callback can be used to remove a dirty page from the dirty page table, once it has been flushed to the disk, it is called atomically with the clearing of the dirty bit on the page frame inside the Bufferpool.

 * Both of the above functions are called with global lock and atleast a reader lock on the page frame held. *AND YOU MUST NOT RELEASE ANY OF THEM, FOR CORRECT OPERATION*

## Setup instructions

**Install dependencies :**

 * [Cutlery](https://github.com/RohanVDvivedi/Cutlery)

 * [PosixUtils](https://github.com/RohanVDvivedi/PosixUtils)

 * [BoomPar](https://github.com/RohanVDvivedi/BoomPar)

 * [ReaderWriterLock](https://github.com/RohanVDvivedi/ReaderWriterLock)

**Download source code :**

 * `git clone https://github.com/RohanVDvivedi/Bufferpool.git`

**Build from source :**

 * `cd Bufferpool`

 * `make clean all`

**Install from the build :**

 * `sudo make install`

 * ***Once you have installed from source, you may discard the build by*** `make clean`

## Using The library

 * add `-lbufferpool -lrwlock -lboompar -lpthread -lcutlery` linker flag, while compiling your application

 * do not forget to include appropriate public api headers as and when needed. this includes

   * `#include` -> read this to get the gist of the API

   * `#include` -> defines the structure of read/write I/O callback function struct

## Instructions for uninstalling library

**Uninstall :**

 * `cd Bufferpool`

 * `sudo make uninstall`