https://github.com/maxi-k/btrblocks

BtrBlocks: Efficient Columnar Compression for Data Lakes (SIGMOD 2023 Paper)
https://github.com/maxi-k/btrblocks

compression data-lake databases research

Last synced: 2 months ago
JSON representation

BtrBlocks: Efficient Columnar Compression for Data Lakes (SIGMOD 2023 Paper)

• Host: GitHub
• URL: https://github.com/maxi-k/btrblocks
• Owner: maxi-k
• License: mit
• Created: 2023-05-22T15:15:22.000Z (about 2 years ago)
• Default Branch: master
• Last Pushed: 2025-04-07T10:12:31.000Z (2 months ago)
• Last Synced: 2025-04-09T15:08:02.976Z (2 months ago)
• Topics: compression, data-lake, databases, research
• Language: C++
• Homepage:
• Size: 247 KB
• Stars: 238
• Watchers: 7
• Forks: 20
• Open Issues: 8
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# BtrBlocks - Efficient Columnar Compression for Data Lakes

[![Paper](https://img.shields.io/badge/paper-SIGMOD%202023-green)](https://bit.ly/btrblocks)

[![Build](https://github.com/maxi-k/btrblocks/actions/workflows/test.yml/badge.svg?event=push)](https://github.com/maxi-k/btrblocks/actions/workflows/test.yml)

- [Paper](https://bit.ly/btrblocks) (two-column version)

- [Video](https://dl.acm.org/doi/10.1145/3589263) (SIGMOD 2023 presentation)

## Usage

After [building](#building) the library, follow one of the [examples](./tools/examples) to get started.

## Components

- `btrblocks/`: the compression library, schemes, utilities, ...

- `btrfiles/`: helper library for binary files and yaml schema information

- `tools/`: various conversion, measurement and benchmarking tools

- `test/`: rudimentary tests for the library

![Dependency Graph](./doc/dependencies.svg)

## Building

We currently require an x86 platform.

This library was built and tested on linux only.

``` sh

mkdir build 

cd build

cmake ..

```

Then, depending on your usecase, build only the library or any of the tools:

- build everything: `make`

- install static library and headers on your system: `sudo make install`

- build the compression library only: `make btrblocks`

- build the tests `make tester`

- build the in-memory decompression speed benchmark: `make decompression_speed`

- ...

For a list of all valid targets, run `make help`.

Library was built and tested on Linux (x86, ARM) and MacOS (ARM).

## Contributors

Adnan Alhomssi

David Sauerwein

Maximilian Kuschewski

## License

MIT - See [License File](LICENSE)