https://github.com/durner/anyblob
AnyBlob - A Universal Cloud Object Storage Download Manager Built For Cost-Throughput Optimal Analytics!
https://github.com/durner/anyblob
blob-storage cloud cloud-storage downloader object object-storage storage
Last synced: 22 days ago
JSON representation
AnyBlob - A Universal Cloud Object Storage Download Manager Built For Cost-Throughput Optimal Analytics!
- Host: GitHub
- URL: https://github.com/durner/anyblob
- Owner: durner
- License: mpl-2.0
- Created: 2023-05-24T09:20:33.000Z (almost 2 years ago)
- Default Branch: master
- Last Pushed: 2025-03-12T11:21:37.000Z (about 2 months ago)
- Last Synced: 2025-03-12T12:26:22.276Z (about 2 months ago)
- Topics: blob-storage, cloud, cloud-storage, downloader, object, object-storage, storage
- Language: C++
- Homepage: http://anyblob.com
- Size: 325 KB
- Stars: 121
- Watchers: 2
- Forks: 13
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# AnyBlob - Universal Cloud Object Storage Download Manager
In this repository, we present AnyBlob.
AnyBlob is a universal download manager that allows to retrieve and upload objects to different cloud object stores.
Our download manager uses less CPU resources than cloud-vendor provided libraries while retaining maximum throughput performance.
AnyBlob leverages IO\_uring for superior performance per core.
For experimental results, please visit our research paper at [PVLDB 16](https://www.vldb.org/pvldb/vol16/p2769-durner.pdf).
## Building AnyBlob
AnyBlob relies on some third-party libraries:
- uring
- openssl
- jemalloc
For Ubuntu 22.04+ the following command installs the third-party libraries:
```
sudo apt update && sudo apt install liburing-dev openssl libssl-dev libjemalloc-dev lld g++ cmake
```
For building, use the following commands:
```
git clone --recursive https://github.com/durner/AnyBlob
mkdir -p build/Release
cd build/Release
cmake -DCMAKE_BUILD_TYPE=Release ../..
make -j16
```
## Using AnyBlob
AnyBlob is a modern C++ library that can be linked to your program.
We provide two examples how to use AnyBlob in your project.
Please find a simple example that does only download a single object from AWS, and a more sophisticated example that runs download benchmarks.
There, you can also find information on how to easily integrate the building process into existing CMake projects for an automated compilation and static linkage of AnyBlob.
The anyblob.cmake should be used in your project to integrate this repository as external project.
## Contribution
If you have bug fixes or improvement, please do not hesitate to open a new merge request.
For coverage testing you can simply `make coverage` and open the coverage report found in the build directory at coverage/index.html.
## Cite this work
If you are using AnyBlob in your scientific work, please cite:
```
Exploiting Cloud Object Storage for High-Performance Analytics
Dominik Durner, Viktor Leis, and Thomas Neumann
PVLDB 16 (11), 2023, 49th International Conference on Very Large Data Bases
```