Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/jnidzwetzki/bboxdb

BBoxDB is a scalable, highly available, and distributed data store for multi-dimensional big data. The software supports operations like multi-dimensional range queries and spatial joins. In addition, data streams are supported.
https://github.com/jnidzwetzki/bboxdb

bigdata data-streams datastream distributed-storage-manager gis key-value-database key-value-store multi-dimensional multi-dimensional-data multidimensional-data nosql nosql-database partitioning range-query spatial-data spatial-join sstables storage-engine storage-manager

Last synced: 17 days ago
JSON representation

Host: GitHub
URL: https://github.com/jnidzwetzki/bboxdb
Owner: jnidzwetzki
License: apache-2.0
Created: 2015-11-07T10:28:59.000Z (about 9 years ago)
Default Branch: master
Last Pushed: 2024-05-13T04:26:36.000Z (6 months ago)
Last Synced: 2024-10-12T10:09:57.090Z (about 1 month ago)
Topics: bigdata, data-streams, datastream, distributed-storage-manager, gis, key-value-database, key-value-store, multi-dimensional, multi-dimensional-data, multidimensional-data, nosql, nosql-database, partitioning, range-query, spatial-data, spatial-join, sstables, storage-engine, storage-manager
Language: Java
Homepage: http://bboxdb.org
Size: 32.8 MB
Stars: 54
Watchers: 4
Forks: 8
Open Issues: 11
Metadata Files:
- Readme: README.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
- Code of conduct: CODE_OF_CONDUCT.md

Awesome Lists containing this project

README

__Please Note:__ The master branch may be in an unstable state during development. Please use our releases for productive environments.

# What is BBoxDB?
BBoxDB is a highly available distributed storage manager designed to handle multi-dimensional big data. In contrast to existing key-value stores, BBoxDB can handle multi-dimensional efficiently. Existing key-value stores are using one-dimensional keys to address the values. Finding a proper key for multi-dimensional data is challenging and often impossible; this is especially true when the data has an extent (non-point data / regions). To retrieve multi-dimensional data from a key-value store, a full data scan is often required. BBoxDB was developed to avoid the expensive full data scan and to make the work with multi-dimensional data more convenient. User-defined filters are supported to process custom data formats, and BBoxDB also supports the handling of data streams.

## Key features
* ✅ A distributed and fault-tolerant data store for n-dimensional data.
* ✅ Data (point and non-point) of any dimension is supported.
* ✅ The data is indexed, which enables efficient range query processing.
* ✅ BigData is supported by spreading the data across a cluster of systems. Each node stores only a small part of the whole dataset.
* ✅ Multi-dimensional shards are created dynamically on the actual distribution of the data (automatically scale-up/scale-down).
* ✅ Data of multiple tables is stored co-partitioned, and spatial-joins can be executed efficiently without data shuffling between nodes.
* ✅ Data are re-distributed in the background without any service interruption.
* ✅ Multi-dimensional data streams can be processed and continuous queries (range queries and spatial joins) are supported.
* ✅ User-defined filters for query processing on custom data types.

## Documentation
The documentation of the project is located at [https://bboxdb.org/documentation/](https://bboxdb.org/documentation/). The documentation also contains the [changelog](https://bboxdb.org/documentation/changelog/) of the project.

## Getting started
For a guided tour through the features of BBoxDB, see the [getting started](https://bboxdb.org/documentation/gettingstarted/) chapter in the documentation. We also recommend reading the [creating client code section](https://bboxdb.org/documentation/client/). The [install guide](https://bboxdb.org/documentation/installation/) explains the needed steps to deploy an own BBoxDB cluster. The guide also describes how you can setup a virtualized cluster with 5 BBoxDB nodes in under two minutes, by using [Docker](https://hub.docker.com/r/jnidzwetzki/bboxdb/) and [Docker Compose](https://docs.docker.com/compose/).

## Screenshots
BBoxDB ships with a GUI that allows observing the global index structure. Below you find two screenshots of the GUI. The screenshots show how the space is partitioned. In addition, some details about the discovered nodes are shown. When two-dimensional bounding boxes with [WGS 84](https://de.wikipedia.org/wiki/World_Geodetic_System_1984) coordinates are used, a map overlay visualization is supported by the GUI. On the top right picture, some spatial data about Germany was imported and the Figure shows, how Germany in partitioned after the data was imported. In addition, the GUI provides operations to explore two dimensional GeoJSON encoded data.

(The screenshots contain content from OpenStreetMap - CC-BY-SA 2.0)

BBoxDB is also able to handle data streams. The first screenshot shows the busses in Sydney fetched from a real-time GTFS feed. The data is provided by the [Transport for New South Wales Website](https://opendata.transport.nsw.gov.au/). The second screenshot shows the aircraft traffic in the area of Berlin. The data is fetched from the Automatic Dependent Surveillance–Broadcast (ADS–B) data feed from the [ADSBHub Website](https://www.adsbhub.org/). For more details about that, see our [tutorial](https://bboxdb.org/documentation/data-stream-tutorial/) on the handling of real-world data streams.

(The screenshots contain content from OpenStreetMap - CC-BY-SA 2.0)

## Contact / Stay informed
* Visit our [website](https://bboxdb.org)
* Read our research papers:
* [Technical report](https://ub-deposit.fernuni-hagen.de/receive/mir_mods_00001277) about the basic ideas of the software.
* [Demo paper](https://dl.acm.org/citation.cfm?id=3269208) about partitioning (presented at [CIKM 2018](https://www.cikm2018.units.it/)).
* [Research paper](https://link.springer.com/article/10.1007/s10619-019-07275-w) about the architecture of BBoxDB at Springer Distributed and Parallel Databases.
* [Demo paper](https://ieeexplore.ieee.org/document/9005999) about user defined filters (presented at [IEEE Big Spatial Data 2019](http://cse.ucdenver.edu/~BSD2019/); received the best paper award).
* [Demo paper](https://edbt2021proceedings.github.io/docs/p170.pdf) about BBoxDB Streams (presented at [EDBT 2021](https://edbticdt2021.cs.ucy.ac.cy)).
* [Research Paper](https://doi.org/10.1007/s10619-022-07408-8) about the architecture of BBoxDB Streams at Springer Distributed and Parallel Databases.
* [PhD Thesis](https://ub-deposit.fernuni-hagen.de/receive/mir_mods_00001836) about the design of BBoxDB.
* Join our chat at [gitter](https://gitter.im/bboxdb/Lobby)
* Follow us on Twitter: [@BBoxDB](https://twitter.com/BBoxDB)
* Subscribe our mailing list at [Google Groups](https://groups.google.com/forum/#!forum/bboxdb)
* Visit our [bug tracking system](https://github.com/jnidzwetzki/bboxdb/issues)
* Read the [source code](https://github.com/jnidzwetzki/bboxdb) and the [documentation](https://bboxdb.org/documentation/)
* For contributing, see our [contributing guide](https://github.com/jnidzwetzki/bboxdb/blob/master/CONTRIBUTING.md)
* If you like the project, please star it on GitHub!

## License
BBoxDB is licensed under the Apache 2.0 license. See the [LICENSE](./LICENSE) file for details.