Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/k-tomaszewski/eternal-db

Eternal DB is a time series database with a data retention policy based on disk space, thus allowing to collect data eternally.
https://github.com/k-tomaszewski/eternal-db

database document-database embedded-database java java-21 json nosql time-series tsdb

Last synced: about 1 month ago
JSON representation

Eternal DB is a time series database with a data retention policy based on disk space, thus allowing to collect data eternally.

Awesome Lists containing this project

README 

        
# eternal-db

Eternal DB is an embedded time series database/data storage engine with a data retention policy based on a disk space. 

As oldest records are removed when needed to reclaim disk space, this database allows to collect data eternally. It gives unmatched control

on a disk resource usage. Moreover, data are easily accessible for any tools as they are kept in regular disk files on a given file system.

It's a schema-less document database as records are just JSON documents.



Some possible use cases:

- collecting measurement data

- collecting metrics

- collecting market data



In above cases the most important thing for a running system is to keep some time window of latest data. Eternal DB allows to express this 

time window in terms of a disk space.



**Please note, that this is not a classic database system like RDBMS.**

It doesn't support SQL, users, transactions, constraints, relations, triggers, stored procedures, etc.

Maybe the best term is a "data storage engine".



As this is designed to be used under Linux, it requires Coreutils: https://www.gnu.org/software/coreutils/



NOTE: This project is a work-in-progress (WIP).



## Project goals

1. It is an embedded database (data store) that is easy to add to a program written in Java.

2. It has a disk space based data retention policy. This is meant to keep as much data as possible within a defined disk space limit.

3. When reclaiming disk space, oldest data records are removed.

4. Records are kept in a text format in files on a given filesystem and these files are easily readable by other programs.

5. Records are considered documents, like in document databases. No fixed data structure is forced.

6. A record always has a timestamp and time is considered an indexed attribute.

7. It is meant to run in a Linux environment.

8. It is thread-safe.

9. It is fault-tolerant. A single error should not block you from reading all your data.



## Non-goals

1. This is not meant to run as a standalone server enabling multiple clients to connect and use.

2. Ability to configure other data retention policy is not required.

3. Support for any other operating system is not required if such an operating system cannot provide Linux-compatible runtime environment.

4. Strict schema validation for data is not a goal.

5. Strict transactions support is not a goal.

6. Data access control is not a goal. A user is responsible to configure desired access rights to a directory defined for data files.

7. Valid use of a single database storage from many database instances is not a goal.

8. Support for efficient searching of records based on any other criteria than time period is not a goal.



## Design decisions

1. Project is developed in Java 21.

2. Records are kept formatted as single line JSON, thus a single data file has a format similar to JSON Lines: https://jsonlines.org/

3. Each line in a data file is beginning with timestamp being number of millis from the Epoch (as given by `System.currentTimeMillis()`)

   and formatted with radix of 32 (to use less disk space), followed by the tab character, followed by JSON.

4. Data are persisted as a disk files in directory tree that is representing periods of time.

5. Project is avoiding unnecessary dependencies. No Vavr, Project Reactor, Guava, etc.

6. The database should be easy to use in Spring Boot based application.



## Usage

### Dependency

Add dependency to eternal-db library (use the latest version):

```xml



  io.github.k_tomaszewski

  eternal-db

  1.0.0



```

As of now the library is published only in GiHub Packages repository, so you need to add additional Maven repository in your `pom.xml` like this:

```xml



    

        github_k-tomaszewski_eternal-db

        https://maven.pkg.github.com/k-tomaszewski/eternal-db

    



```

It seems that GitHub Packages repository requires authentication even for public

artifacts, so you need to have a GitHub account and you need to set up your credentials

(GitHub username and access token) for Maven in the `settings.xml` file. Ref:

- https://maven.apache.org/guides/mini/guide-multiple-repositories.html

- https://maven.apache.org/guides/mini/guide-deployment-security-settings.html

- https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-apache-maven-registry#authenticating-with-a-personal-access-token



### Opening a database

To use it you create a single object of class `io.github.k_tomaszewski.eternaldb.Database`.

Such object gives access to operations on a single data store.



Please note that the `Database` is a generic class and has a type parameter. This is used for type

checking only, so a compiler can detect when a wrong object is passed for writing method.



To create instance of the `Database` class you need to provide a `DatabaseProperties` object, which contains following 

configuration properties:

- path to a directory where data files are going to be stored

- limit of disk space to use, given as integer number of megabytes (MB)

- object being strategy for serialization and deserialization, the library provides `JacksonSerialization` instance as a default

- object being strategy for naming data files, the library provides `BasicFileNaming` instance as a default

- timestamp supplier - object implementing `ToLongFunction` that returns timestamp, possibly based on a record being written. This is optional.



Example:

```java

Database db = new Database<>(new DatabaseProperties<>(Path.of("/home/db"), 100));

```

You need one such object for a single data storage directory. Instances of `Database` class are

thread-safe. You must not use many instances of `Database` class that are configured to use

the same directory or subdirectories of each other.



### Writing data

A basic method for writing data: `void write(T record, long recordMillis)`. To use it you

need to give record timestamp when writing as this is a time series database. Example:

```java

db.write(myRecord, System.currentTimeMillis());

```



If a database is configured with a timestamp supplier (an object implementing `ToLongFunction` interface),

then another data writting method can be used: `void write(T record)`. Calling `write(x)` is equivalent

of calling `write(x, timestampSupplier.applyAsLong(x))`. This can be useful when your domain model

already contains a timestamp attribute.



### Reading data

There is just one method for reading data: `Stream> read(Class type, Long minMillis, Long maxMillis)`.

This is designed to read a set of records with timestamps matching a given range [minMillis, maxMillis].

Both ends of a time range are optional. Example of reading all records with timestamps starting

with given time:

```java

List entities = db.read(MyRecord.class, fromMillis, null)

        .map(Timestamped::record).toList();

```

Here the given type (`MyRecord.class` in the example above) is used for deserialization purpose.



### Closing a database

After all interactions with the database are done, for example at the end of your program, 

you should close the database object:

```java

db.close();

```

You don't need to close a database before the end of your program. It's a lightweight object

and has rather small memory usage.



### Customization of JSON serialization/deserialization

The library provides a default class for serlialization/deserialization strategy: `io.github.k_tomaszewski.eternaldb.JacksonSerialization`.

It uses its own instance of Jackson ObjectMapper (precisely: JsonMapper). One can add customization by

creating this object and passing `Consumer` to the constructor. Example:

```java

Consumer customizer = objectMapper -> objectMapper.registerModule(new JavaTimeModule())

        .disable(SerializationFeature.WRITE_DATES_AS_TIMESTAMPS)

        .enable(SerializationFeature.WRITE_DATES_WITH_ZONE_ID)

        .disable(DeserializationFeature.ADJUST_DATES_TO_CONTEXT_TIME_ZONE);



Database db = new Database<>(new DatabaseProperties<>(Path.of("/home/db"), 100)

        .setSerialization(new JacksonSerialization(customizer)));

```



## Contributions

Contributions are welcome. If you want to contribute, just make a pull request. Please contact me before to discuss your idea:

krzysztof.tomaszewski (at) gmail.com