https://github.com/bitgn/layers

FoundationDB Layers
https://github.com/bitgn/layers

Last synced: about 5 hours ago
JSON representation

FoundationDB Layers

• Host: GitHub
• URL: https://github.com/bitgn/layers
• Owner: bitgn
• License: other
• Created: 2018-05-06T12:42:27.000Z (over 6 years ago)
• Default Branch: master
• Last Pushed: 2018-05-27T07:15:42.000Z (over 6 years ago)
• Last Synced: 2024-06-20T05:25:19.004Z (5 months ago)
• Language: Go
• Size: 104 KB
• Stars: 6
• Watchers: 6
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.org
  • License: LICENSE

Awesome Lists containing this project

README

        
* Benchmarking

Currently the benchmark is done by compiling [[go/benchmark/][golang binary]] and copying

it to the target load tester machine. I'm using the one with the

[[var%20(%0A%09jsonKey%20=%20append(%5B%5Dbyte{255,%20255},%20%5B%5Dbyte("/status/json")...)%0A)%0A][pre-baked FDB cluster on AWS]].

This works fine, however there are limitations:

- Benchmark tool is hard to port to the other languages (coupled with

  the golang).

- FDB apps are limited by the throughput of the single networking

  thread of `fdb_c` native library, so you need to run multiple

  clients against larger clients; current bench can run in parallel,

  but doesn't aggregate the results.

I think, one of the approaches would be to break the current bench

into the following parts:

- *load generator* - a simple native app that does the work and pushes

  telemetry;

- *FDB statistics collector* - a simple app that pushes telemetry

  about the FDB (to be replaced with some native solution);

- Prometheus/InfluxDB - to gather all statistics (replacing TSV

  files);

- Grafana - chart cluster performance in real-time.

Re Prometheus vs InfluxDB for this case. Prometheus pros:

- better long-term solution;

- understands histograms and can aggregate them;

InfluxDB pros:

- I've used it for a long time and know how it works;

- Easy to send custom events (like injected cluster faults).

* Event Store Layer

Event store layer is an implementation of append-only log which also

maintains indexes all messages by their stream name. It is frequently

used to implement _Aggregates with Event Sourcing_ pattern.

Key requirements:

- large messages are split into chunks;

- global stream and event streams are changed within a single

  transaction;

- Events within a stream are stored by an incrementing version number;

- Events in the global stream are stored by the version stamp.

Global event range 

This design document outlines Event Store version 1, without any

optimizations to reduce storage space.

** Key ranges

We designate following key ranges:

| Prefix | Description                                                   |

|      0 | Settings for this Event Store                                 |

|      1 | All events ordered by the version stamp                       |

|      2 | Events stored by streams and ordered by version within stream |

** Methods 

*** AppendToStream

Arguments:

- stream name (string)

- stream version (long) - expected stream version, -1 to skip version check

- events (list of byte[]) - message values

*** ReadFromStream

Arguments:

- stream name (string)

- starting from (long)

- max count (int)

returns: list of stream data

*** ReadAll

Arguments:

- starting from (long)

- max count (int)

returns: list of stream data

*** GetStoreVersion

Returns: store version (long)

* Language-specific implementations

** .NET

.NET Methods are to be implemented as ~async~ returning ~Task~ and

accepting ~CancellationToken~ as the last optional parameter.