{"id":18563111,"url":"https://github.com/bitgn/layers","last_synced_at":"2025-04-10T03:32:34.336Z","repository":{"id":64318831,"uuid":"132337334","full_name":"bitgn/layers","owner":"bitgn","description":"FoundationDB Layers","archived":false,"fork":false,"pushed_at":"2018-05-27T07:15:42.000Z","size":107,"stargazers_count":6,"open_issues_count":0,"forks_count":0,"subscribers_count":5,"default_branch":"master","last_synced_at":"2025-03-24T16:02:50.695Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bitgn.png","metadata":{"files":{"readme":"README.org","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-05-06T12:42:27.000Z","updated_at":"2021-03-29T00:21:32.000Z","dependencies_parsed_at":"2023-01-15T12:00:47.262Z","dependency_job_id":null,"html_url":"https://github.com/bitgn/layers","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bitgn%2Flayers","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bitgn%2Flayers/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bitgn%2Flayers/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bitgn%2Flayers/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bitgn","download_url":"https://codeload.github.com/bitgn/layers/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248151327,"owners_count":21056071,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-06T22:11:50.962Z","updated_at":"2025-04-10T03:32:31.648Z","avatar_url":"https://github.com/bitgn.png","language":"Go","readme":"* Benchmarking\n\n\nCurrently the benchmark is done by compiling [[go/benchmark/][golang binary]] and copying\nit to the target load tester machine. I'm using the one with the\n[[var%20(%0A%09jsonKey%20=%20append(%5B%5Dbyte{255,%20255},%20%5B%5Dbyte(\"/status/json\")...)%0A)%0A][pre-baked FDB cluster on AWS]].\n\nThis works fine, however there are limitations:\n\n- Benchmark tool is hard to port to the other languages (coupled with\n  the golang).\n- FDB apps are limited by the throughput of the single networking\n  thread of `fdb_c` native library, so you need to run multiple\n  clients against larger clients; current bench can run in parallel,\n  but doesn't aggregate the results.\n\n\nI think, one of the approaches would be to break the current bench\ninto the following parts:\n\n- *load generator* - a simple native app that does the work and pushes\n  telemetry;\n- *FDB statistics collector* - a simple app that pushes telemetry\n  about the FDB (to be replaced with some native solution);\n- Prometheus/InfluxDB - to gather all statistics (replacing TSV\n  files);\n- Grafana - chart cluster performance in real-time.\n\nRe Prometheus vs InfluxDB for this case. Prometheus pros:\n\n- better long-term solution;\n- understands histograms and can aggregate them;\n\nInfluxDB pros:\n\n- I've used it for a long time and know how it works;\n- Easy to send custom events (like injected cluster faults).\n\n\n* Event Store Layer\n\nEvent store layer is an implementation of append-only log which also\nmaintains indexes all messages by their stream name. It is frequently\nused to implement _Aggregates with Event Sourcing_ pattern.\n\nKey requirements:\n\n- large messages are split into chunks;\n- global stream and event streams are changed within a single\n  transaction;\n- Events within a stream are stored by an incrementing version number;\n- Events in the global stream are stored by the version stamp.\n\nGlobal event range \n\nThis design document outlines Event Store version 1, without any\noptimizations to reduce storage space.\n\n** Key ranges\n\nWe designate following key ranges:\n\n| Prefix | Description                                                   |\n|      0 | Settings for this Event Store                                 |\n|      1 | All events ordered by the version stamp                       |\n|      2 | Events stored by streams and ordered by version within stream |\n\n** Methods \n\n*** AppendToStream\n\nArguments:\n- stream name (string)\n- stream version (long) - expected stream version, -1 to skip version check\n- events (list of byte[]) - message values\n\n*** ReadFromStream\n\nArguments:\n\n- stream name (string)\n- starting from (long)\n- max count (int)\n\nreturns: list of stream data\n\n*** ReadAll\n\nArguments:\n\n- starting from (long)\n- max count (int)\n\nreturns: list of stream data\n\n*** GetStoreVersion\n\nReturns: store version (long)\n\n\n\n\n\n\n* Language-specific implementations\n\n** .NET\n\n.NET Methods are to be implemented as ~async~ returning ~Task~ and\naccepting ~CancellationToken~ as the last optional parameter.\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbitgn%2Flayers","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbitgn%2Flayers","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbitgn%2Flayers/lists"}