{"id":23564193,"url":"https://github.com/selectel/yawndb","last_synced_at":"2026-03-16T01:47:24.612Z","repository":{"id":14421215,"uuid":"17132258","full_name":"selectel/yawndb","owner":"selectel","description":"In-memory circular array database","archived":false,"fork":false,"pushed_at":"2017-02-03T11:57:51.000Z","size":51,"stargazers_count":129,"open_issues_count":0,"forks_count":12,"subscribers_count":26,"default_branch":"master","last_synced_at":"2025-06-12T16:09:31.983Z","etag":null,"topics":["community","database","erlang","time-series"],"latest_commit_sha":null,"homepage":"","language":"Erlang","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/selectel.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2014-02-24T10:15:45.000Z","updated_at":"2025-04-09T20:53:43.000Z","dependencies_parsed_at":"2022-07-22T12:32:55.232Z","dependency_job_id":null,"html_url":"https://github.com/selectel/yawndb","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/selectel/yawndb","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/selectel%2Fyawndb","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/selectel%2Fyawndb/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/selectel%2Fyawndb/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/selectel%2Fyawndb/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/selectel","download_url":"https://codeload.github.com/selectel/yawndb/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/selectel%2Fyawndb/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30558414,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-15T23:30:23.986Z","status":"ssl_error","status_checked_at":"2026-03-15T23:28:43.564Z","response_time":61,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["community","database","erlang","time-series"],"created_at":"2024-12-26T17:13:19.446Z","updated_at":"2026-03-16T01:47:24.572Z","avatar_url":"https://github.com/selectel.png","language":"Erlang","funding_links":[],"categories":[],"sub_categories":[],"readme":"YAWNDB\n======\n\n[![Build Status](https://travis-ci.org/selectel/yawndb.svg?branch=master)](https://travis-ci.org/selectel/yawndb)\n\nYAWNDB is an in-memory circular array database written in Erlang.  This language is\nbased upon the lightweight process model allowing to process large amounts of\ndata without consuming extra system resources.  In YAWNDB the data are stored\nin a circular buffer named a conveyor in YAWNDB terminology.  All the incoming\ndata are dispersed and put into the archives named buckets according to certain\nrules.  A rule is a set of properties for a set of statistical data (i.e. data\nsize, data collection frequency, etc.)\n\nThe data in YAWNDB are represented as triplets composed of time, value and key.\nThe key (in YAWNDB terminology it is also called path) is a sequence of\nlowercase letters and numeric symbols defining on which conveyor the triplet\nwill be put.\n\nOur approach allows to gain the following benefits:\n\n* the obsolete data are deleted without excessive consumption of resources;\n* fixed memory usage for a fixed number of keys;\n* fixed access time for any records.\n\nArchitecture\n------------\n\nYAWNDB is based on a circular array algorithm implemented in Ecirca library\nwritten in C. YAWNDB modules written in Erlang interact with Ecirca via native\nimplemented functions (NIF). Data retention is provided by Bitcask application\nwritten in Erlang. The REST API is based on Cowboy web server. The data is\nwritten via the socket and read via the REST API interface.\n\nInstallation\n------------\n\nClone the repository:\n\n```\n$ git clone https://github.com/selectel/yawndb.git\n```\n\nand then execute the following commands:\n\n```\n$ cd yawndb\n$ make all\n```\n\nthen\n\n```\n$ ./start.sh\n```\n\nCopy the configuration file:\n\n```\ncp priv/yawndb.yml.example priv/yawndb.yml\n```\n\nor create an appropriate symlink.\n\nConfiguration\n-----------\n\nAll YAWNDB settings are kept in the configuration file `yawndb.yml`.\n\n#### Data Saving Settings\n\n| Parameter            | Type    | Description                                                                        |\n|----------------------|---------|------------------------------------------------------------------------------------|\n| flush_enabled        | boolean | Enabling/disabling saving to disk                                                  |\n| flush_dir            | string  | Directory to save data                                                             |\n| flush_period         | integer | Data saving frequency (in seconds)                                                 |\n| flush_late_threshold | float   | Write to log if saving takes more than flush_period * flush_late_threshold seconds |\n| flush_cache          | integer | Read cache                                                                         |\n| flush_write_buffer   | integer | Write cache                                                                        |\n| flush_block_size     | integer | Block size for saving to disk                                                      |\n\n\n#### Obsolete data deletion settings\n\n| Parameter            | Type    | Description                           |\n|----------------------|---------|---------------------------------------|\n| cleanup_enabled      | boolean | Enable/disable obsolete data deletion |\n| cleanup_period       | integer | Deletion frequency, in seconds        |\n\n#### User API settings\n\n| Parameter            | Type    | Description                                             |\n|----------------------|---------|---------------------------------------------------------|\n| listen_jsonapi_req   | boolean | Enable/disable the user API                             |\n| jsonapi_iface        | string  | Specify which network interface that will be used       |\n| jsonapi_port         | integer | Specify which local port number to use                  |\n| jsonapi_prespawned   | integer | Number of workers previously created (can be increased) |\n\n#### Administrative API settings\n\n| Parameter            | Type    | Description                                             |\n|----------------------|---------|---------------------------------------------------------|\n| listen_tcpapi_req    | boolean | Enabling/disabling the administrator API                |\n| tcpapi_iface         | string  | Specify which network interface that will be used       |\n| tcpapi_port          | integer | Specify which local port number to use                  |\n| tcpapi_prespawned    | integer | Number of workers previously created (can be increased) |\n\n#### Data storing settings\n\n| Parameter            | Type    | Description                         |\n|----------------------|---------|-------------------------------------|\n| max_slice            | integer | Maximum size of the slice to choose |\n| rules                | list    | Rules regulating data retention     |\n\n#### Data storing rules\n\n| Parameter            | Type    | Description                           |\n|----------------------|---------|---------------------------------------|\n| name                 | string  | The name of the rule                  |\n| prefix               | string  | The prefix referred by the rule       |\n| limit                | integer | Conveyor size                         |\n| split                | string  | Define the way the value will be divided in case it does not match the bucket: proportional - the value will be proportionally divided between the two nearest buckets; equal - the value will be divided into halves between the two nearest buckets; forward/backward - the whole value wil be put into the previous (or the nearest) bucket |\n| type                 | string  | Define the value to be saved in buckets: last - only the last value will be saved; max - the maximal incoming value will be saved; min - the minimal incoming value will be saved; sum - the sum of the incoming values will be saved; avg - the average value will be saved |\n| value_size           | integer | Define the value size for a bucket (16, 32 и 64 bits), possible values: small, medium, large\n| additional_values    | list    | A list of \"foobar: weak/strong\" pairs from 0 to 14 symbols in length.  Sometimes it is necessary to save the additional values not being numbers ('timeout', for example).  The list of additional values is used for this purpose.  The values can be either strong or weak.  The weak values are updated if the bucket is updated while the strong values are never updated. |\n| autoremove           | boolean | Enable/disable conveyor removal in case of data deterioration |\n\nGet data API\n---------------\n\nThe data are captured with the HTTP API. There are two types of API in YAWNDB\n- the administrative API and the user API. All API settings are specified in\nthe configuration file in the sections Admin JSON API and User JSON API.\n\nA typical API response is given in the following form:\n\n```json\n{\n    \"status\": \"ok\",\n    \"code\": \"ok\",\n    \"answer\": {}\n}\n```\n\nThe status field may take the values `ok` or `error`.\nThe code field contains \"ok\" on success.  The answer is displayed in the answer\nfield. If an error occurs, its code will be displayed in the code field; the\nhuman-readable description of the error will be displayed in the answer field.\n\n#### Get a list of rules for a selected path\n\n`GET /paths/:path/rules`\n\nExample of an answer:\n\n```json\n{\n    \"status\": \"ok\",\n    \"code\": \"ok\",\n    \"answer\": [\"stat\"]\n}\n```\n\n#### Get a list of time-value pairs for a given period:\n\n`POST /paths/slice`\n\nThe conveyors are indicated in the body of the request: ```paths=path1/rule,path2/rule2,path3/rule3```.\n\nRequest parameters:\n\n| Parameter | Type    | Description                 |\n|-----------|---------|-----------------------------|\n| from      | integer | Initial time, UTC timestamp |\n| to        | integer | End time, UTC timestamp     |\n\nExample of an answer:\n\n```json\n{\n \"status\": \"ok\",\n \"code\": \"ok\",\n \"answer\": [\n    [63555098820, 10, 12],\n    [63555098888, 16, \"empty\"],\n    [63555098940, 12, 10],\n    [63555099000, 3, 8],\n    [63555099060, 10, 12],\n    [63555099120, 2, 9],\n    [63555099180, 7, 12],\n    [63555099240, 3, 4],\n    [63555099300, 20, 10],\n    [63555099360, 2, 11]\n ]\n}\n```\n\n#### Get a list of the last time-value pairs for a given period:\n\n`GET /paths/:path/:rule/slice`\n\nRequest parameters:\n\n| Parameter | Type    | Description                |\n|-----------|---------|----------------------------|\n| from      | integer | Initial time UTC timestamp |\n| to        | integer | End time, UTC timestamp    |\n\nExample of an answer:\n\n```json\n{\n \"status\": \"ok\",\n \"code\": \"ok\",\n \"answer\": [\n    [63555098820, 12],\n    [63555098880, \"empty\"],\n    [63555098940, 10],\n    [63555099000, \"empty\"],\n    [63555099060, 15],\n    [63555099120, \"empty\"],\n    [63555099180, 12],\n    [63555099240, \"empty\"],\n    [63555099300, 10],\n    [63555099360, \"empty\"]\n ]\n}\n```\n\nThe answer contains a list of time-value pairs for a given period; if there is\nno value for a certain moment, the string `empty` is returned.\n\n#### Get a list of the last time-value pairs for a conveyor:\n\nRequest parameters:\n\n| Parameter | Type    | Description               |\n|-----------|---------|---------------------------|\n| n         | integer | The number of last values |\n\nExample of an answer:\n\n\n```json\n{\n \"status\": \"ok\",\n \"code\": \"ok\",\n \"answer\": [\n    [63555103140, 11],\n    [63555103200, 10],\n    [63555103260, 10],\n    [63555103320, 12],\n    [63555103380, 10],\n    [63555103440, 10],\n    [63555103500, 11],\n    [63555103560, 10],\n    [63555103620, 10],\n    [63555103680, 14]\n ]\n}\n```\nThe answer contains a list of time-value pairs for a given period; if there is\nno value for a certain moment, the string `empty` is returned.\n\nAdministrative API\n------------------\n\n#### Get a server status:\n\n`GET /status`\n\nExample of an answer:\n\n```json\n{\n \"status\": \"ok\",\n \"code\": \"ok\",\n \"answer\": {\n    \"read_rpm\": 37,\n    \"write_rpm\": 816042,\n    \"read_rps\": 1,\n    \"write_rps\": 13601,\n    \"processes_now\": 162836,\n    \"processes_max\": 300000,\n    \"paths_count\": 162256,\n    \"dirty_paths_count\": 19864\n }\n}\n```\n\nAnswer fields:\n\n| Field             | Type    | Description                                  |\n|-------------------|---------|----------------------------------------------|\n| read_rpm          | integer | Read requests per minute                     |\n| write_rpm         | integer | Write requests per minute                    |\n| read_rps          | integer | Read requests per second                     |\n| write_rps         | integer | Write requests per second                    |\n| processes_now     | integer | Number of active processes                   |\n| processes_max     | integer | Maximal number of active processes permitted |\n| paths_count       | integer | Number of paths                              |\n| dirty_paths_count | integer | Number of the unsaved paths                  |\n\n\n#### Get the list of all paths saved\n\n`GET /paths/all`\n\nExample of an answer:\n\n```json\n{\n \"status\": \"ok\",\n \"code\": \"ok\",\n \"answer\": [\n    \"some-stats-a\",\n    \"some-stats-b\"\n ]\n}\n```\n\n#### Delete a path\n\n`DELETE /paths/:path`\n\nExample of an answer:\n\n```json\n{\n \"status\": \"ok\",\n \"code\": \"ok\",\n \"answer\": \"deleted\"\n}\n```\n\n#### Get an aggregated list of time-value pairs\n\n`POST /aggregate`\n\nThe paths are indicated in the body of the request: ```paths=path1,path2,path3```.\n\nParameters of the request\n\n| Parameter | Type    | Description                               |\n|-----------|---------|-------------------------------------------|\n| from      | integer | Initial time, UTC timestamp               |\n| to        | integer | Ending time, UTC timestamp                |\n| rule      | string  | The rule used                             |\n| aggregate | string  | Aggregation function (max, min, avg, sum) |\n\nExample of an answer:\n\n```json\n{\n \"status\": \"ok\",\n \"code\": \"ok\",\n \"answer\": [\n    [63555096900, 12],\n    [63555096960, \"empty\"],\n    [63555097020, 10],\n    [63555097080, 16],\n    [63555097140, 10],\n    [63555097200, \"empty\"],\n    [63555097260, 11],\n    [63555097320, 17],\n    [63555097380,16],\n    [63555097440, \"empty\"]\n ]\n}\n```\n\n### Error codes\n\n| Error          | HTTP-code | Description                                                                        |\n|----------------|-----------|------------------------------------------------------------------------------------|\n| no_aggregate   | 400       | The aggregate request parameter is not indicated or improperly formulated          |\n| no_path        | 400       | The path request parameter is not indicated or improperly formulated               |\n| no_rule        | 400       | The rule request parameter is not indicated or improperly formulated               |\n| no_from        | 400       | The from request parameter is not indicated or improperly formulated               |\n| no_to          | 400       | The to request parameter is not indicated or improperly formulated                 |\n| no_n           | 400       | The n request parameter is not indicated or improperly formulated                  |\n| no_paths       | 400       | The paths request parameter is not indicated or improperly formulated              |\n| no_body        | 400       | The request body is expected but absent                                            |\n| no_fun         | 400       | Unknown request                                                                    |\n| from_to_order  | 400       | The value of the from parameter is greater than that of the to parameter           |\n| page_not_found | 404       | Requested path not found                                                           |\n| rule_not_found | 404       | The requested rule for the given path not found                                    |\n| slice_too_big  | 413       | The number of request results is greater than the value of the max_slice parameter |\n| process_limit  | 500       | Limit of active processes achieved                                                 |\n| memory_limit   | 500       | Write limit achieved                                                               |\n| internal       | 500       | Internal error                                                                     |\n\nSave data API\n---------\n\nThis API is used for saving new data to YAWNDB. To save the data connect to\nthe TCP port indicated in the configuration file (TCP API options section) and\nsend the packets in the indicated format. No reply from the server is expected.\n\n#### The main types of data used\n\n| Type   | Description                          |\n|--------|--------------------------------------|\n| uint8  | Unsigned 8-bit number                |\n| uint16 | Unsigned 16-bit number in big-endian |\n| uint64 | Unsigned 64-bit number in big-endian |\n| byte*  | Byte sequence                        |\n\n#### Packet structure\n\n| Field  | Value                                                         |\n|--------|---------------------------------------------------------------|\n| uint16 | Packet size (except this field)                               |\n| uint8  | Protocole version                                             |\n| uint8  | Flag indicating special values (0 - the value is not special) |\n| unit64 | Number of seconds from the beginning of UNIX era              |\n| unit64 | Value (or the number of the special value)                    |\n| byte*  | Byte sequence                                                 |\n\n#### Constructing a packet in Python\n\n```python\ndef encode_yawndb_packet(is_special, path, time, value):\n    \"\"\"\n    Construct a packet to be sent to yawndb.\n\n    :param bool is_special: special value? used for additional_values\n    :param str path: statistics identifier\n    :param int value: statistics value\n    \"\"\"\n    is_special_int = 1 if is_special else 0\n    pck_tail = struct.pack(\"\u003eBBQQ\", YAWNDB_PROTOCOL_VERSION, is_special_int,\n                           time, value) + path\n    pck_head = struct.pack(\"\u003eH\", len(pck_tail))\n    return pck_head + pck_tail\n```\n\n#### Constructing a packet in C\n\n```c\n// indicate the protocol version\n#define YAWNDB_PROTOCOL_VERSION 3\n\n// describe the packet structure\nstruct yawndb_packet_struct {\n    uint16_t    length;\n    uint8_t     version;\n    int8_t      isSpecial;\n    uint64_t    timestamp;\n    uint64_t    value;\n    char        path[];\n};\n\n// construct the packet\nyawndb_packet_struct *encode_yawndb_packet(int8_t isSpecial,\n    uint64_t timestamp, uint64_t value, const char * path)\n{\n    yawndb_packet_struct *packet;\n    uint16_t length;\n\n    length = sizeof(uint8_t) + sizeof(int8_t) + sizeof(uint64_t) + sizeof(uint64_t) + strlen(path);\n    packet = malloc(length + sizeof(uint16_t));\n    packet-\u003elength = htobe16(length);\n    packet-\u003eversion = YAWNDB_PROTOCOL_VERSION;\n    packet-\u003eisSpecial = isSpecial;\n    packet-\u003etimestamp = htobe64(timestamp);\n    packet-\u003evalue = htobe64(value);\n    strncpy(packet-\u003epath, path, strlen(path));\n\n    return packet;\n}\n```\n\nAuthors\n-------\n\nYAWNDB was written by [Dmitry Groshev](https://github.com/si14/) in 2012.\n[Alexander Neganov](https://github.com/ikkeps),\n[Kagami Hiiragi](https://github.com/kagami),\n[Maxim Mitroshin](https://github.com/rocco66),\n[Pavel Abalikhin](https://github.com/tnt-dev) and\n[Fedor Gogolev](https://github.com/knsd) have also contributed to the project.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fselectel%2Fyawndb","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fselectel%2Fyawndb","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fselectel%2Fyawndb/lists"}