{"id":18462579,"url":"https://github.com/jhuebert/iotfsdb","last_synced_at":"2025-08-11T00:18:22.859Z","repository":{"id":258132374,"uuid":"858273864","full_name":"jhuebert/iotfsdb","owner":"jhuebert","description":"Time series database that leverages the unique properties of IOT to efficiently store and retrieve data.","archived":false,"fork":false,"pushed_at":"2025-06-19T22:52:02.000Z","size":430,"stargazers_count":1,"open_issues_count":4,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-06-19T23:29:01.276Z","etag":null,"topics":["bootstrap","chartjs","database","htmx","iot","java","jte","springboot"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jhuebert.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-09-16T16:02:29.000Z","updated_at":"2025-06-10T19:26:56.000Z","dependencies_parsed_at":"2024-11-04T01:20:53.103Z","dependency_job_id":"0c8b278c-ac79-4429-ba8d-8d2a2c808ebb","html_url":"https://github.com/jhuebert/iotfsdb","commit_stats":null,"previous_names":["jhuebert/iotfsdb"],"tags_count":20,"template":false,"template_full_name":null,"purl":"pkg:github/jhuebert/iotfsdb","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jhuebert%2Fiotfsdb","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jhuebert%2Fiotfsdb/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jhuebert%2Fiotfsdb/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jhuebert%2Fiotfsdb/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jhuebert","download_url":"https://codeload.github.com/jhuebert/iotfsdb/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jhuebert%2Fiotfsdb/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":262900086,"owners_count":23381657,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bootstrap","chartjs","database","htmx","iot","java","jte","springboot"],"created_at":"2024-11-06T09:03:00.047Z","updated_at":"2025-07-01T05:03:45.102Z","avatar_url":"https://github.com/jhuebert.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# iotfsdb\n\nTime series database that leverages the unique properties of IOT to efficiently store and retrieve\ndata.\n\n**Getting Started**\n\n- [Running](#running)\n\n## Goals\n\n* Simple data storage and directory structure\n    * Human navigable and updatable\n    * Easy backup\n    * Easy data pruning\n    * Memory mapped files\n* Efficient storage of data\n    * No timestamp storage\n    * Null value support\n    * Fast query performance\n* Easy to use\n    * REST API\n    * Spring Boot\n    * Docker Image\n\n### Leveraged IOT Properties\n\n- Regular interval of data\n- Interval often limited to one value per second\n- Data is of limited resolution\n\n### Drawbacks\n\n- Exact moment of a value is lost as it is placed in a bucket for the interval it belongs to\n\n## Number Type\n\nEach series value has a data type that defines how it is stored in a file. The data type defines the range\nand type of values supported. Pick the smallest type that represents your series for minimal storage,\nfaster processing, and quicker retrieval.\n\nThe number after each type name indicates how many bytes are used to store each value.\n\n| Name       | Minimum                 | Maximum                | Null Value           | Bytes |\n|------------|-------------------------|------------------------|----------------------|-------|\n| `CURVED1`  | -1.7976931348623157E308 | 1.7976931348623157E308 | -128                 | 1     |\n| `CURVED2`  | -1.7976931348623157E308 | 1.7976931348623157E308 | -32768               | 2     |\n| `CURVED4`  | -1.7976931348623157E308 | 1.7976931348623157E308 | -2147483648          | 4     |\n| `FLOAT2`   | -65504                  | 65504                  | NaN                  | 2     |\n| `FLOAT4`   | -3.4028235E38           | 3.4028235E38           | NaN                  | 4     |\n| `FLOAT8`   | -1.7976931348623157E308 | 1.7976931348623157E308 | NaN                  | 8     |\n| `INTEGER1` | -127                    | 127                    | -128                 | 1     |\n| `INTEGER2` | -32767                  | 32767                  | -32768               | 2     |\n| `INTEGER4` | -2147483647             | 2147483647             | -2147483648          | 4     |\n| `INTEGER8` | -9223372036854775807    | 9223372036854775807    | -9223372036854775808 | 8     |\n| `MAPPED1`  | -1.7976931348623157E308 | 1.7976931348623157E308 | -128                 | 1     |\n| `MAPPED2`  | -1.7976931348623157E308 | 1.7976931348623157E308 | -32768               | 2     |\n| `MAPPED4`  | -1.7976931348623157E308 | 1.7976931348623157E308 | -2147483648          | 4     |\n\n### Integer Types\n\n`INTEGER1`, `INTEGER2`, `INTEGER4`, `INTEGER8` store signed integer values. Null values are represented as the minimum value for the\nbase type. This means that any attempt to store that specific value will result in null being\nreturned when querying data.\n\n### Float Types\n\n`FLOAT2`, `FLOAT4`, `FLOAT8` store floating point values in IEEE-754 format. Null values are represented by `NaN`.\nThis means that any attempt to store that specific value will result in null being returned when\nquerying data.\n\n### Mapped Types\n\n`MAPPED1`, `MAPPED2`, `MAPPED4` store floating point values that are backed by the corresponding sized integer\ntypes. This is accomplished by utilizing a minimum and maximum value range that is specified in the\nseries definition. This range is used the map to and from the floating point range to the\ncorresponding integer range.\n\nThis results in increased floating point resolution and a reduction of space if the range of data is\nknown. A downside is that reduced resolution results in the saved value not being identical to the\nvalue that is retrieved (see [example](#mapping-example))\n\n#### Mapping Example\n\n**Series Definition**\n\n| Property      | Value     |\n|---------------|-----------|\n| Number Type   | `MAPPED1` |\n| Minimum Value | `-10.0`   |\n| Maximum Value | `10.0`    |\n\n**Value Mapping**\n\n| Property                   | Value                |\n|----------------------------|----------------------|\n| Input Value                | `-5.3`               |\n| Value mapped to `INTEGER1` | `-67.31`             |\n| `INTEGER1` value stored    | `-67`                |\n| Restored Value             | `-5.275590551181103` |\n\n### Curved Types\n\n`CURVED1`, `CURVED2`, `CURVED4` are identical to the `MAPPED` types except that a curve is used to\nmap it to the underlying integer type. This means that instead of a hard limit at the minimum and\nmaximum value the value can still be stored but at reduced resolution based on an exponential curve.\nThe main reason to use this one over a `MAPPED` type is that it better handles situations where the\nrange is not fully known.\n\n## Directory Layout\n\nThe database root directory contains only directories where each directory represents a single series.\nThe series directory name matches the ID the series is created with.\n\n### Series Directory\n\nThe series directory name and ID must match the regular expression `[a-z0-9][a-z0-9._-]{0,127}`.\nEach series directory contains a `series.json` and period partitioned data files. The `series.json`\nfile contains the series definition and metadata. The only mutable field in the json is the\nmetadata.\n\nThe other files in the series directory are the data files. The name of the files is based on the\npartition selected for the series and represent a period of time. The data is stored in sequential\ntime.\n\n### Example Layout\n\n```\nroot\n├── series-1\n│   ├── series.json\n│   ├── 2023\n│   └── 2024\n├── series-2\n│   ├── series.json\n│   ├── 202306\n│   ├── 202307\n│   └── 202410\n└── series-3\n    ├── series.json\n    ├── 20230611\n    ├── 20230704\n    └── 20241006\n```\n\n## Partition\n\nThe partitioning scheme allows a specific range of time to be represented in a file. Each\npartitioned file represents a calendar date range of time. The partitioning scheme in combination\nwith the type size allows us to jump directly to a data value.\n\n| Name    | Format     | Example    | Description                                                                                   |\n|---------|------------|------------|:----------------------------------------------------------------------------------------------|\n| `DAY`   | `yyyyMMdd` | `20241101` | Represents 00:00 through 23:59 of a single date                                               |\n| `MONTH` | `yyyyMM`   | `202411`   | Represents 00:00 of the first day of the month through 23:59 of the last day of a given month |\n| `YEAR`  | `yyyyMM`   | `2024`     | Represents 00:00 of the first day of January through 23:59 of December 31 of a given year     |\n\n### Partition Example\n\n#### Series Configuration\n\n| Property         | Value    |\n|------------------|----------|\n| Partition Period | `MONTH`  |\n| Interval (ms)    | `60000`  |\n| Number Type      | `FLOAT4` |\n\nWe would like to fetch the value for `2024-12-13T12:34:56`. Because we know the partition period is\n`MONTH`, we know that the data will be located in a file named `202412`. If that file doesn't exist,\nwe know the data doesn't exist and the result is `null`.\n\nSince we know that the data interval for the file is `60000ms`, we can calculate which bytes the\ndata value is located in. The data has an index of `18034` in the array of `float` values. We multiply that by `4`\nto get the byte offset of the data in the file (`72136`).\n\n#### File Contents\n\n| Byte Index | Stored Value | Timestamp Range Represented                  | Type Representation |\n|------------|--------------|----------------------------------------------|---------------------|\n| `0`        | `0x7FC00000` | `[2024-12-01T00:00:00, 2024-12-01T00:01:00)` | `null`              |\n| ...        | ...          | ...                                          | ...                 |\n| `72136`    | `0x3E9E0652` | `[2024-12-13T12:34:00, 2024-12-13T12:35:00)` | `1.2345679`         |\n| ...        | ...          | ...                                          | ...                 |\n| `178556`   | `0x7FC00000` | `[2024-12-31T23:59:00, 2025-01-01T00:00:00)` | `null`              |\n\n## API\n\nThe OpenAPI specification can be viewed at http://localhost:8080/swagger-ui/index.html.\nYou can download the OpenAPI specification from http://localhost:8080/v3/api-docs.yaml.\n\n## Authentication\n\nSince the database exposes a REST API, authentication can be handled by a proxy.\n\n## Building\n\n```shell\n./gradlew build\n```\n\n## Running\n\n### Command Line\n\n```shell\njava -jar iotfsdb.jar\n```\n\n### Docker\n\n[Image on Docker Hub](https://hub.docker.com/repository/docker/jhuebert/iotfsdb/general)\n\n```bash\ndocker run -it -p 8080:8080 -v iotfsdb-data:/data jhuebert/iotfsdb:2\n```\n\n#### Docker Compose\n\n```yaml\nservices:\n  iotfsdb:\n    image: jhuebert/iotfsdb:2\n    volumes:\n      - ./data:/data\n    ports:\n      - 8080:8080\n```\n\n### Properties\n\n| Java Property             | Environment Variable      | Description                                                              | Default Value                                       | Docker Default Value                                |\n|---------------------------|---------------------------|--------------------------------------------------------------------------|-----------------------------------------------------|-----------------------------------------------------|\n| `iotfsdb.root`            | `IOTFSDB_ROOT`            | Root data directory for the database                                     | `memory`                                            | `/data`                                             |\n| `iotfsdb.read-only`       | `IOTFSDB_READ_ONLY`       | Indicates whether any changes to the database are allowed                | `false`                                             | `false`                                             |\n| `iotfsdb.max-query-size`  | `IOTFSDB_MAX_QUERY_SIZE`  | Maximum number of values returned for any series query                   | `1000`                                              | `1000`                                              |\n| `iotfsdb.partition-cache` | `IOTFSDB_PARTITION_CACHE` | Maximum amount of time to keep a series partition file open after access | `expireAfterAccess=5m,maximumSize=10000,softValues` | `expireAfterAccess=5m,maximumSize=10000,softValues` |\n| `iotfsdb.ui`              | `IOTFSDB_UI`              | Indicates whether the web UI will be available                           | `true`                                              | `true`                                              |\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjhuebert%2Fiotfsdb","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjhuebert%2Fiotfsdb","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjhuebert%2Fiotfsdb/lists"}