{"id":16226901,"url":"https://github.com/abbychau/gtsdb","last_synced_at":"2025-10-29T05:30:20.097Z","repository":{"id":243576416,"uuid":"812795197","full_name":"abbychau/gtsdb","owner":"abbychau","description":"A Dead Simple Timeseries Database. Aims to provide basic methods for embedding.","archived":false,"fork":false,"pushed_at":"2025-01-26T13:05:06.000Z","size":276,"stargazers_count":8,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-01-26T14:19:27.567Z","etag":null,"topics":["database","edge-computing","golang","iot","lightweight","self-hosted","time-series","timeseries-database"],"latest_commit_sha":null,"homepage":"https://gtsdb.abby.md","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/abbychau.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-06-09T22:28:47.000Z","updated_at":"2025-01-26T13:05:09.000Z","dependencies_parsed_at":"2024-06-09T23:40:33.636Z","dependency_job_id":"453175da-a762-4b08-859f-1f72c9f3ae13","html_url":"https://github.com/abbychau/gtsdb","commit_stats":null,"previous_names":["abbychau/gtsdb"],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/abbychau%2Fgtsdb","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/abbychau%2Fgtsdb/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/abbychau%2Fgtsdb/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/abbychau%2Fgtsdb/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/abbychau","download_url":"https://codeload.github.com/abbychau/gtsdb/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":238772152,"owners_count":19527880,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["database","edge-computing","golang","iot","lightweight","self-hosted","time-series","timeseries-database"],"created_at":"2024-10-10T12:50:56.452Z","updated_at":"2025-10-29T05:30:20.091Z","avatar_url":"https://github.com/abbychau.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Golang Dead Simple Timeseries Database\n\n\u003cimg width=\"768\" height=\"768\" alt=\"image\" src=\"https://github.com/user-attachments/assets/1eb70120-4748-443f-b358-0bad20b43004\" /\u003e\n\n\n[![coverage](/docs/coverage.png)](./docs/coverage-full.png)\n\n## Introduction\n\nThis is a simple timeseries database written in Golang. It is designed to be simple and easy to use. It is designed to be used in IoT and other applications where you need to store timeseries data.\n\nSimple is not only in terms of usage but also in terms of a fundamentally new ways of storing data.\n\nLet's compare it with other databases, usually, a database do these things:\n1. Write data to WAL (Write Ahead Log) into disk\n2. Stream IO to memory\n3. Do some internal processing like indexing and caching\n4. Write data to disk by blocks, fsync to disk for durability\n5. Update or erase WAL cursor (maybe an offset or ID)\n6. When disaster strikes, read data from disk and replay WAL to recover data\n7. Read data from index and file blocks\n\nI think in a different way. I think, if WAL is anyway needed for a production grade database, why not just use WAL for everything:\n1. Write data to WAL\n2. Create indexes conditionally\n3. Read data from WAL and indexes\n\nThis way, we saves a lot of IO and MEMORY USAGE. That's why I call it a dead simple timeseries database. I don't scarifice durability. And reading from Index is still O(log n).\n\nI don't want to say this is perfect. So here is the tradeoff: \"GTSDB needs more file handles than other databases\" because it keeps WAL open for reading and writing, but I think it's worth it, especially for weak hardwares. And more importantly, we can make a LRU of file handles when it is REALLY needed.\n\nYou can see the performance of this database in the performance section. (It is way better than you can expect by this design even it's just in Golang)\n\nAm I going to write it in Rust/C++/Zig? Yes. I love Rust but GTSDB is still highly experimental and I want to make it more stable before I write it in Rust. I still have a lot of ideas to implement and they sometimes contradict each other. So... I'm still in Golang. Even so, I made it in very high code coverage and with some different internal and end-to-end benchmarks. If you want to use it in production, you can use it. Just make sure to checkout a git hash that is with a green tick in the CI. \n\n\n## Run / Compile\n\n```bash\ngo run .\ngo build .\n```\n\n## Usage\n\n### HTTP API (POST to :5556)\n\n```json\n# Write data\n# POST /\n{\n    \"operation\":\"write\",\n    \"Write\": {\n        \"key\" : \"a_sensor1\",\n        \"Value\": 32242424243333333333.3333\n    }\n}\n\n# Read data\n# POST /\n{\n    \"operation\":\"read\",\n    \"Read\": {\n        \"key\" : \"a_sensor1\",\n        \"start_timestamp\": 1717965210,\n        \"end_timestamp\": 1717965211,\n        \"downsampling\": 3\n    }\n}\n{\n    \"operation\":\"read\",\n    \"Read\": {\n        \"key\" : \"a_sensor1\",\n        \"lastx\": 1\n    }\n}\n\n# Get all keys\n# POST /\n{\n    \"operation\":\"ids\"\n}\n\n# Subscribe to a key\n# POST /\n{\n  \"operation\": \"subscribe\",\n  \"key\": \"sensor1\"\n}\n\n# Unsubscribe to a key\n# POST /\n{\n  \"operation\": \"unsubscribe\",\n  \"key\": \"sensor1\"\n}\n\n# Patch data points for a key\n# POST /\n{\n    \"operation\": \"data-patch\",\n    \"key\": \"sensor1\",\n    \"data\": \"1717965210,123.45\\n1717965211,123.46\\n1717965212,123.47\"\n}\n```\n\n## Performance\n\n### Database Performance\n\n[Benchmark](https://github.com/abbychau/gtsdb/blob/main/main_test.go#L65)\n\n- Run: `go test -benchmem -run=^$ -bench ^BenchmarkMain$ -benchtime=5s`\n\n```\ngoos: windows\ngoarch: amd64\npkg: gtsdb\ncpu: 13th Gen Intel(R) Core(TM) i7-13700KF\nBenchmarkMain-24         6158864             19707 ns/op         4245 B/op           5 allocs/op\nPASS\nok      gtsdb   141.340s\n```\n\n\nExplanation:\n- This benchmark does 50% read and 50% write operations to 100 different keys(devices).\n- It performs 249386 operations in 6 seconds. The operations include read and write operations.\n\n\n### Concurrent Internals Performance\n\n- Run: `make benchmark`\n\n```\ngoos: windows\ngoarch: amd64\npkg: gtsdb/concurrent\ncpu: 13th Gen Intel(R) Core(TM) i7-13700KF\nBenchmarkMap/Sequential_Store-24        12363549               477.2 ns/o     154 B/op           6 allocs/op\nBenchmarkMap/Sequential_Load-24         44461207               209.5 ns/o       7 B/op           0 allocs/op\nBenchmarkMap/Concurrent_Store-24        36248278               256.2 ns/o      56 B/op           4 allocs/op\nBenchmarkMap/Concurrent_Load-24         63952998               216.3 ns/o       7 B/op           0 allocs/op\nBenchmarkMap/Concurrent_Mixed-24        18266863               427.2 ns/o      94 B/op           3 allocs/op\nBenchmarkSet_Add-24                     60995383               208.0 ns/o      52 B/op           0 allocs/op\nBenchmarkSet_Contains-24                484518542               12.36 ns/op              0 B/op          0 allocs/op\nBenchmarkSet_ConcurrentAdd-24           43510735               213.0 ns/o      37 B/op           0 allocs/op\nPASS\nok      gtsdb/concurrent        147.859s\n```\n\n\n## Generate Test Coverage Report\n\n```bash\nmake GenerateTest\n```\n\n## License\n\nMIT\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fabbychau%2Fgtsdb","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fabbychau%2Fgtsdb","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fabbychau%2Fgtsdb/lists"}