{"id":19285973,"url":"https://github.com/chenjiandongx/tszlist","last_synced_at":"2025-06-12T11:03:19.560Z","repository":{"id":57549222,"uuid":"305914038","full_name":"chenjiandongx/tszlist","owner":"chenjiandongx","description":"🔗  tszlist 是一种时序数据线程安全链表","archived":false,"fork":false,"pushed_at":"2020-10-30T09:26:10.000Z","size":22,"stargazers_count":11,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-06-12T11:02:30.654Z","etag":null,"topics":["tsdb","tsz"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/chenjiandongx.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-10-21T05:05:47.000Z","updated_at":"2023-08-13T14:40:50.000Z","dependencies_parsed_at":"2022-08-27T00:40:16.254Z","dependency_job_id":null,"html_url":"https://github.com/chenjiandongx/tszlist","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/chenjiandongx/tszlist","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chenjiandongx%2Ftszlist","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chenjiandongx%2Ftszlist/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chenjiandongx%2Ftszlist/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chenjiandongx%2Ftszlist/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/chenjiandongx","download_url":"https://codeload.github.com/chenjiandongx/tszlist/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chenjiandongx%2Ftszlist/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":259453470,"owners_count":22860083,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["tsdb","tsz"],"created_at":"2024-11-09T21:47:33.634Z","updated_at":"2025-06-12T11:03:19.508Z","avatar_url":"https://github.com/chenjiandongx.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# tszlist\n\n🔗 tszlist 是一种时序数据线程安全链表。\n\n[![GoDoc](https://godoc.org/github.com/chenjiandongx/tszlist?status.svg)](https://godoc.org/github.com/chenjiandongx/tszlist)\n[![Travis](https://travis-ci.org/chenjiandongx/tszlist.svg?branch=master)](https://travis-ci.org/chenjiandongx/tszlist)\n[![Go Report Card](https://goreportcard.com/badge/github.com/chenjiandongx/tszlist)](https://goreportcard.com/report/github.com/chenjiandongx/tszlist)\n[![License](https://img.shields.io/badge/License-MIT-brightgreen.svg)](https://opensource.org/licenses/MIT)\n\n## Motivation\n\n时序数据库（TSDB）在最近几年受到了广泛的关注，而时序数据的压缩算法很大程度上决定了 TSDB 的性能。目前在业界被广泛使用的压缩算法是来自 Fackbook 的一篇论文 [Gorilla: A fast, scalable, in-memory time series database](http://www.vldb.org/pvldb/vol8/p1816-teller.pdf)，包括 [Prometheus](https://prometheus.io/), [InfluxDB](https://www.influxdata.com/), [M3](https://m3db.io/), [TimescaleDB](https://www.timescale.com/) 在内的多种开源时序数据库都使用了 Gorilla 中提到的 delta-of-delta 和 XOR 来分别压缩 Timestamp（int64）和 Value（float64）数据，该算法极大的压缩了存储数据点所需要的内存，收益明显。\n\nTSDB 大多数时候都是为了满足监控场景的需求，这里先介绍两个概念：\n* 数据点: 时序数据的数据点是一个包含 (timestamp, value) 的二元组。\n* 时间线：不同 tag 的组合称为不同的时间线，比如 `{\"__name__\": \"netspeed\", \"host\": \"localhost\", \"iface\": \"eth0\"}`, `{\"__name__\": \"netspeed\", \"host\": \"localhost\", \"iface\": \"eth1\"}`。\n\n我司内部的秒级监控系统（Neo）师承 [open-falcon](https://github.com/open-falcon)，不过进行了大量的重构和优化，其中的 neo-judge 组件承担着判定数据数据是否需要告警的任务，也就是说，该组件需要存储不同时间线最近 N 个数据点用来与告警规则做判定。**这是一种有限长度的缓存策略，超过长度限制时淘汰旧数据。** 且一般来讲，判定规则只需要查询最近 n 个点（n \u003c\u003c N）。\n\ntszlist 是一种对以上特殊场景进行优化的数据结构，数据按 block 存储，链表冗余多一个 block, 该 block 使用 Golang 标准库 List 来存储，一旦 block 大小达到阈值（Overflow），冻结（Frozen）该 block，并使用 Gorilla 算法进行压缩，并追加至全局链表中，整体链表长度达到设置的 limit 阈值的话，删除尾部 block。\n\n## Installation\n\ngo get\n```shell\n$ go get -u github.com/chenjiandongx/tszlist\n```\n\ngo mod\n```shell\nrequire github.com/chenjiandongx/tszlist\n```\n\n## Performance\n\n所有测试代码均位于 [tszlist_test.go](./tszlist_test.go)。\n\n### Write-Operation\n\ntszlist 写性能与标准库实现相差不大。\n\n```shell\n# List limit: 200000\n\nBenchmarkTszListWrite-12         7361260               151 ns/op             102 B/op          2 allocs/op\nBenchmarkStdListWrite-12         7649355               150 ns/op             102 B/op          2 allocs/op\n```\n\n### Read-Operation\n\ntszlist 在读取最近 n 个点时（如果 n 个点都在头部 internallist 中），略快于标准库实现；当需要读取的数据已经 frozen 时，decode block 时效率慢于标准库实现。\n\n不过这个很大程度上取决于 Overflow 和 Limit 阈值的设置，不同比例的阈值会有极大的性能差异。\n\n**Round1: Tsz win**\n```shell\n# List limit: 239, overflow: 30; search: rand.Int63(30)\n# overflow 为 30，limit 为 239，所以此时落在头部 internallist 的数据点数量为 239 % 30 = 29\n# 因此随机取前 30 个数据时均命中热区数据，性能较好\n\nBenchmarkTszListRead-12         10523815               114 ns/op             235 B/op          0 allocs/op\nBenchmarkStdListRead-12          4615876               270 ns/op             644 B/op          4 allocs/op\n```\n\n**Round2: Std win**\n```shell\n# List limit: 240, overflow: 30; search: rand.Int63(30)\n# overflow 为 30，limit 为 239，所以此时落在头部 internallist 的数据点数量为 240 % 30 = 0\n# 所有查询对于 tszlist 来说都要对 block 进行 decode 操作，性能较差\n\nBenchmarkTszListRead-12           502519              2389 ns/op            2618 B/op          8 allocs/op\nBenchmarkStdListRead-12          4683820               262 ns/op             644 B/op          4 allocs/op\n```\n\n**Round3: Std win**\n```shell\n# List limit: 240, overflow: 20; search: rand.Int63(30)\n# overflow 为 30，limit 为 239，所以此时落在头部 internallist 的数据点数量为 240 % 25 = 15\n# 一半的查询需要对 block 进行 decode 操作，性能优于 overflow 阈值为 30 时候的情况\n\nBenchmarkTszListRead-12          1000000              1071 ns/op            1237 B/op          4 allocs/op\nBenchmarkStdListRead-12          4389102               269 ns/op             644 B/op          4 allocs/op\n```\n\n### Memory-Compression\n\ntszlist 最大的优势是内存占用要明显小于使用标准库链表实现（每条时间线 240/20 个数据点是我司的真实使用场景）。\n\n|   | 时间线数量 | 每条时间线数据点 | 内存占用 | 压缩比例 |\n| - | -------- | -------------- | ------ | ------- |\n| StdList | 20w | 240 | 3144M | 0% |\n| TszList | 20w | 240（Overflow: 30）| 1127M | 64.15% |\n| TszList | 20w | 240（Overflow: 25）| 1492M | 52.54% |\n| StdList | 20w | 20 | 282M | 0% |\n| TszList | 20w | 20（Overflow: 8）| 280M | ~0% |\n| TszList | 20w | 20（Overflow: 15） | 208M | 26.24% |\n\n## Advantage\n\n线上环境使用时，CPU 和内存使用率均出现了较为明显的下降。计算资源使用率为（内存、CPU）。\n\n**平均降采样：整体下降**\n\n![Avg](https://user-images.githubusercontent.com/19553554/97100072-d6548400-16ca-11eb-9116-e846ced74ae2.jpg)\n\n**LTTB 降采样：CPU 峰值波动变小**\n\n\u003e LTTB（Largest-Triangel-Three-Bucket）是一种时序数据的降采样绘图方式，算法本身不会对数据的数值进行任何修改，尽量保证绘图时波峰波谷细节。论文地址：[DOWNSAMPLING TIME SERIES FOR VISUAL\nREPRESENTATION](https://skemman.is/bitstream/1946/15343/3/SS_MSthesis.pdf)\n\n![LTTB](https://user-images.githubusercontent.com/19553554/97100076-db193800-16ca-11eb-86df-97e06b847a9d.jpg)\n\n## Usage\n\n```golang\npackage main\n\nimport (\n\t\"fmt\"\n\t\"time\"\n\n\t\"github.com/chenjiandongx/tszlist\"\n)\n\nfunc main() {\n\tl := tszlist.NewList(240, tszlist.WithOverflow(30))\n\n\tnow := time.Now().Unix()\n\tfor i := 0; i \u003c 10; i++ {\n\t\tl.Push(now, float64(i))\n\t\tnow += 5\n\t}\n\n\tfmt.Println(\"front5:\",l.GetN(5))\n\tfmt.Println(\"l.len:\", l.Len())\n\tfmt.Println(\"l.cap:\", l.Cap())\n}\n\n// Output:\n// front5: [{1603604967 9} {1603604962 8} {1603604957 7} {1603604952 6} {1603604947 5}]\n// l.len: 10\n// l.cap: 270\n```\n\n## License\n\nMIT [©chenjiandongx](https://github.com/chenjiandongx)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchenjiandongx%2Ftszlist","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fchenjiandongx%2Ftszlist","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchenjiandongx%2Ftszlist/lists"}