{"id":15067769,"url":"https://github.com/haoel/downsampling","last_synced_at":"2025-10-16T23:37:39.879Z","repository":{"id":43432387,"uuid":"206958069","full_name":"haoel/downsampling","owner":"haoel","description":"downsampling time series data algorithm in Go","archived":false,"fork":false,"pushed_at":"2023-04-24T08:10:20.000Z","size":1717,"stargazers_count":204,"open_issues_count":0,"forks_count":31,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-03-17T13:11:37.202Z","etag":null,"topics":["downsampling-algorithm","golang","ltb","ltob","lttb","series-data"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/haoel.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-09-07T11:25:05.000Z","updated_at":"2025-02-18T13:55:48.000Z","dependencies_parsed_at":"2024-06-18T21:33:22.862Z","dependency_job_id":"d6a76b31-379e-4bcf-a552-2b5e5d167898","html_url":"https://github.com/haoel/downsampling","commit_stats":null,"previous_names":["haoel/downsampling-algorithm"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/haoel%2Fdownsampling","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/haoel%2Fdownsampling/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/haoel%2Fdownsampling/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/haoel%2Fdownsampling/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/haoel","download_url":"https://codeload.github.com/haoel/downsampling/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245767356,"owners_count":20668826,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["downsampling-algorithm","golang","ltb","ltob","lttb","series-data"],"created_at":"2024-09-25T01:27:15.601Z","updated_at":"2025-10-16T23:37:39.752Z","avatar_url":"https://github.com/haoel.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Downsampling Algorithm\n\nThe Golang implementation for downsampling time series data algorithm\n\n- [Downsampling Algorithm](#downsampling-algorithm)\n  - [Background](#background)\n  - [Acknowledgment](#acknowledgment)\n  - [Algorithms Library](#algorithms-library)\n  - [Demo](#demo)\n  - [Performance](#performance)\n    - [Profiling](#profiling)\n    - [Benchmark](#benchmark)\n  - [Further Reading](#further-reading)\n  - [License](#license)\n\n## Background\n\nWhile monitoring the online system, there could be so many metrics' time series data will be stored in the ElasticSearch or NoSQL database for analysis. As time passed, storing every piece of historical data is not a very effective way, and those huge data could impact the analysis performance and the cost of storage.\n\nOne solution just simply deletes the aged historical data(e.g. only keep the latest 6 months' data), but there is a solution we can compressing those data to a small size with good resolution.\n\nHere is the Go library to demonstrate how to downsamping the time series data from 7500 points to 500 points (Actually, you can downsample it to 200 or 300 points).\n\n\n## Acknowledgment\n\n- All of the algorithms are based on Sveinn Steinarsson's 2013 paper [Downsampling Time Series for Visual Representation](\nhttps://skemman.is/bitstream/1946/15343/3/SS_MSthesis.pdf)\n\n- This implementation refers to Ján Jakub Naništa's [implementation by Typescript](https://github.com/janjakubnanista/downsample)\n\n- The test data I borrow from one of Python implementation which is [here](https://github.com/devoxi/lttb-py/)\n\n\n## Algorithms Library\n\n[Sveinn Steinarsson's paper](\nhttps://skemman.is/bitstream/1946/15343/3/SS_MSthesis.pdf) mentioned 3 types of algorithms:\n\n- Largest triangle three buckets (LTTB)\n- Largest triangle one bucket (LTOB)\n- Largest triangle dynamic (LTD)\n\nYou can find all of these implementations under `core` directory.\n\nAnd you can import the library by:\n\n```go\nimport \"github.com/haoel/downsampling/core\"\n```\n\n## Demo\n\n\nFollowing the below instructions compile and run this repo.\n\n```shell\nmake\n./demo/build/bin/main\n```\n\nIf everything goes fine, you will see the following message\n\n```\n2019/09/07 18:34:42 Reading the testing data...\n2019/09/07 18:34:42 Downsampling the data from 7501 to 500...\n2019/09/07 18:34:42 Downsampling data - LTOB algorithm done!\n2019/09/07 18:34:42 Downsampling data - LTTB algorithm done!\n2019/09/07 18:34:42 Downsampling data - LTD algorithm done!\n2019/09/07 18:34:42 Creating the diagram file...\n2019/09/07 18:34:43 Successfully created the diagram - ....../data/downsampling.chart.png\n```\n\nYou can go to the `./demo/build/data/` directory to check the diagram and the CVS files.\n\nThe diagram picture as below\n- The first black chart at the top is the raw data with 7500 points\n- The second, third, and fourth respectively are LTOB, LTTB, and LTD downsampling data with 500 points\n- The last one at the bottom is just put all together.\n\n![](./demo/data/downsampling.chart.png?raw=true)\n\n## Performance\n\nYou can use the following makefile target to analyze the performance of these algorithms.\n\n### Profiling\n\n```\nmake prof\n```\n\n### Benchmark\n\n```\nmake bench\n```\n\n## Further Reading\n\n* [The Billion Data Point [Challenge](https://eng.uber.com/billion-data-point-challenge/) by the Uber Engineering team\n* [Visualize Big Data on Mobile](http://dduraz.com/2019/04/26/data-visualization-mobile/) by dduraz\n* [Sampling large datasets in d3fc](http://blog.scottlogic.com/2015/11/16/sampling-large-data-in-d3fc.html) by William Ferguson\n* [Downsampling algorithms](http://www.adrian.idv.hk/2018-01-24-downsample/) by Adrian S. Tam\n\n\nEnjoy it!\n\n## License\n\n[MIT License](./LICENSE)","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhaoel%2Fdownsampling","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhaoel%2Fdownsampling","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhaoel%2Fdownsampling/lists"}