{"id":22772864,"url":"https://github.com/instaclustr/atsc","last_synced_at":"2026-02-07T13:31:19.988Z","repository":{"id":266558837,"uuid":"638070860","full_name":"instaclustr/atsc","owner":"instaclustr","description":"ATSC is a approach to lossy timeseries compression. Exchanging some precision for high compression ratios.","archived":false,"fork":false,"pushed_at":"2025-04-16T14:38:37.000Z","size":6803,"stargazers_count":10,"open_issues_count":6,"forks_count":2,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-10-20T06:57:15.058Z","etag":null,"topics":["netapp-confidential"],"latest_commit_sha":null,"homepage":"","language":"HTML","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/instaclustr.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2023-05-09T02:48:52.000Z","updated_at":"2025-04-16T14:38:34.000Z","dependencies_parsed_at":"2025-03-27T11:24:00.318Z","dependency_job_id":"bc2a2512-8880-48d2-a7f4-7b2a7b1a7026","html_url":"https://github.com/instaclustr/atsc","commit_stats":null,"previous_names":["instaclustr/atsc"],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/instaclustr/atsc","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/instaclustr%2Fatsc","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/instaclustr%2Fatsc/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/instaclustr%2Fatsc/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/instaclustr%2Fatsc/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/instaclustr","download_url":"https://codeload.github.com/instaclustr/atsc/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/instaclustr%2Fatsc/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29195059,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-07T12:38:28.597Z","status":"ssl_error","status_checked_at":"2026-02-07T12:38:23.888Z","response_time":63,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["netapp-confidential"],"created_at":"2024-12-11T17:09:56.371Z","updated_at":"2026-02-07T13:31:19.955Z","avatar_url":"https://github.com/instaclustr.png","language":"HTML","readme":"# ATSC - Advance Time Series Compressor\n\n**NOTE:** This is still under development. Current status is unsupported!\n\n## Table of Contents\n\n1. [TL;DR;](#tldr)\n2. [What is ATSC?](#what-is-atsc)\n3. [When to use ATSC?](#when-to-use-atsc)\n4. [Documentation](#documentation)\n5. [Building ATSC](#building-atsc)\n6. [ATSC Usage](#atsc-usage)\n7. [Releases](#releases)\n8. [Roadmap](#roadmap)\n9. [Support Status](#support-status)\n\n## TL;DR\n\nThe fastest way to test ATSC is with a CSV file!\n\n1. Download the latest [release](https://github.com/instaclustr/atsc/releases)\n2. Pick a CSV file from [tests folder](https://github.com/instaclustr/atsc/tree/main/atsc/tests/csv) (Those will have the expected internal format).  \n3. Execute the following command:\n\n    ```bash\n    cargo run --release -- --csv \u003cinput-file\u003e\n    ```\n\n4. You have a compressed timeseries!\n\n## What is ATSC?\n\nAdvanced Time Series Compressor (in short: ATSC), is a configurable, *lossy* compressor that uses the characteristics of a time-series to create a function approximation of the time series.\n\nThis way, ATSC only needs to store the parametrization of the function and not the data.\n\nATSC draws inspiration from established compression and signal analysis techniques, achieving significant compression ratios.\n\nIn internal testing ATSC compressed from 46x to 880x the monitoring timeseries of our databases with a fitting error within 1% of the original time-series.\n\nIn some cases, ATSC would produce highly compressed data without any data loss (Perfect fitting functions).\nATSC is meant to be used with long term storage of time series, as it benefits from more points to do a better fitting.\n\nThe decompression of data is faster (up to 40x) vs a slower compression speed, as it is expected that the data might be compressed once and decompressed several times.\n\nInternally ATSC uses the following methods for time series fitting:\n\n* FFT (Fast Fourier Transforms)\n* Constant\n* Interpolation - Catmull-Rom\n* Interpolation - Inverse Distance Weight\n* RLE (Run Length Encoder)\n\nFor a more detailed insight into ATSC read the paper here: [ATSC - A novel approach to time-series compression](https://github.com/instaclustr/atsc/tree/main/paper/ATCS-AdvancedTimeSeriesCompressor.pdf)\n\nATSC input can be an internal format developed to process time series (WBRO), or a CSV. It outputs a compressed format (BRO). A CSV to WBRO format is available here: [CSV Compressor](https://github.com/instaclustr/atsc/tree/main/csv-compressor)\n\n## When to use ATSC?\n\nATSC fits in any place that needs space reduction in trade for precision.\nATSC is to time series what JPG/MP3 is to image/audio.\nIf there is no need of absolute precision of the output vs the original input, you could probably use ATSC.\n\nExample of use cases:\n\n* In places where time series are rolled over, ATSC is a perfect fit. It would probably offer more space savings without any meaningful loss in precision.\n* Time series that are under sampled (e.g. once every 20sec). With ATSC you can greatly increase sample rate (e.g. once per second) without losing space.\n* Long, slow moving data series (e.g. Weather data). Those will most probably follow an easy to fit pattern\n* Data that is meant to be visualized by humans and not machine processed (e.g. Operation teams). With such a small error, under 1%, it shouldn't impact analysis.\n\n## Documentation\n\nFor full documentation please go to [Docs](https://github.com/instaclustr/atsc/tree/main/docs)\n\n## Building ATSC\n\n1. Clone the repository:\n\n   ```bash\n   git clone https://github.com/instaclustr/atsc\n   cd atsc\n   ```\n\n2. Build the project:\n\n   ```bash\n   cargo build --release\n   ```\n\n## ATSC Usage\n\n### Prerequisites\n\n* Ensure you have [Rust](https://www.rust-lang.org/tools/install) installed on your system.\n\n### Usage\n\nATSC relies on files with a WBRO extension to operate, learn more about that here: [WBRO - A time series format](https://github.com/instaclustr/atsc/tree/main/wavbrro)\nYou can also compress from CSV with the provided [CSV tool](https://github.com/instaclustr/atsc/tree/main/csv-compressor)\nThose files would work as input for the compressor.\n\nCompressor usage:\n\n```txt\nUsage: atsc [OPTIONS] \u003cINPUT\u003e\n\nArguments:\n  \u003cINPUT\u003e  input file\n\n      --compressor \u003cCOMPRESSOR\u003e\n          Select a compressor, default is auto [default: auto] [possible values: auto, noop, fft, constant, polynomial, idw]\n  -e, --error \u003cERROR\u003e\n          Sets the maximum allowed error for the compressed data, must be between 0 and 50. Default is 5 (5%).\n          0 is lossless compression\n          50 will do a median filter on the data.\n          In between will pick optimize for the error [default: 5]\n  -u\n          Uncompresses the input file/directory\n  -c, --compression-selection-sample-level \u003cCOMPRESSION_SELECTION_SAMPLE_LEVEL\u003e\n          Samples the input data instead of using all the data for selecting the optimal compressor.\n          Only impacts speed, might or not increased compression ratio. For best results use 0 (default).\n          Only works when compression = Auto.\n          0 will use all the data (slowest)\n          6 will sample 128 data points (fastest) [default: 0]\n      --verbose\n          Verbose output, dumps everysample in the input file (for compression) and in the ouput file (for decompression)\n      --csv\n          Defines user input as a CSV file\n      --no-header\n          Defines if the CSV has no header\n      --fields \u003cFIELDS\u003e\n          Defines names of fields in CSV file. It should follow this format:\n            --fields=TIME_FIELD_NAME,VALUE_FIELD_NAME\n          It assumes that the one before comma is a name of time field and the one\n          after comma is value field. [default: time,value]\n  -h, --help\n          Print help\n  -V, --version\n          Print version\n```\n\n#### Compress a File\n\nTo compress a file using ATSC, run:\n\n```bash\natsc \u003cinput-file\u003e\n```\n\n#### Decompress a File\n\nTo decompress a file, use:\n\n```bash\natsc -u \u003cinput-file\u003e\n```\n\n## Releases\n\n### v0.7 - 20/11/2024\n\n* Added CSV Support\n* Greatly improved documentation\n* Improved Benchmark and testing\n* Improved FFT compression\n* Improved Polynomial compression\n* Demo files and generation scripts\n* Several fixes and cleanups\n\n### v0.6 - 09/11/2024\n\n* Internal release\n\n### v0.5 - 30/11/2023\n\n* Added Polynomial Compressor (with 2 variants)\n* Created and Integrated a proper file type (wbro)\n* Benchmarks of the different compressors\n* Integration testing\n* Several fixes and cleanups\n\n## Roadmap\n\n* Frame expansion (Allowing new data to be appended to existing frames)\n* Dynamic function loading (e.g. providing more functions without touching the whole code base)\n* Global/Per frame error storage\n* Efficient error\n\n## Support Status\n\nPlease see https://www.instaclustr.com/support/documentation/announcements/instaclustr-open-source-project-status/ for Instaclustr support status of this project","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Finstaclustr%2Fatsc","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Finstaclustr%2Fatsc","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Finstaclustr%2Fatsc/lists"}