{"id":21714222,"url":"https://github.com/mingyi850/tsaggcache","last_synced_at":"2026-05-09T19:03:35.809Z","repository":{"id":226115724,"uuid":"767802056","full_name":"mingyi850/TSAggCache","owner":"mingyi850","description":"Time-series cache poc along various dimensions to improve read performance","archived":false,"fork":false,"pushed_at":"2024-08-12T15:34:00.000Z","size":8808,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-07-11T18:51:55.513Z","etag":null,"topics":["database","influxdb","python","time-series"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mingyi850.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2024-03-05T23:22:12.000Z","updated_at":"2024-08-12T15:34:04.000Z","dependencies_parsed_at":"2024-03-06T00:29:34.889Z","dependency_job_id":"05155f42-3e5f-4193-a1f0-258bfac66b64","html_url":"https://github.com/mingyi850/TSAggCache","commit_stats":null,"previous_names":["mingyi850/tsaggcache"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/mingyi850/TSAggCache","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mingyi850%2FTSAggCache","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mingyi850%2FTSAggCache/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mingyi850%2FTSAggCache/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mingyi850%2FTSAggCache/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mingyi850","download_url":"https://codeload.github.com/mingyi850/TSAggCache/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mingyi850%2FTSAggCache/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32831504,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-08T08:22:46.396Z","status":"online","status_checked_at":"2026-05-09T02:00:06.633Z","response_time":123,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["database","influxdb","python","time-series"],"created_at":"2024-11-26T00:33:18.805Z","updated_at":"2026-05-09T19:03:35.760Z","avatar_url":"https://github.com/mingyi850.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# TSAgg Cache\n\n## Abstract\nTime-series databases are the backbone to modern monitoring systems. While time-series databases are typically optimized for fast reads, the increasing number of users of these databases and the need to analyze large amounts of correlated time series prompts the need for an efficient solution to reduce load on time series databases. We present TSAggCache, a platform-agnostic, in-memory caching solution which exploits the similarities in time-series queries, as well as the properties of aggregated time-series data to improve read performance of time-series queries and reduce the load on time-series databases. We have developed a translation DSL, efficient cache indexing structure, and layout which allows the re-use of cached data over multiple dimensions. Our experiments show that TSAggCache can significantly improve client query performance, reducing latency by up to 91% while providing correctness guarantees.\n\n## Problem\n\nTime series data is huge, with Twitter's Observability stack collecting 170 million metrics every minute and serving 200 million queries per day. Time series databases are typically used for metrics and log storage for large systems and applications, and are typically append-only (due to the historical nature of the data)\nOne common use of Timeseries data is dashboarding, where users are able to create visualizations to track specific metrics over time.\n\nHowever, many clients achieve this by simply re-querying data repeatedly or on startup. This repeated queries can put a strain on the time series database by repeatedly querying and aggregating the same data over and over again, which is not required in most cases.\n\n## Components \nWe propose a cache layer in-between the Time Series database which allows for re-use of previously aggregated results and reduces the amount of queries the database needs to handle.\nWe provide the following:\n\n### Query DSL\nTSAggCache utilizes an internal DSL to build and process queries. This allows us to extend the functionality of TSAggCache, allowing it to be used as a cache for other Time-Series database implementations such as Prometheus and Timescale DB in future. We achieve this by providing an extensible DSL which provides abstractions for core features of time series queries such as measurements, filters, tags, and aggregation functions. This allows us to translate queries between different languages and serialization formats, such as from FluxQL to InfluxQL and to JSON. This allows us to build queries using a unified DSL for multiple platforms.\n\n### Cache Server\nThe server is implemented using a simple Python Flask server, which supports 3 main endpoints \n```\nGET /api/health -\u003e Performs healtcheck on cache service\nPOST /api/reset -\u003e Clears all data in the cache\nPOST /api/query -\u003e Queries data in the cache\n```\n\n## TSAggCache Usage\n\n1. The TS Aggregation Cache is built on-top of a key-value system.\n2. The main idea of this design is to store a materialized view of a previous query in-memory, and check to see which parts of the query we can reuse.\n![alt text](image-1.png)\n    - This results in a smaller query needed to be made to the Time Series database in the case of a partial hit\n    - In the case of a complete cache hit, we can simply manipulate the existing data we have in the cache to return the result to the user.\n3. The cache works in these rough steps to produce the output required\n![alt text](image-2.png)\n\n\n## Running the Cache / Experiments\n\n### Dependencies\n```\npip install influxdb3-python\npip install pandas\npip install flask\n```\n\n### Starting the TestServer:\n```\npython3 influxv3/testServer.py\n```\n\n### Provide configs for Influx Connection\nModify the cacheConfig.ini file, providing token, org and host\n```\n[influx]\ntoken=\"\" #Replace \"\" with token\norg=\"\" #Replace \"\" with org name\nhost=\"\" # Replace \"\" with hostname of influx server\n```\n### Running the test data generation\n```\npython3 influxv3/testDataGen3.py\n```\n### Construct queries using the query DSL\n```\nfrom influxv3.queryDSL import InfluxQueryBuilder, QueryAggregation, QueryFilter\n\nqueryBuilder = (InfluxQueryBuilder()\n               .withBucket(\"Test\")\n               .withMeasurements([\"cpu_usage\", \"temperature\"])\n               .withTable(\"system_metrics\")\n               .withFilter(QueryFilter(\"platform\", \"mac_os\").OR(QueryFilter(\"platform\", \"windows\")))\n               .withAggregate(QueryAggregation(\"10m\", \"mean\", False))\n               .withRelativeRange('300m', None)\n               .withGroupKeys([\"host\", \"platform\"])\n       )\n```\n\n### Convert Query DSL to JSON request and send request to server\n```\nrequestJson = queryBuilder.buildJson()\nresponse = requests.post(cacheUrlJson, json=queryJson)\n```\n\n## Utilities\n\n### Cache Structure\nWe construct the cache as a Key-Value store, with each value stored as a tree. Each value is indexed by the constructed key using the hard filters based on a combination of the table, aggregation function and hard filters.\n\n### Measurements and Tracing\nWe provide tracing of operations to profile the time taken for each operation in the cache. This can be activated by providing argument `queryBuilder.buildJson(doTrace=True)` when building the request Json. The response object will contain a json which details the time taken for each operation in the cache.\n\n### Paper\nA writeup for the implementation details of the cache can be found [here](/TsAggCachePaper.pdf)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmingyi850%2Ftsaggcache","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmingyi850%2Ftsaggcache","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmingyi850%2Ftsaggcache/lists"}